Sitemap Generator Forum

Generator takes 18 hours to generate 70,000 URLs.
This seems very excessive. Google SitemapGenerator would run all 500,000 urls in an hour or so.

We own a dedicated server with 4 multi-core processors and 32GB RAM, but I still can't have this running constantly..

What settings should I check to fix this?

Hello,


The crawling time itself depends on the website page generation time mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page, then 1000 pages will be crawled in about 16 minutes.