Sitemap Generator Forum

Can you give me an idea what it takes for

50k pages
100k pages
200k pages
500k pages

What is the typical time in hours that it takes to crawl?

Im asking because I am crawling my site which has about 400k pages and it is taking weeks at this point. Any advice

Hello,

The crawling time itself depends on the website page generation time mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page, then 1000 pages will be crawled in about 16 minutes.

Some of the real-world examples of big db-driven websites:
about 35,000 URLs indexed - 1h 40min total generation time
about 200,000 URLs indexed - 38hours total generation time

Seems like mine has been going on over 200 hours. I am not getting errors but it is so slow. Are you saying that it is due to the slow load time of the pages on our site?

Hello,

this is possible, do you get 200 hours reported in sitemap generator crawling progress page? If you have started it 200 hours ago, may be it was just interrupted at some point and was not actually working?