Sitemap Generator Forum

I started generator 2 days ago, max execution time is over, but bookmark "Crawling" still dont open. Help please. Data of ftp I have allready sent you according this topic https://www.xml-sitemaps.com/forum/index.php/topic,4587.html . Generator is situated here [ External links are visible to forum administrators only ]

What characteristics should have a dedicated server to scan 400,000 pages?

Hello,

in case if generator interface is not opening for you, I'd recommend to try just closing and reopening the browser (sometimes it keeps open connection with site and doesn't open new one until previous is closed).

with website of this size the best option is to create a limited sitemap - with "Maximum depth" or "Maximume URLs" option limited so that it would gather about 100-150,000 URLs, which would be main pages representing "roadmap" sitemap for search engines.

The crawling time itself depends on the website page generation time mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page, then 1000 pages will be crawled in about 16 minutes.

Some of the real-world examples of big db-driven websites:
about 35,000 URLs indexed - 1h 40min total generation time
about 200,000 URLs indexed - 38hours total generation time

With "Max urls" options defined it would be much faster than that.