Sitemap Generator Forum

Hi,

New customer.  I do seo and one of my clients is going to be a week to crawl and prepare a sitemap so I'd like to do it here and have the capability myself.

The site is about a million pages.

Is this practical?

Will my host freak out?

Can I do it in bits?

Thanks!

Hello,

with website of this size the best option is to create a limited sitemap - with "Maximum depth" or "Maximume URLs" option limited so that it would gather about 100-150,000 URLs, which would be main pages representing "roadmap" sitemap for search engines.

The crawling time itself depends on the website page generation time mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page, then 1000 pages will be crawled in about 16 minutes.

Please also note that generator license allows to use it on own sites only, i.e. sitemaps cannot be created for 3rd party sites or provided as a service (additional licenses need to be purchased for that): https://www.xml-sitemaps.com/license.html