Sitemap Generator Forum

Hi,

I have a website that has over 60,000 pages, probably more if the generator considers duplicate pages... I am running it for the first time on this new server, but, I am into my 3rd hour of generating and still have a long ways to go; will future runs of XML-Sitemap take less time?

I don't remember it ever taking so long. I got to thru depth 3 at over 22,000 pages after 12 hours and stopped it. I have reset it for a max depth of 3, which was about where 22,000 pages were. I"m just going to let it run as cron, rather than watch it.

Will they generate quicker after the first one?

Hello,

The crawling time itself depends on the website page generation time mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page, then 1000 pages will be crawled in about 16 minutes.

Basically, it depends on how fast the target website is working.

Generator always starts the process from the scratch (to be able to find all changes on the website), so it will work in the same way next time.
You can improve the crawling speed using "Add directly in sitemap" and "Exclude URLs" settings.

I set it up to run from cron and it ran for 20 minutes and only 600 pages were crawled before it stopped. The day before, after starting and stopping a couple of times, I got up to 6000 pages. (One time, it ran without stopping for about 4 hours.)

What do you suppose is stopping it from completing? Would this considered particularly slow? I have a reseller account, each cpanel account has its own resources. Could I install XML Sitemap Generator in its own cpanel account and run against another cpanel account to offload some of the system demands?

Hello,

it means that your site started responding slower than before, hence crawling speed decreased.