Sitemap Generator Forum

Any suggestions to make the crawl a lot faster?  There are certain strings that I don't want in the sitemap, hoping to make it faster, but didn't get much better.  Around 50k pages.

I used program on another site without a datafeed that was around 700 pages and it worked like a dream.

Any suggestions are helpful.

Thanks,
Jeremy

Hello,

you can use "Exclude URLs" setting to avoid crawling the pages that you don't need in sitemap.

Thank you Oleg.

Right now I have it running in the "background" and am unable to access the Crawl tab to see the status.  I am assuming it is still running (it has been about 24 hours), so would love to see how far it has come so far.

I did use exclude string.  Does that make the overall crawl faster?

thanks,
Jeremy

After making changes in configuration you should restart generator. Please try to close browser and reopen it if you cannot open crawling page.

Hi Oleg,

Crawl has been running for 7 days now (roughly).  Up to 145k pages.  I'm guessing there might be some pages getting spidered that I don't want.  How can I take what I got so far as the sitemap.xml while I make adjustments to hopefully make it go a lot faster.  Is this the normal pace for this many pages so far?

Thanks,
Jeremy

Hello,

you can set "maximum URLs" setting in "crawler limitations" section to 145000 and start generator.

The crawling time itself depends on the website page generation time mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page, then 1000 pages will be crawled in about 16 minutes.

Do I interrupt the crawl, make the cap of 145k, then continue?

Or, can I just update the configuration while it is running?

You should stop generator and continue it again.