Sitemap Generator Forum

Hello,

how can it be that with the free Online Sitemap Generator it takes 100 seconds to crawl 500 sites, but with the Unlimited Sitemap Generator it takes more then 15 minutes (!!!) to crawl less then 500 sites! (same domain of course)
I have a few big sites with more then 20 thousand links each, it would take forever to crawl these sites!

Does anybody have the same problem, and what can i do to speed this thing up?

Hello,

it should not take that long unless your server was overloaded and running slowly at that moment. Please let me know your generator URL via private message to check that.

Hello and thanks to your reply.

There was no load or speed problem with the server for sure, i did these tests quite a few times in a row.
The free crawler was always fast and the unlimited crawler slow.

But it seems i found a way to get full speed with the unlimited crawler, i just set the websites IP in the advanced options. Now the crawler does 15.000 pages in less then 30 min

Could you tell me from where the unlimited crawler crawls the site when i don't set an IP?

Hello,

I'm glad that worked so well for you!
When IP is not specified, the domain lookup is performed according to your server configuration to find out IP from domain name.