Sitemap Generator Forum

Can anyone explain why it takes a lot of time for crawling?
I'm already on it for 3 hours and still lacking about 6 hours to complete.
And I have to do the manual crowling every 15 minutes!
 


Pages added to sitemap: 1637
Pages scanned: 4146 (303,328.7 KB)
Pages left: 7940 (+ 2686 queued for the next depth level)
Time passed: 3:07:20
Time left: 5:58:45
Memory usage: 14,559.8 Kb
Resuming the last session (last updated: 2013-08-31 10:59:15)

Hello,

The crawling time itself depends on the website page generation time mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page, then 1000 pages will be crawled in about 16 minutes.

Thanks for your reply...
What does this mean? (2686 + queued for the next depth level)
By now has almost 5000 in queue!
And Is always increasing.
Regards...

Now I can not crawling and appears every time this information!
Can you help me please ...

Already in progress. Current process state is displayed:

Crawling already in progress.
Last log access time: 2013-08-31 13:13:08
Click here to interrupt it.
-------------------------------------------------------------------------


Pages added to sitemap: 2138
Pages scanned: 5650 (400,052.0 KB)
Pages left: 6436 (+ 5253 queued for the next depth level)
Time passed: 4:16:00
Time left: 4:51:36
Memory usage: 16,491.8 Kb

I have just finished one depth and now I have another to complete!
I'm in the 4th depht, when it ends?
I'm still waiting for a reply!
Regards...

Hello,

sitemap is created when crawling is completely finished (i.e. all pages found on your site are visited by generator bot).

I am getting 130000 pages in 23 minutes. Used to having 130000 pages in days.
Hint was in Narrow Indexed Pages Set. Try Exclusion preset. I am now using this software without a hassle.