Sitemap Generator Forum

Hello,

I have encountered a very strange problem. The sitemap generator used to process the nearly 60,000 pages in just over 150 minutes but it is currently crawling at 1000 pages per hour. No setting has been changed on the generator or the server. It seems that something on the server is slowing things down. I have tried rebooting it but nothing has changed. Usually when I run the crawler the loads spike a little to say 1.5 - 2.0 but now they seem stuck at below 1 and it also does not use the fully memory. Does anyone have any experience with this? I am at a complete loss.

Thanks,

Wouter

Hello,

sitemap generator crawls your site similar to normal visitors, i.e. opening every page of the site to find nested links. So, working time is mostly related to the time it takes to open a page on your site. i.e. if your pages (without images/css/etc) are opened in average of 1 second, crawler will get 3,600 pages in hour. Correspondingly, if your site serves pages faster it can crawl more pages. Please check how long (approximately) it takes to load pages on your site, may be it just started working slower.

I have not noticed a drop in page load times. It seems like something on the server is capping the script off at very slow speed.

Check that you don't have "Make delay after each X request" enabled in generator configuration. Also, what is your website URL? (you can PM it to me)

I am getting 130000 pages in 23 minutes. Used to having 130000 pages in days.
Hint was in Narrow Indexed Pages Set. Try Exclusion preset. I am now using this software without a hassle.