Competitor Analysis Tool: Analyze rival websites and gain valuable SEO and PPC insights
« previous next »
Pages: 1

william1

*
  • *
  • 3
Crawling one page every 1 or 2 seconds
« on: March 19, 2010, 12:15:20 AM »
Hi All

My crawls are taking forever and after looking at teh access logs i see that a page is being requested every 1 or 2 seconds. how can i speed this up?

Ww
Logged

XML-Sitemaps Support

*
  • *****
  • 11781
Re: Crawling one page every 1 or 2 seconds
« Reply #1 on: March 19, 2010, 12:30:39 PM »
Hello,

sitemap generator has a built-in crawler and it requests the page after the previous one has been processed, so in case if page load time is about 1 second, then that's a time period between requests.
Logged

william1

*
  • *
  • 3
Re: Crawling one page every 1 or 2 seconds
« Reply #2 on: March 19, 2010, 03:26:32 PM »
Thanks for the pointers, i will see what i can do to improve me site speed.

Is there a way of disabling this as my server has very little load on it at the moment.

Ww
Logged

XML-Sitemaps Support

*
  • *****
  • 11781
Re: Crawling one page every 1 or 2 seconds
« Reply #3 on: March 20, 2010, 08:17:03 AM »
That is the way generator uses to find all pages on the site, it cannot be disabled.
Logged
Pages: 1
« previous next »