On Page SEO Checker: Get a list of improvements for your website for free in less than 5 minutes!
« previous next »
Pages: 1

sdeeth

*
  • *
  • 1
Generator starts and stops, have 500k pages to crawl
« on: December 27, 2009, 08:09:06 PM »
I have a windows server.  I have the generator to pause 60 seconds after each 1000 pages.  It crawls for a while then stops.  I see other people have had same issue, but then you pm them for help.  Can you help me with this?
Logged

XML-Sitemaps Support

*
  • *****
  • 11783
Re: Generator starts and stops, have 500k pages to crawl
« Reply #1 on: December 27, 2009, 10:36:23 PM »
Hello,

having 60 seconds delay is too long, a few seconds is sufficient.
Logged
Pages: 1
« previous next »