Competitor Analysis Tool: Analyze rival websites and gain valuable SEO and PPC insights
« previous next »
Pages: 1

ONC Hosting

*
  • *
  • 9
Am I crazy?
« on: March 25, 2007, 02:57:26 PM »
Is it my imagination or does the script use more resources and get slower as it crawls...

My site is about 1.5 million pages.   I am on 200,000 pages in my crawl and it is going MUCH slower than in the early pages...

Am I right?
Logged

XML-Sitemaps Support

*
  • *****
  • 11781
Re: Am I crazy?
« Reply #1 on: March 25, 2007, 11:33:55 PM »
Hello,

yes, it is possible that crawling becomes slower when significant amount of pages are already crawled, the actual decrease in speed depends on the number of pages and server configuration.
Logged
Pages: 1
« previous next »