Competitor Analysis Tool: Analyze rival websites and gain valuable SEO and PPC insights
« previous next »
Pages: 1

stefan2

*
  • *
  • 2
3:48:37s processing 84047 pages??
« on: August 13, 2009, 06:13:58 AM »
Hello,

I have the generator installed at [ External links are visible to forum administrators only ]  . The website is increasing the number of pages from a day to another with aprox 1000 pages. But what is concerning me is the time for processing them. It starts at 3am and finishes after 6am . We don't want the server to be solicited when in the day, but the time increaseas from a day to another...Is this a bug? Is there some option to add only the new pages? Is there other posibility?
Logged

XML-Sitemaps Support

*
  • *****
  • 11782
Re: 3:48:37s processing 84047 pages??
« Reply #1 on: August 13, 2009, 10:58:01 PM »
Hello,

The crawling time itself depends on the website page generation time mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page, then 1000 pages will be crawled in about 16 minutes.

In many cases it is possible to insrease crawling speed with "Do not parse"/"Exclude URLs" options.
Logged
Pages: 1
« previous next »