Competitor Analysis Tool: Analyze rival websites and gain valuable SEO and PPC insights
« previous next »
Pages: 1

pspglobalnet

*
  • *
  • 1
After 8 hours and some 35,000 pages crawling suddenly stops
« on: August 18, 2011, 11:27:03 AM »
Hello!

Our site has aproximately 90,000 pages. After 8 hours and some 35,000 pages crawling suddenly stops.

Trying to resume I select:
"Continue the interrupted session (2011-08-18 06:50:15, URLs added: 35685, estimated URLs left in a queue: 20148)"

Yet, crawling does not continue.
At this point crawl_dump.log is already 34 MB in size.

sitemap.xml and sitemap.xml.gz are set to 666 and data folder is set to 777.


I have enclosed my configuration file and teh screen shots of the two crawling pages.

Please advise.
Thanks,

peter
Logged

XML-Sitemaps Support

*
  • *****
  • 11781
Re: After 8 hours and some 35,000 pages crawling suddenly stops
« Reply #1 on: August 18, 2011, 02:10:13 PM »
Hello,

it looks like your server configuration doesn't allow to run the script long enough to create full sitemap. Please try to increase memory_limit and max_execution_time settings in php configuration at your host (php.ini file) or contact hosting support regarding this.
Logged
Pages: 1
« previous next »