Competitor Analysis Tool: Analyze rival websites and gain valuable SEO and PPC insights
« previous next »
Pages: 1

cal

*
  • *
  • 3
Crawler Reaches a Particular URL and Locks-up
« on: December 26, 2011, 04:04:59 PM »
The crawler has been crawling my site for the past 10 hours. Every time it reaches one particular URL it stalls, restarts, stalls, restarts. It has been doing this endlessly.

Here is the info displayed:

Links depth: 4
Current page: gallery/photo.php?exhibition=61&photo=3501&ee_lang=eng
Pages added to sitemap: 11695
Pages scanned: 11720 (171,319.6 KB)
Pages left: 48308 (+ 0 queued for the next depth level)
Time passed: 0:44:44
Time left: 3:04:26
Memory usage: 51,680.1 Kb

I checked the URL and it seems unremarkable. It's just another photo in a gallery app.

Thanks.

Cal
Logged

XML-Sitemaps Support

*
  • *****
  • 11781
Re: Crawler Reaches a Particular URL and Locks-up
« Reply #1 on: December 27, 2011, 11:04:45 AM »
Hello,

it looks like your server configuration doesn't allow to run the script long enough to create full sitemap. Please try to increase memory_limit and max_execution_time settings in php configuration at your host (php.ini file) or contact hosting support regarding this.
Logged
Pages: 1
« previous next »