Crawling process stalls out
« on: August 22, 2013, 04:51:35 PM »
I've been trying to create a sitemap for the past week but the crawling process never fully gets completed (even if I opt to crawl in the background). I have tried to adjust the php.ini settings but have not seen any improvement. Any guidance will be much appreciated.


Already in progress. Current process state is displayed:

Links depth: 2
Current page: ftr-8519p-5a/
Pages added to sitemap: 895
Pages scanned: 1003 (78,045.7 KB)
Pages left: 1172 (+ 1433 queued for the next depth level)
Time passed: 0:24:27
Time left: 0:28:34
Memory usage: 5,158.1 Kb
Re: Crawling process stalls out
« Reply #1 on: August 25, 2013, 08:12:58 AM »
Hello,

it looks like your server configuration doesn't allow to run the script long enough to create full sitemap. Please try to increase memory_limit and max_execution_time settings in php configuration at your host (php.ini file) or contact hosting support regarding this.
 
Re: Crawling process stalls out
« Reply #2 on: September 14, 2013, 07:32:32 PM »
I am having this same issue.  I've increased the memory_limit and max_execution_time and no change.  It seems to fail once the Memory usage gets to around 3,000kb.  memory limit is set to 64mb.  I thought the issue may be the site file path as the hint shows up as C:\Inetpub\...\...\httpdocs\generator/ whereas the document root i believe is just /httpsdocs/sitemap.xml

i've tried multiple variants of the file path and no change in behavior

Links depth: 2
Current page: viewitem.asp?idCategory=52&idProduct=359&priceRange=0x999999
Pages added to sitemap: 443
Pages scanned: 500 (9,114.5 KB)
Pages left: 168 (+ 2274 queued for the next depth level)
Time passed: 0:01:34
Time left: 0:00:31
Memory usage: 3,004.4 Kb
Re: Crawling process stalls out
« Reply #3 on: September 15, 2013, 04:35:31 PM »
Hello,

in case of Windows server it might be needed to increase script timeout in IIS configuration as described on:
[ External links are visible to logged in users only ]
[ External links are visible to logged in users only ]
 
Re: Crawling process stalls out
« Reply #4 on: September 16, 2013, 09:48:04 PM »
Here was the reply from HostGator:

"Thank you for contacting HostGator. We appreciate your patience and apologize for any inconvenience. Unfortunately, we would be unable to modify the memory limit or time limit for a process to execute. We have these listed in a special support article in addition to all other directives which are unavailable for modification."

I suppose I should have checked compatibility with HostGator prior to purchasing this Sitemap software

randy

*
  • *
  • 30
Re: Crawling process stalls out
« Reply #5 on: October 08, 2013, 03:55:34 PM »
Could he use ini_set in the script or increase php_value in .htaccess?
**********