Tools for Any SEO Challenge: Get more traffic to your website with versatile solutions for every step of your SEO journey.
« previous next »
Pages: 1 2

ed11

*
  • *
  • 9
Sitemap generation always stops. and Crawling page broke
« on: February 09, 2009, 01:54:02 AM »
Everyting is setup fine but the crawler always stops after 10 minutes or so.

Now my Crawling tab only displays the Run in background checkbox.

Config seems fine and it does crawl for a while. I always resume but it does not work for very long.

The site i'm trying to crawl has 50-75K pages so.....

I also get this occasionally:
Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.


Ideas?

Ed

« Last Edit: February 09, 2009, 02:01:19 AM by ed11 »
Logged

XML-Sitemaps Support

*
  • *****
  • 11781
Re: Sitemap generation always stops. and Crawling page broke
« Reply #1 on: February 09, 2009, 11:21:16 PM »
Hello,

it looks like your server configuration doesn't allow to run the script long enough to create full sitemap. Please try to increase memory_limit and max_execution_time settings in php configuration at your host (php.ini file) or contact hosting support regarding this.
Logged

ed11

*
  • *
  • 9
Re: Sitemap generation always stops. and Crawling page broke
« Reply #2 on: February 10, 2009, 03:25:03 AM »
What do you recommend for the settings. My php max_execution_time is set to 30 and the memory limit is set to 64M

thanks for your help. I will have to put in a support ticket to have this changed.

Ed
Logged

XML-Sitemaps Support

*
  • *****
  • 11781
Re: Sitemap generation always stops. and Crawling page broke
« Reply #3 on: February 11, 2009, 03:05:01 AM »
I'd try to increase to 1200 and 256M correspondingly.
Logged

ed11

*
  • *
  • 9
Re: Sitemap generation always stops. and Crawling page broke
« Reply #4 on: February 14, 2009, 04:59:25 AM »
I have made the changes and it still runs for less than 5 minutes.

Also, I host at HostGator.com if that helps any.

Ed
Logged

XML-Sitemaps Support

*
  • *****
  • 11781
Re: Sitemap generation always stops. and Crawling page broke
« Reply #5 on: February 15, 2009, 09:24:37 PM »
Are you sure the new settings were applied? Try to create file named phpinfo.php in generator folder with:
Code: [Select]
<?phpphpinfo();?>
and open it in browser to see your PHP settings.
Logged

ed11

*
  • *
  • 9
Re: Sitemap generation always stops. and Crawling page broke
« Reply #6 on: February 26, 2009, 02:37:24 AM »
Interesting. It is set in my php.ini QuickConfig but when I check with phpinfo.php, it is not set there.

I've opened a ticket with HostGator to fix this for me.

I'll follow up when it is working.

Ed
Logged

XML-Sitemaps Support

*
  • *****
  • 11781
Re: Sitemap generation always stops. and Crawling page broke
« Reply #7 on: February 26, 2009, 03:06:14 AM »
Ok, let us know how it goes.
Logged

ed11

*
  • *
  • 9
Re: Sitemap generation always stops. and Crawling page broke
« Reply #8 on: February 27, 2009, 03:38:25 AM »
It still stops. Seems like it runs longer. The crawling tab doesn't fully load anymore. I attached a screen shot. I now can't do anything. I checked my running processes on HostGator and it is not running.
Logged

XML-Sitemaps Support

*
  • *****
  • 11781
Re: Sitemap generation always stops. and Crawling page broke
« Reply #9 on: March 01, 2009, 01:38:20 PM »
What are your new PHP settings values? Please PM me the link to your phpinfo() page.
Logged

ed11

*
  • *
  • 9
Re: Sitemap generation always stops. and Crawling page broke
« Reply #10 on: March 02, 2009, 01:49:30 AM »
Oleg,

I've sent you the info...

Ed
Logged

vera

*
  • *
  • 1
Re: Sitemap generation always stops. and Crawling page broke
« Reply #11 on: June 12, 2009, 04:19:03 AM »
Oleg, I have the same problem, can you help me it. I have followed the instruction written, but it is still the same.

Thank you!
Logged

XML-Sitemaps Support

*
  • *****
  • 11781
Re: Sitemap generation always stops. and Crawling page broke
« Reply #12 on: June 12, 2009, 11:44:17 AM »
Please PM me your generator URL as well.
Logged

alan9

*
  • *
  • 4
Re: Sitemap generation always stops. and Crawling page broke
« Reply #13 on: June 26, 2009, 10:27:49 AM »
Hi,

Was there any resolution to this, as I'm now suffering from the same issue?  My crawl will last 10 mins and then just stop - resuming does nothing.

My host server settings I'm unable to change - however on a test site with the max_memory and max_execution_time limits set lower than my ISP, it completes satisfactorily!

Any ideas?
Logged

XML-Sitemaps Support

*
  • *****
  • 11781
Re: Sitemap generation always stops. and Crawling page broke
« Reply #14 on: June 26, 2009, 10:28:16 PM »
Please PM me your generator URL, crawler settings usually should be configured depending on the site.
Logged
Pages: 1 2
« previous next »