Crawling keeps stopping at 5000 ish
« on: October 17, 2007, 09:13:01 AM »
Hiya,

I keep trying to fully XML-Sitemap my site, but the script keep stopping around the 5000 links mark.

Any ideas?  Need to see anything from my site?


Many thanks.
Re: Crawling keeps stopping at 5000 ish
« Reply #1 on: October 17, 2007, 11:57:46 AM »
Hello,

please try to increase max_execution_time and memory_limit settings in php configuration on your host.
Re: Crawling keeps stopping at 5000 ish
« Reply #2 on: October 17, 2007, 06:12:04 PM »
Hiya,

I did do. I read a lot of forum posted first to try and work it out myself. Settings are:-

max_execution_time = 9000
memory_limit = 500M

The script seems to run for about 5 - 10 mins. I then have to "Resume last session" for it continue. This general does work, but it stops when it gets to approx 5000 links. Its currently stuck on "Continue the interrupted session (2007-10-13 16:04:16, URLs added: 4783, estimated URLs left: 17363)".

Any ideas?
Re: Crawling keeps stopping at 5000 ish
« Reply #4 on: October 18, 2007, 12:25:53 PM »
PM sent.

Thanks.