Arrghh!! Why is my crawl stopping? It cannot finish!
« on: August 21, 2008, 10:00:04 PM »
I have restarted the crawl several times already.  I tried both checking the box "running in the background" and unchecked.  It still does not finish.  It just stopped at 2002 pages with "Pages left: 292 (+ 8702 queued for the next depth level)".

I have also tried adding the .htaccess  file as suggested on another post.  When I add this file though, the page comes up as "500 internal server error".

Your immediate assistance is highly appreciated.
 
Re: Arrghh!! Why is my crawl stopping? It cannot finish!
« Reply #2 on: August 22, 2008, 02:23:36 AM »
How do I send you the URL?  Thanks.

flo

*
  • *
  • 3
Re: Arrghh!! Why is my crawl stopping? It cannot finish!
« Reply #4 on: August 25, 2008, 07:09:38 PM »
Report finally generated!  Thanks!!!

flo

*
  • *
  • 3
Re: Arrghh!! Why is my crawl stopping? It cannot finish!
« Reply #5 on: August 27, 2008, 06:50:37 AM »
Ooopppsss!  There it goes again.  I added products to my website and tried to crawl it again and it stopped 1/3 along the way.  I did not change any of the settings that you have made.  Please help!!!
Re: Arrghh!! Why is my crawl stopping? It cannot finish!
« Reply #6 on: August 28, 2008, 09:45:07 PM »
How many URLs did it crawl before it stopped and how many URLs you got in sitemap before?

flo

*
  • *
  • 3
Re: Arrghh!! Why is my crawl stopping? It cannot finish!
« Reply #7 on: August 29, 2008, 05:00:12 AM »
I had 7890 when you finished the crawl for me.  Now it has "URLs added: 5739, estimated URLs left: 8907".  I added a lot of products as you can see.  Please assist.  Thanks.
Re: Arrghh!! Why is my crawl stopping? It cannot finish!
« Reply #9 on: March 31, 2010, 10:50:56 PM »
Hi, I'm getting similar problems to this. It gets so far and just stops with Error 500. I've tried setting high memory_limit and max_execution_time's but no joy. Can you take a look please: [ External links are visible to forum administrators only ]

Cheers
Re: Arrghh!! Why is my crawl stopping? It cannot finish!
« Reply #10 on: March 31, 2010, 11:17:22 PM »
Hello,

in case if you are running on IIS, it might be required to update script timeout as described in [ External links are visible to logged in users only ]