Hi
We bought the Unlimited version after playing with the Demo and finding this product to be great for it's speed and results. I have been messing with this for three days and my boss is going to have a cow if I don't get this done and start the other projects on my plate soon, so I hope there is a fix. We generate a HUGE google base feed via php and have never run into memory or timeout issues, so I'm not sure the issue is actually memory or exec time.
I searched the forums and double checked my permissions and php.ini, which have the following settings:
max_execution_time = 18000
max_input_time = 600
memory_limit = 512M
upload_max_filesize = 256M
The first time it ran, it got stuck after 6000 links. I manually restarted it over and over until it got down to 7 links left, then it refused to finish. The Crawler just sits at whatever point it's stuck at and returns back to the resume/run page at random intervals. I figured it was a fluke so I reloaded the entire package and started from scratch again, making sure all the permissions were correct just to be on the safe side.
I let it run and it got stuck at only 400 links. I tried running it as a CRON and it still sticks. I blew it out and ran it again, babysitting it all night and resuming whenever it would return to the resume/run screen. This morning it is completely stuck and no longer updates anything but the mod time on the crawl_state.log file. The crawl_dump.log still has the mod time of the last failure.
Links depth: 4
Current page: sierra/sierra-1537C
Pages added to sitemap: 10232
Pages scanned: 18840 (849,266.3 KB)
Pages left: 20 (+ 16166 queued for the next depth level)
Time passed: 2:16:30
Time left: 0:00:08
Memory usage: 20,762.5 Kb
I searched the forums and tried all recommended fixes, but most of these issues were resolved via PM and never have a posted solution. If someone can get this to run, I'll owe you a case of beer because my boss is going to kill me when he finds out how much time I have wasted on the simple task of generating a sitemap. I'll save a bit of time and PM my login, since that seems to be the most common next step
Guest