Crawler stops at certain point
« on: September 21, 2010, 08:15:14 AM »
Hello !!

After I've changed my hosting I am not able to crawl my site fully. Crawler stops after scanning 420 pages:
-----------------------------------------------------
Links depth: 2
Current page: talladega_logo-NA-25.php
Pages added to sitemap: 379
Pages scanned: 420 (20,490.6 KB)
Pages left: 140 (+ 1959 queued for the next depth level)
Time passed: 0:04:54
Time left: 0:01:38
Memory usage: 2,005.3 Kb
-----------------------------------------------------

I've checked with hosting company and they have modified following two settings in php.ini for me but still the result is same:
mamory_limit = 512M (It was 256M before)
max_execution_time = 9000 (It was 300 before)

Whether I run in Background mode or normal mode, it's not going forward after scanning 420 pages.

Any help please !!
Re: Crawler stops at certain point
« Reply #1 on: September 21, 2010, 10:16:27 AM »
Hello,

please let me know your generator URL/login in private message to check this.
Re: Crawler stops at certain point
« Reply #2 on: September 21, 2010, 02:28:38 PM »
Where do I send you my generator username/password? In this forum - isn't it open for public?
Re: Crawler stops at certain point
« Reply #4 on: September 21, 2010, 02:33:44 PM »
Ok, Done - Sent.
Re: Crawler stops at certain point
« Reply #5 on: September 22, 2010, 07:48:31 PM »
Hi !!

Did you get username/password for my Generator? Any updates?
Re: Crawler stops at certain point
« Reply #7 on: September 23, 2010, 06:22:36 PM »
Hi !!

I am really sorry to trouble you again but can you resend the mail as I think either I've not received it or deleted it by mistake.

Thank you in advance.