Problem with website crawling
« on: October 10, 2013, 11:27:38 AM »
Can not crawl the website. Get this error:

HTTP Error 500.0 - Internal Server Error
d:\php\php-cgi.exe - The FastCGI process exceeded configured request timeout
Detailed Error Information
Module   FastCgiModule
Notification   ExecuteRequestHandler
Handler   php-5.2.6
Error Code   0x80070102
Requested URL   [ External links are visible to forum administrators only ]
Physical Path   D:\Domains\CharityJobSC\Website\generator\index.php
Logon Method   Anonymous
Logon User   Anonymous
« Last Edit: October 10, 2013, 11:59:51 AM by steve96 »
Re: Problem with website crawling
« Reply #1 on: October 11, 2013, 08:42:53 AM »
Hello,

in case of Windows server it might be needed to increase script timeout in IIS configuration as described on:
[ External links are visible to logged in users only ]
[ External links are visible to logged in users only ]
 
Re: Problem with website crawling
« Reply #2 on: October 11, 2013, 12:23:31 PM »
Thanks Oleg
We increased script timeout using FastCGI Settings, also increased Activity Timeout.
Now when I run crawling it starts working (no progress bar! but Page Title says Crawling:XML,ROR, Text...) and it never stops. How can I be sure its working properly? Should it list the directions which have already been crawled?

Thanks
Olga
Re: Problem with website crawling
« Reply #3 on: October 12, 2013, 02:12:14 PM »
Hello,

depending on server configuration the progress details could not be visible.
You can open generator crawling page in another browser and see progress details on that page.
Re: Problem with website crawling
« Reply #4 on: October 14, 2013, 10:41:07 AM »
Hi Oleg

We increased the timeout but get the error again. Also when I try to change settings in Configuration tab I get the timeout error as well and no settings were saved.
Does it mean that script is not working? I understand that configuration settings should be saved straight away?
Re: Problem with website crawling
« Reply #5 on: October 16, 2013, 02:27:03 PM »
Hello,

> Does it mean that script is not working?

That usually means that your browser keeps the session open with the website if you have started generator. It would work if you open another browser or completely close your browser (not just a tab) and open it again.
Re: Problem with website crawling
« Reply #6 on: October 16, 2013, 03:00:55 PM »
Hi Oleg

I do not think its a session problem. I closed the browser and now trying to save configuration settings. Its trying to connect but nothing happening and the settings are not saved. In few minutes it returns a timeout error.
Re: Problem with website crawling
« Reply #7 on: October 19, 2013, 08:22:10 AM »
Hello,

please let me know your generator URL/login in private message to check this.