Sitemap Generator Forum

Hi
I have just installed the script with no problem and manualy started the first crawl but this stopped after 5 minutesI have set the  Save the script state, every X seconds: to 140 and Make a delay between requests, X seconds after each N requests: 5 - 20
The sript is running on a windows server.

   Regards
      Gary

Hello Gary,

sometimes the server configuration limits the maximum execution time for the scripts and interrupts the script when time exceeds.
You can either:
1. restart script execution multiple times (with "Resume session" checkbox enabled) until full sitemap is created
OR
2. modify your max_execution_time  setting in php.ini (increase this value). In case if you are running IIS, you should also increase "Maximum script execution time" setting in IIS admin configuration.

Hi
Thanks for the quick response Re: "Resume session" check box I cant find this in the config page only the save the script state.

    Regards
        Gary

Hello,

you will see this checkbox at the Crawling page after your initial session will be interrupted by server (timed out), assuming that you have set the "Save state" option.

Hi
The server dosen't seem to time out the crawl just stops, I then need to close the browser and re open the page and then continue the Continue the interrupted session.

   Gary

Hello Gary,

when the crawling just stops, it means that the script timed out (there is no special "timeout" message  sent by the server). And you should resume the generation after that (or modify php configuration as described above).

I'm having the same problem. I've set the crawler time out in the config for the generator, and gone into php.ini and modified the max_execution_time, adjusted the script execution timeout, and connection timeout to ridiculously high numbers and it's still stopping after a couple minutes. Do you have to do anything for PHP to use the updated settings in php.ini? We've got a site with 75000 pages, and really can't sit around all day restarting the script.

Hi
As I do not have access to php.ini and my server would not increase or change the time out, I have se up a scheduled task (cron) as I am on a windows server to run periodicaly which seems to work great with no great loads on the sever.

    ATS

So to get the  new php.ini loaded you have to recycle the DefaultAppPool, but even doing this it still just stops. With a site as large as ours, setting up a cron job to run every 5 minutes isn't going to work for getting the job done anytime soon. Two days turns into four days....

So to get the  new php.ini loaded you have to recycle the DefaultAppPool, but even doing this it still just stops. With a site as large as ours, setting up a cron job to run every 5 minutes isn't going to work for getting the job done anytime soon. Two days turns into four days....

Hello,

you should not set the cron task to every 5 minutes! The daily task is more than enough to keep your sitemap fresh (and it's perfectly fine to have it generated weekly for larger sites).

If it's timing out every couple minutes is the cron job running once a day really going to accomplish anything worthwhile?

When script is executed by the cron, it is usually configured with other options that allows it to run longer (i.e., Apache doesn't interrupt the php process in this case).
Could you please try to execute generator from the command line? (via SSH)

Will try

Gotten to 10,000 and it's still going. So I'm hopeful it'll finish.

did you get this to work out?  I am having the same troubles.  The generator kept stopping... and I was just moving save state to save state.  BUT  the script stopped going far enough forward for the save state to save (I had it set every 300 seconds.. or 5 min). 

So I tried running the script in the uninterrupt mode... but I don't know if it's still going.... The save states aren't continually being made.... so how do I know if it's still running?

thanks.
BV