Sitemap Generator Forum

I am using a max. of 64 MB PHP memory.

I am getting stuck at about 2XXX pages at depth 3. Sometimes I get an error like "Out of memory (allocated 443285504)".

My settings for "Make a delay between requests, X seconds after each N requests:" is 3s every 50 requests. Maximum execution time = 0, Save the script state, every X seconds: = 30, Maximum pages: = 0, Maximum depth level: = 0,

I only selected the Yahoo sitemaps as the only additional sitemap to create. Don't understand how much resources this script is going to need to complete 1 sitemap generation run and how much it is going to need to update the sitemap daily for my site.

I increased my max php memory to 128 mb..

Links depth: 2
Current page: ----
Pages added to sitemap: 2680
Pages scanned: 2680 (192,330.4 Kb)
Pages left: 3798 (+ 6015 queued for the next depth level)
Time passed: 10:50
Time left: 15:22
Memory usage: -
Resuming the last session (last updated: 2007-07-24 10:14:06)
Fatal error: Out of memory (allocated 685244416) (tried to allocate 1833823 bytes) in pages/class.grab.inc.php(2) : eval()'d code on line 300

-------------

I really don't get it..... How much more RAM do I need for this script to work properly on my site? I have already created a database driven sitemap page to import the links to the XML Sitemaps script.

Hello,

try to restart Apache, it looks like your php config changes did not take effect.

I did restart.

My crawler went dead after a while and I clicked on the interrupt link and nothing much happened. I tried to start a new manual crawl without resuming but it just won't start.

I think I found the problem. I set the saving option at 30 s and the script seems to hang at exactly 30 s every time.

I am finally generating more than 3000 pages but I had to turn off the "Save every X seconds" option to make it work. It was causing the script to stop at every X seconds instead of saving it and continuing.

Hello,

you should increase max_execution_time setting in php configuration to solve that.

Ok, I shall increase max execution time. I managed to crawl over 20,000 pages in a period of about 12 hours + but nothing was saved in the end when the crawler crashed.

I extended my max execution time to 80s and told the script to save every 60s but it still stops at 60s.

Running this script without saving is stupid. But I can't run this script properly if I enable saving. What can I do now? My max ram = 64 MB, max execution = 80 seconds. What more must I do??

I get the error out of memory when my script reaches the save point. My save point is still way below the max execution time limit so I don't get it. If I turn off the save mode, I can run the script beyond my previous save point. Therefore, I do not think this is an issue with the max php memory or max execution time limit.

I just tested the script to save every 10 seconds and after the 1st 10 seconds was over, the script crashed with an out of memory error. This does not make sense. I have 64 mb max memory right now and 80s max execution time.

Hello,

please PM me with server access details for troubleshooting.

I have given you FTP access to my sitemap folder and the username/password to the script. What have you discovered and how do I resolve this problem?