Sitemap Generator Forum

Hey,
I am getting the below fatal error. On seeing your previous post I tried changing memory_limit and max_execution_time settings in php configuration in php.ini file.
I still get the same exception. Mine is large site and has more than 75000 pages.
Please advice..

Links depth: 2
Current page: profile/Suzanne-Bradley1/22974
Pages added to sitemap: 4733
Pages scanned: 4740 (122,741.7 KB)
Pages left: 7015 (+ 38128 queued for the next depth level)
Time passed: 15:05
Time left: 22:20
Memory usage: 20,005.8 Kb
Resuming the last session (last updated: 2009-03-11 10:43:16)
Fatal error: Out of memory (allocated 46137344) (tried to allocate 44 bytes) in /home/gailm/public_html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 6

Admin - PM'ed you login details.

Thanks,

Hello,

I've optimized your crawler settings and sitemap has been created successfully, please check that.

I would be very interested to know what the crawler settings are that you need to optimise to prevent this problem.
Regards, Kenneth.

Hello,

in most cases this issue is resolved with a custom setup of "Do not parse"/"Exclude URLs"/"Memory limit" and other settings. That depends on the specific site structure though.

Hi Admin,
I setup cron before 5 days, and still my sitemap is not complete.
Every till i check "Crawling" in Generator, its failed and I see "Do not interrupt the script even after closing the browser window until the crawling is complete"
The Sitemap has not completed even 30%.

Please help me out. Please check my previous PM's for my login details.

Thanks,

Replied to your pM.

Hi Admin,
My sitemap generation is totally messed up. When i try to run the cron, even that doesn't run.

Could you please help me out . Check my PM ..

Thanks,

Hello Admin,
Please check your PM for the reply from my Host.

Thanks,

I you provided full details of your problem instead of referring to PM messages then maybe someone on the forum could also help you and the answer could also benefit others with the same problem..
Regards, Kenneth

Hi Kenneth,
Here is the problem.

First of all I tried running it through webpage to complete the first run which was a failure with the below exception.

Links depth: 2
Current page: profile/SuzanneBr
Pages added to sitemap: 4733
Pages scanned: 4740 (122,741.7 KB)
Pages left: 7015 (+ 38128 queued for the next depth level)
Time passed: 15:05
Time left: 22:20
Memory usage: 20,005.8 Kb
Resuming the last session (last updated: 2009-03-11 10:43:16)
Fatal error: Out of memory (allocated 46137344) (tried to allocate 44 bytes) in /home/gailm/public_html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 6

Then as per the Admin's suggestion I tried running the cron command through SSH which was also a failure.
I contacted my Host, see the below reply.
==================
I ran the generator in your jailshell account to see the error, and the script is indeed running up on the memory limit.  There is a 64M limit for php processes.  You will need to contact the developer of the script to see about reducing its memory consumption.

As a note, I ran the script with a raised memory limit and it seems to have worked, though I have no way of knowing for sure.  It didn't produce a memory error.

 php -d memory_limit=128M public_html/generator/runcrawl.php

If you use this command, please be aware that if you run it often (more than once every hour or so), you run the risk of having your account suspended for over-utilization of system resources.
==================

Even when I tried to run the command by raising the Memory Limit to 128 was a failure.
See the below fatal error.

[20-Mar-2009 11:06:12] PHP Fatal error:  Allowed memory size of 67108864 bytes exhausted (tried to allocate 12877744 bytes) in /home/gailm/public_html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 6
[20-Mar-2009 11:07:20] PHP Fatal error:  Allowed memory size of 67108864 bytes exhausted (tried to allocate 12796362 bytes) in /home/gailm/public_html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 6

So, as of now my sitemap is not complete.

It would be great if someone solves my problem.

Thanks in advance,

64 meg limit is not enough, will your host not increase this on a permanent basis.
Also try running as a cron instead of the generator interface.
Over 4000 pages added seems a lot but not knowing your url it is hard to be sure, are you indexing some sort of online calendar, if so try excluding some url's and see what happens.
Regards, Kenneth.

Hi Kenneth,
Thanks for your reply. I am hosted at HostGator.
My Site has more than 1 Lack pages which has to be indexed. I also tried setting up cron and even that failed.

Got to think about someother way.

Thanks,

I just check with my host about increasing php limit. See the below reply from my host.

===============
No, it is not possible to increase the limit permanently.  The system-wide default is 64M.  Please let us know if you have any other questions.
Sincerely,
================

Maybe you should be thinking about changing your host then.
If you post your url I will see if I can suggest anything else to cure your problem.
Regards, Kenneth.