Sitemap Generator Forum

I have a rather large comparison shopping website that I am trying to create an XML sitemap for, but I keep getting the following fatal error:
------------------------------------------------------------------------------------------
Links depth: 2
Current page: best_price_shopping.php?intCatId=1897&strOfferSortType=price&strOfferSortOrder=asc&arrAttr[]=lg
Pages added to sitemap: 398
Pages scanned: 400 (22,755.5 KB)
Pages left: 66 (+ 53030 queued for the next depth level)
Time passed: 0:10:09
Time left: 0:01:40
Memory usage: 33,008.8 Kb
Auto-restart monitoring: Sun Feb 27 14:03:21 EST 2011 (5 second(s) since last update)Resuming the last session (last updated: 2011-02-27 18:59:50)
Fatal error: Out of memory (allocated 74448896) (tried to allocate 17301513 bytes) in /home/w2look/public_html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 27
------------------------------------------------------------------------------------------
I have read other posts in the forum that recommended that I check my php.ini settings, so I did and made changes. I increased the max_execution time to 180 sec and the the memory_limit to a whopping 2048M. I have also checked to make sure that my Crawler Limitations setting for max memory usage was set. I tried both 2048M and 0 for unlimited, but I am still getting this error.

Can someone please let me know how to fix this?

Hello,

please let me know your generator URL/login in private message to check this.

Ok, I have sent you the message that includes my generator's url, username and password. Please let me know what you find out as soon as possible.

OK, so I sent you my generator url and pass like you suggested, you sent me a pm back saying everything was good, so I ran the crawl again making sure to check the run in background box. Everything seemed to be working correctly, so I left the page and closed the browser only to return a few hours later and see this:

Links depth: 4
Current page: compare_the_best_prices_on-Dual+Electronics+XDMA6355+Car+CD~+MP3+Player-103030217
Pages added to sitemap: 16526
Pages scanned: 16560 (283,906.3 KB)
Pages left: 13972 (+ 6400 queued for the next depth level)
Time passed: 1:52:18
Time left: 1:34:45
Memory usage: 36,267.4 Kb
Auto-restart monitoring: Mon Feb 28 15:37:37 EST 2011 (19 second(s) since last update)Resuming the last session (last updated: 2011-02-28 20:32:44)
Fatal error: Out of memory (allocated 74448896) (tried to allocate 44 bytes) in /home/w2look/public_html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 27

So, I went back and checked the configuration settings and no setting had been changed since I ran it the first time. I then clicked on the crawling tab, selected the run in background box and the resume previously interrupted session box, then clicked run. It starts the crawl again from a point where only 16155 pages have been added to the sitemap, but then I get:

Links depth: 4
Current page: compare_the_best_prices_on-Pyle+PLWCH12D+12
Pages added to sitemap: 16466
Pages scanned: 16500 (283,586.1 KB)
Pages left: 14032 (+ 6334 queued for the next depth level)
Time passed: 1:52:16
Time left: 1:35:28
Memory usage: 36,162.0 Kb
Auto-restart monitoring: Mon Feb 28 15:46:09 EST 2011 (10 second(s) since last update)Resuming the last session (last updated: 2011-02-28 20:32:44)
Fatal error: Out of memory (allocated 74448896) (tried to allocate 44 bytes) in /home/w2look/public_html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 27
 
Any suggestions?

By the way, this almost seems like it's a different crawl page because the link to stop the crawl has disappeared from the page. Does it use different crawl pages when it tries to re-start an interrupted session?

I'm not sure if my private messaging is working. I have replied to your PM a few times without a response and I have tried sending you another PM through your profile page, but when I look in my Sent Items, I only see one sent message. Please let me know what is going on. I am getting fustrated with this thing not working.

Hello,

try to increase max_execution_time setting as well (currently it's set to 180 seconds which is not enough fopr large site).
Also, if you have ssh access to your host I'd recomment to run generator in command line instead.

You will see PMs in sent list if you enable "Save a copy in my outbox" checkbox.

OK, I'll try increasing the max execution time, but this looks like a memory allocation error to me, not a time-out error.

Do you have any suggestions for how long I should increase the execution time?

Do I need to run the script from the start again or can I restart a previously interuppted session?

Did you get my message about the sitemap.xml.gz files? If so, do you have an answer for me?

Please let me know as much info as you can.

You can try to set it to 3000. Running command line manually could be a solution (not a cron task but manually start it to track the progress).

If you have gzip compression enabled in generator config, you need to manually create mepty files named:
sitemap.xml.gz, sitemap1.xml.gz, sitemap2.xml.gz etc depending on how many URLs are found.
without gzip option you should create
sitemap.xml, sitemap1.xml, sitemap2.xml etc

OK, this is getting ridiculous!

I set the max_execution time to 3000 and the memory_limit to 512 in my php.ini file. Then I set up a cron job to run the script this morning. It starts up just fine but within a few seconds it crashes. I tried changing the start time on the cron job a few times (8:35am, 9:55am, 10am) to see if I could get it to run, but no luck. The error log in my generator folder states:

[01-Mar-2011 08:35:51] PHP Fatal error:  Allowed memory size of 67108864 bytes exhausted (tried to allocate 12487694 bytes) in /home/w2look/public_html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 27
[01-Mar-2011 09:55:35] PHP Fatal error:  Allowed memory size of 67108864 bytes exhausted (tried to allocate 12259711 bytes) in /home/w2look/public_html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 27
[01-Mar-2011 10:00:34] PHP Fatal error:  Allowed memory size of 67108864 bytes exhausted (tried to allocate 12321293 bytes) in /home/w2look/public_html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 27

The only other thing I can try is running it from the command line. I don't currently have SSH access to my server, but I am working on getting it. I guess I will try to run it from the command line if I can get access. If I can't, I'm going to have to request my money back because I can't get the product to perform as advertised. Unless, you have some other idea as to how I can get this script to run to completion.

Ok, please also try this when running in command line:
/usr/local/bin/php -i|grep memory_limit

to make sure that memory_limit is actually set to 512M (since from the error message above it looks like if it was still at 64M)

yeah, I thought it looked like it had not been reset either. I will contact my host and see if I can get the server settings reset. I will let you know the outcome.

OK, so I talked to support on my hosting account and apparently the max memory on my account is only 64M because it is a shared server. So, unless you have a way to keep the memory usage under 64M I am going to have to issue a refund request through Paypal. There is nothing in the system requirements that state the program will not work unless the server has X amount of available memory. It simply states that memory requirements will vary. The only other option I have is to upgrade to a dedicated server which would triple my hosting costs. This is not an option just to be able to run one script. It's too bad, I think you have a really good script here and your service has been really good up to this point. I just wish there was some way to get it working, but unless you have some other solution, I will have to request the refund. Sorry.

You can set "Maximum URLs" setting in generator cofniguration to keep smaller sitemap and correspondingly less memory usage.

I'll give you one more chance. I will try to come up with a way to limit my URLs that I want in the sitemap. If it works, I will keep the product and not request a refund. I will let you know how it turns out.