Sitemap Generator Forum

I'm finding that Standalone XML Sitemap Generator keeps stopping. If I start a job with "only" a few hundred pages it goes fine. When I start the job with 20,000 pages it stops itself after anywhere from 2 minutes to 10 minutes. I don't see any error or any problem; it's just stopped. When I go back to the page, it asked me if I want to resume.

Almost all of the 20,000 pages are the same page name except different database variables complete the page based upon the variables in the URL. Is it possible that the system is trying to access the same page too frequently and shutting itself down??

Hi,

the crawler tries to access EVERY page it find link to (unless youspecified it in the exclusion options).
So, if you have a larger site than can be crawled with your current php settings, try to increase max_execution_time and memory_limit settings in your php.ini (if you have access to it) and restart apache.

Included php.ini are the following two lines:
max_execution_time = 30     ; Maximum execution time of each script, in seconds
memory_limit = 8M      ; Maximum amount of memory a script may consume (8MB)

What would you suggest??

Is there any way to overcome this limit without changing the "max_execution_time"?? Perhaps by placing some refresh line on Standalone XML Sitemap Generator crawler page??

Before I purchased Standalone XML Sitemap Generator, I ran a 500 page site map against my site via your online system and it worked great. Could there be some other setting in Standalone XML Sitemap Generator that's preventing this from running??

My website is [ External links are visible to forum administrators only ]. Can you try running a larger sitemap on it and see what happens?? FYI-I have about 24,000 pages.

Hello,

obviously it takes more time to crawl a big site. Can't say the exact values for you, but you may try to find the better values by yourself: increase them somehow and see if it is enough (then increase gain if it still fails).

Some php configurations allows to increase max_execution_time within script and Generator script always tries to do this, but probably this doesn't work in your case.

You can also run generator in shell if you have SSH access to the host (runcrawl.php file should be executed) and setup cron job to do this on scheduled base.

I do have SSH access. We use a piece of software called "Putty" to access the site. By the way, I only need to do this once (vs. setting it up on schedule).

Unfortunately, I'm not very technical. Specifically what command would I type into Putty to execute "runcrawl.php"? Is there any way to stop the process once it starts if for some reason it causes a problem?

At the "Crawling" page of sitemap generator you will find a simple instruction on what to use to setup cron job/how to execute the script in ssh.

For instance:
https://www.xml-sitemaps.com/generator-demo/index.php?op=crawl

Cron job setup
You can use the following command line to setup the cron job for sitemap generator:
/usr/bin/php /home/xmlsites/public_html/generator-demo/runcrawl.php

And you can stop the crawler process any time using shell in the same way as any other process (using "kill" shell command).

You're being very patient with me. Thank you.

I'm now running the program though SSH. When I run it for 500 pages as a test, it runs perfectly. I then ran it for 25,000 pages maximum (as we have about 22,000). I ran it twice with the same results. Both times it ran for 19 minutes and then stopped. Nothing was output to the sitemap.

On the following page, I've placed (a) the permissions for the various files so you can see if that's right and (b) the output from running the program for 19 minutes:
[ External links are visible to forum administrators only ]

I then set it to save every 300 seconds. When I do that, the job just stops sooner and then restarts from the beginning.

Thanks

Hi,

first of all, you should disable "save run state" option if you execute it from command line (it just take additional load which is not necessary).

It's strange that it is interrupted with no error message displayed - what is displayed after the last line in your output dump (4980 | 16636 | 101,817.2 | 18:57 | 63:19 | 4 | - | 4976 | 99 | 0). The command line prompt is shown?

Well.. generally, I can suggest to update PHP version from 4.1.2 (your current one) to the more recent release. It is possible that there is some bug that appears when processing large data arrays or similar.

Hello,

I have the unlimited version and set up a cron job to run...I just noticed after 7 months that my sitemap.xml file has not been updated since 1/08...so I went in to the generator and tried to run it but when I go to the crawl page this is all I see....


No "RUN" button or anything...so I came on here and saw an update...uploaded the update, only index.php and "pages" folder and still nothing...

All this time I assume this was running in the background...

Thanks

Hello,

it looks like your server configuration doesn't allow to run the script long enough to create full sitemap. Please try to increase memory_limit and max_execution_time settings in php configuration at your host (php.ini file) or contact hosting support regarding this.

Is there something else that could be causing this problem? 

I have the following:

max_execution_time = 3000     ; Maximum execution time of each script, in seconds
max_input_time = 120   ; Maximum amount of time each script may spend parsing request data
memory_limit = 1024M      ; Maximum amount of memory a script may consume (8MB)

i ran it up this high just to see where it would start to work...and all the way at 1024 and 3000 it's still not working...these are extremely excessive values...

also I notice when I click on the tab "view sitemap"

I have this error down at the bottom
Fatal error: Allowed memory size of 20971520 bytes exhausted (tried to allocate 16 bytes) in /home/velocity/public_html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 40

Which is obvious that there is some memory_limit...but I can't see where it is....

It looks like PHP settings you quoted are not applied on your server, please try to create a phpinfo script in generator folder and open it in browser to make sure that those settings are applied.

No I restarted apache yesterday when I made the changes and check it with phpinfo and it confirms the new setting...

Did you put the phpinfo script in generator folder? Please let me know the link in private message.

I just wanted to check if you got my pm...I sent you one...but not sure if you got it because it doesn't show in my outbox..

Thanks