Sitemap Generator Forum

Hi bought the stand alone gen yesterday. After much desifering of the instructions maneged to install it. I set it up with the default settings in the set up section and the go to the crawl page click the run in back ground, set if off and it starts crawling but when it get to around the (is it depth)? 3 it stops working. when i goto the xml file theres nothing in it. any clues how to fix would be good
thanks JC

Hello,

as discussed in this topic: https://www.xml-sitemaps.com/forum/index.php/topic,124.html,
there are the following options in this case:
- increase max_execution_time setting in php.ini file at your server and restart apache
- execute sitemap generator from command line (if you have ssh access to your server)
- use "Save state" option at Sitemap Generator configuration page and execute the crawler multiple times with "Resume generation" enabled until full sitemap is created
- limit the number of pages to index in Sitemap Generator configuration

HI Oleg,

Thank you for installing the program for me. You apparently got the program to run throught the entire process on 04/24/09, unfortunately I am suffering the stalling phenomenon.

I increased the run time to 180 seconds and memory to 128 in the php.ini file, my host said I do not need to restart apache. It stalled again after 5 minutes. I will continue to restartt he program manually through the day but I would really like to find a more elegant solution.
--

I set the chron job with my hosts help, that is not scheduled to run until a few days from now.
--

In trying to run the program from the terminal (ssh) with my hosts tech support person Tom holding my hand (at 7:00 AM on a Sunday morning (Thanks, Tom)) we got this:

-bash: runcrawl.php: line 2: syntax error near unexpected token `base64_decode'
-bash: runcrawl.php: line 2: `eval(base64_decode('Pz48P3BocAokb3AgPSAkX1JFUVVFU1RbJ29wJ10gPSAnY3Jhd2xwcm9jJzsKaWYoaXNzZXQoJF9TRVJWRVJbJ1JFUVVFU1RfTUVUSE9EJ10pKQp7CmVjaG8gJ1RoaXMgdG9vbCBjYW4gYmUgZXhlY3V0ZWQgaW4gY29tbWFuZCBsaW5lIG1vZGUgb25seSc7CmV4aXQ7Cn0KJFhOVWZDTndoWTRUID0gdHJ1ZTsKY2hkaXIoZGlybmFtZShfX0ZJTEVfXykpOwokX1JFUVVFU1RbJ2JnJ10gPSB0cnVlOwokX1JFUVVFU1RbJ3Jlc3VtZSddID0gdHJ1ZTsKaW5jbHVkZSAnLi9pbmRleC5waHAnOwo/Pg=='));'
My host said this is beyond their ken and that I should refer back to you. The tech did tell me to specify that they use a Linux based system and said that might be important.

That's all I know. Do you have further instructions or requests for more information?

Thanks,
Dean

Hello,

make sure that you include "php" in command line when running in SSH, like:
/usr/bin/php /path/to/generator/runcrawl.php

I did so and this is what was returned to me.

[~/public_html/generator]# . runcrawl.php
-bash: ?php: No such file or directory
-bash: runcrawl.php: line 2: syntax error near unexpected token `base64_decode'
-bash: runcrawl.php: line 2: `eval(base64_decode('Pz48P3BocAokb3AgPSAkX1JFUVVFU1RbJ29wJ10gPSAnY3Jhd2xwcm9jJzsKaWYoaXNzZXQoJF9TRVJWRVJbJ1JFUVVFU1RfTUVUSE9EJ10pKQp7CmVjaG8gJ1RoaXMgdG9vbCBjYW4gYmUgZXhlY3V0ZWQgaW4gY29tbWFuZCBsaW5lIG1vZGUgb25seSc7CmV4aXQ7Cn0KJFhOVWZDTndoWTRUID0gdHJ1ZTsKY2hkaXIoZGlybmFtZShfX0ZJTEVfXykpOwokX1JFUVVFU1RbJ2JnJ10gPSB0cnVlOwokX1JFUVVFU1RbJ3Jlc3VtZSddID0gdHJ1ZTsKaW5jbHVkZSAnLi9pbmRleC5waHAnOwo/Pg=='));'

well I tried to manually run the site map but I got an out oif memory error.

Links depth: 10
Current page: gallery/displayimage.php?album=topn&cat=15&pos=1320
Pages added to sitemap: 31010
Pages scanned: 31080 (538,140.0 KB)
Pages left: 404 (+ 6140 queued for the next depth level)
Time passed: 135:32
Time left: 1:45
Memory usage: 52,001.5 Kb
Resuming the last session (last updated: 2009-05-17 16:21:35)
Fatal error: Out of memory (allocated 106954752) (tried to allocate 71 bytes) in /home4/**********/public_html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 6

It's strange Oleg because you ran it just fine. I didn't change any of the settings from when you installed so I am at  a complete loss.

I hope you can shed some light on this perplexing problem.

I ran the script as a Chron job last night and it apparently worked to some extent.

Unfortunately I find there are now three site map files sitemap.xml, sitemap1.xml and sitemap2.xml the sizes are 1k, 9.19m and 4.34m respectively where the sitemap you generated on installation was only 1.88m

The ror.xml file is 24.4m but the one you generated was 2.14m

Oleg, I haven't changed any of the settings of the sitemap generator from what it was when you installed it. I made changes to the php.ini file as specified earlier in this thread.

I remembered that I had to dlete a SEO plugin for my photo gallery because it's .htaccess file, which rewrote the urls, caused conflicts with the 301 redirects I had to install.

Could this naming convention change for the 1000's of URLs in my galley have cause the size increase in the sitemap files?

If so then that situation can be considerd resolved, however we still have the manualk function and the ssh script failure that I would like to address and resolve.

Thanks, Oleg

Dean

Hello,

yes, it's possible that you recent changes resulted in significant increase in number of URLs and sitemap generator has splitten your sitemap on multiple files as a result.

Yes, removing the SEO freindly .htaccess file was the cause for the increase in size. I played with the setting sof the program and found an acceptable compromise between program function and which files it catalogs.

In the process I discovered the reason the generator was locking up in manual mode was because of the vast size of my site. Oleg could probably explain the technical reasons for this, all I care about is the functional aspects because I'm not a tech. Regardless restricting the number of directories caused the program to run fast and effectively, I am glad to know that it functioned wonderfully as a Chron Job even when I had it indexing the entire 58K page site.

The script issue in ssh mode has not been tested using a smaller sampling but I don't have a need or time for that.

The end result is that this problem was solved by restricting the number of directories indexed.

This is a good program, Oleg. I thank you for writing/developng it and look forward to gaining t=the advantages it provides for many years to come.

Dean

[ External links are visible to forum administrators only ]

Thank you for your kind words, Dean!

Hello,

Just bought the unlimited a day or so ago. Having the same problem where there is activity and then it seems to stop. I left it running overnight and it did not go past where it stopped. Best I can tell from the Forum, the problem is that the server ran out of resources.

I am running on a shared managed server so changing server configuration is not an option. The only possible solution is to change your program settings.

What do you suggest changing?

I thought about reducing the link depth but can't find a setting on your configuration to do that.

Hello,

you can limit maximum crawling depth in "Crawler Limitations, Finetune" setion on Configuration page (click section title to expand the settings list)