Sitemap Generator Forum

Hi i was hoping someone could recognize what is wrong with my setup,

The server is all setup and seems to be fine, crawl settings save and kicks off with log populating as it runs through the site but it doesn't complete.   

I have tried many times with different tweaks, permissions have been left very open just to try and get around it but it just gets to 'Saving Dump' and does nothing, no error, no sitemap created , the gui still says crawl is in progress but if i click off the page and back (with the do not interrupt box checked) it shows details under resume last session , but trying to resume also produces the same issue.

It is writing data to the ./data directory in a crawl file but nothing is generated, i have tried several output types but same result.

The only log i can see if the debug log , which is shown in the progress dialog box so no further info from that unfortunately

I think it runs for about an hour progressing pretty quickly and gets to about 50MB memory according to logs  , the server is a 2 x vcpu with 3.75GB ram , Amazon Linux AMI


Any help much appreciated

Thanks

Hello,

sitemap files are created only after the crawling is completely finished. You need to run generator again with "Resume" checkbox enabled until the process is completed.

Thanks i have tried this but nothing seems to happen,

monitoring the system processes i see no relevant activity, previously while running it was showing live progress on crawl tab and the php processes were visibly working

It says its in progress but log doesn't go further than saving dump , and if i click of and back onto the crawl page (ran with the 'do not interrupt' option) it shows it stopped again.   

I did even leave it over night even though it didn't appear to be doing anything , no progress by the morning , no further entries in the log.

Doesn't it do the 'saving dump' at the end of the script ?

it seems to me this command is having difficulties ,  im not sure what it could be though as permissions are open.

is there anything else i can provide to help work it out ?

Thanks

Hello,  i

"Saving dump" is done at specified intervals ("Save state" setting in generator configuration).

I would recommend to run generator in command line if you have ssh access to your server.

Thank you for the advice, running via command line highlighted the issue - script had maxed out allowed memory.

For reference here is how i fixed this

from the generator directory run
sudo php ./runcrawl.php [ External links are visible to forum administrators only ]

This displayed the error when it stopped
PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 18568240 bytes) in /ebs/var/www/generator/pages/class.utils.inc.php on line 102

PHP limits script memory to 128MB as default , adjust it in here  (i upped mine to 1024 just to be sure , its a dedicated server)

edited memory setting in /etc/php.ini   (default location)



Its running now and progressing further so thanks again !

I'm glad it works now.