Sitemap Generator Forum

Hallo

I have crawling my website every day for one week and each time I have added maximum paged with 2000.

Everytime the website pages are expanding and growing and now I have:

25259 pages
Processing time: 28445,05 s
Pages size: 2,181.91Mb
And now the staus is:
006-07-09 08:56:28, URLs added: 25295, estimated URLs left: 92495)

Why does it newer end?

Best regards
Staffan /Sweden

Hello,

do you mean that the crawling stops at this point?
If so, your server limits the maximum script execution time and/or memory for the scripts and you modify your php configuration:
increase memory_limit and max_execution_time settings in php.ini and restart apache.

also discussed here: https://www.xml-sitemaps.com/forum/index.php/topic,318.html

Hallo again

Now I have crawled one domain for 2 days. I have put the configuration to "unlimited time" and save every 3600 sec. Everytime it stops and are not finished and everytime there is still 25000 more pages to crawl. It looks like it will never end. Is it some kind of loop or what?

Best regards
Staffan

Hello Staffan,

the number of pages depends on your site fully, do you mean that there is much less than 25000 URLs at your site?
Did you increased the memory_limit and max_execution_time settings in php.ini ? (the script will not work for unlimited time in case if server limits it).

Hallo again

Yes, we increaded the memory limits and the time. We have about 10.000 URL:s on the site. We have tried to let it crawled on the smallest part that is about 2000 URL:s. It took 3 days before the limits of 512 Mb was ending the program.

It seem it will only crawl round and round.

What shall I do?

Best regards
Staffan

Hello Staffan,

please let me know your generator instance URL via private message.

Hallo

My computer guy says: "There is no time limit since its being executed from CLI PHP, the memory limit is set at 512MB and is reached after approx 3 days of crawling"

What do you want to know in a private message? (is that your e-mail address?)

/Staffan

Hello,

it definitely should not take 3 days to crawl 2000 URLs and this amount of pages should not take 512Mb of memory, that's why I want to check your configuration. I need the URL of your sitemap generator instance so that I can see its config in browser.

you can send me a private message here: https://www.xml-sitemaps.com/forum/index.php?action=pm;sa=send;u=1

I am having what seems to be the same issue.  Crawler runs and runs - even if I tell it to timeout after 90 seconds... but nothing is getting saved.
:-(

Hello Josh,

replied to your private message.

I was having the same problem with my sitemap not completing i.e. stopping near the end

I set an interrupt of 60 secs per 1000 requests and it now completes successully 12,000+ pages.

Steve

Suggestion:
Docs say "When the generator script is running (either with cron or using "Run in background" feature), you will see it's progress state on "Crawling" page. There you will also find the link to stop the script, which is very useful for big sites because you don't have to wait until it is finished if you want to modify the configuration and re-run the script."

That STOP link should be prominent on every script page, regardless of mode = cron, background, or foreground.
Right now, I've canceled a background run, restarted manually, and see no such STOP link.

Hello,

thank you for suggestion.

I have the same problem. Crawling goes on forever and there's no stop button.

You can manually upload empty "interrupt.log" file into data/ folder to stop the generator.