Sitemap Generator Forum

My generator stop running after about crawling 40-50k pages (site has about half million pages) and when I try to start it again, I don't receive any message, just thrown back to ssh command line.

I've increase memory to 1GB and running crawler from ssh.

Hello,

with website of this size the best option is to create a limited sitemap - with "Maximum depth" or "Maximume URLs" option limited so that it would gather about 200-300,000 URLs, which would be main pages representing "roadmap" sitemap for search engines.

Ok. But how to get 200-300k links when generator stops working on 40-50k and whatever I try, it just got back to command line without any report.

P.S. This 40-50k links are in depth of 3.

Hello,

please let me know your generator URL/login in private message to check this.