Sitemap Generator Forum

I placed the script on one of my servers for a very-very large website that I have (with lots of data). I ran the script and it took up my entire servers virtual memory and overloaded the server. I had to (it took time because the server had 99% of it's virtual memory full) kill the process and restart the server.

Keep in mind that I have around 40 million pages that need to be crawled. Is there anything in the setting where it allows me to specify the maximum amount of memory the script is allowed to use on the server.

How long do you think this would take to crawl and build a sitemap for 40,000,000  links/pages? Days? Weeks?

It might take weeks to create sitemap of that size with the crawling method - sitemap generator crawls your site similar to normal visitors, so you should multiple the number of URL with the average time that it takes to open the page on your server (with no images/css). It would make more sense to create a smaller sitemap with the most important URLs included.

Thanks,

The main problem was the script overloading my server and taking up a good 90% of the virtual memory. Is there a file where I can specify a memory limit, etc.?

You can specify the maximum memory_limit in PHP configuration (php.ini).