Sitemap Generator Forum

I just moved my site from an aging dedicated server to a cloud server.  I have total server control and can change anything I need.

After moving I noticed maybe one page every 30 seconds was being indexed and the process would stop after a few hours.  On the old server it would scan my 50,000 pages fast, stop at 40,000, but then I would restart it and it would finish up.  Total time would be maybe 6 hours.

I read through the and it seems the PHP.ini file needs to be tweaked.  I changed my settings to:

max_execution_time = 86400
max_input_time = 86400
memory_limit = 1024

But this had absolutely no effect.  All settings in the program are exactly the same as before.  Any ideas?

Hello,


The crawling time itself depends on the website page generation time mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page, then 1000 pages will be crawled in about 16 minutes.

Some of the real-world examples of big db-driven websites:
about 35,000 URLs indexed - 1h 40min total generation time
about 200,000 URLs indexed - 38hours total generation time

I reset the items in my php.ini and it is not stopping but it is so slow.  I restarted it from where it left off this morning and here are the stats:

Links depth: 3
Current page: articles.detail.php/16160/209/Finance/3/If_You_Want_to_Make_Money
Pages added to sitemap: 18289
Pages scanned: 29180 (466,023.5 KB)
Pages left: 31282 (+ 27393 queued for the next depth level)
Time passed: 227:08
Time left: 243:29
Memory usage: 84,780.0 Kb

On my last, much slower server, it would have been finished adding about 49,000 pages to the site map hours ago.  What can I do to speed it up?  The site is not slow at all.  According to my Pingdom server monitoring my average page load time is 572 ms

Hello,

please let me know your generator URL/login in private message to check this.

It is still going.  I'll get you the log in details but after many many hours it seems close to finished.

Links depth: 7
Current page: articles.detail.php/3126/189/Motivational/Publishing/45/E-Bay_Gets_from_Sellers
Pages added to sitemap: 49878
Pages scanned: 114520 (1,312,408.4 KB)
Pages left: 2645 (+ 13149 queued for the next depth level)
Time passed: 1676:00
Time left: 38:42
Memory usage: 137,108.9 Kb

I don't know if you did anything but my site maps fly now:


Links depth: 4
Current page: articles.detail.php/1802/180/Home_Business/Business/1/Success...Bottoms-Up
Pages added to sitemap: 60679
Pages scanned: 60680 (93,919.2 KB)
Pages left: 12277 (+ 0 queued for the next depth level)
Time passed: 74:20
Time left: 15:02
Memory usage: 103,453.3 Kb

1 hour 14 minutes ans 60,679 pages added.  nice

Great, I'm glad it works better now!