is this normal ??
« on: August 30, 2008, 06:07:55 PM »
Links depth: 3
Current page: search-Bios Utilities.html
Pages added to sitemap: 10000
Pages scanned: 12645 (570,260.8 KB)
Pages left: 0 (+ 35537 queued for the next depth level)
Time passed: 202:31
Time left: 0:00
Memory usage: 71,442.9 Kb
Completed
Total pages indexed: 10000

i own a very strong server but look at the time passed ! and memory usage !!!

p.s. i just include some certain links not all and not all links just 10,000 !!!

is it normal ?!
Re: is this normal ??
« Reply #1 on: August 31, 2008, 09:55:18 PM »
Hello,

sitemap processing time depends on the page generation time mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page (it may be less or more), then 1000 pages will be crawled in about 16 minutes.

Some of the real-world examples of big db-driven websites:
about 35,000 URLs indexed - 1h 40min total generation time
about 200,000 URLs indexed - 38hours total generation time
Re: is this normal ??
« Reply #2 on: September 01, 2008, 12:05:20 PM »
so compare with about 35,000 URLs indexed - 1h 40min total generation time

mine is not going well , right ?
10,000 URLs indexed Time passed: 202:31min

is it about the server power ? or is it because of the bad SEO of my websites ?!

do u mind if i send you the url and admin panel to check these things if u think something should be fix somewhere?
thanks
Re: is this normal ??
« Reply #3 on: September 01, 2008, 08:41:41 PM »
It's not related to "bad seo of your website", but rather the time it takes to generate page on your site. You can get a rough idea on how long it takes to download your pages (html only) using http headers viewer (check "Download time:" row).