Question - Will SiteMap Generator re-crawl the same pages?
« on: March 03, 2010, 03:11:24 AM »
Hello,

Beginner question.

I purchased the Sitemap generator, installed and ran it.  Over 40,000 pages were crawled.

Now my question is, when I run the crawl again, will it go over the 40,000 pages again?  

Because those 40,000 + are permanent, they won't be changed.  I  have a folder called "news_of_2010" and that is the only folder updated daily with new links.

Will Google still crawl the old pages? Will Sitemap run over the same pages over and over?
How do I make it only crawl the new folde rthat's updated daily, since the other 40,000 + files are permanent and non-editable.

Thanks in advance.

« Last Edit: March 03, 2010, 03:13:48 AM by webmaster267 »
Re: Question - Will SiteMap Generator re-crawl the same pages?
« Reply #1 on: March 04, 2010, 12:51:33 AM »
Hello,

you can keep existing sitemap unchanged, and create a second sitemap for that folder with new files (change sitemap filename and Starting URL accordingly)