Sitemap Generator Forum

I'm not sure what is causing this, but take a look at the generated html sitemap here [ External links are visible to forum administrators only ]

Everything goes great, then suddenly the page title isn't used anymore. I can't figure out what is causing it. At first I thought it was because of a HTML 4.0 transitional page, but I excluded and no parsed the URL and I still get the problem.

Any ideas? Everything else seems to be working great.

Hello,

if you have some of the URLs defined in "Do not parse" option, they will have no title extracted, since pages are not fetched from the site.

That makes sense. However, the contents of my do not parse is identical to the contents of my excluded URLs. So unparsed URLs should not be in the final sitemap. 

I think I've found the problem, though. I had set crawl depth to 4. and that last level really flies through. Now I've set my crawl depth to 5 with the following result:

51 new URLs found - which are getting the title problem described above, and
the ones that did have a title problem are now fine.

So the problem seems to come from the way the last set level is dealt with in the crawl.

This experiment also means I need to rethink a couple of settings, not least of which is the depth of some of the pages!!

I ended up solving the problem by creating seperate xml sitemaps for .html pages and for non-.html pages. Trying to put it all into one sitemap wasn't solving the problem no matter how deep I crawled. Maybe a vBulletin thing 

Hello Dave,

in case if you have crawling depth limited, all pages from the last depth level are NOT fetched as well for better performance.