When a site uses relative urls, the generator can get stuck in a loop because it finds pages that don't exist because it ignores the <base href="[ External links are visible to forum administrators only ]"> tag.
i.e.
you are at the URL "[ External links are visible to forum administrators only ]"
there are links to just "info.asp" "contact.asp" which do not exist in /cat/sub/
The crawler detects [ External links are visible to forum administrators only ] and contact.asp as a URL.
I have asked our developers to change "info.asp" to "/info.asp" which is how it should be done for good code, but they argue that <base href="[ External links are visible to forum administrators only ]"> is valid code.
How can I turn on base url support in XML-Sitemaps?
Due to this issue, the crawler hits the site of 700 odd pages, and finds 90,000+ when running 4 levels deep.
Anyhelp please? other than to fire the developers!?!