Sitemap Generator Forum

hi!

all my pages i don't want to be in the sitemap.xml have a
<meta name="robots" content="noindex,nofollow" />
tag

but some of them are in the sitemap.xml.
is there something wrong with the meta tag or how does this work?
thanks
mat

Hello,

what is an example URL that is included? Don't you have it specified in "Doi not parse" setting in sitemap generator configuration? (in this case it's still included)

hi!
an example would be [ External links are visible to forum administrators only ]
it has the meta tag in the source code: robots noindex, nofollow.

i automatically include this meta tag when a category page has 0 results. i don't want the searchengines to index these pages and so i don't want them to be in the sitemap too.

i can't add them manually to the exclude filter as this meta tag is only on pages with 0 results and they can differ from time to time as categories may have items in it in the future and others that have items now will not have them in the future maybe.

so the question is, does the generator look at the meta-tag, or ist the robots info on the page irrelevant. as i read, that robots.txt rules are followed by the generator i hoped that the page specific meta-tags are too

thanks for your info about this!
mathias

when i look at this post https://www.xml-sitemaps.com/forum/index.php/topic,1868.html
it seems that the meta robots tag should also work for the generator
why is it not working for me?
m

Hello,

I just checked the URL you mentioned above and I don't see the robots meta tag in it's html source, please check that.

alright, try this one [ External links are visible to forum administrators only ] - i see the meta tag in the source code
maybe the other page was updatet in the meantime, i don't know any more which url it was as it his just visible for admins
thanks for recheck!

Possibly the meta tag was not there at the moment when sitemap generator was crawloing your site? You can regenerate sitemap to check that.

i rechecked. the tag was there... what can the problem else be?

You can regenerate sitemap to check that.

yeah, you already said that!
i can recheck and find what it is. i just want to knwo if it is a feature of the generator to automatically exclude pages with a robots noindex attribute. if it should work like this - i will find out as the problem seems to be on my side...

Yes, definitely - sitemap generator excludes pages with robots noindex meta tag (unless you have those pages specified in "do not parse" setting, as mentioend above)