Sitemap Generator Forum

I have a number of urls in my dynamic (database driven) site, which if they have segments like the following, I want to disallow.

like: ?PHPSESSID=

as in the following
[ External links are visible to forum administrators only ]
[ External links are visible to forum administrators only ]
[ External links are visible to forum administrators only ]
[ External links are visible to forum administrators only ]
[ External links are visible to forum administrators only ]

and like: all/?orderby=

as in the following:
[ External links are visible to forum administrators only ]
[ External links are visible to forum administrators only ]
[ External links are visible to forum administrators only ]
[ External links are visible to forum administrators only ]
[ External links are visible to forum administrators only ]
[ External links are visible to forum administrators only ]

How do I disallow these "pages." I am tempted to enter into the disallow field strings like:
?PHPSESSID=
all/?orderby=
but I am not sure this will work. Please advise.

Hello,

yes, that should work. Add this to "Exclude URLs" input field in generator configuration.

Thanks. I tried it and the number of pages indexed were 1,004, which is less than the previous indexing. So I have to believe it worked.

Looks good in /sitemap.xml, however I am unable to find /sitemap_generator/data/urllist.txt or /sitemap_generator/data/sitemap1.html. I just converted to 2.0. Is there some setting I'm missing to create these files?

Perhaps you have disabled the "Create Text Sitemap" and "Create HTML Sitemap" at configuration page?

Ooooops! I thought I had checked those. Looking back, I see they were unchecked. Thanks.

One more question. I also have a robots.txt file. Will this conflict with the sitemaps as far as spiders are concerned? With sitemaps in place is a robots.txt file redundant and no longer necessary?

Hi,

robots.txt file is required to exclude some of your pages from being indexed by search engines. So, if you had it at your site, you should leave it there.
The sitemap generator script is checking robots.txt and doesn't included the disallowed URLs into site map.

Re: "The sitemap generator script is checking robots.txt and doesn't included the disallowed URLs into site map."

Aha! So because I have already disallowed URLs in sitemap generator, does it hurt to also disallow them in robots.txt as well?

What's the best strategy/practice here for Google, Yahoo and other search engines? Shall I keep robots.txt and NOT disallow in sitemap generator the URLs which I have already disallowed in robots.txt? Disallow URLs in both places?

Would disallowing ?PHPSESSID=, etc. work in robots.txt?

Hi!

You should keep your robots.txt file in place AND exclude the same pages in sitemap generator.

generally, "?PHPSESSID=" type of links are not excluded by robots.txt protocol
more details on robots.txt: [ External links are visible to logged in users only ]

Thanks admin.