Sitemap Generator Forum

Hello. For some reason my unlimited sitemap generator crawls my forum and store sub directories, but not my articles and blog sub directories. They are linked on the home page just like the rest of my directories, but they just will not show up in my sitemap. Also, my articles and blog sub directories have an htaccess file in them, that re-writes URL's to be more search engine friendly. Could this be why?

Any help would be appreciated, thank you.

Sincerely,
Michael Cottier

Hello,

please make sure that you don't have them excluded in robots.txt or via <meta name="robots"> tag in html sorce code.

Well on all of my pages I do not have any meta name="robots" tag, and in my robots file I have this:

User-agent: *
Disallow: /cgi-bin/

Do you know what I should do or change to fix it?

Please let me know your generator URL and an example URL that is not included in sitemap via private message.