Sitemap Generator Forum

In this topic, we were advised that an entire folder can be excluded:

https://www.xml-sitemaps.com/forum/index.php/topic,200.html

...such as:

[ External links are visible to forum administrators only ]

...but it appears that the crawling still includes the folder, thus taking a long time to crawl the folder anyway.

Is this right? An excluded folder is crawled but not included in the sitemap?

Don Strack

Hello Don,

you should include it in both "Do not parse URLs" and "Exclude URLs" options for this.

Thanks. But I must be doing it wrong. The folder is still being crawled. And takes at least 10 times as long because it is crawling this folder.

I have included the following in both the "Do Not Parse URL" field, and in the "Exclude URL" field:

[ External links are visible to forum administrators only ]

Apparently, a wildcard does not work either.

Please help.

Don Strack

It's working now, so to close this thread, here is what I learned.

I was adding the entire "http:// mydomain.com/folder/" address, instead of just the folder, like this: "folder/".

I did a search on "exclude", rather than "folder" and found a link to the documentation.

Now I have some issues with the markup of the html sitemap, but that's for another thread.

Don Strack

Great, thank you for the follow-up!