Sitemap Generator Forum

I ran into a few small issues that I would appreciate clarification on.

First - I had the system crawl my entire site, it was over 40,000 articles so it took a long time - ended up changing the depth because I had left it at unlimited. (But, it had reached 60,000+ which was odd).

Anyway - after I changed the depth to 100, now there's only 6519 pages crawled... (strange!)

This is an enormous article directory and has been live since 1998 - so obviously I'd like the articles to be showing properly.

Also - I told it to EXCLUDE the /cgi-bin/ directory, but it still provided many urls linked in it! THAT is a no no to me. So how do I get it to exclude completely the cgi-bin folders? I have over 120 categories that I want to be included... and not much else. Do I really have to manually add all 120+? I'd rather post the few "exclude" urls - it's faster, but obviously something isn't right.


Amber

Hello,

1. you can reverse the depth level to 150 in this case.

2. how did you specify it in Exclude URLs setting and what is an example URL that is still added in sitemap?