Sitemap Generator Forum

I ahve installed the program and it seems to work to some degree. But it does not cover the whole site. I try to tell it not to look in the cgi-bin but it seems to have a mind of it's own. Hope someone can help.

thanks


Dan

If you use the latest version, you can add a robots.txt file to the root and the crawler should obey what you restict in there...

hope this helps

Hi,
I want to confirm what raramuridesign just said about robots.txt - the script works as the Google crawler does for your site.
But it's also possible to add special restrictions at the generator configuration page: "Exclude files" textarea input lets you define the exclusion substrings, like "cgi-bin/".

Hello Again:

Perhaps I didn't explain myself well. I would like to index my whole site but it doesn't. Only 107 of over 1000 files. The fact that it insists on indexing my cgi-bin is an annoyance because the program gives me the impression it can handle that

Hi,

please let me know the url to your site either here or in PM.

Hello:

The site is [ External links are visible to forum administrators only ]

I tried the generator to crawl your site and seems to work ok. Are you sure you didn't added anything into 'Exclude URLs' field? (in this case the part of urls is not included in sitemap) Otherwise please let me know the url for your generator instance (via PM)  so I can check you confi settings.