Sitemap Generator Forum

I just want to index our root web. For example: rootweb.com but when I try the free site generator it also indexes all of our subwebs and even our forums (exceeding 500 pages) all located in our public_html directory.

How do I prevent (exclude) subwebs from being indexed? I couldn't see how to set the "parameters."

Thanks,

Kerry

Hello,

using free online sitemap generator you can exclude them with robots.txt file on your server.

With standalone generator version you can define that in configuration.

Hi admin,

Just to be clear: If I add "exclude" certain pages and subwebs in a robots.txt file on my server and then run the  free site generator that it I won't get the "exceeded 500 pages" message?

Thanks,

Kerry

Update:

If I add subwebs in my robots.txt file doesn't that mean that those subwebs won't be searched/indexed  by search engines?

Also, where do I find the "parameters" option mentioned in step 1:

Enter your full website URL and some optional parameters in the form below.

Perhaps that doesn't apply for the free version?

Kerry

Hello,

if you exclude certain URLs in robots.txt then search engines will NOT index them as well, so you should be very careful with that option.

With free online generator the only optional parameters are: frequency, last modification time, priority attribute.

The standalone script is a much more advanced tool, you can check the demo here: https://www.xml-sitemaps.com/generator-demo/

Thanks Oleg,

I understand.

Kerry