Hello,
In my robots.txt I have some folders that I do not want to be indexed by Google and by the sitemap.
The problem is that the generator can not pass the robots.txt file and so, even if my site has over 2.000 pages, the generator indexes only 35.
Is there a way to make the generator crawl all my pages except the ones mentioned in the robots.txt?
Thanks.
Guest