Competitor Analysis Tool: Analyze rival websites and gain valuable SEO and PPC insights
« previous next »
Pages: 1

allen5

*
  • *
  • 11
Honour robots.txt ?
« on: January 06, 2011, 11:42:38 AM »
Hi,

In our site we have disallowed a form component through robots.txt, the script still crawls these pages and adds them to the sitemap.

Is there a way of excluded certain pages or honouring the robots.txt file?
Logged

allen5

*
  • *
  • 11
Re: Honour robots.txt ?
« Reply #1 on: January 06, 2011, 11:46:20 AM »
I think I found the answer, would I take the exclusion rules such as:

Disallow: /components/

and Add this to "exclude urls". It would be nice if it automatically honoured robots.txt
Logged

XML-Sitemaps Support

*
  • *****
  • 11781
Re: Honour robots.txt ?
« Reply #2 on: January 06, 2011, 09:05:10 PM »
Hello,

robots.txt exclusion should work as well.
Logged
Pages: 1
« previous next »