Google have 11 000 URL restricted by robots
« on: May 29, 2011, 08:27:56 AM »
I have 11 000 url restricted by robots is that normal?


Google webmaster is blocking this kind of threads


/tags.php?tag=investment+in+thailand URL restricted by robots.txt

/threads/2207-Obama-Visit-To-Indonesia?p=2389 URL restricted by robots.txt
Re: Google have 11 000 URL restricted by robots
« Reply #1 on: May 29, 2011, 09:06:40 AM »
Hello,

if you don't want those pages to be indexed by search engines, you can keep the exclusion in robots.txt.
Re: Google have 11 000 URL restricted by robots
« Reply #2 on: May 29, 2011, 12:01:47 PM »
Yeah i know but should i?
Re: Google have 11 000 URL restricted by robots
« Reply #3 on: May 30, 2011, 06:51:20 AM »
Perhaps you would want to index tags.php pages, as for the second example it looks like a duplicate URL for the topic.