I have a blog website which generates articles from database.
Every url in my site is like this:
<a href src="/xxx"> having a trailing slash (/) at the start, to determine the root.
I have urls like:
/post/article-titlerewritten from
post.php?id=article-titlewith .htaccess
and so we have:
the actual link of a file is this: /contact.php
the generator, crawls /contact.php, /post/contact.php, /xxx/contact.php and so on.
ATM I have to exclude those links from getting indexed.
Guest