Hi,
I understand how to exclude folders when they come right after the main domain in the URL. For example, if I want to exclude [ External links are visible to forum administrators only ] I know to exclude /privatecontent. However, how do I exclude crawling content (such as pagination & sorts) that do not easily fall into that category?
For example, I DO want to crawl [ External links are visible to forum administrators only ]. But I don't want to crawl [ External links are visible to forum administrators only ] or
[ External links are visible to forum administrators only ].
How can I exclude URLs that contain /sort/title or sort/created_datetime.
Another example: I do want to crawl [ External links are visible to forum administrators only ].
But i don't want to crawl [ External links are visible to forum administrators only ]
or
[ External links are visible to forum administrators only ]
How can I exclude URLs that contain name_ASC or gender_ASC?
Thank you,
Jason