Excluding Page & Sort URLs
« on: May 06, 2011, 10:28:58 PM »
Hi,

I understand how to exclude folders when they come right after the main domain in the URL. For example, if I want to exclude [ External links are visible to forum administrators only ] I know to exclude /privatecontent. However, how do I exclude crawling content (such as pagination & sorts) that do not easily fall into that category?

For example, I DO want to crawl [ External links are visible to forum administrators only ]. But I don't want to crawl [ External links are visible to forum administrators only ] or
[ External links are visible to forum administrators only ].

How can I exclude URLs that contain /sort/title or sort/created_datetime.

Another example: I do want to crawl [ External links are visible to forum administrators only ].
But i don't want to crawl [ External links are visible to forum administrators only ]
or
[ External links are visible to forum administrators only ]

How can I exclude URLs that contain name_ASC or gender_ASC?

Thank you,
Jason
Re: Excluding Page & Sort URLs
« Reply #1 on: May 07, 2011, 08:24:58 AM »
Hello,

it can be achieved by adding this in "Exclude URLs" setting:
Code: [Select]
sort/
_ASC
that will remove all links that have that included as a substring.