Crawl only URLs containing a specific path
« on: February 26, 2021, 08:10:08 PM »
Hello,

Is it possible to setup a configuration in the sitemap where it crawls and adds to the sitemap only a specific path in the url?

Lets say my url is [ External links are visible to forum administrators only ], but I want the crawler only to add the urls that contain [ External links are visible to forum administrators only ]
[ External links are visible to forum administrators only ]
[ External links are visible to forum administrators only ]

So the key here would be "blue-pets" in the url.

Is that possible?
Re: Crawl only URLs containing a specific path
« Reply #1 on: February 27, 2021, 05:33:59 AM »
Hello,

you can add "blue-pets/" in "Parse only" and "Include only" setting OR you can just use:
Code: [Select]
mydomain .com/blue-pets/ as a Starting URL.
Re: Crawl only URLs containing a specific path
« Reply #2 on: February 27, 2021, 09:21:11 PM »
Some parts of your message say "[ External links are visible to forum administrators only ] " How can I see it?