Competitor Analysis Tool: Analyze rival websites and gain valuable SEO and PPC insights
« previous next »
Pages: 1

ajgilbertson

*
  • *
  • 2
Crawler reading Calandar links
« on: May 01, 2016, 06:45:03 PM »
How do I go about stopping the crawling of calendar years that are not even here yet?
Example 2020, 2022. I am showing a link depth of 91 currently "Current page: calendar/2023/09/21/"
Logged

XML-Sitemaps Support

*
  • *****
  • 11781
Re: Crawler reading Calandar links
« Reply #1 on: May 03, 2016, 09:47:17 AM »
Hello,

you can add them in Exclude URLs setting:
calendar/2022/
calendar/2023/



or
calendar/202
Logged

ajgilbertson

*
  • *
  • 2
Re: Crawler reading Calendar links
« Reply #2 on: May 03, 2016, 01:37:10 PM »
Thanks -
I will give this a try.
The way it is going I may want to exclude the whole Calendar. Crawling the year 1923 now.
Logged
Pages: 1
« previous next »