Competitor Analysis Tool: Analyze rival websites and gain valuable SEO and PPC insights
« previous next »
Pages: 1

coolbits

*
  • *
  • 2
Sitemap ignores the robot.txt
« on: June 19, 2019, 07:39:13 PM »
Sitemap ignores the robot.txt
In robots.txt prohibited:
Disallow: /*&v=list

Sitemap:
/catalog/teplye-kolgotki?page=2&v=list&s=hit&d=desc

Option "Use robots.txt" enabled, but do not work.
« Last Edit: June 19, 2019, 07:48:47 PM by coolbits »
Logged

XML-Sitemaps Support

*
  • *****
  • 11781
Re: Sitemap ignores the robot.txt
« Reply #1 on: June 20, 2019, 09:03:35 PM »
Hello,

please let me know your generator URL/login in private message to check this.

Also, you can add this in Exclude URLs setting:
v=list
Logged
Pages: 1
« previous next »