On Page SEO Checker: Get a list of improvements for your website for free in less than 5 minutes!
« previous next »
Pages: 1

yacosta

*
  • *
  • 1
Blocking specific pages/directories from getting crawled
« on: July 25, 2007, 06:55:42 PM »
Hi is there a way to block specific pages/directories from being crawled?
Logged

XML-Sitemaps Support

*
  • *****
  • 11781
Re: Blocking specific pages/directories from getting crawled
« Reply #1 on: July 26, 2007, 01:24:58 AM »
Hello,

yes, you can block them using robots.txt or just define them in "Exclude pages" option in Sitemap generator configuration.
Logged
Pages: 1
« previous next »