Competitor Analysis Tool: Analyze rival websites and gain valuable SEO and PPC insights
« previous next »
Pages: 1

soulesschild

*
  • *
  • 1
Crawling pages that require authentication
« on: May 09, 2012, 12:22:01 AM »
Is there a way to set the crawler up in a way that it can use a user name/password to crawl through my site that has specific authenticated pages that require login?

Thanks
Logged

XML-Sitemaps Support

*
  • *****
  • 11782
Re: Crawling pages that require authentication
« Reply #1 on: May 10, 2012, 08:44:16 PM »
Generator bot crawls pages similar to search engine bots, so it can't access pages if they are password protected.
Logged
Pages: 1
« previous next »