Competitor Analysis Tool: Analyze rival websites and gain valuable SEO and PPC insights
« previous next »
Pages: 1

pgmason

*
  • *
  • 1
How does XML-sitemap generator identify itself in logs?
« on: January 27, 2009, 11:49:13 AM »
Hi

How does the online generator identify itself in traffic logs?  What user agent name does it use?  I want to make sure we don't block it in our robots.txt file, as we typically block unidientified spiders.

Thanks

Pete
Logged

XML-Sitemaps Support

*
  • *****
  • 11783
Re: How does XML-sitemap generator identify itself in logs?
« Reply #1 on: January 27, 2009, 11:29:10 PM »
Hello,

sitemap generator User-agent header is:
Mozilla/5.0 (XML Sitemaps Generator 1.0)

It checks the disallow directives in robots.txt for googlebot and "User-agent: *".
Logged
Pages: 1
« previous next »