Sitemap Generator Forum

I am running the site map generator on our development server to test it, and no pages are being indexed.  It is my educated guess that it has to do with our robots.txt that disallows all robots.  I can fix this by adding the site map generator robot as User agent, but I don't know what name to use for the robot.  Can you help me with this?

Hello,

please check our search engine robot simulator to see if your site is "crawlable": https://www.xml-sitemaps.com/se-bot-simulator.html
In case if it's related to robots.txt, I'd recommend to update it to allow crawling.

Yes.  I know I need to update the robots.txt to allow the crawler, but I only want to allow the xml-sitemap generator crawler and no others.  In order to do that, I need the name of the crawler.

I already tested the simulator on our live site, and I know it works just fine.  This is a development site not available to the public or search engine crawlers.

Can someone give me the actual name of the crawler to reference in robots.txt?

You can manually disable (set to 0) 'xs_robotstxt' option in config.inc.php file for that.

Thank you.  That did the job.