Sitemap Generator Forum

Hi,

I am trying to run XML Sitemap crawler on our non-production website. But for some reason it doesn't get any results. I think it is because our non-production robots.txt is not allowed to be crawled. I modified the robots.txt file as below but still not able to crawl the site using the XML Sitemap crawler.  What am I doing wrong here?

User-agent: XML-Sitemaps
Disallow:

User-agent: *
Disallow: /

Hello,

sitemap generator checks for "User-agent: *" and "User-agent: googlebot" entries in robots.txt.
You can set xs_robotstxt setting in generator/data/generator.conf file to "0" to avoid this.

Hi,

I made the suggested changes and the crawler ran fine. But, I am running into another issue. My purpose was to crawl the site to detect all the Canonical URL's on the website. I checked "enable canonical URLs" under Advanced Settings and then ran the crawler. It just gave an xml sitemap file with all the URL's it detected. I dont see a seperate coloumn which lists the canonical URL's detected. Does this tool provide the output of the Canonical Link for all the URL's that it detected?

> Does this tool provide the output of the Canonical Link for all the URL's that it detected?

There is no separate field for that. In case if generator find canonical meta tag on your page, it will use that canonical URL instead of the link that is currently crawled.