Sitemap Generator Forum

Hi,

I tried to run the script to generate an XML sitemap for my site but get the following error,

There was an error while retrieving the URL specified: [ External links are visible to forum administrators only ]
HTTP Code:

HTTP headers:
x_csize: 0

HTTP output:


currently, my site is blocked by robots.txt because of some reason.
It can be a problem to crawl for XML Sitemap generation?

Please, let me know.

MJ Kang

Hello,

yes, sitemap generator respects rules defined in robots.txt and will not crawl the site if that is disallowed.