Standalone Sitemap Generator will not crawl site
« on: March 27, 2011, 09:39:37 PM »
Hello - when I click the Run button for starting the generator it says Completed but does not crawl any pages - shows a return as follows:

Sitemap details
Request date:
27 March 2011, 20:26
Processing time:
0:00:01s
Pages indexed:
0
Sitemap files:
0
Pages size:
0.00Mb

Our website has about 700-800 pages.

The sitemap is there but dates back to February 2011.

Can anyone help? Ive checked all the permissions for the relevant folders.
Re: Standalone Sitemap Generator will not crawl site
« Reply #2 on: March 28, 2011, 12:36:41 PM »
Hello - discovered the problem. at some stage my robots.txt file on my site for some unknown reason changed from

User-agent: *
Disallow:

to

User-agent: *
Disallow: /

Which meant that Sitemap Generator could not access any files! not entirely sure why this was the case but would recommend anyone having the same problem to check the robots.txt file...