robots.txt file
« on: August 09, 2017, 03:29:45 PM »
I have just installed version 7.2 on my subdomain. When I  run the generator, it runs for about 2 seconds and then finishes and no urls in the .xml file.
I have looked around your forum and it looks like the issue is with my robots.txt file which disallows any crawling with this
User-agent: *
Disallow: /
Is there a way to stop bots crawling my site but allow the generator to run?

Thanks
Re: robots.txt file
« Reply #2 on: August 10, 2017, 09:56:34 AM »
Thanks for that, I didn't know about that setting.

I found 'xs_robotstxt' => '1' in the config file so I have tried changing it to '0' and '' but still the same issue

Thanks
Re: robots.txt file
« Reply #3 on: August 10, 2017, 12:10:56 PM »
Hello,

please let me know your generator URL/login in private message to check this.
Re: robots.txt file
« Reply #4 on: August 10, 2017, 12:33:33 PM »
I have sent you a private message as requested
Re: robots.txt file
« Reply #5 on: August 15, 2017, 11:35:05 AM »
Have you had a chance ye to look at this issue ?
Re: robots.txt file
« Reply #6 on: August 15, 2017, 10:27:59 PM »
Hello,

a reply was sent to you the same day, please check the inbox.