I recently added a redirection to the main site that redirects to a mobile version. For some reason, the crawl of the site aborts because the redirect occurs i.e. the crawl is being identified as coming from a bot. Can you tell me the USER_AGENT id that is used by the generator? I turn it off (the redirect) to generate the map now, but for that time the redirect is disabled. Not that it is a long time but it would be nice simply have a check to avoid the problem.
Thanks. Jack
Guest