Crawling Fails on https Website with 400 Bad Request
« on: September 29, 2019, 05:34:10 AM »
Converted a site to https a while back using an htaccess rewrite condition. All good. However, on attempting to run the generator I now repeatedly get the following error.  Note: I did try and remove the htaccess rewrite and am still seeing failures. Any help appreciated.

There was an error while retrieving the URL specified: [ External links are visible to forum administrators only ]
HTTP Code:
HTTP/1.1 400 Bad Request
HTTP headers:
server: nginx
date: Sun, 29 Sep 2019 04:29:01 GMT
content-type: text/html
content-length: 166
connection: close
x_csize: 166

HTTP output:
400 Bad Request 400 Bad Request nginx
Re: Crawling Fails on https Website with 400 Bad Request
« Reply #1 on: September 29, 2019, 07:39:15 AM »
Hello,

please make sure that you are running the latest version of sitemap generator (v8.0)
Re: Crawling Fails on https Website with 400 Bad Request
« Reply #2 on: September 29, 2019, 09:01:33 PM »
Downloaded and running now.  Looks good so far.  Will let you know.
Re: Crawling Fails on https Website with 400 Bad Request
« Reply #3 on: September 29, 2019, 09:22:20 PM »
Hmmm... just got this on crawling page.

The webpage at [ External links are visible to forum administrators only ] might be temporarily down or it may have moved permanently to a new web address.

Attempting to navigate to configuration page failed.  Opened Chrome Incognito tab and could get to options but don't see anything obvious to me. Looks like browser is still working though on both tabs.  Worth mentioning that I did follow all of the PDF instructions, double-checked permissions, etc.
Re: Crawling Fails on https Website with 400 Bad Request
« Reply #4 on: September 29, 2019, 10:28:25 PM »
Was eventually able to get to crawling options, set it to keep running as a process.  Just got this again:

The webpage at [ External links are visible to forum administrators only ] might be temporarily down or it may have moved permanently to a new web address.

Thoughts?
Re: Crawling Fails on https Website with 400 Bad Request
« Reply #5 on: September 30, 2019, 05:10:52 AM »
Hello,

it looks like your server configuration doesn't allow to run the script long enough to create full sitemap. Please try to increase memory_limit and max_execution_time settings in php configuration at your host (php.ini file) or contact hosting support regarding this.