Another piece of info
this comes up in the log file once the crawling is done, which btw takes hours???
</style>
<title>502 Bad Gateway</title>
</head>
<body bgcolor=#ffffff link=#CC0D2F vlink=#CC0D2F alink=#CC0D2F>
<table width=400>
<tr>
<td>
<h1>Bad Gateway</h1>
An error occurred while communicating with another application or
an upstream server.