Remote end closed connection without response

[selimt]

Occasionally we are getting the following traceback:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/elastic_transport/_node/_http_urllib3.py", line 167, in perform_request
    response = self.pool.urlopen(
  File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 841, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.10/site-packages/urllib3/util/retry.py", line 449, in increment
    raise reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.10/site-packages/urllib3/util/util.py", line 38, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
  File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 534, in _make_request
    response = conn.getresponse()
  File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 565, in getresponse
    httplib_response = super().getresponse()
  File "/usr/lib64/python3.10/http/client.py", line 1375, in getresponse
    response.begin()
  File "/usr/lib64/python3.10/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python3.10/http/client.py", line 287, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

elastic-transport==9.1.0
elasticsearch==9.1.1
urllib3==2.5.0
celery=5.5.3

After the retry, this goes away but I am curious as to why this happens. ES node logs don't show anything out of the ordinary.

The ES requests are submitted through Celery workers so not sure if it is a bad interaction between Celery and urllib3.

I know it is not a timeout error since the the request returns after a short time:

[2025-09-17 19:23:19,693: INFO/ForkPoolWorker-78] POST http://data-elasticsearch.service.consulcp:9200/catalog-qa-rolodex_swblt/_refresh [status:N/A duration:0.870s]

[miguelgrinberg]

I assume this is the bottom part of a longer stack trace? From the portion that you submitted I can't really make any guesses. Does this happen always on the same operation? I also find it odd that there are no traces of this request in the server logs, are you positive that these failed requests do not show up there?

[selimt]

I am positive, the server logs don't have any warnings or errors around the time that the error occurred. It looks like it happened twice on one of our worker containers overnight.

Here is the full stack trace (although some of those are from multiple workers):

[2025-09-18 20:42:10,457: INFO/ForkPoolWorker-17] POST http://data-elasticsearch.service.consulcp:9200/catalog-rolodex_editdroid/_search [status:N/A duration:0.002s]
[2025-09-18 20:42:10,457: WARNING/ForkPoolWorker-17] Node <Urllib3HttpNode(http://data-elasticsearch.service.consulcp:9200)> has failed for 1 times in a row, putting on 1 second timeout
[2025-09-18 20:42:10,457: INFO/ForkPoolWorker-1] POST http://data-elasticsearch.service.consulcp:9200/catalog-rolodex_editdroid/_search [status:N/A duration:0.001s]
[2025-09-18 20:42:10,457: WARNING/ForkPoolWorker-1] Node <Urllib3HttpNode(http://data-elasticsearch.service.consulcp:9200)> has failed for 1 times in a row, putting on 1 second timeout
[2025-09-18 20:42:10,457: INFO/ForkPoolWorker-2] POST http://data-elasticsearch.service.consulcp:9200/catalog-rolodex_editdroid/_refresh [status:400 duration:0.002s]
[2025-09-18 20:42:10,458: INFO/ForkPoolWorker-18] POST http://data-elasticsearch.service.consulcp:9200/catalog-rolodex_editdroid/_refresh [status:400 duration:0.003s]
[2025-09-18 20:42:10,457: WARNING/ForkPoolWorker-1] Retrying request after failure (attempt 0 of 10)
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
  File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 534, in _make_request
    response = conn.getresponse()
  File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 565, in getresponse
    httplib_response = super().getresponse()
  File "/usr/lib64/python3.10/http/client.py", line 1375, in getresponse
    response.begin()
  File "/usr/lib64/python3.10/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python3.10/http/client.py", line 287, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/elastic_transport/_node/_http_urllib3.py", line 167, in perform_request
    response = self.pool.urlopen(
  File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 841, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.10/site-packages/urllib3/util/retry.py", line 449, in increment
    raise reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.10/site-packages/urllib3/util/util.py", line 38, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
  File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 534, in _make_request
    response = conn.getresponse()
  File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 565, in getresponse
    httplib_response = super().getresponse()
  File "/usr/lib64/python3.10/http/client.py", line 1375, in getresponse
    response.begin()
  File "/usr/lib64/python3.10/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python3.10/http/client.py", line 287, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/elastic_transport/_transport.py", line 344, in perform_request
    resp = node.perform_request(
  File "/usr/local/lib/python3.10/site-packages/elastic_transport/_node/_http_urllib3.py", line 202, in perform_request
    raise err from e
elastic_transport.ConnectionError: Connection error caused by: ProtocolError(('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')))
[2025-09-18 20:42:10,457: WARNING/ForkPoolWorker-17] Retrying request after failure (attempt 0 of 10)
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
  File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 534, in _make_request
    response = conn.getresponse()
  File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 565, in getresponse
    httplib_response = super().getresponse()
  File "/usr/lib64/python3.10/http/client.py", line 1375, in getresponse
    response.begin()
  File "/usr/lib64/python3.10/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python3.10/http/client.py", line 287, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/elastic_transport/_node/_http_urllib3.py", line 167, in perform_request
    response = self.pool.urlopen(
  File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 841, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.10/site-packages/urllib3/util/retry.py", line 449, in increment
    raise reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.10/site-packages/urllib3/util/util.py", line 38, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
  File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 534, in _make_request
    response = conn.getresponse()
  File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 565, in getresponse
    httplib_response = super().getresponse()
  File "/usr/lib64/python3.10/http/client.py", line 1375, in getresponse
    response.begin()
  File "/usr/lib64/python3.10/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python3.10/http/client.py", line 287, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/elastic_transport/_transport.py", line 344, in perform_request
    resp = node.perform_request(
  File "/usr/local/lib/python3.10/site-packages/elastic_transport/_node/_http_urllib3.py", line 202, in perform_request
    raise err from e
elastic_transport.ConnectionError: Connection error caused by: ProtocolError(('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')))
[2025-09-18 20:42:10,461: ERROR/ForkPoolWorker-18] Task sync_connection[97be9acf-e133-476a-b371-2e75e80f4b32] raised unexpected: UnpickleableExceptionWrapper('elasticsearch', 'BadRequestError', ('illegal_argument_exception',), "BadRequestError('illegal_argument_exception', meta=ApiResponseMeta(status=400, http_version='1.1', headers={'X-elastic-product': 'Elasticsearch', 'content-type': 'application/json', 'content-length': '203'}, duration=0.0025582313537597656, node=NodeConfig(scheme='http', host='data-elasticsearch.service.consulcp', port=9200, path_prefix='', headers={'user-agent': 'elasticsearch-py/9.1.1 (Python/3.10.14; elastic-transport/9.1.0)'}, connections_per_node=10, request_timeout=10.0, http_compress=False, verify_certs=True, ca_certs=None, client_cert=None, client_key=None, ssl_assert_hostname=None, ssl_assert_fingerprint=None, ssl_version=None, ssl_context=None, ssl_show_warn=True, _extras={})), body={'error': {'root_cause': [{'type': 'illegal_argument_exception', 'reason': 'text is empty (possibly HTTP/0.9)'}], 'type': 'illegal_argument_exception', 'reason': 'text is empty (possibly HTTP/0.9)'}, 'status': 400})")
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/celery/app/trace.py", line 453, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/celery/app/trace.py", line 736, in __protected_call__
    return self.run(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/catalog/tasks/sync/tasks.py", line 246, in sync_connection
    r_id = get_rolodex_by_namespace_and_taggable(
  File "/usr/local/lib/python3.10/site-packages/catalog/utils.py", line 90, in retry_operation_on_exception_wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/catalog/tasks/sync/tasks.py", line 176, in get_rolodex_by_namespace_and_taggable
    Index(name=rolodex_index).refresh()
  File "/usr/local/lib/python3.10/site-packages/elasticsearch/dsl/_sync/index.py", line 304, in refresh
    return self._get_connection(using).indices.refresh(index=self._name, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/elasticsearch/_sync/client/utils.py", line 415, in wrapped
    return api(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/elasticsearch/_sync/client/indices.py", line 4597, in refresh
    return self.perform_request(  # type: ignore[return-value]
  File "/usr/local/lib/python3.10/site-packages/elasticsearch/_sync/client/_base.py", line 422, in perform_request
    return self._client.perform_request(
  File "/usr/local/lib/python3.10/site-packages/elasticsearch/_sync/client/_base.py", line 271, in perform_request
    response = self._perform_request(
  File "/usr/local/lib/python3.10/site-packages/elasticsearch/_sync/client/_base.py", line 351, in _perform_request
    raise HTTP_EXCEPTIONS.get(meta.status, ApiError)(
celery.utils.serialization.UnpickleableExceptionWrapper: BadRequestError('illegal_argument_exception', meta=ApiResponseMeta(status=400, http_version='1.1', headers={'X-elastic-product': 'Elasticsearch', 'content-type': 'application/json', 'content-length': '203'}, duration=0.0025582313537597656, node=NodeConfig(scheme='http', host='data-elasticsearch.service.consulcp', port=9200, path_prefix='', headers={'user-agent': 'elasticsearch-py/9.1.1 (Python/3.10.14; elastic-transport/9.1.0)'}, connections_per_node=10, request_timeout=10.0, http_compress=False, verify_certs=True, ca_certs=None, client_cert=None, client_key=None, ssl_assert_hostname=None, ssl_assert_fingerprint=None, ssl_version=None, ssl_context=None, ssl_show_warn=True, _extras={})), body={'error': {'root_cause': [{'type': 'illegal_argument_exception', 'reason': 'text is empty (possibly HTTP/0.9)'}], 'type': 'illegal_argument_exception', 'reason': 'text is empty (possibly HTTP/0.9)'}, 'status': 400})
[2025-09-18 20:42:10,461: ERROR/ForkPoolWorker-2] Task sync_connection[64ff920b-72ea-4fff-9cd1-2da5c9f39da4] raised unexpected: UnpickleableExceptionWrapper('elasticsearch', 'BadRequestError', ('x_content_parse_exception',), 'BadRequestError(\'x_content_parse_exception\', meta=ApiResponseMeta(status=400, http_version=\'1.1\', headers={\'X-elastic-product\': \'Elasticsearch\', \'content-type\': \'application/vnd.elasticsearch+json;compatible-with=9\', \'content-length\': \'1247\'}, duration=0.0019087791442871094, node=NodeConfig(scheme=\'http\', host=\'data-elasticsearch.service.consulcp\', port=9200, path_prefix=\'\', headers={\'user-agent\': \'elasticsearch-py/9.1.1 (Python/3.10.14; elastic-transport/9.1.0)\'}, connections_per_node=10, request_timeout=10.0, http_compress=False, verify_certs=True, ca_certs=None, client_cert=None, client_key=None, ssl_assert_hostname=None, ssl_assert_fingerprint=None, ssl_version=None, ssl_context=None, ssl_show_warn=True, _extras={})), body={\'error\': {\'root_cause\': [{\'type\': \'x_content_parse_exception\', \'reason\': \'[1:6] Unrecognized token \\\'POST\\\': was expecting (JSON String, Number, Array, Object or token \\\'null\\\', \\\'true\\\' or \\\'false\\\')\\n at [Source: (byte[])"POST /catalog-rolodex_editdroid/_refresh HTTP/1.1\\r\\nHost: data-elasticsearch.service.consulcp:9200\\r\\nAccept-Encoding: identity\\r\\nContent-Length: 0\\r\\nuser-agent: elasticsearch-dsl-py/9."; line: 1, column: 6]\'}], \'type\': \'x_content_parse_exception\', \'reason\': \'[1:6] Unrecognized token \\\'POST\\\': was expecting (JSON String, Number, Array, Object or token \\\'null\\\', \\\'true\\\' or \\\'false\\\')\\n at [Source: (byte[])"POST /catalog-rolodex_editdroid/_refresh HTTP/1.1\\r\\nHost: data-elasticsearch.service.consulcp:9200\\r\\nAccept-Encoding: identity\\r\\nContent-Length: 0\\r\\nuser-agent: elasticsearch-dsl-py/9."; line: 1, column: 6]\', \'caused_by\': {\'type\': \'json_parse_exception\', \'reason\': \'Unrecognized token \\\'POST\\\': was expecting (JSON String, Number, Array, Object or token \\\'null\\\', \\\'true\\\' or \\\'false\\\')\\n at [Source: (byte[])"POST /catalog-rolodex_editdroid/_refresh HTTP/1.1\\r\\nHost: data-elasticsearch.service.consulcp:9200\\r\\nAccept-Encoding: identity\\r\\nContent-Length: 0\\r\\nuser-agent: elasticsearch-dsl-py/9."; line: 1, column: 6]\'}}, \'status\': 400})')
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/celery/app/trace.py", line 453, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/celery/app/trace.py", line 736, in __protected_call__
    return self.run(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/catalog/tasks/sync/tasks.py", line 246, in sync_connection
    r_id = get_rolodex_by_namespace_and_taggable(
  File "/usr/local/lib/python3.10/site-packages/catalog/utils.py", line 90, in retry_operation_on_exception_wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/catalog/tasks/sync/tasks.py", line 176, in get_rolodex_by_namespace_and_taggable
    Index(name=rolodex_index).refresh()
  File "/usr/local/lib/python3.10/site-packages/elasticsearch/dsl/_sync/index.py", line 304, in refresh
    return self._get_connection(using).indices.refresh(index=self._name, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/elasticsearch/_sync/client/utils.py", line 415, in wrapped
    return api(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/elasticsearch/_sync/client/indices.py", line 4597, in refresh
    return self.perform_request(  # type: ignore[return-value]
  File "/usr/local/lib/python3.10/site-packages/elasticsearch/_sync/client/_base.py", line 422, in perform_request
    return self._client.perform_request(
  File "/usr/local/lib/python3.10/site-packages/elasticsearch/_sync/client/_base.py", line 271, in perform_request
    response = self._perform_request(
  File "/usr/local/lib/python3.10/site-packages/elasticsearch/_sync/client/_base.py", line 351, in _perform_request
    raise HTTP_EXCEPTIONS.get(meta.status, ApiError)(
celery.utils.serialization.UnpickleableExceptionWrapper: BadRequestError('x_content_parse_exception', meta=ApiResponseMeta(status=400, http_version='1.1', headers={'X-elastic-product': 'Elasticsearch', 'content-type': 'application/vnd.elasticsearch+json;compatible-with=9', 'content-length': '1247'}, duration=0.0019087791442871094, node=NodeConfig(scheme='http', host='data-elasticsearch.service.consulcp', port=9200, path_prefix='', headers={'user-agent': 'elasticsearch-py/9.1.1 (Python/3.10.14; elastic-transport/9.1.0)'}, connections_per_node=10, request_timeout=10.0, http_compress=False, verify_certs=True, ca_certs=None, client_cert=None, client_key=None, ssl_assert_hostname=None, ssl_assert_fingerprint=None, ssl_version=None, ssl_context=None, ssl_show_warn=True, _extras={})), body={'error': {'root_cause': [{'type': 'x_content_parse_exception', 'reason': '[1:6] Unrecognized token \'POST\': was expecting (JSON String, Number, Array, Object or token \'null\', \'true\' or \'false\')\n at [Source: (byte[])"POST /catalog-rolodex_editdroid/_refresh HTTP/1.1\r\nHost: data-elasticsearch.service.consulcp:9200\r\nAccept-Encoding: identity\r\nContent-Length: 0\r\nuser-agent: elasticsearch-dsl-py/9."; line: 1, column: 6]'}], 'type': 'x_content_parse_exception', 'reason': '[1:6] Unrecognized token \'POST\': was expecting (JSON String, Number, Array, Object or token \'null\', \'true\' or \'false\')\n at [Source: (byte[])"POST /catalog-rolodex_editdroid/_refresh HTTP/1.1\r\nHost: data-elasticsearch.service.consulcp:9200\r\nAccept-Encoding: identity\r\nContent-Length: 0\r\nuser-agent: elasticsearch-dsl-py/9."; line: 1, column: 6]', 'caused_by': {'type': 'json_parse_exception', 'reason': 'Unrecognized token \'POST\': was expecting (JSON String, Number, Array, Object or token \'null\', \'true\' or \'false\')\n at [Source: (byte[])"POST /catalog-rolodex_editdroid/_refresh HTTP/1.1\r\nHost: data-elasticsearch.service.consulcp:9200\r\nAccept-Encoding: identity\r\nContent-Length: 0\r\nuser-agent: elasticsearch-dsl-py/9."; line: 1, column: 6]'}}, 'status': 400})

[miguelgrinberg]

There are two error messages in the deeper levels of your stack trace:

• text is empty (possibly HTTP/0.9)
• Unrecognized token 'POST': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')

This suggests you are sending invalid bodies to Elasticsearch and this is causing the errors.

[selimt]

I haven't seen this in a while so I am going to chalk it up to some network instability. We also use a load balancer called consul and its DNS resolution can be flaky.

Do you suggest that we specify the host names directly in the connection arguments, rather than the load balancer DNS ? We currently have 3 nodes per site that serve the requests.