anyio.ClosedResourceError during concurrent HTTP2 GET requests

[DXist]

I use async httpx client to issue multiple HTTP2 GET requests (10 workers per host) and sometimes get anyio.ClosedResourceError exception (see traceback below).

The service I connect to fails often and returns 500 errors. Some requests just get timeouts.

Default timeout on the client is 10, read timeout is 60. Sometimes I see log messages about happened ReadTimeout before the exception happens. I connect to a service behind Nginx reverse proxy and 60 seconds is a default proxy connect/read timeout for Nginx.

httpx version is 0.18.2, httpcore version is 0.13.6. I found a commit in httpcore related to connection close and read timeouts. I didn't get this error before this change but haven't tested the previous version extensively.

File "/root/.local/lib/python3.9/site-packages/httpx/_client.py", line 1722, in get
return await self.request(
File "/root/.local/lib/python3.9/site-packages/httpx/_client.py", line 1481, in request
response = await self.send(
File "/root/.local/lib/python3.9/site-packages/httpx/_client.py", line 1568, in send
response = await self._send_handling_auth(
File "/root/.local/lib/python3.9/site-packages/httpx/_client.py", line 1604, in _send_handling_auth
response = await self._send_handling_redirects(
File "/root/.local/lib/python3.9/site-packages/httpx/_client.py", line 1640, in _send_handling_redirects
response = await self._send_single_request(request, timeout)
File "/root/.local/lib/python3.9/site-packages/httpx/_client.py", line 1681, in _send_single_request
) = await transport.handle_async_request(
File "/root/.local/lib/python3.9/site-packages/httpx/_transports/default.py", line 283, in handle_async_request
) = await self._pool.handle_async_request(
File "/root/.local/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
response = await connection.handle_async_request(
File "/root/.local/lib/python3.9/site-packages/httpcore/_async/connection.py", line 148, in handle_async_request
return await self.connection.handle_async_request(
File "/root/.local/lib/python3.9/site-packages/httpcore/_async/http2.py", line 165, in handle_async_request
return await h2_stream.handle_async_request(
File "/root/.local/lib/python3.9/site-packages/httpcore/_async/http2.py", line 344, in handle_async_request
status_code, headers = await self.receive_response(timeout)
File "/root/.local/lib/python3.9/site-packages/httpcore/_async/http2.py", line 419, in receive_response
event = await self.connection.wait_for_event(self.stream_id, timeout)
File "/root/.local/lib/python3.9/site-packages/httpcore/_async/http2.py", line 242, in wait_for_event
await self.receive_events(timeout)
File "/root/.local/lib/python3.9/site-packages/httpcore/_async/http2.py", line 249, in receive_events
data = await self.socket.read(self.READ_NUM_BYTES, timeout)
File "/root/.local/lib/python3.9/site-packages/httpcore/_backends/anyio.py", line 60, in read
return await self.stream.receive(n)
File "/root/.local/lib/python3.9/site-packages/anyio/streams/tls.py", line 152, in receive
data = await self._call_sslobject_method(self._ssl_object.read, max_bytes)
File "/root/.local/lib/python3.9/site-packages/anyio/streams/tls.py", line 109, in _call_sslobject_method
data = await self.transport_stream.receive()
File "/root/.local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 1091, in receive
raise ClosedResourceError from None
anyio.ClosedResourceError

[DXist]

On the weekend I received another error:

Traceback (most recent call last):
File "/root/.local/lib/python3.9/site-packages/h2/connection.py", line 224, in process_input
func, target_state = self._transitions[(self.state, input_)]
KeyError: (<ConnectionState.CLOSED: 3>, <ConnectionInputs.RECV_HEADERS: 9>)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/root/.local/lib/python3.9/site-packages/click/core.py", line 763, in invoke
return __callback(*args, **kwargs)

... omitted

asyncio.run(
File "/opt/conda/lib/python3.9/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/opt/conda/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
return future.result()

... omitted

response = await client.get(url)
File "/root/.local/lib/python3.9/site-packages/httpx/_client.py", line 1722, in get
return await self.request(
File "/root/.local/lib/python3.9/site-packages/httpx/_client.py", line 1481, in request
response = await self.send(
File "/root/.local/lib/python3.9/site-packages/httpx/_client.py", line 1568, in send
response = await self._send_handling_auth(
File "/root/.local/lib/python3.9/site-packages/httpx/_client.py", line 1604, in _send_handling_auth
response = await self._send_handling_redirects(
File "/root/.local/lib/python3.9/site-packages/httpx/_client.py", line 1640, in _send_handling_redirects
response = await self._send_single_request(request, timeout)
File "/root/.local/lib/python3.9/site-packages/httpx/_client.py", line 1681, in _send_single_request
) = await transport.handle_async_request(
File "/root/.local/lib/python3.9/site-packages/httpx/_transports/default.py", line 283, in handle_async_request
) = await self._pool.handle_async_request(
File "/root/.local/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
response = await connection.handle_async_request(
File "/root/.local/lib/python3.9/site-packages/httpcore/_async/connection.py", line 148, in handle_async_request
return await self.connection.handle_async_request(
File "/root/.local/lib/python3.9/site-packages/httpcore/_async/http2.py", line 165, in handle_async_request
return await h2_stream.handle_async_request(
File "/root/.local/lib/python3.9/site-packages/httpcore/_async/http2.py", line 344, in handle_async_request
status_code, headers = await self.receive_response(timeout)
File "/root/.local/lib/python3.9/site-packages/httpcore/_async/http2.py", line 419, in receive_response
event = await self.connection.wait_for_event(self.stream_id, timeout)
File "/root/.local/lib/python3.9/site-packages/httpcore/_async/http2.py", line 242, in wait_for_event
await self.receive_events(timeout)
File "/root/.local/lib/python3.9/site-packages/httpcore/_async/http2.py", line 253, in receive_events
events = self._h2_state.receive_data(data)
File "/root/.local/lib/python3.9/site-packages/h2/connection.py", line 1463, in receive_data
events.extend(self._receive_frame(frame))
File "/root/.local/lib/python3.9/site-packages/h2/connection.py", line 1486, in _receive_frame
frames, events = self._frame_dispatch_table[frame.__class__](frame)
File "/root/.local/lib/python3.9/site-packages/h2/connection.py", line 1554, in _receive_headers_frame
events = self.state_machine.process_input(
File "/root/.local/lib/python3.9/site-packages/h2/connection.py", line 228, in process_input
raise ProtocolError(
h2.exceptions.ProtocolError: Invalid input ConnectionInputs.RECV_HEADERS in state ConnectionState.CLOSED

I'll try to downgrade httpcore.

[DXist]

The second error KeyError: (<ConnectionState.CLOSED: 3>, <ConnectionInputs.RECV_HEADERS: 9>) is reproduced with httpcore==0.13.5.

[lovelydinosaur]

Okay, thanks @DXist - do you think you'd be able to draw up a minimum reproducible case that helps reproduce this?

Ideally a single script that you can show here that we can just run directly, and that's pointing at external real-world URLs.

[DXist]

@tomchristie , I've uploaded a Python wheel of a dataset preparation tool and the input files archive with YouTube video ids that I process using this tool.

The tool requires ffmpeg to get audio but it's possible just to remove the call to ffmpeg in youtube_dataset/dataset.py.

The tool connects to Invidious servers, YouTube blocks them and requests start to fail and timeout.

The tool can be installed in user site-packages via pip install --user <path_to_the_wheel>

Unpack dataset.tar.gz on the disk to <dataset_path> directory.

Make sure that httpcore version is 0.13.6 (pip list | grep httpcore)

Then run

youtube-dataset download-dataset <dataset_path> <dataset_path> ""

[mazzi]

I'm suffering the same scenario. There's any update on this discussion?
Thanks @DXist !

[Dantenerosas]

I'm encountering same issue using both threads and asyncio (KeyError: (<ConnectionState.CLOSED: 3>, <ConnectionInputs.SEND_HEADERS: 0>) etc for threads and same as @DXist for asyncio) but only when reusing same client instance, if creating new client instance per requests everything works fine. I'm making hundreds requests per second to same set of hosts (1-2 hosts per instance), max 300 connections + max 100 keepalive in both.
Httpx version is 0.20.0 (same happened on 0.19.0) and httpcore version is 0.13.7. Dunno when this started to occur because until today we haven't been reusing clients for httpx in particular

UPD: Tested some lower versions, it's present at least down to httpcore 0.13.3 so it seems it runs deeper than that

Full text of error for sync version, the request is made like resp = session.request(request_type, real_url, headers=headers, follow_redirects=allow_redirects):

File "/usr/local/lib/python3.9/dist-packages/httpx/_client.py", line 787, in request
return self.send(
File "/usr/local/lib/python3.9/dist-packages/httpx/_client.py", line 878, in send
response = self._send_handling_auth(
File "/usr/local/lib/python3.9/dist-packages/httpx/_client.py", line 908, in _send_handling_auth
response = self._send_handling_redirects(
File "/usr/local/lib/python3.9/dist-packages/httpx/_client.py", line 947, in _send_handling_redirects
response = self._send_single_request(request, timeout)
File "/usr/local/lib/python3.9/dist-packages/httpx/_client.py", line 983, in _send_single_request
(status_code, headers, stream, extensions) = transport.handle_request(
File "/usr/local/lib/python3.9/dist-packages/httpx/_transports/default.py", line 180, in handle_request
status_code, headers, byte_stream, extensions = self._pool.handle_request(
File "/usr/local/lib/python3.9/dist-packages/httpcore/_sync/http_proxy.py", line 127, in handle_request
return self._tunnel_request(
File "/usr/local/lib/python3.9/dist-packages/httpcore/_sync/http_proxy.py", line 278, in _tunnel_request
) = connection.handle_request(
File "/usr/local/lib/python3.9/dist-packages/httpcore/_sync/connection.py", line 148, in handle_request
return self.connection.handle_request(
File "/usr/local/lib/python3.9/dist-packages/httpcore/_sync/http2.py", line 165, in handle_request
return h2_stream.handle_request(
File "/usr/local/lib/python3.9/dist-packages/httpcore/_sync/http2.py", line 339, in handle_request
self.send_headers(method, url, headers, has_body, timeout)
File "/usr/local/lib/python3.9/dist-packages/httpcore/_sync/http2.py", line 398, in send_headers
self.connection.send_headers(self.stream_id, headers, end_stream, timeout)
File "/usr/local/lib/python3.9/dist-packages/httpcore/_sync/http2.py", line 271, in send_headers
self._h2_state.send_headers(stream_id, headers, end_stream=end_stream)
File "/usr/local/lib/python3.9/dist-packages/h2/connection.py", line 766, in send_headers
self.state_machine.process_input(ConnectionInputs.SEND_HEADERS)
File "/usr/local/lib/python3.9/dist-packages/h2/connection.py", line 228, in process_input
raise ProtocolError(
h2.exceptions.ProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED

Full error for asyncio version, the response is awaited like resp = await session.request(request_type, real_url, headers=headers, follow_redirects=allow_redirects):

File "/usr/local/lib/python3.9/dist-packages\httpx_client.py", line 1483, in request
response = await self.send(
File "/usr/local/lib/python3.9/dist-packages\httpx_client.py", line 1571, in send
response = await self._send_handling_auth(
File "/usr/local/lib/python3.9/dist-packages\httpx_client.py", line 1599, in _send_handling_auth
response = await self._send_handling_redirects(
File "/usr/local/lib/python3.9/dist-packages\httpx_client.py", line 1636, in _send_handling_redirects
response = await self._send_single_request(request)
File "/usr/local/lib/python3.9/dist-packages\httpx_client.py", line 1673, in _send_single_request
response = await transport.handle_async_request(request)
File "/usr/local/lib/python3.9/dist-packages\httpx_transports\default.py", line 276, in handle_async_request
) = await self._pool.handle_async_request(
File "/usr/local/lib/python3.9/dist-packages\httpcore_async\http_proxy.py", line 127, in handle_async_request
return await self._tunnel_request(
File "/usr/local/lib/python3.9/dist-packages\httpcore_async\http_proxy.py", line 278, in _tunnel_request
) = await connection.handle_async_request(
File "/usr/local/lib/python3.9/dist-packages\httpcore_async\connection.py", line 148, in handle_async_request
return await self.connection.handle_async_request(
File "/usr/local/lib/python3.9/dist-packages\httpcore_async\http2.py", line 165, in handle_async_request
return await h2_stream.handle_async_request(
File "/usr/local/lib/python3.9/dist-packages\httpcore_async\http2.py", line 339, in handle_async_request
await self.send_headers(method, url, headers, has_body, timeout)
File "/usr/local/lib/python3.9/dist-packages\httpcore_async\http2.py", line 398, in send_headers
await self.connection.send_headers(self.stream_id, headers, end_stream, timeout)
File "/usr/local/lib/python3.9/dist-packages\httpcore_async\http2.py", line 274, in send_headers
await self.socket.write(data_to_send, timeout)
File "/usr/local/lib/python3.9/dist-packages\httpcore_backends\anyio.py", line 77, in write
return await self.stream.send(data)
File "/usr/local/lib/python3.9/dist-packages\anyio\streams\tls.py", line 159, in send
await self._call_sslobject_method(self._ssl_object.write, item)
File "/usr/local/lib/python3.9/dist-packages\anyio\streams\tls.py", line 125, in _call_sslobject_method
await self.transport_stream.send(self._write_bio.read())
File "/usr/local/lib/python3.9/dist-packages\anyio_backends_asyncio.py", line 1116, in send
raise ClosedResourceError
anyio.ClosedResourceError

We don't close httpx.Client\AsyncClient instances ourselves whatsoever (nor any other pool\session-like objects we use from urllib3, aiohttp etc) until app is killing self (the only exception is when creating new httpx.Client\request.Session instance in "no pooling" variant, we let GC handle that because we have increased connections limit anyway), so if there is nothing wrong with our code (most likely not because aiohttp and urllib3 dont share same behavior when reusing client\pool) it seems like this error stems from httpcore or h2 most likely because httpx built on top of them, AFAIK. I've encountered this error mostly after some requests have failed (e.g. couldnt connect to proxy) or some requests timed out using same reused httpx.Client

[Dantenerosas]

Seems like recent httpcore release 0.14.* resolved this issue for us so most likely this is resolved now, thanks to @tomchristie 's rewrite of httpcore. @DXist have you tried the most recent versions of httpx and httpcore and if so has the problem disappeared for you too?

[DXist]

@Dantenerosas No, I haven't. I switched projects and don't use httpx/httpcore any more.

[zeitderforschung]

Today, I got a similar effect with httpx[http2] 0.23.3 and httpx-sse 0.1.0. I use a shared global httpx.AsyncClient created in a FastAPI lifespan method and call only client.get, client.post and client.head in my FastAPI routes. The client is never explicitly closed. But at some point it stops working and every call fails with the following exception:

2023-03-22 20:28:10.180 | ERROR    | uvicorn.protocols.http.h11_impl:run_asgi:434 - Exception in ASGI application
Traceback (most recent call last):

  File "/opt/conda/lib/python3.9/site-packages/h2/connection.py", line 224, in process_input
    func, target_state = self._transitions[(self.state, input_)]
                         |    |             |    |      -> <ConnectionInputs.SEND_HEADERS: 0>
                         |    |             |    -> <ConnectionState.CLOSED: 3>
                         |    |             -> <h2.connection.H2ConnectionStateMachine object at 0x7fbcba1f7d60>
                         |    -> {(<ConnectionState.IDLE: 0>, <ConnectionInputs.SEND_HEADERS: 0>): (None, <ConnectionState.CLIENT_OPEN: 1>), (<ConnectionState...
                         -> <h2.connection.H2ConnectionStateMachine object at 0x7fbcba1f7d60>

KeyError: (<ConnectionState.CLOSED: 3>, <ConnectionInputs.SEND_HEADERS: 0>)

...

File "/opt/conda/lib/python3.9/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
          |          -> '<ConnectionTerminated error_code:ErrorCodes.NO_ERROR, last_stream_id:5, additional_data:None>'
          -> <class 'httpx.RemoteProtocolError'>

httpx.RemoteProtocolError: <ConnectionTerminated error_code:ErrorCodes.NO_ERROR, last_stream_id:5, additional_data:None>

Maybe it is related to encode/httpcore#87

[zanieb]

See python-hyper/h2#1199 and python-hyper/h2#1181

I've also gotten lots of reports of this in PrefectHQ/prefect#7442 but I don't think this discussion is relevant for your issue.

[zeitderforschung]

Thank you. So, I guess there is currently no workaround to use a global AsyncClient with HTTP/2. I will go on with http2 disabled.