CancelledError with anyio 3.7.0

[sondrelg]

After upgrading dependencies, requests using httpx.AsyncClient now fail with:

CancelledError                            Traceback (most recent call last)
File ~/some/path.py:60, in _send_request(...)
---> 60     response = await session.get(url, params=url_params)

File ~/path/.venv/lib/python3.11/site-packages/httpx/_client.py:1757, in AsyncClient.get(self, url, params, headers, cookies, auth, follow_redirects, timeout, extensions)
   1752 """
   1753 Send a `GET` request.
   1754 
   1755 **Parameters**: See `httpx.request`.
   1756 """
-> 1757 return await self.request(
   1758     "GET",
   1759     url,
   1760     params=params,
   1761     headers=headers,
   1762     cookies=cookies,
   1763     auth=auth,
   1764     follow_redirects=follow_redirects,
   1765     timeout=timeout,
   1766     extensions=extensions,
   1767 )

File ~/path/.venv/lib/python3.11/site-packages/httpx/_client.py:1530, in AsyncClient.request(self, method, url, content, data, files, json, params, headers, cookies, auth, follow_redirects, timeout, extensions)
   1517 request = self.build_request(
   1518     method=method,
   1519     url=url,
   (...)
   1528     extensions=extensions,
   1529 )
-> 1530 return await self.send(request, auth=auth, follow_redirects=follow_redirects)

File ~/path/.venv/lib/python3.11/site-packages/httpx/_client.py:1617, in AsyncClient.send(self, request, stream, auth, follow_redirects)
   1615 auth = self._build_request_auth(request, auth)
-> 1617 response = await self._send_handling_auth(
   1618     request,
   1619     auth=auth,
   1620     follow_redirects=follow_redirects,
   1621     history=[],
   1622 )
   1623 try:

File ~/path/.venv/lib/python3.11/site-packages/httpx/_client.py:1645, in AsyncClient._send_handling_auth(self, request, auth, follow_redirects, history)
   1644 while True:
-> 1645     response = await self._send_handling_redirects(
   1646         request,
   1647         follow_redirects=follow_redirects,
   1648         history=history,
   1649     )
   1650     try:

File ~/path/.venv/lib/python3.11/site-packages/httpx/_client.py:1682, in AsyncClient._send_handling_redirects(self, request, follow_redirects, history)
   1680     await hook(request)
-> 1682 response = await self._send_single_request(request)
   1683 try:

File ~/path/.venv/lib/python3.11/site-packages/httpx/_client.py:1719, in AsyncClient._send_single_request(self, request)
   1718 with request_context(request=request):
-> 1719     response = await transport.handle_async_request(request)
   1721 assert isinstance(response.stream, AsyncByteStream)

File ~/path/.venv/lib/python3.11/site-packages/httpx/_transports/default.py:353, in AsyncHTTPTransport.handle_async_request(self, request)
    352 with map_httpcore_exceptions():
--> 353     resp = await self._pool.handle_async_request(req)
    355 assert isinstance(resp.stream, typing.AsyncIterable)

File ~/path/.venv/lib/python3.11/site-packages/httpcore/_async/connection_pool.py:261, in AsyncConnectionPool.handle_async_request(self, request)
    260     await self.response_closed(status)
--> 261     raise exc
    262 else:

File ~/path/.venv/lib/python3.11/site-packages/httpcore/_async/connection_pool.py:245, in AsyncConnectionPool.handle_async_request(self, request)
    244 try:
--> 245     response = await connection.handle_async_request(request)
    246 except ConnectionNotAvailable:
    247     # The ConnectionNotAvailable exception is a special case, that
    248     # indicates we need to retry the request on a new connection.
   (...)
    252     # might end up as an HTTP/2 connection, but which actually ends
    253     # up as HTTP/1.1.

File ~/path/.venv/lib/python3.11/site-packages/httpcore/_async/connection.py:69, in AsyncHTTPConnection.handle_async_request(self, request)
     68 try:
---> 69     stream = await self._connect(request)
     71     ssl_object = stream.get_extra_info("ssl_object")

File ~/path/.venv/lib/python3.11/site-packages/httpcore/_async/connection.py:117, in AsyncHTTPConnection._connect(self, request)
    116 async with Trace("connect_tcp", logger, request, kwargs) as trace:
--> 117     stream = await self._network_backend.connect_tcp(**kwargs)
    118     trace.return_value = stream

File ~/path/.venv/lib/python3.11/site-packages/httpcore/backends/auto.py:31, in AutoBackend.connect_tcp(self, host, port, timeout, local_address, socket_options)
     30 await self._init_backend()
---> 31 return await self._backend.connect_tcp(
     32     host,
     33     port,
     34     timeout=timeout,
     35     local_address=local_address,
     36     socket_options=socket_options,
     37 )

File ~/path/.venv/lib/python3.11/site-packages/httpcore/backends/asyncio.py:114, in AsyncIOBackend.connect_tcp(self, host, port, timeout, local_address, socket_options)
    113 with anyio.fail_after(timeout):
--> 114     stream: anyio.abc.ByteStream = await anyio.connect_tcp(
    115         remote_host=host,
    116         remote_port=port,
    117         local_host=local_address,
    118     )
    119     # By default TCP sockets opened in `asyncio` include TCP_NODELAY.

File ~/path/.venv/lib/python3.11/site-packages/anyio/_core/_sockets.py:221, in connect_tcp(remote_host, remote_port, local_host, tls, ssl_context, tls_standard_compatible, tls_hostname, happy_eyeballs_delay)
    220         with move_on_after(happy_eyeballs_delay):
--> 221             await event.wait()
    223 if connected_stream is None:

File ~/path/.venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py:1778, in Event.wait(self)
   1777 async def wait(self) -> None:
-> 1778     if await self._event.wait():
   1779         await checkpoint()

File ~/.pyenv/versions/3.11.2/lib/python3.11/asyncio/locks.py:213, in Event.wait(self)
    212 try:
--> 213     await fut
    214     return True

CancelledError: 

It took me a while to identify that anyio was the issue, since there's no error included. I'm also not sure if this should be opened here, in anyio, or elsewhere. Thought I would try here first 🙇

Basically any request fails right away, when run. I'm using httpx 0.24.1 (latest).

[graingert]

Can you paste the code in ~/some/path.py?

[sondrelg]

Sure! It amounts to this:

async def _send_request(...) -> Response:
    with AsyncClient() as client:
        response = await client.get(url, timeout=REQUEST_TIMEOUT, params=url_params)

I'll quickly see if I'm able to produce a minimal reproducible code sample 👍

[sondrelg]

Ok this is a weird one. It seems to only happen when I interact with redis (4.6.0) first:

import asyncio

from httpx import AsyncClient
from redis.asyncio import Redis

redis = Redis.from_url('redis://127.0.0.1:6379')


async def main():
    await redis.get('INFO')  # <-- if you remove this, it's fine
    async with AsyncClient() as session:
        await session.get("https://google.com")


asyncio.run(main())

[graingert]

I tried running this and I got no output as expected

I ran in this virtualenv:

anyio==3.7.0
certifi==2023.5.7
h11==0.14.0
httpcore==0.17.2
httpx==0.24.1
idna==3.4
redis==4.6.0
sniffio==1.3.0

[sondrelg]

I'll try to set up a fresh project and report back 👍

[sondrelg]

Ok, it actually seems to be a <3.11.3 issue, because of this: https://github.com/redis/redis-py/blob/master/redis/asyncio/connection.py#L31. Your example reproduces for me on 3.11.2.

if sys.version_info >= (3, 11, 3):
    from asyncio import timeout as async_timeout
else:
    from async_timeout import timeout as async_timeout

[sondrelg]

This does it

pyenv local 3.11.2 
python -m venv .venv      
source .venv/bin/activate
pip install redis httpx anyio
python main.py

Smells less and less like an httpx issue though 🙂

[sondrelg]

Seem relevant: redis/redis-py#2633 (comment)

[sondrelg]

Reading redis/redis-py#2633 (comment), it looks like I've been very unlucky. I'm happy to close this and upgrade to 3.11.4 🙂

[sondrelg]

thanks for the help @graingert!