Investigating botocore.exceptions.HTTPClientError errors

[Apakottur]

I'm working on a project that uses aiobotocore to access many differnet AWS functions, and in the last few days, out of nowhere, we started getting a lot of spurious errors like this one:

...
, in _send_request
    while await self._needs_retry(
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "***/aiobotocore/endpoint.py", line 271, in _needs_retry
    responses = await self._event_emitter.emit(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "***/aiobotocore/hooks.py", line 66, in _emit
    response = await resolve_awaitable(handler(**kwargs))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "***/aiobotocore/_helpers.py", line 6, in resolve_awaitable
    return await obj
           ^^^^^^^^^
  File "***/aiobotocore/retryhandler.py", line 107, in _call
    if await resolve_awaitable(self._checker(**checker_kwargs)):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "***/aiobotocore/_helpers.py", line 6, in resolve_awaitable
    return await obj
           ^^^^^^^^^
  File "***/aiobotocore/retryhandler.py", line 126, in _call
    should_retry = await self._should_retry(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "***/aiobotocore/retryhandler.py", line 152, in _should_retry
    return await resolve_awaitable(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "***/aiobotocore/_helpers.py", line 6, in resolve_awaitable
    return await obj
           ^^^^^^^^^
  File "***/aiobotocore/retryhandler.py", line 174, in _call
    checker(attempt_number, response, caught_exception)
  File "***/botocore/retryhandler.py", line 247, in __call__
    return self._check_caught_exception(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "***/botocore/retryhandler.py", line 416, in _check_caught_exception
    raise caught_exception
  File "***/aiobotocore/endpoint.py", line 181, in _do_get_response
    http_response = await self._send(request)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "***/aiobotocore/endpoint.py", line 294, in _send
    return await self.http_session.send(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "***/aiobotocore/httpsession.py", line 273, in send
    raise HTTPClientError(error=e)
botocore.exceptions.HTTPClientError: An HTTP Client raised an unhandled exception: unhandled errors in a TaskGroup (1 sub-exception)

The code which results in this error is for example this:

from aiobotocore.session import get_session

async def func():
    async with get_session().create_client("s3") as client:
        await client.get_object_attributes( # error occurs here
            Bucket="my-bucket",
            Key="my-key",
            ObjectAttributes=["ObjectSize"],
        )

It happens in our production environment once every ~1000 calls to the AWS API, on different AWS API functions. I am unable to reproduce this locally consistently.

How can I debug this further? Has anyone had a similar issue?

[Apakottur]

Found the problem, it was a broken version of a subdependency called aiohappyeyeballs. Updating it to version 2.4.3 fixes the issue.