Content-MD5 error in recursive delete - recent versions only

[hutch3232]

Describe the bug

Running the following script with uv run <script> builds a clean environment that produces this error. When I switch the version of aiobotocore to a lower version, like 2.17.* it runs successfully.

# /// script
# requires-python = ">=3.10"
# dependencies = [
#     "aiobotocore==2.18.*",
#     "s3fs",
# ]
# ///
import s3fs
 
s3 = s3fs.S3FileSystem()
s3.rm(
    "s3://my-bucket/users/hutch/logs/",
    recursive=True,
)
 
# Traceback (most recent call last):
#   File "/mnt/imported/data/mt/uv/environments-v2/blah-68eb1b6ef32670d7/lib/python3.10/site-packages/s3fs/core.py", line 114, in _error_wrapper
#     return await func(*args, **kwargs)
#   File "/mnt/imported/data/mt/uv/environments-v2/blah-68eb1b6ef32670d7/lib/python3.10/site-packages/aiobotocore/client.py", line 412, in _make_api_call
#     raise error_class(parsed_response, operation_name)
# botocore.exceptions.ClientError: An error occurred (BadDigest) when calling the DeleteObjects operation (reached max retries: 4): The Content-MD5 you specified did not match what we received.
 
# The above exception was the direct cause of the following exception:
 
# Traceback (most recent call last):
#   File "/mnt/code/blah.py", line 15, in <module>
#     "s3://res-paas-mmi/users/hutch/logs/",
#   File "/mnt/imported/data/mt/uv/environments-v2/blah-68eb1b6ef32670d7/lib/python3.10/site-packages/fsspec/asyn.py", line 118, in wrapper
#     return sync(self.loop, func, *args, **kwargs)
#   File "/mnt/imported/data/mt/uv/environments-v2/blah-68eb1b6ef32670d7/lib/python3.10/site-packages/fsspec/asyn.py", line 103, in sync
#     raise return_result
#   File "/mnt/imported/data/mt/uv/environments-v2/blah-68eb1b6ef32670d7/lib/python3.10/site-packages/fsspec/asyn.py", line 56, in _runner
#     result[0] = await coro
#   File "/mnt/imported/data/mt/uv/environments-v2/blah-68eb1b6ef32670d7/lib/python3.10/site-packages/s3fs/core.py", line 2071, in _rm
#     out = await _run_coros_in_chunks(
#   File "/mnt/imported/data/mt/uv/environments-v2/blah-68eb1b6ef32670d7/lib/python3.10/site-packages/fsspec/asyn.py", line 267, in _run_coros_in_chunks
#     result, k = await done.pop()
#   File "/mnt/imported/data/mt/uv/environments-v2/blah-68eb1b6ef32670d7/lib/python3.10/site-packages/fsspec/asyn.py", line 244, in _run_coro
#     return await asyncio.wait_for(coro, timeout=timeout), i
#   File "/mnt/imported/data/mt/uv-python/cpython-3.10.16-linux-x86_64-gnu/lib/python3.10/asyncio/tasks.py", line 408, in wait_for
#     return await fut
#   File "/mnt/imported/data/mt/uv/environments-v2/blah-68eb1b6ef32670d7/lib/python3.10/site-packages/s3fs/core.py", line 2045, in _bulk_delete
#     out = await self._call_s3(
#   File "/mnt/imported/data/mt/uv/environments-v2/blah-68eb1b6ef32670d7/lib/python3.10/site-packages/s3fs/core.py", line 371, in _call_s3
#     return await _error_wrapper(
#   File "/mnt/imported/data/mt/uv/environments-v2/blah-68eb1b6ef32670d7/lib/python3.10/site-packages/s3fs/core.py", line 146, in _error_wrapper
#     raise err
# OSError: [Errno 22] The Content-MD5 you specified did not match what we received.

Checklist

• I have reproduced in environment where pip check passes without errors
• I have provided pip freeze results
  • No, but this is a reproducible script. I can go back to my work computer if this is needed.
• I have provided sample code or detailed way to reproduce
• I have tried the same code in botocore to ensure this is an aiobotocore specific issue
  • Well, I am not sure, i only changed the version of aiobotocore
• I have tried similar code in aiohttp to ensure this is is an aiobotocore specific issue
  • Well, I am not sure, i only changed the version of aiobotocore
• I have checked the latest and older versions of aiobotocore/aiohttp/python to see if this is a regression / injection

pip freeze results

Environment:

• Python Version: 3.10
• OS name and version: Ubuntu 20.04

Additional context
I am running this on an AWS S3-compatibile storage, Scality. I have seen this issue: fsspec/s3fs#931 and have used the recommended solution there successfully to eliminate these errors for other issues by setting this in my environment:

AWS_REQUEST_CHECKSUM_CALCULATION=WHEN_REQUIRED
AWS_RESPONSE_CHECKSUM_VALIDATION=WHEN_REQUIRED

Despite these, this recursive delete is still showing up on newer aiobotocore.