-
-
Notifications
You must be signed in to change notification settings - Fork 200
Description
(note had AI write a bunch of this g)
aiobotocore enables asyncio support by layering async I/O on top of botocore, which is a synchronous, low-level AWS SDK designed for boto3. It does this by patching internal components and substituting key behaviors with async-compatible implementations using aiohttp or httpx.
While this has extended the usefulness of botocore in the async world, it comes with significant downsides:
• Fragility: aiobotocore depends on internal botocore APIs, which change frequently and cause frequent breakages.
• Incomplete Async Behavior: Many operations still run synchronously or with blocking I/O, limiting concurrency.
• Maintenance Overhead: Keeping up with botocore releases and protocol changes is a continuous and complex burden.
• Performance Constraints: Python-level async adds overhead compared to true native async implementations.
A New Direction
Rather than continuing to patch around botocore, this proposal suggests building a native async Python client that wraps the AWS SDK for C++, offering:
• ✅ True async I/O and thread safety backed by native code
• ✅ Higher performance for high-throughput services (e.g., S3, DynamoDB streams)
• ✅ Long-term stability from using officially supported AWS SDKs
• ✅ Elimination of fragile monkey-patching and compatibility layers
Why This Is Feasible Now
Previously, this approach was dismissed as too complex due to the cost of:
• Writing Pybind11 or CFFI wrappers for many AWS services
• Managing cross-platform builds of the AWS SDK for C++
• Designing clean, Pythonic async interfaces on top of C++ code
Today, with the help of AI-assisted tooling, these barriers are lower than ever:
• AI can generate and maintain binding code, docstrings, and type hints
• Glue code and examples can be iterated on rapidly with LLM assistance
Goals
• Build a proof-of-concept async Python wrapper for the AWS C++ SDK (e.g. S3 or STS).
• Benchmark it against aiobotocore in terms of latency, throughput, and memory usage.
• Maintain idiomatic asyncio interfaces (e.g., async for, async with, await client.get_object(...)).
• Explore possible adoption paths for production workloads.