Skip to content

Replace aiobotocore with Native Async Python Bindings to the AWS SDK for C++ #1375

@thehesiod

Description

@thehesiod

(note had AI write a bunch of this g)

aiobotocore enables asyncio support by layering async I/O on top of botocore, which is a synchronous, low-level AWS SDK designed for boto3. It does this by patching internal components and substituting key behaviors with async-compatible implementations using aiohttp or httpx.

While this has extended the usefulness of botocore in the async world, it comes with significant downsides:
• Fragility: aiobotocore depends on internal botocore APIs, which change frequently and cause frequent breakages.
• Incomplete Async Behavior: Many operations still run synchronously or with blocking I/O, limiting concurrency.
• Maintenance Overhead: Keeping up with botocore releases and protocol changes is a continuous and complex burden.
• Performance Constraints: Python-level async adds overhead compared to true native async implementations.

A New Direction

Rather than continuing to patch around botocore, this proposal suggests building a native async Python client that wraps the AWS SDK for C++, offering:
• ✅ True async I/O and thread safety backed by native code
• ✅ Higher performance for high-throughput services (e.g., S3, DynamoDB streams)
• ✅ Long-term stability from using officially supported AWS SDKs
• ✅ Elimination of fragile monkey-patching and compatibility layers

Why This Is Feasible Now

Previously, this approach was dismissed as too complex due to the cost of:
• Writing Pybind11 or CFFI wrappers for many AWS services
• Managing cross-platform builds of the AWS SDK for C++
• Designing clean, Pythonic async interfaces on top of C++ code

Today, with the help of AI-assisted tooling, these barriers are lower than ever:
• AI can generate and maintain binding code, docstrings, and type hints
• Glue code and examples can be iterated on rapidly with LLM assistance

Goals
• Build a proof-of-concept async Python wrapper for the AWS C++ SDK (e.g. S3 or STS).
• Benchmark it against aiobotocore in terms of latency, throughput, and memory usage.
• Maintain idiomatic asyncio interfaces (e.g., async for, async with, await client.get_object(...)).
• Explore possible adoption paths for production workloads.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions