aiobotocore to copy files in a bucket

[eduard93]

I have a following lambda:

import asyncio
from timeit import default_timer as timer
from aiobotocore.session import get_session

def lambda_handler(event, context):
    source = event.get("source", "my-bucket")
    source_prefix = event.get("source_prefix", "source/")
    destination = event.get("destination", source)
    destination_prefix = event.get("destination_prefix", "dest/")

    if not source_prefix.endswith("/"):
        source_prefix += "/"
    if not destination_prefix.endswith("/"):
        destination_prefix += "/"

    asyncio.run(main(source, source_prefix, destination, destination_prefix))


async def main(source, source_prefix, destination, destination_prefix):
    start = timer()
    session = get_session()
    count = 0
    async with session.create_client("s3") as s3:
        paginator = s3.get_paginator("list_objects_v2")
        async for page in paginator.paginate(Bucket=source, Prefix=source_prefix):
            for file in page.get("Contents", []):
                count += 1
                source_key = file["Key"]
                copy_source = {'Bucket': source, 'Key': source_key}
                await s3.copy_object(
                    Bucket=destination,
                    CopySource=copy_source,
                    Key=destination_prefix + source_key.split("/")[-1],
                )

    end = timer()
    timing = end - start
    print(f"Sent files: {count}")
    print(f"Took: {timing:.2f} s")
    print(f"Speed: {count/timing:.2f} files/s")

It does the following:

1. Iterates source bucket/prefix.
2. For each file copies it to destination bucket/prefix.

Each file is below 1Kb in size. I expected this code to work much (or at least somewhat) faster than the basic boto3 version but it's really slow (~10.6 files per second on 1vCPU 1769 Mb RAM). What am I doing wrong? Files can be copied in any order.

[thehesiod]

you're not doing things in parallel :) How many files are we talking about? I created https://gist.github.com/thehesiod/7081ab165b9a0d4de2e07d321cc2391d to help in doing this kind of thing