Skip to content

Enable faster s3 multipart download #125

@LeSim

Description

@LeSim

For large files, boto3 uses multipart download. It issues multiple HTTP GET requests with different range headers.

In ds_proxy's fetch.rs, for every range request, a download from the start is launched, and then only the selected range is extracted and sent to the client.

To enable faster multipart downloads, we need to cache the encryption header located in the first HEADER_V2_SIZE bytes, then compute the wrapping chunks, and within them, the corresponding position of the requested range.

boto3 code :

import boto3
s3 = boto3.client('s3')

with open('filename', 'wb') as data:
    s3.download_fileobj('amzn-s3-demo-bucket', 'mykey', data)

A faire avec un fichier de plus de 8 Mo.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    Status

    Frigo

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions