For large files, boto3 uses multipart download. It issues multiple HTTP GET requests with different range headers.
In ds_proxy's fetch.rs, for every range request, a download from the start is launched, and then only the selected range is extracted and sent to the client.
To enable faster multipart downloads, we need to cache the encryption header located in the first HEADER_V2_SIZE bytes, then compute the wrapping chunks, and within them, the corresponding position of the requested range.
boto3 code :
import boto3
s3 = boto3.client('s3')
with open('filename', 'wb') as data:
s3.download_fileobj('amzn-s3-demo-bucket', 'mykey', data)
A faire avec un fichier de plus de 8 Mo.
For large files, boto3 uses multipart download. It issues multiple HTTP GET requests with different range headers.
In ds_proxy's fetch.rs, for every range request, a download from the start is launched, and then only the selected range is extracted and sent to the client.
To enable faster multipart downloads, we need to cache the encryption header located in the first
HEADER_V2_SIZEbytes, then compute the wrapping chunks, and within them, the corresponding position of the requested range.boto3 code :
A faire avec un fichier de plus de 8 Mo.