Skip to content

using dynamic chunksize when uploading large files? #971

@skshetry

Description

@skshetry

S3Filesystem currently uses a default chunksize of 50MiB. Given AWS's limit of 10,000 parts per upload, this means the maximum file size that can be uploaded out of the box is about 488GiB (less than 5TiB size limit of an object in S3):

s3fs/s3fs/core.py

Line 1233 in ec57f88

chunksize=50 * 2**20,

The motivation of this is this issue in dvc:

While we could handle this in DVC by adjusting the chunksize, I wanted to check if there's interest in making the chunksize dynamic within s3fs itself. That way, users can upload larger files without needing to configure this manually.

Would this be something you'd consider? Would that be considered a breaking change?
Happy to help with a PR if it sounds reasonable. If not, feel free to close this.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions