-
Notifications
You must be signed in to change notification settings - Fork 290
Open
Description
S3Filesystem currently uses a default chunksize of 50MiB. Given AWS's limit of 10,000 parts per upload, this means the maximum file size that can be uploaded out of the box is about 488GiB (less than 5TiB size limit of an object in S3):
Line 1233 in ec57f88
chunksize=50 * 2**20, |
The motivation of this is this issue in dvc:
While we could handle this in DVC by adjusting the chunksize, I wanted to check if there's interest in making the chunksize dynamic within s3fs itself. That way, users can upload larger files without needing to configure this manually.
Would this be something you'd consider? Would that be considered a breaking change?
Happy to help with a PR if it sounds reasonable. If not, feel free to close this.
Metadata
Metadata
Assignees
Labels
No labels