Skip to content

fix: handle multipart uploads with >1000 parts for AWS/RadosGW#42

Merged
egabancho merged 1 commit intoinveniosoftware:masterfrom
oarepo:contribution-huge-files
Aug 5, 2025
Merged

fix: handle multipart uploads with >1000 parts for AWS/RadosGW#42
egabancho merged 1 commit intoinveniosoftware:masterfrom
oarepo:contribution-huge-files

Conversation

@mesemus
Copy link
Contributor

@mesemus mesemus commented Jul 27, 2025

Description

This PR fixes failing multipart upload for very large files to AWS or RadosGW.

  • For very large files (hundreds of GB), multipart uploads in the current UI uploader can exceed
    1000 parts
  • Unlike MinIO, AWS and RadosGW part listings return at most 1000 parts per response. Code
    now handles pagination to retrieve all parts
  • Added manual test to verify the fix (disabled by default). MinIO's implementation differs by
    returning full listings unconditionally and cannot be used to test the fix.

→ See test_multipart.py for manual test execution details

Checklist

Ticks in all boxes and 🟢 on all GitHub actions status checks are required to merge:

Frontend

Reminder

By using GitHub, you have already agreed to the GitHub’s Terms of Service including that:

  1. You license your contribution under the same terms as the current repository’s license.
  2. You agree that you have the right to license your contribution under the current repository’s license.

@mesemus mesemus force-pushed the contribution-huge-files branch 2 times, most recently from 79f94a0 to 10fe04c Compare July 28, 2025 05:19
* For very large files (hundreds of GB), multipart uploads
  in the current UI uploader can exceed 1000 parts
* Unlike MinIO, AWS and RadosGW part listings return
  at most 1000 parts per response. Code now handles pagination
  to retrieve all parts
* Added manual test to verify the fix (disabled by default).
  MinIO's implementation differs by returning full listings
  unconditionally and cannot be used to test the fix.

→ See test_multipart.py for manual test execution details
@mesemus mesemus force-pushed the contribution-huge-files branch from 10fe04c to 99d7497 Compare July 28, 2025 05:22
Copy link
Contributor

@wgresshoff wgresshoff left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks very good to me, great you added a test for >parts as well!

@egabancho egabancho merged commit e3946cc into inveniosoftware:master Aug 5, 2025
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants