Feat S3 Transfer Manager v2 DownloadDirectory #3163
+952
−23
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lucix-aws
reviewed
Aug 12, 2025
|
|
||
| // A callback function to allow customers to update individual | ||
| // GetObjectInput that the S3 Transfer Manager generates | ||
| Callback GetRequestCallback |
There was a problem hiding this comment.
You seem to be missing a few inputs from the SEP
- recursive
- failurePolicy
There was a problem hiding this comment.
Talked offline, design for these seems to be somewhat in-flux so we can defer for now.
lucix-aws
reviewed
Aug 12, 2025
| return i.downloadDirectory(ctx) | ||
| } | ||
|
|
||
| type directoryDownloader struct { |
There was a problem hiding this comment.
Progress listener support?
lucix-aws
reviewed
Aug 12, 2025
| if !ok { | ||
| break | ||
| } | ||
| if d.getErr() != nil { |
There was a problem hiding this comment.
You should likely also be checking for context cancellation here (after the channel read).
lucix-aws
reviewed
Aug 12, 2025
| break | ||
| } | ||
|
|
||
| err = os.MkdirAll(filepath.Dir(data.path), os.ModePerm) |
There was a problem hiding this comment.
os.ModePerm is 0777, i.e. anyone on the system can read/write/execute/cd into it. I doubt that's what we want to set here. I don't see a set of default permissions mentioned in the SEP, I would seek clarification from the authors.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Implement v2 s3 transfer manager's DownloadDirectory api bound to single union client which mimics normal service client's initialization and api call. User could now download a bucket's objects to local directory with customized filtering and file naming in a single operation call.
Test: This PR passed unit tests for DownloadDirectory which covers use cases with different config and objects naming to confirm this key-prefix downloader could find valid objects, generate correct file path and pass download requests to S3 concurrently. It also passed integration test which uploads objects with different sizes to S3 via S3 client, then downloads objects back using new directory downloader and checks their content consistency