-
Notifications
You must be signed in to change notification settings - Fork 931
Add S3 Pre-signed URL GET design document #6203
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
joviegas
merged 12 commits into
aws:master
from
jencymaryjoseph:feature/presignedUrl-GET
Jun 27, 2025
Merged
Changes from 7 commits
Commits
Show all changes
12 commits
Select commit
Hold shift + click to select a range
f362ac1
Add S3 Pre-signed URL GET design document
jencymaryjoseph ab2f8c2
Update Design.md
jencymaryjoseph f5a97f6
Update DecisionLog.md
jencymaryjoseph 6799c87
Updated Design.md
jencymaryjoseph 073245f
Updated DecisionLog.md
jencymaryjoseph 531c911
Remove images folder
jencymaryjoseph f4d7592
Merge branch 'feature/presignedUrl-GET' of github.com:jencymaryjoseph…
jencymaryjoseph 6a77d7e
Updated Design.md - removed codegen annotations
jencymaryjoseph a3d2bef
Update Design.md - addressed comments
jencymaryjoseph bc65e50
Merge branch 'master' into feature/presignedUrl-GET
joviegas 2358558
Merge branch 'master' into feature/presignedUrl-GET
joviegas e16e386
Merge branch 'master' into feature/presignedUrl-GET
joviegas File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,32 @@ | ||
|
||
# S3 Pre-signed URL GET - Decision Log | ||
|
||
## Review Meeting: 06/17/2024 | ||
**Attendees**: Alban, John, Zoe, Dongie, Bole, Ran, Saranya | ||
|
||
### Closed Decisions | ||
|
||
1. Create a new PresignedUrlGetObjectResponse specifically for pre-signed URLs, or use the existing GetObjectResponse? Decided to use the existing GetObjectResponse for pre-signed URL operations as the HTTP response from a pre-signed URL GET is same as a standard S3 GetObject response. | ||
|
||
2. Use the existing SDK and S3 exceptions or implement specialized exceptions for validation errors like expired URLs? Decided to utilize existing SDK exceptions rather than creating specialized ones for pre-signed URL operations. | ||
|
||
3. Provide additional client-side validation with server-side validation as fallback or just rely entirely on server-side validation from S3? No additional client-side validation will be implemented for pre-signed URLs. | ||
|
||
### Discussions Addressed | ||
|
||
1. Are there alternative methods to skip signing, such as using NoOpSigner(), instead of setting additional Execution attributes? Added the use of NoOpSigner() in the design doc. | ||
|
||
2. Does the S3 response include a checksum? If so, should checksum-related support be implemented in this project, or deferred until after Transfer Manager support is added? S3 Response doesn't include checksum. | ||
|
||
3. What should we name the Helper API? Options include PresignedURLManager or PresignedUrlExtension. Will be addressed in the Surface API Review. | ||
|
||
## Review Meeting: 06/23/2024 | ||
**Attendees**: John, Zoe, Dongie, Bole, Ran, Saranya, Alex, David | ||
|
||
### Decisions Addressed | ||
|
||
1. Should PresignedUrlGetObjectRequest extend S3Request/SdkRequest? Decided to use a standalone request class with minimal parameters (presignedUrl, rangeStart, rangeEnd) to avoid exposing incompatible configurations like credentials and signers. Internally convert to S3Request for ClientHandler compatibility. | ||
|
||
2. Replace IS_DISCOVERED_ENDPOINT execution attribute with a more semantically appropriate solution. Decided to introduce new SKIP_ENDPOINT_RESOLUTION execution attribute specifically for presigned URL scenarios where endpoint resolution should be bypassed, as IS_DISCOVERED_ENDPOINT is tied to deprecated endpoint discovery feature. | ||
|
||
3. Use separate rangeStart/rangeEnd fields vs single range string parameter. Decided to use separate rangeStart and rangeEnd Long fields for better user experience, as start/end is more intuitive than string parsing. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,188 @@ | ||
# Design Document (S3 Pre-signed URL GET) | ||
|
||
## Introduction | ||
|
||
This design introduces S3 object downloads using pre-signed URLs in AWS SDK Java v2, providing feature parity with [v1](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/transfer/PresignedUrlDownload.html). Some customers have described a need for downloading S3 objects through pre-signed URLs while maintaining Client side SDK benefits like automatic retries, metrics collection, and typed response objects. | ||
|
||
This document proposes how this functionality should be implemented in the Java SDK v2, addressing customer-requested features ([GitHub Issue #2731](https://github.com/aws/aws-sdk-java-v2/issues/2731), [GitHub Issue #181](https://github.com/aws/aws-sdk-java-v2/issues/181)) by reducing complexity and improving usability for temporary access scenarios. | ||
|
||
## Design Review | ||
|
||
Look at decision log here: [Decision Log Section](DecisionLog.md) | ||
|
||
The Java SDK team has decided to implement a separate `PresignedUrlManager`. The team chose the helper API pattern over direct `S3Client` integration to maintain clean separation of concerns while preserving SDK functionality. | ||
|
||
## Overview | ||
|
||
The design introduces new helper APIs `AsyncPresignedUrlManager` and `PresignedUrlManager` which can be instantiated via the existing `S3AsyncClient` and `S3Client` respectively. These managers provide a clean abstraction layer that preserves SDK benefits while handling the unique requirements of pre-signed URL requests. | ||
|
||
This design will implement only the GET /download function for presigned URLs. | ||
|
||
|
||
|
||
## Proposed APIs | ||
|
||
The v2 SDK will support a presigned URL manager for both sync and async clients that can leverage pre-signed URL downloads. | ||
|
||
### Instantiation | ||
Instantiating from existing client: | ||
|
||
```java | ||
// Async Presigned URL Manager | ||
S3AsyncClient s3Client = S3AsyncClient.create(); | ||
AsyncPresignedUrlManager presignManager = s3Client.presignedManager(); | ||
|
||
// Sync Presigned URL Manager | ||
S3Client s3Client = S3Client.create(); | ||
PresignedUrlManager presignManager = s3Client.presignedManager(); | ||
``` | ||
|
||
### General Usage Examples | ||
|
||
```java | ||
// Create presigned URL request | ||
PresignedUrlGetObjectRequest request = PresignedUrlGetObjectRequest.builder() | ||
.presignedUrl(presignedUrl) | ||
.rangeStart(0L) | ||
.rangeEnd(1024L) | ||
.build(); | ||
|
||
// Async usage | ||
S3AsyncClient s3Client = S3AsyncClient.create(); | ||
AsyncPresignedUrlManager presignManager = s3Client.presignedManager(); | ||
CompletableFuture<GetObjectResponse> response = presignManager.getObject(request); | ||
|
||
// Sync usage | ||
S3Client s3Client = S3Client.create(); | ||
PresignedUrlManager presignManager = s3Client.presignedManager(); | ||
GetObjectResponse response = presignManager.getObject(request); | ||
``` | ||
|
||
### AsyncPresignedUrlManager Interface | ||
|
||
```java | ||
/** | ||
* Presigned URL Manager class that implements presigned URL features for an async client. | ||
*/ | ||
@SdkPublicApi | ||
@Generated("software.amazon.awssdk:codegen") | ||
public interface AsyncPresignedUrlManager { | ||
|
||
/** | ||
* Downloads S3 objects using pre-signed URLs. Bypasses normal authentication | ||
* and endpoint resolution while maintaining SDK benefits like retries and metrics. | ||
* | ||
* @param request the presigned URL request containing URL and optional range parameters. | ||
* @return a CompletableFuture of the corresponding GetObjectResponse. | ||
*/ | ||
CompletableFuture<GetObjectResponse> getObject(PresignedUrlGetObjectRequest request); | ||
jencymaryjoseph marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
/** | ||
* Downloads S3 objects using pre-signed URLs with custom response transformation. | ||
* | ||
* @param request the presigned URL request. | ||
* @param responseTransformer custom transformer for processing the response. | ||
* @return a CompletableFuture of the transformed response. | ||
*/ | ||
<T> CompletableFuture<T> getObject(PresignedUrlGetObjectRequest request, | ||
AsyncResponseTransformer<GetObjectResponse, T> responseTransformer); | ||
|
||
// Additional getObject() overloads for file downloads, byte arrays, etc. | ||
// Standard Builder interface with client() and overrideConfiguration() methods | ||
} | ||
``` | ||
|
||
### PresignedUrlManager Interface | ||
|
||
```java | ||
/** | ||
* Presigned URL Manager class that implements presigned URL features for a sync client. | ||
jencymaryjoseph marked this conversation as resolved.
Show resolved
Hide resolved
|
||
*/ | ||
@SdkPublicApi | ||
@Generated("software.amazon.awssdk:codegen") | ||
jencymaryjoseph marked this conversation as resolved.
Show resolved
Hide resolved
|
||
public interface PresignedUrlManager { | ||
|
||
/** | ||
* Downloads S3 objects using pre-signed URLs. Bypasses normal authentication | ||
* and endpoint resolution while maintaining SDK benefits like retries and metrics. | ||
* | ||
* @param request the presigned URL request containing URL and optional range parameters. | ||
* @return the GetObjectResponse. | ||
*/ | ||
GetObjectResponse getObject(PresignedUrlGetObjectRequest request); | ||
|
||
/** | ||
* Downloads S3 objects using pre-signed URLs with custom response transformation. | ||
* | ||
* @param request the presigned URL request. | ||
* @param responseTransformer custom transformer for processing the response. | ||
* @return the transformed response. | ||
*/ | ||
<T> T getObject(PresignedUrlGetObjectRequest request, | ||
ResponseTransformer<GetObjectResponse, T> responseTransformer); | ||
|
||
// Additional getObject() overloads for file downloads, byte arrays, etc. | ||
// Standard Builder interface with client() and overrideConfiguration() methods | ||
} | ||
``` | ||
|
||
### PresignedUrlGetObjectRequest | ||
|
||
```java | ||
/** | ||
* Request object for presigned URL GET operations. | ||
*/ | ||
@SdkPublicApi | ||
@Immutable | ||
@ThreadSafe | ||
public final class PresignedUrlGetObjectRequest | ||
implements ToCopyableBuilder<PresignedUrlGetObjectRequest.Builder, PresignedUrlGetObjectRequest> { | ||
|
||
private final String presignedUrl; | ||
private final Long rangeStart; | ||
private final Long rangeEnd; | ||
|
||
// Standard getters: presignedUrl(), rangeStart(), rangeEnd() | ||
// Standard builder methods: builder(), toBuilder() | ||
// Standard Builder class with presignedUrl(), rangeStart(), rangeEnd() setter methods | ||
} | ||
``` | ||
|
||
## FAQ | ||
|
||
### Why don't we implement presigned URL download/GET feature directly on the S3Client? | ||
|
||
Three approaches were considered: | ||
|
||
1. **Dedicated PresignedUrlManager (CHOSEN)**: Separate manager accessed via `s3Client.presignedManager()` | ||
- **Pros**: Clean separation, preserves SDK features, follows v2 patterns | ||
- **Cons**: New API surface for users to learn | ||
|
||
2. **Direct S3Client Integration**: Add presigned URL methods directly to S3Client | ||
- **Pros**: Familiar interface, direct migration path from v1 | ||
- **Cons**: Requires core interceptor changes, complex integration | ||
jencymaryjoseph marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
3. **S3Presigner Extension**: Extend existing S3Presigner to execute URLs | ||
- **Pros**: Logical extension of presigner concept | ||
- **Cons**: Breaks current stateless presigner patterns | ||
|
||
**Decision**: Option 1 provides clean separation while preserving SDK benefits and following established v2 utility patterns.cutePresignedGetObject(presignedRequest); | ||
|
||
### Why doesn't PresignedUrlGetObjectRequest extend S3Request? | ||
|
||
While extending S3Request would provide access to RequestOverrideConfiguration, many of these configurations (like credentials provider, signers) are not supported with presigned URL execution and could cause conflicts. Instead, we use a standalone request with only essential parameters (presignedUrl, rangeStart, rangeEnd). Internally, this gets wrapped in an encapsulated class that extends S3Request for use with ClientHandler. | ||
jencymaryjoseph marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
|
||
## References | ||
|
||
**GitHub feature requests:** | ||
- [S3 Presigned URL Support #2731](https://github.com/aws/aws-sdk-java-v2/issues/2731) | ||
- [Presigned URL GET Support #181](https://github.com/aws/aws-sdk-java-v2/issues/181) | ||
|
||
**AWS Documentation:** | ||
- [S3 Pre-signed URLs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/presigned-urls.html) | ||
|
||
**SDK Documentation:** | ||
- [AWS SDK for Java v1 implementation](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/welcome.html) | ||
- [S3 Client architecture patterns](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) | ||
|
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.