-
Notifications
You must be signed in to change notification settings - Fork 25.6k
Upgrade repository-s3 to AWS SDK v2
#126843
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
elasticsearchmachine
merged 36 commits into
elastic:main
from
DaveCTurner:2025/04/15/repository-s3-sdk-v2
Apr 24, 2025
Merged
Changes from 4 commits
Commits
Show all changes
36 commits
Select commit
Hold shift + click to select a range
c265fb9
Upgrade `repository-s3` to AWS SDK v2
DaveCTurner d9f7439
Update docs/changelog/126843.yaml
DaveCTurner 897557f
Replace changelog
DaveCTurner 252005f
Remove spurious changelog
DaveCTurner 71d2578
Merge branch 'main' into 2025/04/15/repository-s3-sdk-v2
DaveCTurner d39a4bb
Don't run out of permits
DaveCTurner 420f0a8
asInstanceOf
DaveCTurner a8ad71c
randomPurpose
DaveCTurner c30e7f3
TODOs about multipart cleanups
DaveCTurner bce8992
Comment on CoreMetric.API_CALL_SUCCESSFUL loop
DaveCTurner 787d8cc
Fix loadDeprecatedCredentials comment
DaveCTurner 74d7fe6
NOMERGE
DaveCTurner 33db762
NOMERGE
DaveCTurner c6795d9
Fix AwsStsHttpHandlerTests
DaveCTurner 9014c45
Merge branch 'main' into 2025/04/15/repository-s3-sdk-v2
DaveCTurner 0200361
Comment why AWS_REGION env var in ECS test
DaveCTurner 8f83362
Typo
DaveCTurner 1e51dcd
Implementation details of RegionFromEndpointGuesser
DaveCTurner 25c1bc1
thirdPartyAudit comments
DaveCTurner af60136
No need to pre-load regions
DaveCTurner b47aacc
Move to Javadoc
DaveCTurner 3efa003
Merge branch 'main' into 2025/04/15/repository-s3-sdk-v2
DaveCTurner 6e86bd8
Merge branch 'main' into 2025/04/15/repository-s3-sdk-v2
DaveCTurner bfbf664
Update docs/changelog/126843.yaml
DaveCTurner a449250
Partially revert "Update docs/changelog/126843.yaml"
DaveCTurner 6390e9c
Update docs/changelog/126843.yaml
DaveCTurner a4ac0ef
Revert "Update docs/changelog/126843.yaml"
DaveCTurner 62b93ec
Merge branch 'main' into 2025/04/15/repository-s3-sdk-v2
DaveCTurner c5c0b40
Merge branch 'main' into 2025/04/15/repository-s3-sdk-v2
DaveCTurner ba63e10
Merge branch 'main' into 2025/04/15/repository-s3-sdk-v2
DaveCTurner 48b7838
Merge branch 'main' into 2025/04/15/repository-s3-sdk-v2
DaveCTurner ecd2531
Merge branch 'main' into 2025/04/15/repository-s3-sdk-v2
DaveCTurner 870df07
Merge branch 'main' into 2025/04/15/repository-s3-sdk-v2
DaveCTurner e0df223
More info about region inference
DaveCTurner 7c3f82e
Enable cross-region access sometimes
DaveCTurner 1371d2a
Dedup
DaveCTurner File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,81 @@ | ||
| pr: 126843 | ||
| summary: Upgrade `repository-s3` to AWS SDK v2 | ||
| area: Snapshot/Restore | ||
| type: breaking | ||
| issues: | ||
| - 120993 | ||
| breaking: | ||
| title: Upgrade `repository-s3` to AWS SDK v2 | ||
| area: Cluster and node setting | ||
| details: >- | ||
|
|
||
| In earlier versions of {es} the `repository-s3` plugin was based on the AWS | ||
| SDK v1. AWS will withdraw support for this SDK before the end of the life | ||
| of {es} {minor-version} so we must migrate to the newer AWS SDK v2. | ||
|
|
||
| Unfortunately there are several differences between the two AWS SDK | ||
| versions which may require you to adjust your system configuration when | ||
| upgrading to {es} {minor-version} or later. These differences include, but | ||
| may not be limited to, the following items. | ||
|
|
||
| * AWS SDK v2 requires users to specify the region to use for signing | ||
| requests, or else to run in an environment in which it can determine the | ||
| correct region automatically. The older SDK would try to determine the | ||
| region based on the endpoint URL as specified with the | ||
| `s3.client.${CLIENT_NAME}.endpoint` setting, together with other data | ||
| drawn from the operating environment, and would ultimately fall back to | ||
| `us-east-1` if no better value could be found. | ||
|
|
||
| * AWS SDK v2 does not support the EC2 IMDSv1 protocol. | ||
|
|
||
| * AWS SDK v2 does not support the | ||
| `com.amazonaws.sdk.ec2MetadataServiceEndpointOverride` system property. | ||
|
|
||
| * AWS SDK v2 does not permit specifying a choice between HTTP and HTTPS so | ||
| the `s3.client.${CLIENT_NAME}.protocol` setting is deprecated and no longer | ||
| has any effect. | ||
|
|
||
| * AWS SDK v2 does not permit control over throttling for retries, so the | ||
| the `s3.client.${CLIENT_NAME}.use_throttle_retries` setting is deprecated | ||
| and no longer has any effect. | ||
|
|
||
| * AWS SDK v2 requires the use of the V4 signature algorithm, so the | ||
| `s3.client.${CLIENT_NAME}.signer_override` setting is deprecated and no | ||
| longer has any effect. | ||
|
|
||
| * AWS SDK v2 does not support the `log-delivery-write` canned ACL. | ||
|
|
||
| * AWS SDK v2 counts 4xx responses differently in its metrics reporting. | ||
|
|
||
| * AWS SDK v2 always uses the regional STS endpoint, whereas AWS SDK v2 | ||
| could use either a regional endpoint or the global | ||
| `https://sts.amazonaws.com` one. | ||
|
|
||
| impact: >- | ||
|
|
||
| If you use the `repository-s3` module, test your upgrade thoroughly before | ||
| upgrading any production workloads. | ||
|
|
||
| Adapt your configuration to the new SDK functionality. This includes, but | ||
| may not be limited to, the following items. | ||
|
|
||
| * Specify the correct signing region using the | ||
| `s3.client.${CLIENT_NAME}.region` setting on each node. {es} will try and | ||
| determine the correct region based on the endpoint URL and other data | ||
| drawn from the operating environment but cannot guarantee to do so | ||
| correctly in all cases. | ||
|
|
||
| * If you use IMDS to determine the availability zone of a node or to obtain | ||
| credentials for accessing the EC2 API, ensure that it supports the IMDSv2 | ||
| protocol. | ||
|
|
||
| * If applicable, discontinue use of the | ||
| `com.amazonaws.sdk.ec2MetadataServiceEndpointOverride` system property. | ||
|
|
||
| * If applicable, specify that you wish to use the insecure HTTP protocol to | ||
| access the S3 API by setting `s3.client.${CLIENT_NAME}.endpoint` to a URL | ||
| which starts with `http://`. | ||
|
|
||
| * If applicable, discontinue use of the `log-delivery-write` canned ACL. | ||
|
|
||
| notable: true |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what does crt mean?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Apparently it's a different HTTP client implemented in native code, see https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/http-configuration-crt.html. We don't use it. Comment improved in 25c1bc1.