Skip to content

Commit 4e722f2

Browse files
DaveCTurnerDaveCTurner
authored
Upgrade discovery-ec2 to AWS SDK v2 (elastic#125072)
* Upgrade `discovery-ec2` to AWS SDK v2 Backport of elastic#122062 to 8.x * Remove stray UpdateForV10
1 parent 653313b commit 4e722f2

37 files changed

+1057
-1007
lines changed

‎docs/changelog/122062.yaml‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
pr: 122062
2+
summary: Upgrade `discovery-ec2` to AWS SDK v2
3+
area: Discovery-Plugins
4+
type: breaking
5+
issues: []
6+
breaking:
7+
title: Upgrade `discovery-ec2` to AWS SDK v2
8+
area: Cluster and node setting
9+
details: >-
10+
11+
In earlier versions of {es} the `discovery-ec2` plugin was based on the AWS
12+
SDK v1. AWS will withdraw support for this SDK before the end of the life
13+
of {es} {minor-version} so we must migrate to the newer AWS SDK v2.
14+
15+
Unfortunately there are several differences between the two AWS SDK
16+
versions which may require you to adjust your system configuration when
17+
upgrading to {es} {minor-version} or later. These differences include, but
18+
may not be limited to, the following items.
19+
20+
* AWS SDK v2 does not support the EC2 IMDSv1 protocol.
21+
22+
* AWS SDK v2 does not support the `aws.secretKey` or
23+
`com.amazonaws.sdk.ec2MetadataServiceEndpointOverride` system properties.
24+
25+
* AWS SDK v2 does not permit specifying a choice between HTTP and HTTPS so
26+
the `discovery.ec2.protocol` setting is no longer effective.
27+
28+
* AWS SDK v2 does not accept an access key without a secret key or vice
29+
versa.
30+
31+
impact: >-
32+
33+
If you use the `discovery-ec2` plugin, test your upgrade thoroughly before
34+
upgrading any production workloads.
35+
36+
Adapt your configuration to the new SDK functionality. This includes, but
37+
may not be limited to, the following items.
38+
39+
* If you use IMDS to determine the availability zone of a node or to obtain
40+
credentials for accessing the EC2 API, ensure that it supports the IMDSv2
41+
protocol.
42+
43+
* If applicable, discontinue use of the `aws.secretKey` and
44+
`com.amazonaws.sdk.ec2MetadataServiceEndpointOverride` system properties.
45+
46+
* If applicable, specify that you wish to use the insecure HTTP protocol to
47+
access the EC2 API by setting `discovery.ec2.endpoint` to a URL which
48+
starts with `http://`.
49+
50+
* Either supply both an access key and a secret key using the keystore
51+
settings `discovery.ec2.access_key` and `discovery.ec2.secret_key`, or
52+
configure neither of these settings.
53+
54+
notable: true

‎docs/plugins/discovery-ec2.asciidoc‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -83,8 +83,8 @@ The available settings for the EC2 discovery plugin are as follows.
8383

8484
`discovery.ec2.protocol`::
8585

86-
The protocol to use to connect to the EC2 service endpoint, which may be
87-
either `http` or `https`. Defaults to `https`.
86+
This setting has no effect. To use HTTP to connect to the EC2 service
87+
endpoint, set `discovery.ec2.endpoint` to a URL that starts with `http://`.
8888

8989
`discovery.ec2.proxy.host`::
9090

‎gradle/verification-metadata.xml‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4655,6 +4655,11 @@
46554655
<sha256 value="64d8c2bcccd33c20ccdbafa101b01d8e0f750c4e4bd227b0b765046f601eb944" origin="Generated by Gradle"/>
46564656
</artifact>
46574657
</component>
4658+
<component group="software.amazon.awssdk" name="apache-client" version="2.30.38">
4659+
<artifact name="apache-client-2.30.38.jar">
4660+
<sha256 value="ebb1d3d05711ccf2aa9bfc43fcc69fbe32e7be69e006e7952679c2f37d149f4d" origin="Generated by Gradle"/>
4661+
</artifact>
4662+
</component>
46584663
<component group="software.amazon.awssdk" name="auth" version="2.30.38">
46594664
<artifact name="auth-2.30.38.jar">
46604665
<sha256 value="22d59f9af8111be5219eb33ef480d84c616565913da57cb4eac686076fea370e" origin="Generated by Gradle"/>
@@ -4670,6 +4675,11 @@
46704675
<sha256 value="b62be02560a46135181342afc9fb2d99373a9f04f384caf30863e2e7fe5b3892" origin="Generated by Gradle"/>
46714676
</artifact>
46724677
</component>
4678+
<component group="software.amazon.awssdk" name="aws-query-protocol" version="2.30.38">
4679+
<artifact name="aws-query-protocol-2.30.38.jar">
4680+
<sha256 value="bfd558e937de70c3260df2356b47a25b562c59b5ebeded6b199846cc9a354fe5" origin="Generated by Gradle"/>
4681+
</artifact>
4682+
</component>
46734683
<component group="software.amazon.awssdk" name="bedrockruntime" version="2.30.38">
46744684
<artifact name="bedrockruntime-2.30.38.jar">
46754685
<sha256 value="4424437b49fdf263ea460f4da634d3279ada7f4763827d74fea48c0f8f2afea3" origin="Generated by Gradle"/>
@@ -4685,6 +4695,11 @@
46854695
<sha256 value="82d97bcbb18d8f369b00c9971ca8c24ad94769d20836e0c4f86ebcdfea994cdb" origin="Generated by Gradle"/>
46864696
</artifact>
46874697
</component>
4698+
<component group="software.amazon.awssdk" name="ec2" version="2.30.38">
4699+
<artifact name="ec2-2.30.38.jar">
4700+
<sha256 value="a2e52ca80aac79553f9da1463256db0177b12c5c24c1b5660a3bb7874b66222f" origin="Generated by Gradle"/>
4701+
</artifact>
4702+
</component>
46884703
<component group="software.amazon.awssdk" name="endpoints-spi" version="2.30.38">
46894704
<artifact name="endpoints-spi-2.30.38.jar">
46904705
<sha256 value="80620e3020a29871073a8a4efbcaa4d546667eeb92dfd478de808dca7e0500aa" origin="Generated by Gradle"/>
@@ -4720,6 +4735,11 @@
47204735
<sha256 value="e784929d8a51591b6ed51344f41b37f2a68582d2e912e8310ea3e57a56d4d6bf" origin="Generated by Gradle"/>
47214736
</artifact>
47224737
</component>
4738+
<component group="software.amazon.awssdk" name="imds" version="2.30.38">
4739+
<artifact name="imds-2.30.38.jar">
4740+
<sha256 value="1586c80dfee0d09e26ad9cb313529530f1fb75d9360b05b1f1f1ca246f39d1b2" origin="Generated by Gradle"/>
4741+
</artifact>
4742+
</component>
47234743
<component group="software.amazon.awssdk" name="json-utils" version="2.30.38">
47244744
<artifact name="json-utils-2.30.38.jar">
47254745
<sha256 value="823f565bc6d4031e4b3dada05c1e66c1344f34d498344b7186a2f2d048ba01d8" origin="Generated by Gradle"/>

‎plugins/discovery-ec2/build.gradle‎

Lines changed: 94 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ import org.elasticsearch.gradle.internal.info.BuildParams
88
* your election, the "Elastic License 2.0", the "GNU Affero General Public
99
* License v3.0 only", or the "Server Side Public License, v 1".
1010
*/
11+
apply plugin: 'elasticsearch.internal-cluster-test'
1112
apply plugin: 'elasticsearch.internal-java-rest-test'
1213
apply plugin: 'elasticsearch.internal-cluster-test'
1314

@@ -17,30 +18,83 @@ esplugin {
1718
}
1819

1920
dependencies {
20-
api "com.amazonaws:aws-java-sdk-ec2:${versions.awsv1sdk}"
21-
api "com.amazonaws:aws-java-sdk-core:${versions.awsv1sdk}"
22-
api "org.apache.httpcomponents:httpclient:${versions.httpclient}"
23-
api "org.apache.httpcomponents:httpcore:${versions.httpcore}"
24-
api "commons-logging:commons-logging:${versions.commonslogging}"
25-
api "org.apache.logging.log4j:log4j-1.2-api:${versions.log4j}"
26-
api "commons-codec:commons-codec:${versions.commonscodec}"
27-
api "com.fasterxml.jackson.core:jackson-core:${versions.jackson}"
28-
api "com.fasterxml.jackson.core:jackson-databind:${versions.jackson}"
29-
api "com.fasterxml.jackson.core:jackson-annotations:${versions.jackson}"
30-
api "com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:${versions.jackson}"
31-
api "joda-time:joda-time:2.10.10"
21+
22+
implementation "software.amazon.awssdk:annotations:${versions.awsv2sdk}"
23+
implementation "software.amazon.awssdk:apache-client:${versions.awsv2sdk}"
24+
implementation "software.amazon.awssdk:auth:${versions.awsv2sdk}"
25+
implementation "software.amazon.awssdk:aws-core:${versions.awsv2sdk}"
26+
implementation "software.amazon.awssdk:ec2:${versions.awsv2sdk}"
27+
implementation "software.amazon.awssdk:endpoints-spi:${versions.awsv2sdk}"
28+
implementation "software.amazon.awssdk:http-client-spi:${versions.awsv2sdk}"
29+
implementation "software.amazon.awssdk:identity-spi:${versions.awsv2sdk}"
30+
implementation "software.amazon.awssdk:imds:${versions.awsv2sdk}"
31+
implementation "software.amazon.awssdk:retries-spi:${versions.awsv2sdk}"
32+
implementation "software.amazon.awssdk:sdk-core:${versions.awsv2sdk}"
33+
implementation "software.amazon.awssdk:utils:${versions.awsv2sdk}"
34+
35+
runtimeOnly "software.amazon.awssdk:aws-query-protocol:${versions.awsv2sdk}"
36+
runtimeOnly "software.amazon.awssdk:checksums-spi:${versions.awsv2sdk}"
37+
runtimeOnly "software.amazon.awssdk:checksums:${versions.awsv2sdk}"
38+
runtimeOnly "software.amazon.awssdk:http-auth-aws:${versions.awsv2sdk}"
39+
runtimeOnly "software.amazon.awssdk:http-auth-spi:${versions.awsv2sdk}"
40+
runtimeOnly "software.amazon.awssdk:http-auth:${versions.awsv2sdk}"
41+
runtimeOnly "software.amazon.awssdk:json-utils:${versions.awsv2sdk}"
42+
runtimeOnly "software.amazon.awssdk:metrics-spi:${versions.awsv2sdk}"
43+
runtimeOnly "software.amazon.awssdk:profiles:${versions.awsv2sdk}"
44+
runtimeOnly "software.amazon.awssdk:protocol-core:${versions.awsv2sdk}"
45+
runtimeOnly "software.amazon.awssdk:regions:${versions.awsv2sdk}"
46+
runtimeOnly "software.amazon.awssdk:retries:${versions.awsv2sdk}"
47+
runtimeOnly "software.amazon.awssdk:third-party-jackson-core:${versions.awsv2sdk}"
48+
49+
implementation "org.apache.httpcomponents:httpclient:${versions.httpclient}"
50+
51+
runtimeOnly "commons-codec:commons-codec:${versions.commonscodec}"
52+
runtimeOnly "commons-logging:commons-logging:${versions.commonslogging}"
53+
runtimeOnly "joda-time:joda-time:2.10.10"
54+
runtimeOnly "org.apache.httpcomponents:httpcore:${versions.httpcore}"
55+
runtimeOnly "org.apache.logging.log4j:log4j-1.2-api:${versions.log4j}"
56+
runtimeOnly "org.slf4j:slf4j-nop:${versions.slf4j}"
57+
// runtimeOnly("org.apache.logging.log4j:log4j-slf4j-impl:${versions.log4j}") https://github.com/elastic/elasticsearch/issues/93714
58+
runtimeOnly "org.slf4j:slf4j-api:${versions.slf4j}"
59+
runtimeOnly "org.reactivestreams:reactive-streams:${versions.reactive_streams}"
3260

3361
javaRestTestImplementation project(':plugins:discovery-ec2')
3462
javaRestTestImplementation project(':test:fixtures:aws-fixture-utils')
3563
javaRestTestImplementation project(':test:fixtures:aws-ec2-fixture')
3664
javaRestTestImplementation project(':test:fixtures:ec2-imds-fixture')
3765

66+
testImplementation project(':test:fixtures:aws-fixture-utils')
67+
testImplementation project(':test:fixtures:ec2-imds-fixture')
68+
3869
internalClusterTestImplementation project(':test:fixtures:ec2-imds-fixture')
3970
}
4071

4172
tasks.named("dependencyLicenses").configure {
42-
mapping from: /aws-java-sdk-.*/, to: 'aws-java-sdk'
43-
mapping from: /jackson-.*/, to: 'jackson'
73+
mapping from: 'annotations', to: 'aws-sdk-2'
74+
mapping from: 'apache-client', to: 'aws-sdk-2'
75+
mapping from: 'auth', to: 'aws-sdk-2'
76+
mapping from: 'aws-core', to: 'aws-sdk-2'
77+
mapping from: 'aws-query-protocol', to: 'aws-sdk-2'
78+
mapping from: 'checksums', to: 'aws-sdk-2'
79+
mapping from: 'checksums-spi', to: 'aws-sdk-2'
80+
mapping from: 'ec2', to: 'aws-sdk-2'
81+
mapping from: 'endpoints-spi', to: 'aws-sdk-2'
82+
mapping from: 'http-auth', to: 'aws-sdk-2'
83+
mapping from: 'http-auth-aws', to: 'aws-sdk-2'
84+
mapping from: 'http-auth-spi', to: 'aws-sdk-2'
85+
mapping from: 'http-client-spi', to: 'aws-sdk-2'
86+
mapping from: 'identity-spi', to: 'aws-sdk-2'
87+
mapping from: 'imds', to: 'aws-sdk-2'
88+
mapping from: 'json-utils', to: 'aws-sdk-2'
89+
mapping from: 'metrics-spi', to: 'aws-sdk-2'
90+
mapping from: 'profiles', to: 'aws-sdk-2'
91+
mapping from: 'protocol-core', to: 'aws-sdk-2'
92+
mapping from: 'regions', to: 'aws-sdk-2'
93+
mapping from: 'retries', to: 'aws-sdk-2'
94+
mapping from: 'retries-spi', to: 'aws-sdk-2'
95+
mapping from: 'sdk-core', to: 'aws-sdk-2'
96+
mapping from: 'third-party-jackson-core', to: 'aws-sdk-2'
97+
mapping from: 'utils', to: 'aws-sdk-2'
4498
}
4599

46100
esplugin.bundleSpec.from('config/discovery-ec2') {
@@ -69,19 +123,23 @@ tasks.register("writeTestJavaPolicy") {
69123
"permission org.bouncycastle.crypto.CryptoServicesPermission \"exportSecretKey\";",
70124
"permission org.bouncycastle.crypto.CryptoServicesPermission \"exportPrivateKey\";",
71125
"permission java.io.FilePermission \"\${javax.net.ssl.trustStore}\", \"read\";",
72-
"permission java.util.PropertyPermission \"com.amazonaws.sdk.ec2MetadataServiceEndpointOverride\", \"write\";",
73126
"permission java.security.SecurityPermission \"getProperty.jdk.tls.disabledAlgorithms\";",
74127
"permission java.security.SecurityPermission \"getProperty.jdk.certpath.disabledAlgorithms\";",
75128
"permission java.security.SecurityPermission \"getProperty.keystore.type.compat\";",
76129
"permission java.security.SecurityPermission \"getProperty.org.bouncycastle.ec.max_f2m_field_size\";",
130+
"permission java.util.PropertyPermission \"aws.ec2MetadataServiceEndpoint\", \"write\";",
131+
"permission java.util.PropertyPermission \"http.proxyHost\", \"read\";",
132+
"permission java.util.PropertyPermission \"aws.region\", \"read\";",
77133
"};"
78134
].join("\n")
79135
)
80136
} else {
81137
javaPolicy.write(
82138
[
83139
"grant {",
84-
" permission java.util.PropertyPermission \"com.amazonaws.sdk.ec2MetadataServiceEndpointOverride\", \"write\";",
140+
"permission java.util.PropertyPermission \"aws.ec2MetadataServiceEndpoint\", \"write\";",
141+
"permission java.util.PropertyPermission \"http.proxyHost\", \"read\";",
142+
"permission java.util.PropertyPermission \"aws.region\", \"read\";",
85143
"};"
86144
].join("\n"))
87145
}
@@ -93,27 +151,42 @@ tasks.withType(Test).configureEach {
93151
// this is needed for insecure plugins, remove if possible!
94152
systemProperty 'tests.artifact', project.name
95153

96-
// this is needed to manipulate com.amazonaws.sdk.ec2MetadataServiceEndpointOverride system property
154+
// this is needed to manipulate aws.ec2MetadataServiceEndpoint system property
97155
// it is better rather disable security manager at all with `systemProperty 'tests.security.manager', 'false'`
98156
if (buildParams.inFipsJvm){
99157
nonInputProperties.systemProperty 'java.security.policy', "=file://${layout.buildDirectory.asFile.get()}/tmp/java.policy"
100158
} else {
101159
nonInputProperties.systemProperty 'java.security.policy', "file://${layout.buildDirectory.asFile.get()}/tmp/java.policy"
102160
}
161+
162+
systemProperty 'aws.region', 'es-test-region'
103163
}
104164

105165
tasks.named("thirdPartyAudit").configure {
106166
ignoreMissingClasses(
107167
// classes are missing
108-
'com.amazonaws.jmespath.JmesPathExpression',
109-
'com.amazonaws.jmespath.ObjectMapperSingleton',
110168
'javax.servlet.ServletContextEvent',
111169
'javax.servlet.ServletContextListener',
112170
'org.apache.avalon.framework.logger.Logger',
113171
'org.apache.log.Hierarchy',
114172
'org.apache.log.Logger',
115173
'javax.jms.Message',
116-
'javax.xml.bind.DatatypeConverter',
117-
'javax.xml.bind.JAXBContext'
174+
175+
// eventstream not used by the sync client
176+
'software.amazon.eventstream.HeaderValue',
177+
'software.amazon.eventstream.Message',
178+
'software.amazon.eventstream.MessageDecoder',
179+
180+
// crt?
181+
'software.amazon.awssdk.crt.auth.credentials.Credentials',
182+
'software.amazon.awssdk.crt.auth.signing.AwsSigner',
183+
'software.amazon.awssdk.crt.auth.signing.AwsSigningConfig',
184+
'software.amazon.awssdk.crt.auth.signing.AwsSigningConfig$AwsSignatureType',
185+
'software.amazon.awssdk.crt.auth.signing.AwsSigningConfig$AwsSignedBodyHeaderType',
186+
'software.amazon.awssdk.crt.auth.signing.AwsSigningConfig$AwsSigningAlgorithm',
187+
'software.amazon.awssdk.crt.auth.signing.AwsSigningResult',
188+
'software.amazon.awssdk.crt.http.HttpHeader',
189+
'software.amazon.awssdk.crt.http.HttpRequest',
190+
'software.amazon.awssdk.crt.http.HttpRequestBodyStream',
118191
)
119192
}

0 commit comments

Comments
 (0)