Skip to content

Commit a2d98e4

Browse files
DaveCTurnerDaveCTurner
authored
Upgrade discovery-ec2 to AWS SDK v2 (elastic#122062)
1 parent 9866a67 commit a2d98e4

37 files changed

+1059
-1008
lines changed

‎docs/changelog/122062.yaml‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
pr: 122062
2+
summary: Upgrade `discovery-ec2` to AWS SDK v2
3+
area: Discovery-Plugins
4+
type: breaking
5+
issues: []
6+
breaking:
7+
title: Upgrade `discovery-ec2` to AWS SDK v2
8+
area: Cluster and node setting
9+
details: >-
10+
11+
In earlier versions of {es} the `discovery-ec2` plugin was based on the AWS
12+
SDK v1. AWS will withdraw support for this SDK before the end of the life
13+
of {es} {minor-version} so we must migrate to the newer AWS SDK v2.
14+
15+
Unfortunately there are several differences between the two AWS SDK
16+
versions which may require you to adjust your system configuration when
17+
upgrading to {es} {minor-version} or later. These differences include, but
18+
may not be limited to, the following items.
19+
20+
* AWS SDK v2 does not support the EC2 IMDSv1 protocol.
21+
22+
* AWS SDK v2 does not support the `aws.secretKey` or
23+
`com.amazonaws.sdk.ec2MetadataServiceEndpointOverride` system properties.
24+
25+
* AWS SDK v2 does not permit specifying a choice between HTTP and HTTPS so
26+
the `discovery.ec2.protocol` setting is no longer effective.
27+
28+
* AWS SDK v2 does not accept an access key without a secret key or vice
29+
versa.
30+
31+
impact: >-
32+
33+
If you use the `discovery-ec2` plugin, test your upgrade thoroughly before
34+
upgrading any production workloads.
35+
36+
Adapt your configuration to the new SDK functionality. This includes, but
37+
may not be limited to, the following items.
38+
39+
* If you use IMDS to determine the availability zone of a node or to obtain
40+
credentials for accessing the EC2 API, ensure that it supports the IMDSv2
41+
protocol.
42+
43+
* If applicable, discontinue use of the `aws.secretKey` and
44+
`com.amazonaws.sdk.ec2MetadataServiceEndpointOverride` system properties.
45+
46+
* If applicable, specify that you wish to use the insecure HTTP protocol to
47+
access the EC2 API by setting `discovery.ec2.endpoint` to a URL which
48+
starts with `http://`.
49+
50+
* Either supply both an access key and a secret key using the keystore
51+
settings `discovery.ec2.access_key` and `discovery.ec2.secret_key`, or
52+
configure neither of these settings.
53+
54+
notable: true

‎docs/reference/elasticsearch-plugins/discovery-ec2-usage.md‎

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -43,9 +43,6 @@ The available settings for the EC2 discovery plugin are as follows.
4343
`discovery.ec2.endpoint`
4444
: The EC2 service endpoint to which to connect. See [https://docs.aws.amazon.com/general/latest/gr/rande.html#ec2_region](https://docs.aws.amazon.com/general/latest/gr/rande.html#ec2_region) to find the appropriate endpoint for the region. This setting defaults to `ec2.us-east-1.amazonaws.com` which is appropriate for clusters running in the `us-east-1` region.
4545

46-
`discovery.ec2.protocol`
47-
: The protocol to use to connect to the EC2 service endpoint, which may be either `http` or `https`. Defaults to `https`.
48-
4946
`discovery.ec2.proxy.host`
5047
: The address or host name of an HTTP proxy through which to connect to EC2. If not set, no proxy is used.
5148

‎gradle/verification-metadata.xml‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4684,6 +4684,11 @@
46844684
<sha256 value="64d8c2bcccd33c20ccdbafa101b01d8e0f750c4e4bd227b0b765046f601eb944" origin="Generated by Gradle"/>
46854685
</artifact>
46864686
</component>
4687+
<component group="software.amazon.awssdk" name="apache-client" version="2.30.38">
4688+
<artifact name="apache-client-2.30.38.jar">
4689+
<sha256 value="ebb1d3d05711ccf2aa9bfc43fcc69fbe32e7be69e006e7952679c2f37d149f4d" origin="Generated by Gradle"/>
4690+
</artifact>
4691+
</component>
46874692
<component group="software.amazon.awssdk" name="auth" version="2.30.38">
46884693
<artifact name="auth-2.30.38.jar">
46894694
<sha256 value="22d59f9af8111be5219eb33ef480d84c616565913da57cb4eac686076fea370e" origin="Generated by Gradle"/>
@@ -4699,6 +4704,11 @@
46994704
<sha256 value="b62be02560a46135181342afc9fb2d99373a9f04f384caf30863e2e7fe5b3892" origin="Generated by Gradle"/>
47004705
</artifact>
47014706
</component>
4707+
<component group="software.amazon.awssdk" name="aws-query-protocol" version="2.30.38">
4708+
<artifact name="aws-query-protocol-2.30.38.jar">
4709+
<sha256 value="bfd558e937de70c3260df2356b47a25b562c59b5ebeded6b199846cc9a354fe5" origin="Generated by Gradle"/>
4710+
</artifact>
4711+
</component>
47024712
<component group="software.amazon.awssdk" name="bedrockruntime" version="2.30.38">
47034713
<artifact name="bedrockruntime-2.30.38.jar">
47044714
<sha256 value="4424437b49fdf263ea460f4da634d3279ada7f4763827d74fea48c0f8f2afea3" origin="Generated by Gradle"/>
@@ -4714,6 +4724,11 @@
47144724
<sha256 value="82d97bcbb18d8f369b00c9971ca8c24ad94769d20836e0c4f86ebcdfea994cdb" origin="Generated by Gradle"/>
47154725
</artifact>
47164726
</component>
4727+
<component group="software.amazon.awssdk" name="ec2" version="2.30.38">
4728+
<artifact name="ec2-2.30.38.jar">
4729+
<sha256 value="a2e52ca80aac79553f9da1463256db0177b12c5c24c1b5660a3bb7874b66222f" origin="Generated by Gradle"/>
4730+
</artifact>
4731+
</component>
47174732
<component group="software.amazon.awssdk" name="endpoints-spi" version="2.30.38">
47184733
<artifact name="endpoints-spi-2.30.38.jar">
47194734
<sha256 value="80620e3020a29871073a8a4efbcaa4d546667eeb92dfd478de808dca7e0500aa" origin="Generated by Gradle"/>
@@ -4749,6 +4764,11 @@
47494764
<sha256 value="e784929d8a51591b6ed51344f41b37f2a68582d2e912e8310ea3e57a56d4d6bf" origin="Generated by Gradle"/>
47504765
</artifact>
47514766
</component>
4767+
<component group="software.amazon.awssdk" name="imds" version="2.30.38">
4768+
<artifact name="imds-2.30.38.jar">
4769+
<sha256 value="1586c80dfee0d09e26ad9cb313529530f1fb75d9360b05b1f1f1ca246f39d1b2" origin="Generated by Gradle"/>
4770+
</artifact>
4771+
</component>
47524772
<component group="software.amazon.awssdk" name="json-utils" version="2.30.38">
47534773
<artifact name="json-utils-2.30.38.jar">
47544774
<sha256 value="823f565bc6d4031e4b3dada05c1e66c1344f34d498344b7186a2f2d048ba01d8" origin="Generated by Gradle"/>

‎plugins/discovery-ec2/build.gradle‎

Lines changed: 94 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@
66
* your election, the "Elastic License 2.0", the "GNU Affero General Public
77
* License v3.0 only", or the "Server Side Public License, v 1".
88
*/
9+
apply plugin: 'elasticsearch.internal-cluster-test'
910
apply plugin: 'elasticsearch.internal-java-rest-test'
1011
apply plugin: 'elasticsearch.internal-cluster-test'
1112

@@ -15,30 +16,83 @@ esplugin {
1516
}
1617

1718
dependencies {
18-
api "com.amazonaws:aws-java-sdk-ec2:${versions.awsv1sdk}"
19-
api "com.amazonaws:aws-java-sdk-core:${versions.awsv1sdk}"
20-
api "org.apache.httpcomponents:httpclient:${versions.httpclient}"
21-
api "org.apache.httpcomponents:httpcore:${versions.httpcore}"
22-
api "commons-logging:commons-logging:${versions.commonslogging}"
23-
api "org.apache.logging.log4j:log4j-1.2-api:${versions.log4j}"
24-
api "commons-codec:commons-codec:${versions.commonscodec}"
25-
api "com.fasterxml.jackson.core:jackson-core:${versions.jackson}"
26-
api "com.fasterxml.jackson.core:jackson-databind:${versions.jackson}"
27-
api "com.fasterxml.jackson.core:jackson-annotations:${versions.jackson}"
28-
api "com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:${versions.jackson}"
29-
api "joda-time:joda-time:2.10.10"
19+
20+
implementation "software.amazon.awssdk:annotations:${versions.awsv2sdk}"
21+
implementation "software.amazon.awssdk:apache-client:${versions.awsv2sdk}"
22+
implementation "software.amazon.awssdk:auth:${versions.awsv2sdk}"
23+
implementation "software.amazon.awssdk:aws-core:${versions.awsv2sdk}"
24+
implementation "software.amazon.awssdk:ec2:${versions.awsv2sdk}"
25+
implementation "software.amazon.awssdk:endpoints-spi:${versions.awsv2sdk}"
26+
implementation "software.amazon.awssdk:http-client-spi:${versions.awsv2sdk}"
27+
implementation "software.amazon.awssdk:identity-spi:${versions.awsv2sdk}"
28+
implementation "software.amazon.awssdk:imds:${versions.awsv2sdk}"
29+
implementation "software.amazon.awssdk:retries-spi:${versions.awsv2sdk}"
30+
implementation "software.amazon.awssdk:sdk-core:${versions.awsv2sdk}"
31+
implementation "software.amazon.awssdk:utils:${versions.awsv2sdk}"
32+
33+
runtimeOnly "software.amazon.awssdk:aws-query-protocol:${versions.awsv2sdk}"
34+
runtimeOnly "software.amazon.awssdk:checksums-spi:${versions.awsv2sdk}"
35+
runtimeOnly "software.amazon.awssdk:checksums:${versions.awsv2sdk}"
36+
runtimeOnly "software.amazon.awssdk:http-auth-aws:${versions.awsv2sdk}"
37+
runtimeOnly "software.amazon.awssdk:http-auth-spi:${versions.awsv2sdk}"
38+
runtimeOnly "software.amazon.awssdk:http-auth:${versions.awsv2sdk}"
39+
runtimeOnly "software.amazon.awssdk:json-utils:${versions.awsv2sdk}"
40+
runtimeOnly "software.amazon.awssdk:metrics-spi:${versions.awsv2sdk}"
41+
runtimeOnly "software.amazon.awssdk:profiles:${versions.awsv2sdk}"
42+
runtimeOnly "software.amazon.awssdk:protocol-core:${versions.awsv2sdk}"
43+
runtimeOnly "software.amazon.awssdk:regions:${versions.awsv2sdk}"
44+
runtimeOnly "software.amazon.awssdk:retries:${versions.awsv2sdk}"
45+
runtimeOnly "software.amazon.awssdk:third-party-jackson-core:${versions.awsv2sdk}"
46+
47+
implementation "org.apache.httpcomponents:httpclient:${versions.httpclient}"
48+
49+
runtimeOnly "commons-codec:commons-codec:${versions.commonscodec}"
50+
runtimeOnly "commons-logging:commons-logging:${versions.commonslogging}"
51+
runtimeOnly "joda-time:joda-time:2.10.10"
52+
runtimeOnly "org.apache.httpcomponents:httpcore:${versions.httpcore}"
53+
runtimeOnly "org.apache.logging.log4j:log4j-1.2-api:${versions.log4j}"
54+
runtimeOnly "org.slf4j:slf4j-nop:${versions.slf4j}"
55+
// runtimeOnly("org.apache.logging.log4j:log4j-slf4j-impl:${versions.log4j}") https://github.com/elastic/elasticsearch/issues/93714
56+
runtimeOnly "org.slf4j:slf4j-api:${versions.slf4j}"
57+
runtimeOnly "org.reactivestreams:reactive-streams:${versions.reactive_streams}"
3058

3159
javaRestTestImplementation project(':plugins:discovery-ec2')
3260
javaRestTestImplementation project(':test:fixtures:aws-fixture-utils')
3361
javaRestTestImplementation project(':test:fixtures:aws-ec2-fixture')
3462
javaRestTestImplementation project(':test:fixtures:ec2-imds-fixture')
3563

64+
testImplementation project(':test:fixtures:aws-fixture-utils')
65+
testImplementation project(':test:fixtures:ec2-imds-fixture')
66+
3667
internalClusterTestImplementation project(':test:fixtures:ec2-imds-fixture')
3768
}
3869

3970
tasks.named("dependencyLicenses").configure {
40-
mapping from: /aws-java-sdk-.*/, to: 'aws-java-sdk'
41-
mapping from: /jackson-.*/, to: 'jackson'
71+
mapping from: 'annotations', to: 'aws-sdk-2'
72+
mapping from: 'apache-client', to: 'aws-sdk-2'
73+
mapping from: 'auth', to: 'aws-sdk-2'
74+
mapping from: 'aws-core', to: 'aws-sdk-2'
75+
mapping from: 'aws-query-protocol', to: 'aws-sdk-2'
76+
mapping from: 'checksums', to: 'aws-sdk-2'
77+
mapping from: 'checksums-spi', to: 'aws-sdk-2'
78+
mapping from: 'ec2', to: 'aws-sdk-2'
79+
mapping from: 'endpoints-spi', to: 'aws-sdk-2'
80+
mapping from: 'http-auth', to: 'aws-sdk-2'
81+
mapping from: 'http-auth-aws', to: 'aws-sdk-2'
82+
mapping from: 'http-auth-spi', to: 'aws-sdk-2'
83+
mapping from: 'http-client-spi', to: 'aws-sdk-2'
84+
mapping from: 'identity-spi', to: 'aws-sdk-2'
85+
mapping from: 'imds', to: 'aws-sdk-2'
86+
mapping from: 'json-utils', to: 'aws-sdk-2'
87+
mapping from: 'metrics-spi', to: 'aws-sdk-2'
88+
mapping from: 'profiles', to: 'aws-sdk-2'
89+
mapping from: 'protocol-core', to: 'aws-sdk-2'
90+
mapping from: 'regions', to: 'aws-sdk-2'
91+
mapping from: 'retries', to: 'aws-sdk-2'
92+
mapping from: 'retries-spi', to: 'aws-sdk-2'
93+
mapping from: 'sdk-core', to: 'aws-sdk-2'
94+
mapping from: 'third-party-jackson-core', to: 'aws-sdk-2'
95+
mapping from: 'utils', to: 'aws-sdk-2'
4296
}
4397

4498
esplugin.bundleSpec.from('config/discovery-ec2') {
@@ -67,19 +121,23 @@ tasks.register("writeTestJavaPolicy") {
67121
"permission org.bouncycastle.crypto.CryptoServicesPermission \"exportSecretKey\";",
68122
"permission org.bouncycastle.crypto.CryptoServicesPermission \"exportPrivateKey\";",
69123
"permission java.io.FilePermission \"\${javax.net.ssl.trustStore}\", \"read\";",
70-
"permission java.util.PropertyPermission \"com.amazonaws.sdk.ec2MetadataServiceEndpointOverride\", \"write\";",
71124
"permission java.security.SecurityPermission \"getProperty.jdk.tls.disabledAlgorithms\";",
72125
"permission java.security.SecurityPermission \"getProperty.jdk.certpath.disabledAlgorithms\";",
73126
"permission java.security.SecurityPermission \"getProperty.keystore.type.compat\";",
74127
"permission java.security.SecurityPermission \"getProperty.org.bouncycastle.ec.max_f2m_field_size\";",
128+
"permission java.util.PropertyPermission \"aws.ec2MetadataServiceEndpoint\", \"write\";",
129+
"permission java.util.PropertyPermission \"http.proxyHost\", \"read\";",
130+
"permission java.util.PropertyPermission \"aws.region\", \"read\";",
75131
"};"
76132
].join("\n")
77133
)
78134
} else {
79135
javaPolicy.write(
80136
[
81137
"grant {",
82-
" permission java.util.PropertyPermission \"com.amazonaws.sdk.ec2MetadataServiceEndpointOverride\", \"write\";",
138+
"permission java.util.PropertyPermission \"aws.ec2MetadataServiceEndpoint\", \"write\";",
139+
"permission java.util.PropertyPermission \"http.proxyHost\", \"read\";",
140+
"permission java.util.PropertyPermission \"aws.region\", \"read\";",
83141
"};"
84142
].join("\n"))
85143
}
@@ -91,27 +149,42 @@ tasks.withType(Test).configureEach {
91149
// this is needed for insecure plugins, remove if possible!
92150
systemProperty 'tests.artifact', project.name
93151

94-
// this is needed to manipulate com.amazonaws.sdk.ec2MetadataServiceEndpointOverride system property
152+
// this is needed to manipulate aws.ec2MetadataServiceEndpoint system property
95153
// it is better rather disable security manager at all with `systemProperty 'tests.security.manager', 'false'`
96154
if (buildParams.inFipsJvm){
97155
nonInputProperties.systemProperty 'java.security.policy', "=file://${layout.buildDirectory.asFile.get()}/tmp/java.policy"
98156
} else {
99157
nonInputProperties.systemProperty 'java.security.policy', "file://${layout.buildDirectory.asFile.get()}/tmp/java.policy"
100158
}
159+
160+
systemProperty 'aws.region', 'es-test-region'
101161
}
102162

103163
tasks.named("thirdPartyAudit").configure {
104164
ignoreMissingClasses(
105165
// classes are missing
106-
'com.amazonaws.jmespath.JmesPathExpression',
107-
'com.amazonaws.jmespath.ObjectMapperSingleton',
108166
'javax.servlet.ServletContextEvent',
109167
'javax.servlet.ServletContextListener',
110168
'org.apache.avalon.framework.logger.Logger',
111169
'org.apache.log.Hierarchy',
112170
'org.apache.log.Logger',
113171
'javax.jms.Message',
114-
'javax.xml.bind.DatatypeConverter',
115-
'javax.xml.bind.JAXBContext'
172+
173+
// eventstream not used by the sync client
174+
'software.amazon.eventstream.HeaderValue',
175+
'software.amazon.eventstream.Message',
176+
'software.amazon.eventstream.MessageDecoder',
177+
178+
// crt?
179+
'software.amazon.awssdk.crt.auth.credentials.Credentials',
180+
'software.amazon.awssdk.crt.auth.signing.AwsSigner',
181+
'software.amazon.awssdk.crt.auth.signing.AwsSigningConfig',
182+
'software.amazon.awssdk.crt.auth.signing.AwsSigningConfig$AwsSignatureType',
183+
'software.amazon.awssdk.crt.auth.signing.AwsSigningConfig$AwsSignedBodyHeaderType',
184+
'software.amazon.awssdk.crt.auth.signing.AwsSigningConfig$AwsSigningAlgorithm',
185+
'software.amazon.awssdk.crt.auth.signing.AwsSigningResult',
186+
'software.amazon.awssdk.crt.http.HttpHeader',
187+
'software.amazon.awssdk.crt.http.HttpRequest',
188+
'software.amazon.awssdk.crt.http.HttpRequestBodyStream',
116189
)
117190
}

0 commit comments

Comments
 (0)