-
Couldn't load subscription status.
- Fork 62
AWS IAM authentication #178
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
Verified on EC2 with a policy connecting to an Amazon MKS Running pipeline as described in the description of this PR. And was able to create and connect client with AWS IAM |
18f8e55 to
bef696e
Compare
|
👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 You are A LIFE SAVER!! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Comment on library versions
build.gradle
Outdated
| implementation 'com.github.luben:zstd-jni:1.5.5-4' | ||
| implementation 'org.lz4:lz4-java:1.8.0' | ||
| implementation 'org.xerial.snappy:snappy-java:1.1.10.5' | ||
| implementation 'software.amazon.msk:aws-msk-iam-auth:1.1.4' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we use more modern versions of these libraries? These are somewhat out of date
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
build.gradle
Outdated
| implementation 'org.lz4:lz4-java:1.8.0' | ||
| implementation 'org.xerial.snappy:snappy-java:1.1.10.5' | ||
| implementation 'software.amazon.msk:aws-msk-iam-auth:2.2.0' | ||
| // transitive deps used by software.amazon.msk:aws-msk-iam-auth |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
😳
This seems like... a lot. Are we concerned that we are going to have issues with dependency clashes with other plugins and Logstash core?
cc @jsvd
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In 4c37ca9 you can see the difference switching from 1.1.4 to 2.2.0. While there is a consistent part of AWS libraries (part of groups software.amazon.awssdk and com.amazonaws) with this new version it adds:
commons-codec:commons-codeccommons-logging:commons-loggingio.netty:netty-*org.reactivestreams:reactive-streamsorg.apache.httpcomponentshttpclientandhttpcore
While Netty http and http2 codecs are present it was added also a dependency on Apache's httpclient so seems redundant.
The risky one is Netty, because used in other Logstash plugins.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
From Logstash core we have:
httpcore-4.4.16.jarvs this4.4.13(no apparent conflict)httpclient-4.5.14.jarvs this4.5.13(no apparent conflict)commons-codec-1.17.0.jarvs this1.15API should be compatiblecommons-logging-1.3.1.jarvs this1.2API should be compatible- Netty in Beats/TCP/HTTP inputs are at
4.1.109vs this4.1.108so not so far and shouldn't break
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The pluggable security classes root in 2 classes:
software.amazon.msk.auth.iam.IAMLoginModulesoftware.amazon.msk.auth.iam.IAMClientCallbackHandler
The IAMClientCallbackHandler explicitly instantiates MSKCredentialProvider at: https://github.com/aws/aws-msk-iam-auth/blob/v2.2.0/src/main/java/software/amazon/msk/auth/iam/IAMClientCallbackHandler.java#L54
The MSKCredentialProvider has direct dependency on 3 main AWSSDK packages:
software.amazon.awssdk.authprovided bysoftware.amazon.awssdk:authsoftware.amazon.awssdk.coreprovided bysoftware.amazon.awssdk:sdk-coresoftware.amazon.awssdk.services.stsprovide bysoftware.amazon.awssdk:sts
Looking at the source of aws-msk-iam-auth the artifact software.amazon.awssdk:sdk-core requires the following dependency tree:
+--- software.amazon.awssdk:sdk-core:2.26.8
+--- software.amazon.awssdk:annotations:2.26.8
+--- software.amazon.awssdk:http-client-spi:2.26.8
| +--- software.amazon.awssdk:annotations:2.26.8
| +--- software.amazon.awssdk:utils:2.26.8 (*)
| +--- software.amazon.awssdk:metrics-spi:2.26.8
| | +--- software.amazon.awssdk:annotations:2.26.8
| | \--- software.amazon.awssdk:utils:2.26.8 (*)
| \--- org.reactivestreams:reactive-streams:1.0.4
+--- software.amazon.awssdk:metrics-spi:2.26.8 (*)
+--- software.amazon.awssdk:endpoints-spi:2.26.8
| \--- software.amazon.awssdk:annotations:2.26.8
+--- software.amazon.awssdk:http-auth-spi:2.26.8
| +--- software.amazon.awssdk:annotations:2.26.8
| +--- software.amazon.awssdk:utils:2.26.8 (*)
| +--- software.amazon.awssdk:http-client-spi:2.26.8 (*)
| +--- org.reactivestreams:reactive-streams:1.0.4
| \--- software.amazon.awssdk:identity-spi:2.26.8
| +--- software.amazon.awssdk:annotations:2.26.8
| \--- software.amazon.awssdk:utils:2.26.8 (*)
+--- software.amazon.awssdk:http-auth-aws:2.26.8
| +--- software.amazon.awssdk:annotations:2.26.8
| +--- software.amazon.awssdk:utils:2.26.8 (*)
| +--- software.amazon.awssdk:identity-spi:2.26.8 (*)
| +--- software.amazon.awssdk:http-client-spi:2.26.8 (*)
| +--- software.amazon.awssdk:http-auth-spi:2.26.8 (*)
| +--- software.amazon.awssdk:checksums-spi:2.26.8
| | \--- software.amazon.awssdk:annotations:2.26.8
| \--- software.amazon.awssdk:checksums:2.26.8
| +--- software.amazon.awssdk:annotations:2.26.8
| \--- software.amazon.awssdk:checksums-spi:2.26.8 (*)
+--- software.amazon.awssdk:checksums-spi:2.26.8 (*)
+--- software.amazon.awssdk:checksums:2.26.8 (*)
+--- software.amazon.awssdk:identity-spi:2.26.8 (*)
+--- software.amazon.awssdk:utils:2.26.8 (*)
+--- software.amazon.awssdk:profiles:2.26.8
| +--- software.amazon.awssdk:utils:2.26.8 (*)
| \--- software.amazon.awssdk:annotations:2.26.8
+--- software.amazon.awssdk:retries-spi:2.26.8
| +--- software.amazon.awssdk:annotations:2.26.8
| \--- software.amazon.awssdk:utils:2.26.8 (*)
+--- software.amazon.awssdk:retries:2.26.8
| +--- software.amazon.awssdk:retries-spi:2.26.8 (*)
| +--- software.amazon.awssdk:annotations:2.26.8
| \--- software.amazon.awssdk:utils:2.26.8 (*)
\--- org.reactivestreams:reactive-streams:1.0.4
software.amazon.awssdk:sts has direct dependency on software.amazon.awssdk:apache-client and software.amazon.awssdk:netty-nio-client which kicks in all the netty related dependencies.
Despite the class MSKCredentialProvider doesn't directly refer to classes contained in software.amazon.awssdk:sso and software.amazon.awssdk:ssooidc, removing those doesn't provide any gain in terms of numerosity of dependencies.
So my conclusion is that we can't strip any jar.
|
any update on this plugin? no updates since august? we are waiting this plugin to integrate logstash ->msk |
|
@bn-npakki due to the large number of dependencies that thisPR transitively kicks in, we are discussing in elastic/logstash#16673 how to isolate from the rest of Logstash classpath. Once we have an agreement on how to manage it, we can move forward with this PR. |
Co-authored-by: Akos Korsos <[email protected]>
… use AWS IAM authentication
…dule to build the uber-jar for AWS IAM
758a37d to
da05745
Compare
|
Hi @robbavey, Do we have any estimation on when this feature will be merged? |
|
Hi @robbavey given that we have the repository https://github.com/elastic/logstash-kafka-iams-packages that hosts the uber jars with all transitive dependencies publicly available, I would ask another round of review for this so that we can move it forward 🙏 |
Co-authored-by: João Duarte <[email protected]>
Co-authored-by: João Duarte <[email protected]>
| | <<plugins-{type}s-{plugin}-sasl_client_callback_handler_class>> |<<string,string>>|No | ||
| | <<plugins-{type}s-{plugin}-sasl_oauthbearer_token_endpoint_url>> |<<string,string>>|No | ||
| | <<plugins-{type}s-{plugin}-sasl_oauthbearer_scope_claim_name>> |<<string,string>>|No | ||
| | <<plugins-{type}s-{plugin}-sasl_iam_jar_paths>> |<<array,array>>|No |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
not ordered alphabetically :D
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sasl_iam_jar_paths should stay before sasl_login_callback_handler_class, or we also fix sasl_oauthbearer* sasl_jaas_config and sasl_kerberos_service_name ordering. Maybe sorting the whole settings section should be done in another PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Release notes
Creation and use of new setting to load uber jars which contains cloud providers IAM authentication SASL client libraries.
What does this PR do?
Introduces the setting
sasl_iam_jar_pathswhich is an array of library paths that points to uber jars containing all the classes needed by specific cloud provider IAM for SASL authentication.During registration phase, uses the Ruby
requireto load the jars configured.Why is it important/What is the impact to the user?
Permit to the users that deploy Logstash in a cloud provider to leverage the cloud provider's IAM to authenticate the client with SASL.
Checklist
[ ] I have made corresponding change to the default configuration files (and/or docker env variables)Author's Checklist
How to test this PR locally
Related issues
sasl.client.callback.handler.classKafka client setting to select a specific SASL Handler class #177