From 7199a8b2e3917f5dfb0d859d5ecbb37fd8e81a4f Mon Sep 17 00:00:00 2001 From: andsel Date: Mon, 19 Aug 2024 17:04:26 +0200 Subject: [PATCH 01/11] Added dependency on AWS IAM jars and updated documentation Co-authored-by: Akos Korsos --- docs/input-kafka.asciidoc | 14 ++++++++++++++ docs/output-kafka.asciidoc | 14 ++++++++++++++ 2 files changed, 28 insertions(+) diff --git a/docs/input-kafka.asciidoc b/docs/input-kafka.asciidoc index ffe67f8..a1e92d9 100644 --- a/docs/input-kafka.asciidoc +++ b/docs/input-kafka.asciidoc @@ -65,6 +65,19 @@ For more information see https://kafka.apache.org/{kafka_client_doc}/documentati Kafka consumer configuration: https://kafka.apache.org/{kafka_client_doc}/documentation.html#consumerconfigs +==== AWS MSK IAM authentication +If you use AWS MSK, the AWS MSK IAM access control enables you to handle both authentication and authorization for your MSK cluster with AWS IAM. +For more information on this AWS MSK feature see the https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html[AWS documentation]. + +To use this Kafka input with AWS MSK IAM authentication, set the following configuration: +``` +security_protocol => "SASL_SSL" +sasl_mechanism => "AWS_MSK_IAM" +sasl_jaas_config => "software.amazon.msk.auth.iam.IAMLoginModule required;" +sasl_client_callback_handler_class => "software.amazon.msk.auth.iam.IAMClientCallbackHandler" +``` +For more IAM authentication configurations, see the https://github.com/aws/aws-msk-iam-auth[AWS MSK IAM authentication library documentation]. + ==== Metadata fields The following metadata from Kafka broker are added under the `[@metadata]` field: @@ -654,6 +667,7 @@ This can be defined either in Kafka's JAAS config or in Kafka's config. http://kafka.apache.org/documentation.html#security_sasl[SASL mechanism] used for client connections. This may be any mechanism for which a security provider is available. +For AWS MSK IAM authentication use `AWS_MSK_IAM`. GSSAPI is the default mechanism. [id="plugins-{type}s-{plugin}-schema_registry_key"] diff --git a/docs/output-kafka.asciidoc b/docs/output-kafka.asciidoc index 64f3dbe..90ba552 100644 --- a/docs/output-kafka.asciidoc +++ b/docs/output-kafka.asciidoc @@ -66,6 +66,19 @@ https://kafka.apache.org/{kafka_client_doc}/documentation.html#producerconfigs NOTE: This plugin does not support using a proxy when communicating to the Kafka broker. +==== AWS MSK IAM authentication +If you use AWS MSK, the AWS MSK IAM access control enables you to handle both authentication and authorization for your MSK cluster with AWS IAM. +For more information on this AWS MSK feature see the https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html[AWS documentation]. + +To use this Kafka input with AWS MSK IAM authentication, set the following configuration: +``` +security_protocol => "SASL_SSL" +sasl_mechanism => "AWS_MSK_IAM" +sasl_jaas_config => "software.amazon.msk.auth.iam.IAMLoginModule required;" +sasl_client_callback_handler_class => "software.amazon.msk.auth.iam.IAMClientCallbackHandler" +``` +For more IAM authentication configurations, see the https://github.com/aws/aws-msk-iam-auth[AWS MSK IAM authentication library documentation]. + [id="plugins-{type}s-{plugin}-options"] ==== Kafka Output Configuration Options @@ -490,6 +503,7 @@ This can be defined either in Kafka's JAAS config or in Kafka's config. http://kafka.apache.org/documentation.html#security_sasl[SASL mechanism] used for client connections. This may be any mechanism for which a security provider is available. +For AWS MSK IAM authentication use `AWS_MSK_IAM`. GSSAPI is the default mechanism. [id="plugins-{type}s-{plugin}-security_protocol"] From 9340525c3907e61a36ebe6ee44722911aab8b09a Mon Sep 17 00:00:00 2001 From: andsel Date: Fri, 23 Aug 2024 17:37:30 +0200 Subject: [PATCH 02/11] Bumped 11.5.2 version and updated Changelog to expose the settings to use AWS IAM authentication --- CHANGELOG.md | 4 ++++ logstash-integration-kafka.gemspec | 2 +- 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e7d6392..f57d13d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,7 @@ +## 11.6.1 + - Expose the SASL client callback class setting to the Logstash configuration [#177](https://github.com/logstash-plugins/logstash-integration-kafka/pull/177) + - Bundles all libraries to use AWS IAM authentication as SASL client [#178](https://github.com/logstash-plugins/logstash-integration-kafka/pull/178) + ## 11.6.0 - Support additional `oauth` and `sasl` configuration options for configuring kafka client [#189](https://github.com/logstash-plugins/logstash-integration-kafka/pull/189) diff --git a/logstash-integration-kafka.gemspec b/logstash-integration-kafka.gemspec index 9c3bf38..0bb1518 100644 --- a/logstash-integration-kafka.gemspec +++ b/logstash-integration-kafka.gemspec @@ -1,6 +1,6 @@ Gem::Specification.new do |s| s.name = 'logstash-integration-kafka' - s.version = '11.6.0' + s.version = '11.6.1' s.licenses = ['Apache-2.0'] s.summary = "Integration with Kafka - input and output plugins" s.description = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline "+ From 9e8ad9f2a6be83126c61b18356b02a985bc0b175 Mon Sep 17 00:00:00 2001 From: andsel Date: Tue, 11 Feb 2025 14:12:19 +0100 Subject: [PATCH 03/11] Introduce setting to load external IAM SASL providers and add a submodule to build the uber-jar for AWS IAM --- build.gradle | 1 + iam-aws-uber/build.gradle | 60 ++++++++++++++++++++++ lib/logstash/inputs/kafka.rb | 3 +- lib/logstash/plugin_mixins/kafka/common.rb | 5 ++ settings.gradle | 1 + 5 files changed, 69 insertions(+), 1 deletion(-) create mode 100644 iam-aws-uber/build.gradle create mode 100644 settings.gradle diff --git a/build.gradle b/build.gradle index 39243c2..f7547e9 100644 --- a/build.gradle +++ b/build.gradle @@ -93,6 +93,7 @@ task vendor { File newJarFile = file("${vendorPathPrefix}/${groupPath}/${dep.name}/${dep.version}/${dep.name}-${dep.version}.jar") newJarFile.mkdirs() Files.copy(f.toPath(), newJarFile.toPath(), REPLACE_EXISTING) + println "Copying ${f} to ${newJarFile}" } } } diff --git a/iam-aws-uber/build.gradle b/iam-aws-uber/build.gradle new file mode 100644 index 0000000..6ac6668 --- /dev/null +++ b/iam-aws-uber/build.gradle @@ -0,0 +1,60 @@ +plugins { + id 'java' + id 'com.gradleup.shadow' version '8.3.6' +} + +group = 'org.logstash' +version = '2.2.0' + +repositories { + // Use Maven Central for resolving dependencies. + mavenCentral() +} + +dependencies { + implementation 'software.amazon.msk:aws-msk-iam-auth:2.2.0' +} + +// Apply a specific Java toolchain to ease working on different environments. +java { + toolchain { + languageVersion = JavaLanguageVersion.of(17) + } +} + +tasks.named('shadowJar', com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar) { + archiveBaseName = 'aws-msk-iam-auth' + archiveClassifier = 'uber' +// archiveVersion = '' + +// enableRelocation = true +// relocationPrefix = 'org.logstash' + +// relocate('software.amazon', 'org.logstash.software.amazon') { +// // these are classes explicitly referred into the Kafka plugin configuration, so need to keep +// // original name +// exclude 'software.amazon.msk.auth.iam.IAMLoginModule' +// exclude 'software.amazon.msk.auth.iam.IAMClientCallbackHandler' +// +// // This provider is used during KafkaConsumer creation +// exclude 'software.amazon.awssdk.auth.credentials.AwsCredentialsProvider' +// exclude 'software.amazon.awssdk.core.exception.SdkException' +// exclude 'software.amazon.awssdk.core.exception.SdkException$Builder' +// exclude 'software.amazon.awssdk.core.exception.SdkException$BuilderImpl' +// exclude 'software.amazon.awssdk.core.exception.SdkClientException' +// exclude 'software.amazon.awssdk.core.exception.SdkClientException$Builder' +// exclude 'software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl' +// } + + relocate('org.apache.http', 'org.logstash.org.apache.http') + relocate('com.fasterxml.jackson', 'org.logstash.com.fasterxml.jackson') + relocate('org.reactivestreams', 'org.logstash.org.reactivestreams') + + // commons-logging and commons-codec are used by org.apache.httpcomponents:httpclient + relocate('org.apache.commons.logging', 'org.logstash.org.apache.commons.logging') + relocate('org.apache.commons.codec', 'org.logstash.org.apache.commons.codec') + + // insert in the uber jar but not relocate packages that are pretty stable across Logstash code base: + // - io.netty + // - org/slf4j +} diff --git a/lib/logstash/inputs/kafka.rb b/lib/logstash/inputs/kafka.rb index aa713c3..91e4887 100644 --- a/lib/logstash/inputs/kafka.rb +++ b/lib/logstash/inputs/kafka.rb @@ -224,7 +224,8 @@ class LogStash::Inputs::Kafka < LogStash::Inputs::Base config :sasl_login_retry_backoff_ms, :validate => :number, :default => 100 # Kafka default # (optional) The maximum duration, in milliseconds, for HTTPS call attempts. config :sasl_login_retry_backoff_max_ms, :validate => :number, :default => 10000 # Kafka default - # http://kafka.apache.org/documentation.html#security_sasl[SASL mechanism] used for client connections. + config :sasl_iam_jar_paths, :validate => :array + # http://kafka.apache.org/documentation.html#security_sasl[SASL mechanism] used for client connections. # This may be any mechanism for which a security provider is available. # GSSAPI is the default mechanism. config :sasl_mechanism, :validate => :string, :default => "GSSAPI" diff --git a/lib/logstash/plugin_mixins/kafka/common.rb b/lib/logstash/plugin_mixins/kafka/common.rb index 6564e80..b562eec 100644 --- a/lib/logstash/plugin_mixins/kafka/common.rb +++ b/lib/logstash/plugin_mixins/kafka/common.rb @@ -49,6 +49,11 @@ def set_sasl_config(props) props.put("sasl.login.read.timeout.ms", sasl_login_read_timeout_ms.to_s) unless sasl_login_read_timeout_ms.nil? props.put("sasl.login.retry.backoff.ms", sasl_login_retry_backoff_ms.to_s) unless sasl_login_retry_backoff_ms.nil? props.put("sasl.login.retry.backoff.max.ms", sasl_login_retry_backoff_max_ms.to_s) unless sasl_login_retry_backoff_max_ms.nil? + if !sasl_iam_jar_paths.nil? && !sasl_iam_jar_paths.empty? + sasl_iam_jar_paths.each do |jar_path| + require jar_path + end + end end def reassign_dns_lookup diff --git a/settings.gradle b/settings.gradle new file mode 100644 index 0000000..d22cf8d --- /dev/null +++ b/settings.gradle @@ -0,0 +1 @@ +include('iam-aws-uber') \ No newline at end of file From da0574567a7b21fee91937e8492d2e7fb420bbc2 Mon Sep 17 00:00:00 2001 From: andsel Date: Tue, 11 Feb 2025 14:13:23 +0100 Subject: [PATCH 04/11] Avoid to relocate AWS classes which generate runtime NoDefClassError --- iam-aws-uber/build.gradle | 20 -------------------- 1 file changed, 20 deletions(-) diff --git a/iam-aws-uber/build.gradle b/iam-aws-uber/build.gradle index 6ac6668..38b6322 100644 --- a/iam-aws-uber/build.gradle +++ b/iam-aws-uber/build.gradle @@ -25,26 +25,6 @@ java { tasks.named('shadowJar', com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar) { archiveBaseName = 'aws-msk-iam-auth' archiveClassifier = 'uber' -// archiveVersion = '' - -// enableRelocation = true -// relocationPrefix = 'org.logstash' - -// relocate('software.amazon', 'org.logstash.software.amazon') { -// // these are classes explicitly referred into the Kafka plugin configuration, so need to keep -// // original name -// exclude 'software.amazon.msk.auth.iam.IAMLoginModule' -// exclude 'software.amazon.msk.auth.iam.IAMClientCallbackHandler' -// -// // This provider is used during KafkaConsumer creation -// exclude 'software.amazon.awssdk.auth.credentials.AwsCredentialsProvider' -// exclude 'software.amazon.awssdk.core.exception.SdkException' -// exclude 'software.amazon.awssdk.core.exception.SdkException$Builder' -// exclude 'software.amazon.awssdk.core.exception.SdkException$BuilderImpl' -// exclude 'software.amazon.awssdk.core.exception.SdkClientException' -// exclude 'software.amazon.awssdk.core.exception.SdkClientException$Builder' -// exclude 'software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl' -// } relocate('org.apache.http', 'org.logstash.org.apache.http') relocate('com.fasterxml.jackson', 'org.logstash.com.fasterxml.jackson') From 16cdd927045bf98cddbd955ed0d63c0069a73e77 Mon Sep 17 00:00:00 2001 From: andsel Date: Wed, 12 Feb 2025 09:37:12 +0100 Subject: [PATCH 05/11] Added sasl_iam_jar_paths setting also to Kafka output and minor changes --- build.gradle | 1 - docs/input-kafka.asciidoc | 1 + docs/output-kafka.asciidoc | 1 + iam-aws-uber/build.gradle | 2 +- lib/logstash/inputs/kafka.rb | 3 ++- lib/logstash/outputs/kafka.rb | 2 ++ 6 files changed, 7 insertions(+), 3 deletions(-) diff --git a/build.gradle b/build.gradle index f7547e9..39243c2 100644 --- a/build.gradle +++ b/build.gradle @@ -93,7 +93,6 @@ task vendor { File newJarFile = file("${vendorPathPrefix}/${groupPath}/${dep.name}/${dep.version}/${dep.name}-${dep.version}.jar") newJarFile.mkdirs() Files.copy(f.toPath(), newJarFile.toPath(), REPLACE_EXISTING) - println "Copying ${f} to ${newJarFile}" } } } diff --git a/docs/input-kafka.asciidoc b/docs/input-kafka.asciidoc index a1e92d9..d8a36b9 100644 --- a/docs/input-kafka.asciidoc +++ b/docs/input-kafka.asciidoc @@ -73,6 +73,7 @@ To use this Kafka input with AWS MSK IAM authentication, set the following confi ``` security_protocol => "SASL_SSL" sasl_mechanism => "AWS_MSK_IAM" +sasl_iam_jar_paths => ["/pth/to/aws_iam_uber.jar"] sasl_jaas_config => "software.amazon.msk.auth.iam.IAMLoginModule required;" sasl_client_callback_handler_class => "software.amazon.msk.auth.iam.IAMClientCallbackHandler" ``` diff --git a/docs/output-kafka.asciidoc b/docs/output-kafka.asciidoc index 90ba552..1e206f4 100644 --- a/docs/output-kafka.asciidoc +++ b/docs/output-kafka.asciidoc @@ -74,6 +74,7 @@ To use this Kafka input with AWS MSK IAM authentication, set the following confi ``` security_protocol => "SASL_SSL" sasl_mechanism => "AWS_MSK_IAM" +sasl_iam_jar_paths => ["/pth/to/aws_iam_uber.jar"] sasl_jaas_config => "software.amazon.msk.auth.iam.IAMLoginModule required;" sasl_client_callback_handler_class => "software.amazon.msk.auth.iam.IAMClientCallbackHandler" ``` diff --git a/iam-aws-uber/build.gradle b/iam-aws-uber/build.gradle index 38b6322..5db45db 100644 --- a/iam-aws-uber/build.gradle +++ b/iam-aws-uber/build.gradle @@ -36,5 +36,5 @@ tasks.named('shadowJar', com.github.jengelman.gradle.plugins.shadow.tasks.Shadow // insert in the uber jar but not relocate packages that are pretty stable across Logstash code base: // - io.netty - // - org/slf4j + // - org.slf4j } diff --git a/lib/logstash/inputs/kafka.rb b/lib/logstash/inputs/kafka.rb index 91e4887..b8b1dbd 100644 --- a/lib/logstash/inputs/kafka.rb +++ b/lib/logstash/inputs/kafka.rb @@ -216,6 +216,8 @@ class LogStash::Inputs::Kafka < LogStash::Inputs::Base config :sasl_oauthbearer_scope_claim_name, :validate => :string, :default => 'scope' # Kafka default # SASL login callback handler class config :sasl_login_callback_handler_class, :validate => :string + # Path to the jar containing client and all dependencies for SASL IAM authentication of specific cloud vendor + config :sasl_iam_jar_paths, :validate => :array # (optional) The duration, in milliseconds, for HTTPS connect timeout config :sasl_login_connect_timeout_ms, :validate => :number # (optional) The duration, in milliseconds, for HTTPS read timeout. @@ -224,7 +226,6 @@ class LogStash::Inputs::Kafka < LogStash::Inputs::Base config :sasl_login_retry_backoff_ms, :validate => :number, :default => 100 # Kafka default # (optional) The maximum duration, in milliseconds, for HTTPS call attempts. config :sasl_login_retry_backoff_max_ms, :validate => :number, :default => 10000 # Kafka default - config :sasl_iam_jar_paths, :validate => :array # http://kafka.apache.org/documentation.html#security_sasl[SASL mechanism] used for client connections. # This may be any mechanism for which a security provider is available. # GSSAPI is the default mechanism. diff --git a/lib/logstash/outputs/kafka.rb b/lib/logstash/outputs/kafka.rb index 8ba0148..5557543 100644 --- a/lib/logstash/outputs/kafka.rb +++ b/lib/logstash/outputs/kafka.rb @@ -149,6 +149,8 @@ class LogStash::Outputs::Kafka < LogStash::Outputs::Base config :security_protocol, :validate => ["PLAINTEXT", "SSL", "SASL_PLAINTEXT", "SASL_SSL"], :default => "PLAINTEXT" # SASL client callback handler class config :sasl_client_callback_handler_class, :validate => :string + # Path to the jar containing client and all dependencies for SASL IAM authentication of specific cloud vendor + config :sasl_iam_jar_paths, :validate => :array # The URL for the OAuth 2.0 issuer token endpoint. config :sasl_oauthbearer_token_endpoint_url, :validate => :string # (optional) The override name of the scope claim. From 97601598f3a0b89feab703281bc513b32747d309 Mon Sep 17 00:00:00 2001 From: andsel Date: Mon, 17 Feb 2025 11:14:03 +0100 Subject: [PATCH 06/11] Removed exported part of uber jar creation --- iam-aws-uber/build.gradle | 40 --------------------------------------- settings.gradle | 1 - 2 files changed, 41 deletions(-) delete mode 100644 iam-aws-uber/build.gradle delete mode 100644 settings.gradle diff --git a/iam-aws-uber/build.gradle b/iam-aws-uber/build.gradle deleted file mode 100644 index 5db45db..0000000 --- a/iam-aws-uber/build.gradle +++ /dev/null @@ -1,40 +0,0 @@ -plugins { - id 'java' - id 'com.gradleup.shadow' version '8.3.6' -} - -group = 'org.logstash' -version = '2.2.0' - -repositories { - // Use Maven Central for resolving dependencies. - mavenCentral() -} - -dependencies { - implementation 'software.amazon.msk:aws-msk-iam-auth:2.2.0' -} - -// Apply a specific Java toolchain to ease working on different environments. -java { - toolchain { - languageVersion = JavaLanguageVersion.of(17) - } -} - -tasks.named('shadowJar', com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar) { - archiveBaseName = 'aws-msk-iam-auth' - archiveClassifier = 'uber' - - relocate('org.apache.http', 'org.logstash.org.apache.http') - relocate('com.fasterxml.jackson', 'org.logstash.com.fasterxml.jackson') - relocate('org.reactivestreams', 'org.logstash.org.reactivestreams') - - // commons-logging and commons-codec are used by org.apache.httpcomponents:httpclient - relocate('org.apache.commons.logging', 'org.logstash.org.apache.commons.logging') - relocate('org.apache.commons.codec', 'org.logstash.org.apache.commons.codec') - - // insert in the uber jar but not relocate packages that are pretty stable across Logstash code base: - // - io.netty - // - org.slf4j -} diff --git a/settings.gradle b/settings.gradle deleted file mode 100644 index d22cf8d..0000000 --- a/settings.gradle +++ /dev/null @@ -1 +0,0 @@ -include('iam-aws-uber') \ No newline at end of file From 749d48597bb22646b74d8706e7fa4087dd5ea21a Mon Sep 17 00:00:00 2001 From: andsel Date: Mon, 17 Feb 2025 12:55:49 +0100 Subject: [PATCH 07/11] Updated documentation --- CHANGELOG.md | 2 +- docs/input-kafka.asciidoc | 6 ++++-- docs/output-kafka.asciidoc | 6 ++++-- 3 files changed, 9 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f57d13d..e988389 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,6 @@ ## 11.6.1 - Expose the SASL client callback class setting to the Logstash configuration [#177](https://github.com/logstash-plugins/logstash-integration-kafka/pull/177) - - Bundles all libraries to use AWS IAM authentication as SASL client [#178](https://github.com/logstash-plugins/logstash-integration-kafka/pull/178) + - Adds a mechanism to load AWS IAM authentication as SASL client libraries at startup [#178](https://github.com/logstash-plugins/logstash-integration-kafka/pull/178) ## 11.6.0 - Support additional `oauth` and `sasl` configuration options for configuring kafka client [#189](https://github.com/logstash-plugins/logstash-integration-kafka/pull/189) diff --git a/docs/input-kafka.asciidoc b/docs/input-kafka.asciidoc index d8a36b9..cc42ea3 100644 --- a/docs/input-kafka.asciidoc +++ b/docs/input-kafka.asciidoc @@ -69,11 +69,13 @@ Kafka consumer configuration: https://kafka.apache.org/{kafka_client_doc}/docume If you use AWS MSK, the AWS MSK IAM access control enables you to handle both authentication and authorization for your MSK cluster with AWS IAM. For more information on this AWS MSK feature see the https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html[AWS documentation]. -To use this Kafka input with AWS MSK IAM authentication, set the following configuration: +To use this Kafka input with AWS MSK IAM authentication, download the uber jar which contains the client library for +this specific cloud vendor and all the transitive dependencies from this https://to.do/[repository]. +Configure the following setting: ``` security_protocol => "SASL_SSL" sasl_mechanism => "AWS_MSK_IAM" -sasl_iam_jar_paths => ["/pth/to/aws_iam_uber.jar"] +sasl_iam_jar_paths => ["/path/to/aws_iam_uber.jar"] sasl_jaas_config => "software.amazon.msk.auth.iam.IAMLoginModule required;" sasl_client_callback_handler_class => "software.amazon.msk.auth.iam.IAMClientCallbackHandler" ``` diff --git a/docs/output-kafka.asciidoc b/docs/output-kafka.asciidoc index 1e206f4..3f79b48 100644 --- a/docs/output-kafka.asciidoc +++ b/docs/output-kafka.asciidoc @@ -70,11 +70,13 @@ NOTE: This plugin does not support using a proxy when communicating to the Kafk If you use AWS MSK, the AWS MSK IAM access control enables you to handle both authentication and authorization for your MSK cluster with AWS IAM. For more information on this AWS MSK feature see the https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html[AWS documentation]. -To use this Kafka input with AWS MSK IAM authentication, set the following configuration: +To use this Kafka input with AWS MSK IAM authentication, download the uber jar which contains the client library for +this specific cloud vendor and all the transitive dependencies from this https://to.do/[repository]. +Configure the following setting: ``` security_protocol => "SASL_SSL" sasl_mechanism => "AWS_MSK_IAM" -sasl_iam_jar_paths => ["/pth/to/aws_iam_uber.jar"] +sasl_iam_jar_paths => ["/path/to/aws_iam_uber.jar"] sasl_jaas_config => "software.amazon.msk.auth.iam.IAMLoginModule required;" sasl_client_callback_handler_class => "software.amazon.msk.auth.iam.IAMClientCallbackHandler" ``` From 803a14907ddc55ce0b9d0481ceecb0a9754854c0 Mon Sep 17 00:00:00 2001 From: andsel Date: Wed, 16 Apr 2025 15:43:52 +0200 Subject: [PATCH 08/11] Mino, set the URL for the repository where to download the uber jars --- docs/input-kafka.asciidoc | 2 +- docs/output-kafka.asciidoc | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/input-kafka.asciidoc b/docs/input-kafka.asciidoc index cc42ea3..06f4551 100644 --- a/docs/input-kafka.asciidoc +++ b/docs/input-kafka.asciidoc @@ -70,7 +70,7 @@ If you use AWS MSK, the AWS MSK IAM access control enables you to handle both au For more information on this AWS MSK feature see the https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html[AWS documentation]. To use this Kafka input with AWS MSK IAM authentication, download the uber jar which contains the client library for -this specific cloud vendor and all the transitive dependencies from this https://to.do/[repository]. +this specific cloud vendor and all the transitive dependencies from this https://github.com/elastic/logstash-kafka-iams-packages/releases[repository]. Configure the following setting: ``` security_protocol => "SASL_SSL" diff --git a/docs/output-kafka.asciidoc b/docs/output-kafka.asciidoc index 3f79b48..eba0894 100644 --- a/docs/output-kafka.asciidoc +++ b/docs/output-kafka.asciidoc @@ -71,7 +71,7 @@ If you use AWS MSK, the AWS MSK IAM access control enables you to handle both au For more information on this AWS MSK feature see the https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html[AWS documentation]. To use this Kafka input with AWS MSK IAM authentication, download the uber jar which contains the client library for -this specific cloud vendor and all the transitive dependencies from this https://to.do/[repository]. +this specific cloud vendor and all the transitive dependencies from this https://github.com/elastic/logstash-kafka-iams-packages/releases[repository]. Configure the following setting: ``` security_protocol => "SASL_SSL" From 5b70999309f39dde35858b1024a4c73b6b5dfd42 Mon Sep 17 00:00:00 2001 From: Andrea Selva Date: Thu, 8 May 2025 14:35:10 +0200 Subject: [PATCH 09/11] Added doc section id for AWS MSK IAM authentication part MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: João Duarte --- docs/input-kafka.asciidoc | 1 + docs/output-kafka.asciidoc | 1 + 2 files changed, 2 insertions(+) diff --git a/docs/input-kafka.asciidoc b/docs/input-kafka.asciidoc index 06f4551..ce98b11 100644 --- a/docs/input-kafka.asciidoc +++ b/docs/input-kafka.asciidoc @@ -65,6 +65,7 @@ For more information see https://kafka.apache.org/{kafka_client_doc}/documentati Kafka consumer configuration: https://kafka.apache.org/{kafka_client_doc}/documentation.html#consumerconfigs +[id="plugins-{type}s-{plugin}-aws_msk_iam_auth"] ==== AWS MSK IAM authentication If you use AWS MSK, the AWS MSK IAM access control enables you to handle both authentication and authorization for your MSK cluster with AWS IAM. For more information on this AWS MSK feature see the https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html[AWS documentation]. diff --git a/docs/output-kafka.asciidoc b/docs/output-kafka.asciidoc index eba0894..8c0752c 100644 --- a/docs/output-kafka.asciidoc +++ b/docs/output-kafka.asciidoc @@ -66,6 +66,7 @@ https://kafka.apache.org/{kafka_client_doc}/documentation.html#producerconfigs NOTE: This plugin does not support using a proxy when communicating to the Kafka broker. +[id="plugins-{type}s-{plugin}-aws_msk_iam_auth"] ==== AWS MSK IAM authentication If you use AWS MSK, the AWS MSK IAM access control enables you to handle both authentication and authorization for your MSK cluster with AWS IAM. For more information on this AWS MSK feature see the https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html[AWS documentation]. From e0512ded3525bce1cf44aad9e257fa96521b91a4 Mon Sep 17 00:00:00 2001 From: Andrea Selva Date: Thu, 8 May 2025 14:35:29 +0200 Subject: [PATCH 10/11] Streamlined iteration MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: João Duarte --- lib/logstash/plugin_mixins/kafka/common.rb | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/lib/logstash/plugin_mixins/kafka/common.rb b/lib/logstash/plugin_mixins/kafka/common.rb index b562eec..238e27a 100644 --- a/lib/logstash/plugin_mixins/kafka/common.rb +++ b/lib/logstash/plugin_mixins/kafka/common.rb @@ -49,11 +49,7 @@ def set_sasl_config(props) props.put("sasl.login.read.timeout.ms", sasl_login_read_timeout_ms.to_s) unless sasl_login_read_timeout_ms.nil? props.put("sasl.login.retry.backoff.ms", sasl_login_retry_backoff_ms.to_s) unless sasl_login_retry_backoff_ms.nil? props.put("sasl.login.retry.backoff.max.ms", sasl_login_retry_backoff_max_ms.to_s) unless sasl_login_retry_backoff_max_ms.nil? - if !sasl_iam_jar_paths.nil? && !sasl_iam_jar_paths.empty? - sasl_iam_jar_paths.each do |jar_path| - require jar_path - end - end + sasl_iam_jar_paths&.each {|jar_path| require jar_path } end def reassign_dns_lookup From 4b3690098496b1649718d7682ffc2b0ec651f445 Mon Sep 17 00:00:00 2001 From: andsel Date: Thu, 8 May 2025 14:50:38 +0200 Subject: [PATCH 11/11] [Doc] Added description and definition for setting sasl_iam_jar_paths --- docs/input-kafka.asciidoc | 9 +++++++++ docs/output-kafka.asciidoc | 9 +++++++++ 2 files changed, 18 insertions(+) diff --git a/docs/input-kafka.asciidoc b/docs/input-kafka.asciidoc index ce98b11..a312eb1 100644 --- a/docs/input-kafka.asciidoc +++ b/docs/input-kafka.asciidoc @@ -150,6 +150,7 @@ See the https://kafka.apache.org/{kafka_client_doc}/documentation for more detai | <> |<>|No | <> |<>|No | <> |<>|No +| <> |<>|No | <> |<>|No | <> |<>|No | <> |<>|No @@ -601,6 +602,14 @@ The URL for the OAuth 2.0 issuer token endpoint. (optional) The override name of the scope claim. +[id="plugins-{type}s-{plugin}-sasl_iam_jar_paths"] +===== `sasl_iam_jar_paths` +* Value type is <> +* There is no default value for this setting. + +Contains the list of paths to jar libraries that contains cloud providers MSK IAM's clients. +There is one jar per provider and can be retrieved as described in <<"plugins-{type}s-{plugin}-aws_msk_iam_auth">>. + [id="plugins-{type}s-{plugin}-sasl_login_callback_handler_class"] ===== `sasl_login_callback_handler_class` * Value type is <> diff --git a/docs/output-kafka.asciidoc b/docs/output-kafka.asciidoc index 8c0752c..ac7c543 100644 --- a/docs/output-kafka.asciidoc +++ b/docs/output-kafka.asciidoc @@ -121,6 +121,7 @@ See the https://kafka.apache.org/{kafka_client_doc}/documentation for more detai | <> |<>|No | <> |<>|No | <> |<>|No +| <> |<>|No | <> |<>|No | <> |<>|No | <> |<>|No @@ -437,6 +438,14 @@ The URL for the OAuth 2.0 issuer token endpoint. (optional) The override name of the scope claim. +[id="plugins-{type}s-{plugin}-sasl_iam_jar_paths"] +===== `sasl_iam_jar_paths` +* Value type is <> +* There is no default value for this setting. + +Contains the list of paths to jar libraries that contains cloud providers MSK IAM's clients. +There is one jar per provider and can be retrieved as described in <<"plugins-{type}s-{plugin}-aws_msk_iam_auth">>. + [id="plugins-{type}s-{plugin}-sasl_login_callback_handler_class"] ===== `sasl_login_callback_handler_class` * Value type is <>