pipeline: input: kafka: document AWS MSK IAM

Signed-off-by: Eduardo Silva <[email protected]>

Author: edsiper

pipeline/inputs/kafka.md

@@ -1,6 +1,8 @@
-# Kafka
+# Kafka Consumer
 
-The _Kafka_ input plugin subscribes to one or more Kafka topics to collect messages from an [Apache Kafka](https://kafka.apache.org/) service.
+The _Kafka_ input plugin enables Fluent Bit to consume messages directly from one or more [Apache Kafka](https://kafka.apache.org/) topics. By subscribing to specified topics, this plugin efficiently collects and forwards Kafka messages for further processing within your Fluent Bit pipeline.
+
+Starting with version 4.0.4, the Kafka input plugin supports authentication with AWS MSK IAM, enabling integration with Amazon MSK (Managed Streaming for Apache Kafka) clusters that require IAM-based access.
 
 This plugin uses the official [librdkafka C library](https://github.com/edenhill/librdkafka) as a built-in dependency.
 
@@ -18,6 +20,15 @@ This plugin uses the official [librdkafka C library](https://github.com/edenhill
 | `rdkafka.{property}` | `{property}` can be any [librdkafka properties](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md) | _none_ |
 | `threaded` | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` |
 
+
+## Configuration parameters for AWS MSK clusters based on IAM authentication
+
+| Property | Description | Type | Required |
+|----------|-------------|------|----------|
+| `aws_msk_iam` | Enable AWS MSK IAM authentication | Boolean | No (default: false) |
+| `aws_msk_iam_cluster_arn` | Full ARN of the MSK cluster for region extraction | String | Yes (when `aws_msk_iam` is true) |
+
+
 ## Get started
 
 To subscribe to or collect messages from Apache Kafka, run the plugin from the command line or through the configuration file as shown below.
@@ -30,7 +41,7 @@ The Kafka plugin can read parameters through the `-p` argument (property):
 $ fluent-bit -i kafka -o stdout -p brokers=192.168.1.3:9092 -p topics=some-topic
 ```
 
-### Configuration file
+### Configuration file (recommended)
 
 In your main configuration file append the following:
 
@@ -44,7 +55,7 @@ pipeline:
           brokers: 192.168.1.3:9092
           topics: some-topic
           poll_ms: 100
-          
+
     outputs:
         - name: stdout
           match: '*'
@@ -83,13 +94,13 @@ pipeline:
           topics: fb-source
           poll_ms: 100
           format: json
-          
+
     filters:
         - name: lua
           match: '*'
           script: kafka.lua
           call: modify_kafka_message
-          
+
     outputs:
         - name: kafka
           brokers: kafka-broker:9092
@@ -128,4 +139,63 @@ Since the payload will be in JSON format, the plugin is configured to parse the
 
 Every message received is then processed with `kafka.lua` and sent back to the `fb-sink` topic of the same broker.
 
-The example can be executed locally with `make start` in the `examples/kafka_filter` directory (`docker/compose` is used).
+The example can be executed locally with `make start` in the `examples/kafka_filter` directory (`docker/compose` is used).
+
+## AWS MSK IAM Authentication
+
+*Available since Fluent Bit v4.0.4*
+
+Fluent Bit supports authentication to Amazon MSK (Managed Streaming for Apache Kafka) clusters using AWS IAM. This allows you to securely connect to MSK brokers with AWS credentials, leveraging IAM roles and policies for access control.
+
+### Prerequisites
+
+**Build Requirements**
+
+If you are compiling Fluent Bit from source, ensure the following requirements are met to enable AWS MSK IAM support:
+
+- The packages `libsasl2` and `libsasl2-dev` must be installed on your build environment.
+
+**Runtime Requirements**
+- **Network Access:** Fluent Bit must be able to reach your MSK broker endpoints (AWS VPC setup).
+- **AWS Credentials:** Provide credentials using any supported AWS method:
+  - IAM roles (recommended for EC2, ECS, or EKS)
+  - Environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`)
+  - AWS credentials file (`~/.aws/credentials`)
+  - Instance metadata service (IMDS)
+
+  Note these credentials are discovery by default when `aws_msk_iam` flag is enabled.
+
+- **IAM Permissions:** The credentials must allow access to the target MSK cluster (see example policy below).
+
+### Configuration Parameters
+
+| Property                  | Description                                         | Type    | Required                      |
+|---------------------------|-----------------------------------------------------|---------|-------------------------------|
+| `aws_msk_iam`             | Enable AWS MSK IAM authentication                   | Boolean | No (default: false)           |
+| `aws_msk_iam_cluster_arn` | Full ARN of the MSK cluster for region extraction   | String  | Yes (if `aws_msk_iam` is true)|
+
+### Example AWS IAM Policy
+
+> **Note:** IAM policies and permissions can be complex and may vary depending on your organization's security requirements. If you are unsure about the correct permissions or best practices, please consult with your AWS administrator or an AWS expert who is familiar with MSK and IAM security.
+
+The AWS credentials used by Fluent Bit must have permission to connect to your MSK cluster. Here is a minimal example policy:
+
+```json
+{
+    "Version": "2012-10-17",
+    "Statement": [
+        {
+            "Sid": "VisualEditor0",
+            "Effect": "Allow",
+            "Action": [
+                "kafka-cluster:*",
+                "kafka-cluster:DescribeCluster",
+                "kafka-cluster:ReadData",
+                "kafka-cluster:DescribeTopic",
+                "kafka-cluster:Connect"
+            ],
+            "Resource": "*"
+        }
+    ]
+}
+```