@@ -9,11 +9,13 @@ This directory contains examples for using Fluent Bit with Apache Kafka, includi
99A simple example demonstrating Kafka input and output with a Lua filter.
1010
1111** Features:**
12+
1213- Kafka consumer input
1314- Lua filter for message transformation
1415- Kafka producer output
1516
1617** Usage:**
18+
1719``` bash
1820docker-compose up
1921```
@@ -23,6 +25,7 @@ docker-compose up
2325Comprehensive examples for AWS MSK with IAM authentication, covering various deployment scenarios.
2426
2527** Scenarios covered:**
28+
2629- Standard MSK cluster (auto-detected region)
2730- MSK via PrivateLink (explicit region)
2831- MSK Serverless (auto-detected region)
@@ -37,13 +40,15 @@ AWS MSK supports IAM authentication, which eliminates the need to manage separat
3740### Configuration
3841
3942Enable MSK IAM authentication by setting:
43+
4044``` ini
4145rdkafka.sasl.mechanism aws_msk_iam
4246```
4347
4448### Region Detection
4549
4650Fluent Bit can automatically detect the AWS region from standard MSK broker hostnames:
51+
4752- ` b-1.example.kafka.us-east-1.amazonaws.com ` → region: ` us-east-1 `
4853- ` boot-abc.kafka-serverless.us-west-2.amazonaws.com ` → region: ` us-west-2 `
4954- ` vpce-123.kafka.eu-west-1.vpce.amazonaws.com ` → region: ` eu-west-1 `
@@ -87,30 +92,33 @@ Your IAM role or user needs the following permissions:
8792 " kafka-cluster:WriteData"
8893 ],
8994 "Resource" : [
90- " arn:aws:kafka:REGION:ACCOUNT:cluster/CLUSTER_NAME/* " ,
91- " arn:aws:kafka:REGION:ACCOUNT:topic/CLUSTER_NAME/*" ,
92- " arn:aws:kafka:REGION:ACCOUNT:group/CLUSTER_NAME/*"
95+ " arn:aws:kafka:REGION:ACCOUNT:cluster/CLUSTER_NAME/CLUSTER_UUID " ,
96+ " arn:aws:kafka:REGION:ACCOUNT:topic/CLUSTER_NAME/CLUSTER_UUID/ *" ,
97+ " arn:aws:kafka:REGION:ACCOUNT:group/CLUSTER_NAME/CLUSTER_UUID/ *"
9398 ]
9499 }
95100 ]
96101}
97102```
98103
104+ ** Note:** The cluster UUID can be found via the AWS Console, the DescribeCluster API, or the AWS CLI (` aws kafka describe-cluster ` ).
105+
99106** Note:** Adjust permissions based on your use case:
107+
100108- Consumers need: ` Connect ` , ` DescribeCluster ` , ` ReadData `
101109- Producers need: ` Connect ` , ` WriteData `
102110
103111## Configuration Parameters
104112
105113### Common Parameters
106114
107- | Parameter | Description | Required |
108- | -----------| -------------| ----------|
109- | ` brokers ` | Comma-separated list of Kafka brokers | Yes |
110- | ` topics ` | Topic name(s) for input or output | Yes |
111- | ` rdkafka.sasl.mechanism ` | Set to ` aws_msk_iam ` for MSK IAM auth | For MSK IAM |
112- | ` aws_region ` | AWS region (auto-detected if not set) | Only for custom DNS |
113- | ` group_id ` | Consumer group ID | For input |
115+ | Parameter | Description | Required |
116+ | ------------------------ | ------------------------------------- | ------------------- |
117+ | ` brokers ` | Comma-separated list of Kafka brokers | Yes |
118+ | ` topics ` | Topic name(s) for input or output | Yes |
119+ | ` rdkafka.sasl.mechanism ` | Set to ` aws_msk_iam ` for MSK IAM auth | For MSK IAM |
120+ | ` aws_region ` | AWS region (auto-detected if not set) | Only for custom DNS |
121+ | ` group_id ` | Consumer group ID | For input |
114122
115123### Additional librdkafka Parameters
116124
@@ -129,17 +137,20 @@ For a complete list of parameters, see the [librdkafka configuration documentati
129137### Local Kafka (Docker)
130138
1311391 . Start the Kafka stack:
140+
132141 ``` bash
133142 cd examples/kafka_filter
134143 docker-compose up -d
135144 ```
136145
1371462 . Run Fluent Bit:
147+
138148 ``` bash
139149 fluent-bit -c kafka.conf
140150 ```
141151
1421523 . Produce test messages:
153+
143154 ``` bash
144155 ./scripts/kafka-produce.sh
145156 ```
@@ -165,6 +176,7 @@ For a complete list of parameters, see the [librdkafka configuration documentati
165176** Error:** ` failed to setup MSK IAM authentication OAuth callback `
166177
167178** Solutions:**
179+
168180- For custom DNS/PrivateLink: Add ` aws_region ` parameter
169181- Verify AWS credentials are available
170182- Check IAM permissions
@@ -175,6 +187,7 @@ For a complete list of parameters, see the [librdkafka configuration documentati
175187
176188** Solution:**
177189Explicitly set the region:
190+
178191``` ini
179192aws_region us-east-1
180193```
@@ -183,6 +196,7 @@ aws_region us-east-1
183196
184197** Solution:**
185198Increase timeout values:
199+
186200``` ini
187201rdkafka.socket.timeout.ms 60000
188202rdkafka.metadata.max.age.ms 180000
@@ -197,5 +211,6 @@ rdkafka.metadata.max.age.ms 180000
197211## Support
198212
199213For issues or questions:
214+
200215- [ Fluent Bit GitHub Issues] ( https://github.com/fluent/fluent-bit/issues )
201216- [ Fluent Bit Slack Community] ( https://fluentbit.io/slack )
0 commit comments