Skip to content

Commit 001e478

Browse files
committed
docs(kafka): improve README formatting and IAM permissions
- Add blank lines for better readability - Fix markdown table formatting - Correct IAM permission ARNs to include cluster UUID - Add note about finding cluster UUID via AWS Console/CLI - Improve section spacing and code block formatting Signed-off-by: Arbin <[email protected]>
1 parent 6bb3ab3 commit 001e478

File tree

1 file changed

+25
-10
lines changed

1 file changed

+25
-10
lines changed

examples/kafka_filter/README.md

Lines changed: 25 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -9,11 +9,13 @@ This directory contains examples for using Fluent Bit with Apache Kafka, includi
99
A simple example demonstrating Kafka input and output with a Lua filter.
1010

1111
**Features:**
12+
1213
- Kafka consumer input
1314
- Lua filter for message transformation
1415
- Kafka producer output
1516

1617
**Usage:**
18+
1719
```bash
1820
docker-compose up
1921
```
@@ -23,6 +25,7 @@ docker-compose up
2325
Comprehensive examples for AWS MSK with IAM authentication, covering various deployment scenarios.
2426

2527
**Scenarios covered:**
28+
2629
- Standard MSK cluster (auto-detected region)
2730
- MSK via PrivateLink (explicit region)
2831
- MSK Serverless (auto-detected region)
@@ -37,13 +40,15 @@ AWS MSK supports IAM authentication, which eliminates the need to manage separat
3740
### Configuration
3841

3942
Enable MSK IAM authentication by setting:
43+
4044
```ini
4145
rdkafka.sasl.mechanism aws_msk_iam
4246
```
4347

4448
### Region Detection
4549

4650
Fluent Bit can automatically detect the AWS region from standard MSK broker hostnames:
51+
4752
- `b-1.example.kafka.us-east-1.amazonaws.com` → region: `us-east-1`
4853
- `boot-abc.kafka-serverless.us-west-2.amazonaws.com` → region: `us-west-2`
4954
- `vpce-123.kafka.eu-west-1.vpce.amazonaws.com` → region: `eu-west-1`
@@ -87,30 +92,33 @@ Your IAM role or user needs the following permissions:
8792
"kafka-cluster:WriteData"
8893
],
8994
"Resource": [
90-
"arn:aws:kafka:REGION:ACCOUNT:cluster/CLUSTER_NAME/*",
91-
"arn:aws:kafka:REGION:ACCOUNT:topic/CLUSTER_NAME/*",
92-
"arn:aws:kafka:REGION:ACCOUNT:group/CLUSTER_NAME/*"
95+
"arn:aws:kafka:REGION:ACCOUNT:cluster/CLUSTER_NAME/CLUSTER_UUID",
96+
"arn:aws:kafka:REGION:ACCOUNT:topic/CLUSTER_NAME/CLUSTER_UUID/*",
97+
"arn:aws:kafka:REGION:ACCOUNT:group/CLUSTER_NAME/CLUSTER_UUID/*"
9398
]
9499
}
95100
]
96101
}
97102
```
98103

104+
**Note:** The cluster UUID can be found via the AWS Console, the DescribeCluster API, or the AWS CLI (`aws kafka describe-cluster`).
105+
99106
**Note:** Adjust permissions based on your use case:
107+
100108
- Consumers need: `Connect`, `DescribeCluster`, `ReadData`
101109
- Producers need: `Connect`, `WriteData`
102110

103111
## Configuration Parameters
104112

105113
### Common Parameters
106114

107-
| Parameter | Description | Required |
108-
|-----------|-------------|----------|
109-
| `brokers` | Comma-separated list of Kafka brokers | Yes |
110-
| `topics` | Topic name(s) for input or output | Yes |
111-
| `rdkafka.sasl.mechanism` | Set to `aws_msk_iam` for MSK IAM auth | For MSK IAM |
112-
| `aws_region` | AWS region (auto-detected if not set) | Only for custom DNS |
113-
| `group_id` | Consumer group ID | For input |
115+
| Parameter | Description | Required |
116+
| ------------------------ | ------------------------------------- | ------------------- |
117+
| `brokers` | Comma-separated list of Kafka brokers | Yes |
118+
| `topics` | Topic name(s) for input or output | Yes |
119+
| `rdkafka.sasl.mechanism` | Set to `aws_msk_iam` for MSK IAM auth | For MSK IAM |
120+
| `aws_region` | AWS region (auto-detected if not set) | Only for custom DNS |
121+
| `group_id` | Consumer group ID | For input |
114122

115123
### Additional librdkafka Parameters
116124

@@ -129,17 +137,20 @@ For a complete list of parameters, see the [librdkafka configuration documentati
129137
### Local Kafka (Docker)
130138

131139
1. Start the Kafka stack:
140+
132141
```bash
133142
cd examples/kafka_filter
134143
docker-compose up -d
135144
```
136145

137146
2. Run Fluent Bit:
147+
138148
```bash
139149
fluent-bit -c kafka.conf
140150
```
141151

142152
3. Produce test messages:
153+
143154
```bash
144155
./scripts/kafka-produce.sh
145156
```
@@ -165,6 +176,7 @@ For a complete list of parameters, see the [librdkafka configuration documentati
165176
**Error:** `failed to setup MSK IAM authentication OAuth callback`
166177

167178
**Solutions:**
179+
168180
- For custom DNS/PrivateLink: Add `aws_region` parameter
169181
- Verify AWS credentials are available
170182
- Check IAM permissions
@@ -175,6 +187,7 @@ For a complete list of parameters, see the [librdkafka configuration documentati
175187

176188
**Solution:**
177189
Explicitly set the region:
190+
178191
```ini
179192
aws_region us-east-1
180193
```
@@ -183,6 +196,7 @@ aws_region us-east-1
183196

184197
**Solution:**
185198
Increase timeout values:
199+
186200
```ini
187201
rdkafka.socket.timeout.ms 60000
188202
rdkafka.metadata.max.age.ms 180000
@@ -197,5 +211,6 @@ rdkafka.metadata.max.age.ms 180000
197211
## Support
198212

199213
For issues or questions:
214+
200215
- [Fluent Bit GitHub Issues](https://github.com/fluent/fluent-bit/issues)
201216
- [Fluent Bit Slack Community](https://fluentbit.io/slack)

0 commit comments

Comments
 (0)