Skip to content

Commit fb3e713

Browse files
esmerelesmerel
authored
Merge branch 'master' into lynettemiles/sc-136206/update-fluent-bit-docs-pipeline-inputs-tail
Signed-off-by: Lynette Miles <[email protected]>
2 parents eb8bb01 + 755a324 commit fb3e713

24 files changed

+1210
-523
lines changed

‎pipeline/inputs/kafka.md

Lines changed: 85 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
1-
# Kafka
1+
# Kafka Consumer
22

3-
The _Kafka_ input plugin subscribes to one or more Kafka topics to collect messages from an [Apache Kafka](https://kafka.apache.org/) service.
3+
The _Kafka_ input plugin enables Fluent Bit to consume messages directly from one or more [Apache Kafka](https://kafka.apache.org/) topics. By subscribing to specified topics, this plugin efficiently collects and forwards Kafka messages for further processing within your Fluent Bit pipeline.
4+
5+
Starting with version 4.0.4, the Kafka input plugin supports authentication with AWS MSK IAM, enabling integration with Amazon MSK (Managed Streaming for Apache Kafka) clusters that require IAM-based access.
46

57
This plugin uses the official [librdkafka C library](https://github.com/edenhill/librdkafka) as a built-in dependency.
68

@@ -30,7 +32,7 @@ The Kafka plugin can read parameters through the `-p` argument (property):
3032
$ fluent-bit -i kafka -o stdout -p brokers=192.168.1.3:9092 -p topics=some-topic
3133
```
3234

33-
### Configuration file
35+
### Configuration file (recommended)
3436

3537
In your main configuration file append the following:
3638

@@ -44,7 +46,7 @@ pipeline:
4446
brokers: 192.168.1.3:9092
4547
topics: some-topic
4648
poll_ms: 100
47-
49+
4850
outputs:
4951
- name: stdout
5052
match: '*'
@@ -83,13 +85,13 @@ pipeline:
8385
topics: fb-source
8486
poll_ms: 100
8587
format: json
86-
88+
8789
filters:
8890
- name: lua
8991
match: '*'
9092
script: kafka.lua
9193
call: modify_kafka_message
92-
94+
9395
outputs:
9496
- name: kafka
9597
brokers: kafka-broker:9092
@@ -128,4 +130,80 @@ Since the payload will be in JSON format, the plugin is configured to parse the
128130

129131
Every message received is then processed with `kafka.lua` and sent back to the `fb-sink` topic of the same broker.
130132

131-
The example can be executed locally with `make start` in the `examples/kafka_filter` directory (`docker/compose` is used).
133+
The example can be executed locally with `make start` in the `examples/kafka_filter` directory (`docker/compose` is used).
134+
135+
## AWS MSK IAM Authentication
136+
137+
*Available since Fluent Bit v4.0.4*
138+
139+
Fluent Bit supports authentication to Amazon MSK (Managed Streaming for Apache Kafka) clusters using AWS IAM. This allows you to securely connect to MSK brokers with AWS credentials, leveraging IAM roles and policies for access control.
140+
141+
### Prerequisites
142+
143+
**Build Requirements**
144+
145+
If you are compiling Fluent Bit from source, ensure the following requirements are met to enable AWS MSK IAM support:
146+
147+
- The packages `libsasl2` and `libsasl2-dev` must be installed on your build environment.
148+
149+
**Runtime Requirements**
150+
- **Network Access:** Fluent Bit must be able to reach your MSK broker endpoints (AWS VPC setup).
151+
- **AWS Credentials:** Provide credentials using any supported AWS method:
152+
- IAM roles (recommended for EC2, ECS, or EKS)
153+
- Environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`)
154+
- AWS credentials file (`~/.aws/credentials`)
155+
- Instance metadata service (IMDS)
156+
157+
Note these credentials are discovery by default when `aws_msk_iam` flag is enabled.
158+
159+
- **IAM Permissions:** The credentials must allow access to the target MSK cluster (see example policy below).
160+
161+
### Configuration Parameters
162+
163+
| Property | Description | Type | Required |
164+
|---------------------------|-----------------------------------------------------|---------|-------------------------------|
165+
| `aws_msk_iam` | Enable AWS MSK IAM authentication | Boolean | No (default: false) |
166+
| `aws_msk_iam_cluster_arn` | Full ARN of the MSK cluster for region extraction | String | Yes (if `aws_msk_iam` is true)|
167+
168+
169+
### Configuration Example
170+
171+
```yaml
172+
pipeline:
173+
inputs:
174+
- name: kafka
175+
brokers: my-cluster.abcdef.c1.kafka.us-east-1.amazonaws.com:9098
176+
topics: my-topic
177+
aws_msk_iam: true
178+
aws_msk_iam_cluster_arn: arn:aws:kafka:us-east-1:123456789012:cluster/my-cluster/abcdef-1234-5678-9012-abcdefghijkl-s3
179+
180+
outputs:
181+
- name: stdout
182+
match: '*'
183+
```
184+
185+
### Example AWS IAM Policy
186+
187+
> **Note:** IAM policies and permissions can be complex and may vary depending on your organization's security requirements. If you are unsure about the correct permissions or best practices, please consult with your AWS administrator or an AWS expert who is familiar with MSK and IAM security.
188+
189+
The AWS credentials used by Fluent Bit must have permission to connect to your MSK cluster. Here is a minimal example policy:
190+
191+
```json
192+
{
193+
"Version": "2012-10-17",
194+
"Statement": [
195+
{
196+
"Sid": "VisualEditor0",
197+
"Effect": "Allow",
198+
"Action": [
199+
"kafka-cluster:*",
200+
"kafka-cluster:DescribeCluster",
201+
"kafka-cluster:ReadData",
202+
"kafka-cluster:DescribeTopic",
203+
"kafka-cluster:Connect"
204+
],
205+
"Resource": "*"
206+
}
207+
]
208+
}
209+
```

‎pipeline/inputs/network-io-metrics.md

Lines changed: 41 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -25,20 +25,36 @@ To monitor network traffic from your system, you can run the plugin from the com
2525

2626
Run Fluent Bit using a command similar to the following:
2727

28-
```bash
29-
bin/fluent-bit -i netif -p interface=eth0 -o stdout
28+
```shell
29+
$ fluent-bit -i netif -p interface=eth0 -o stdout
3030
```
3131

3232
Which returns something the following:
3333

3434
```text
35-
Fluent Bit v1.x.x
36-
* Copyright (C) 2019-2020 The Fluent Bit Authors
37-
* Copyright (C) 2015-2018 Treasure Data
35+
Fluent Bit v4.0.3
36+
* Copyright (C) 2015-2025 The Fluent Bit Authors
3837
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
3938
* https://fluentbit.io
4039
41-
[2017/07/08 23:34:18] [ info] [engine] started
40+
______ _ _ ______ _ _ ___ _____
41+
| ___| | | | | ___ (_) | / || _ |
42+
| |_ | |_ _ ___ _ __ | |_ | |_/ /_| |_ __ __/ /| || |/' |
43+
| _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / /_| || /| |
44+
| | | | |_| | __/ | | | |_ | |_/ / | |_ \ V /\___ |\ |_/ /
45+
\_| |_|\__,_|\___|_| |_|\__| \____/|_|\__| \_/ |_(_)___/
46+
47+
48+
[2025/07/01 14:44:47] [ info] [fluent bit] version=4.0.3, commit=f5f5f3c17d, pid=1
49+
[2025/07/01 14:44:47] [ info] [storage] ver=1.5.3, type=memory, sync=normal, checksum=off, max_chunks_up=128
50+
[2025/07/01 14:44:47] [ info] [simd ] disabled
51+
[2025/07/01 14:44:47] [ info] [cmetrics] version=1.0.3
52+
[2025/07/01 14:44:47] [ info] [ctraces ] version=0.6.6
53+
[2025/07/01 14:44:47] [ info] [input:mem:mem.0] initializing
54+
[2025/07/01 14:44:47] [ info] [input:mem:mem.0] storage_strategy='memory' (memory only)
55+
[2025/07/01 14:44:47] [ info] [sp] stream processor started
56+
[2025/07/01 14:44:47] [ info] [engine] Shutdown Grace Period=5, Shutdown Input Grace Period=2
57+
[2025/07/01 14:44:47] [ info] [output:stdout:stdout.0] worker #0 started
4258
[0] netif.0: [1499524459.001698260, {"eth0.rx.bytes"=>89769869, "eth0.rx.packets"=>73357, "eth0.rx.errors"=>0, "eth0.tx.bytes"=>4256474, "eth0.tx.packets"=>24293, "eth0.tx.errors"=>0}]
4359
[1] netif.0: [1499524460.002541885, {"eth0.rx.bytes"=>98, "eth0.rx.packets"=>1, "eth0.rx.errors"=>0, "eth0.tx.bytes"=>98, "eth0.tx.packets"=>1, "eth0.tx.errors"=>0}]
4460
[2] netif.0: [1499524461.001142161, {"eth0.rx.bytes"=>98, "eth0.rx.packets"=>1, "eth0.rx.errors"=>0, "eth0.tx.bytes"=>98, "eth0.tx.packets"=>1, "eth0.tx.errors"=>0}]
@@ -47,25 +63,9 @@ Fluent Bit v1.x.x
4763

4864
### Configuration file
4965

50-
In your main configuration file append the following `Input` and `Output` sections:
66+
In your main configuration file append the following:
5167

5268
{% tabs %}
53-
{% tab title="fluent-bit.conf" %}
54-
55-
```python
56-
[INPUT]
57-
Name netif
58-
Tag netif
59-
Interval_Sec 1
60-
Interval_NSec 0
61-
Interface eth0
62-
[OUTPUT]
63-
Name stdout
64-
Match *
65-
```
66-
67-
{% endtab %}
68-
6969
{% tab title="fluent-bit.yaml" %}
7070

7171
```yaml
@@ -76,14 +76,31 @@ pipeline:
7676
interval_sec: 1
7777
interval_nsec: 0
7878
interface: eth0
79+
7980
outputs:
8081
- name: stdout
8182
match: '*'
8283
```
8384
85+
{% endtab %}
86+
{% tab title="fluent-bit.conf" %}
87+
88+
```text
89+
[INPUT]
90+
Name netif
91+
Tag netif
92+
Interval_Sec 1
93+
Interval_NSec 0
94+
Interface eth0
95+
96+
[OUTPUT]
97+
Name stdout
98+
Match *
99+
```
100+
84101
{% endtab %}
85102
{% endtabs %}
86103

87104
Which calculates using the formula: `Total interval (sec) = Interval_Sec + (Interval_Nsec / 1000000000)`.
88105

89-
For example: `1.5s = 1s + 500000000ns`
106+
For example: `1.5s = 1s + 500000000ns`

‎pipeline/inputs/nginx.md

Lines changed: 78 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -61,38 +61,82 @@ server {
6161

6262
From the command line you can let Fluent Bit generate the checks with the following options:
6363

64-
```bash
65-
fluent-bit -i nginx_metrics -p host=127.0.0.1 -p port=80 -p status_url=/status -p nginx_plus=off -o stdout
64+
```shell
65+
$ fluent-bit -i nginx_metrics -p host=127.0.0.1 -p port=80 -p status_url=/status -p nginx_plus=off -o stdout
6666
```
6767

6868
To gather metrics from the command line with the NGINX Plus REST API you need to turn on the
6969
`nginx_plus` property:
7070

71-
```bash
72-
fluent-bit -i nginx_metrics -p host=127.0.0.1 -p port=80 -p nginx_plus=on -p status_url=/api -o stdout
71+
```shell
72+
$ fluent-bit -i nginx_metrics -p host=127.0.0.1 -p port=80 -p nginx_plus=on -p status_url=/api -o stdout
7373
```
7474

7575
### Configuration File
7676

77-
In your main configuration file append the following `Input` and `Output` sections:
77+
In your main configuration file append the following:
78+
79+
{% tabs %}
80+
{% tab title="fluent-bit.yaml" %}
81+
82+
```yaml
83+
pipeline:
84+
inputs:
85+
- name: nginx_metrics
86+
nginx_plus: off
87+
host: 127.0.0.1
88+
port: 80
89+
status_URL: /status
90+
91+
outputs:
92+
- name: stdout
93+
match: '*'
94+
```
95+
96+
{% endtab %}
97+
{% tab title="fluent-bit.conf" %}
7898
79-
```python
99+
```text
80100
[INPUT]
81101
Name nginx_metrics
82102
Nginx_Plus off
83103
Host 127.0.0.1
84104
Port 80
85105
Status_URL /status
86-
Nginx_Plus off
87106

88107
[OUTPUT]
89108
Name stdout
90109
Match *
91110
```
92111

112+
{% endtab %}
113+
{% endtabs %}
114+
93115
And for NGINX Plus API:
94116

95-
```python
117+
{% tabs %}
118+
{% tab title="fluent-bit.yaml" %}
119+
120+
```yaml
121+
pipeline:
122+
inputs:
123+
- name: nginx_metrics
124+
nginx_plus: on
125+
host: 127.0.0.1
126+
port: 80
127+
status_URL: /api
128+
129+
130+
outputs:
131+
- name: stdout
132+
match: '*'
133+
```
134+
135+
{% endtab %}
136+
{% tab title="fluent-bit.conf" %}
137+
138+
```text
139+
[INPUT]
96140
Name nginx_metrics
97141
Nginx_Plus on
98142
Host 127.0.0.1
@@ -104,23 +148,43 @@ And for NGINX Plus API:
104148
Match *
105149
```
106150

151+
{% endtab %}
152+
{% endtabs %}
153+
107154
## Test your configuration
108155

109156
You can test against the NGINX server running on localhost by invoking it directly from the command line:
110157

111-
```bash
112-
fluent-bit -i nginx_metrics -p host=127.0.0.1 -p nginx_plus=off -o stdout -p match=* -f 1
158+
```shell
159+
$ fluent-bit -i nginx_metrics -p host=127.0.0.1 -p nginx_plus=off -o stdout -p match=* -f 1
113160
```
114161

115162
Which should return something like the following:
116163

117164
```text
118-
Fluent Bit v2.x.x
119-
* Copyright (C) 2019-2020 The Fluent Bit Authors
120-
* Copyright (C) 2015-2018 Treasure Data
165+
Fluent Bit v4.0.3
166+
* Copyright (C) 2015-2025 The Fluent Bit Authors
121167
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
122168
* https://fluentbit.io
123169
170+
______ _ _ ______ _ _ ___ _____
171+
| ___| | | | | ___ (_) | / || _ |
172+
| |_ | |_ _ ___ _ __ | |_ | |_/ /_| |_ __ __/ /| || |/' |
173+
| _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / /_| || /| |
174+
| | | | |_| | __/ | | | |_ | |_/ / | |_ \ V /\___ |\ |_/ /
175+
\_| |_|\__,_|\___|_| |_|\__| \____/|_|\__| \_/ |_(_)___/
176+
177+
178+
[2025/07/01 14:44:47] [ info] [fluent bit] version=4.0.3, commit=f5f5f3c17d, pid=1
179+
[2025/07/01 14:44:47] [ info] [storage] ver=1.5.3, type=memory, sync=normal, checksum=off, max_chunks_up=128
180+
[2025/07/01 14:44:47] [ info] [simd ] disabled
181+
[2025/07/01 14:44:47] [ info] [cmetrics] version=1.0.3
182+
[2025/07/01 14:44:47] [ info] [ctraces ] version=0.6.6
183+
[2025/07/01 14:44:47] [ info] [input:mem:mem.0] initializing
184+
[2025/07/01 14:44:47] [ info] [input:mem:mem.0] storage_strategy='memory' (memory only)
185+
[2025/07/01 14:44:47] [ info] [sp] stream processor started
186+
[2025/07/01 14:44:47] [ info] [engine] Shutdown Grace Period=5, Shutdown Input Grace Period=2
187+
[2025/07/01 14:44:47] [ info] [output:stdout:stdout.0] worker #0 started
124188
2021-10-14T19:37:37.228691854Z nginx_connections_accepted = 788253884
125189
2021-10-14T19:37:37.228691854Z nginx_connections_handled = 788253884
126190
2021-10-14T19:37:37.228691854Z nginx_http_requests_total = 42045501
@@ -133,4 +197,4 @@ Fluent Bit v2.x.x
133197

134198
## Exported metrics
135199

136-
For a list of available metrics, see the [NGINX Prometheus Exporter metrics documentation](https://github.com/nginxinc/nginx-prometheus-exporter/blob/main/README.md) on GitHub.
200+
For a list of available metrics, see the [NGINX Prometheus Exporter metrics documentation](https://github.com/nginxinc/nginx-prometheus-exporter/blob/main/README.md) on GitHub.

0 commit comments

Comments
 (0)