Merge branch 'master' into lynettemiles/sc-136206/update-fluent-bit-docs-pipeline-inputs-tail

Signed-off-by: Lynette  Miles <[email protected]>

Author: esmerel

pipeline/inputs/kafka.md

@@ -1,6 +1,8 @@
-# Kafka
+# Kafka Consumer
 
-The _Kafka_ input plugin subscribes to one or more Kafka topics to collect messages from an [Apache Kafka](https://kafka.apache.org/) service.
+The _Kafka_ input plugin enables Fluent Bit to consume messages directly from one or more [Apache Kafka](https://kafka.apache.org/) topics. By subscribing to specified topics, this plugin efficiently collects and forwards Kafka messages for further processing within your Fluent Bit pipeline.
+
+Starting with version 4.0.4, the Kafka input plugin supports authentication with AWS MSK IAM, enabling integration with Amazon MSK (Managed Streaming for Apache Kafka) clusters that require IAM-based access.
 
 This plugin uses the official [librdkafka C library](https://github.com/edenhill/librdkafka) as a built-in dependency.
 
@@ -30,7 +32,7 @@ The Kafka plugin can read parameters through the `-p` argument (property):
 $ fluent-bit -i kafka -o stdout -p brokers=192.168.1.3:9092 -p topics=some-topic
 ```
 
-### Configuration file
+### Configuration file (recommended)
 
 In your main configuration file append the following:
 
@@ -44,7 +46,7 @@ pipeline:
           brokers: 192.168.1.3:9092
           topics: some-topic
           poll_ms: 100
-          
+
     outputs:
         - name: stdout
           match: '*'
@@ -83,13 +85,13 @@ pipeline:
           topics: fb-source
           poll_ms: 100
           format: json
-          
+
     filters:
         - name: lua
           match: '*'
           script: kafka.lua
           call: modify_kafka_message
-          
+
     outputs:
         - name: kafka
           brokers: kafka-broker:9092
@@ -128,4 +130,80 @@ Since the payload will be in JSON format, the plugin is configured to parse the
 
 Every message received is then processed with `kafka.lua` and sent back to the `fb-sink` topic of the same broker.
 
-The example can be executed locally with `make start` in the `examples/kafka_filter` directory (`docker/compose` is used).
+The example can be executed locally with `make start` in the `examples/kafka_filter` directory (`docker/compose` is used).
+
+## AWS MSK IAM Authentication
+
+*Available since Fluent Bit v4.0.4*
+
+Fluent Bit supports authentication to Amazon MSK (Managed Streaming for Apache Kafka) clusters using AWS IAM. This allows you to securely connect to MSK brokers with AWS credentials, leveraging IAM roles and policies for access control.
+
+### Prerequisites
+
+**Build Requirements**
+
+If you are compiling Fluent Bit from source, ensure the following requirements are met to enable AWS MSK IAM support:
+
+- The packages `libsasl2` and `libsasl2-dev` must be installed on your build environment.
+
+**Runtime Requirements**
+- **Network Access:** Fluent Bit must be able to reach your MSK broker endpoints (AWS VPC setup).
+- **AWS Credentials:** Provide credentials using any supported AWS method:
+  - IAM roles (recommended for EC2, ECS, or EKS)
+  - Environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`)
+  - AWS credentials file (`~/.aws/credentials`)
+  - Instance metadata service (IMDS)
+
+  Note these credentials are discovery by default when `aws_msk_iam` flag is enabled.
+
+- **IAM Permissions:** The credentials must allow access to the target MSK cluster (see example policy below).
+
+### Configuration Parameters
+
+| Property                  | Description                                         | Type    | Required                      |
+|---------------------------|-----------------------------------------------------|---------|-------------------------------|
+| `aws_msk_iam`             | Enable AWS MSK IAM authentication                   | Boolean | No (default: false)           |
+| `aws_msk_iam_cluster_arn` | Full ARN of the MSK cluster for region extraction   | String  | Yes (if `aws_msk_iam` is true)|
+
+
+### Configuration Example
+
+```yaml
+pipeline:
+  inputs:
+    - name: kafka
+      brokers: my-cluster.abcdef.c1.kafka.us-east-1.amazonaws.com:9098
+      topics: my-topic
+      aws_msk_iam: true
+      aws_msk_iam_cluster_arn: arn:aws:kafka:us-east-1:123456789012:cluster/my-cluster/abcdef-1234-5678-9012-abcdefghijkl-s3
+
+  outputs:
+    - name: stdout
+      match: '*'
+```
+
+### Example AWS IAM Policy
+
+> **Note:** IAM policies and permissions can be complex and may vary depending on your organization's security requirements. If you are unsure about the correct permissions or best practices, please consult with your AWS administrator or an AWS expert who is familiar with MSK and IAM security.
+
+The AWS credentials used by Fluent Bit must have permission to connect to your MSK cluster. Here is a minimal example policy:
+
+```json
+{
+    "Version": "2012-10-17",
+    "Statement": [
+        {
+            "Sid": "VisualEditor0",
+            "Effect": "Allow",
+            "Action": [
+                "kafka-cluster:*",
+                "kafka-cluster:DescribeCluster",
+                "kafka-cluster:ReadData",
+                "kafka-cluster:DescribeTopic",
+                "kafka-cluster:Connect"
+            ],
+            "Resource": "*"
+        }
+    ]
+}
+```

pipeline/inputs/network-io-metrics.md

@@ -25,20 +25,36 @@ To monitor network traffic from your system, you can run the plugin from the com
 
 Run Fluent Bit using a command similar to the following:
 
-```bash
-bin/fluent-bit -i netif -p interface=eth0 -o stdout
+```shell
+$ fluent-bit -i netif -p interface=eth0 -o stdout
 ```
 
 Which returns something the following:
 
 ```text
-Fluent Bit v1.x.x
-* Copyright (C) 2019-2020 The Fluent Bit Authors
-* Copyright (C) 2015-2018 Treasure Data
+Fluent Bit v4.0.3
+* Copyright (C) 2015-2025 The Fluent Bit Authors
 * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
 * https://fluentbit.io
 
-[2017/07/08 23:34:18] [ info] [engine] started
+______ _                  _    ______ _ _             ___  _____
+|  ___| |                | |   | ___ (_) |           /   ||  _  |
+| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __/ /| || |/' |
+|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / /_| ||  /| |
+| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /\___  |\ |_/ /
+\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/     |_(_)___/
+
+
+[2025/07/01 14:44:47] [ info] [fluent bit] version=4.0.3, commit=f5f5f3c17d, pid=1
+[2025/07/01 14:44:47] [ info] [storage] ver=1.5.3, type=memory, sync=normal, checksum=off, max_chunks_up=128
+[2025/07/01 14:44:47] [ info] [simd    ] disabled
+[2025/07/01 14:44:47] [ info] [cmetrics] version=1.0.3
+[2025/07/01 14:44:47] [ info] [ctraces ] version=0.6.6
+[2025/07/01 14:44:47] [ info] [input:mem:mem.0] initializing
+[2025/07/01 14:44:47] [ info] [input:mem:mem.0] storage_strategy='memory' (memory only)
+[2025/07/01 14:44:47] [ info] [sp] stream processor started
+[2025/07/01 14:44:47] [ info] [engine] Shutdown Grace Period=5, Shutdown Input Grace Period=2
+[2025/07/01 14:44:47] [ info] [output:stdout:stdout.0] worker #0 started
 [0] netif.0: [1499524459.001698260, {"eth0.rx.bytes"=>89769869, "eth0.rx.packets"=>73357, "eth0.rx.errors"=>0, "eth0.tx.bytes"=>4256474, "eth0.tx.packets"=>24293, "eth0.tx.errors"=>0}]
 [1] netif.0: [1499524460.002541885, {"eth0.rx.bytes"=>98, "eth0.rx.packets"=>1, "eth0.rx.errors"=>0, "eth0.tx.bytes"=>98, "eth0.tx.packets"=>1, "eth0.tx.errors"=>0}]
 [2] netif.0: [1499524461.001142161, {"eth0.rx.bytes"=>98, "eth0.rx.packets"=>1, "eth0.rx.errors"=>0, "eth0.tx.bytes"=>98, "eth0.tx.packets"=>1, "eth0.tx.errors"=>0}]
@@ -47,25 +63,9 @@ Fluent Bit v1.x.x
 
 ### Configuration file
 
-In your main configuration file append the following `Input` and `Output` sections:
+In your main configuration file append the following:
 
 {% tabs %}
-{% tab title="fluent-bit.conf" %}
-
-```python
-[INPUT]
-    Name          netif
-    Tag           netif
-    Interval_Sec  1
-    Interval_NSec 0
-    Interface     eth0
-[OUTPUT]
-    Name   stdout
-    Match  *
-```
-
-{% endtab %}
-
 {% tab title="fluent-bit.yaml" %}
 
 ```yaml
@@ -76,14 +76,31 @@ pipeline:
           interval_sec: 1
           interval_nsec: 0
           interface: eth0
+          
     outputs:
         - name: stdout
           match: '*'
 ```
 
+{% endtab %}
+{% tab title="fluent-bit.conf" %}
+
+```text
+[INPUT]
+    Name          netif
+    Tag           netif
+    Interval_Sec  1
+    Interval_NSec 0
+    Interface     eth0
+
+[OUTPUT]
+    Name   stdout
+    Match  *
+```
+
 {% endtab %}
 {% endtabs %}
 
 Which calculates using the formula: `Total interval (sec) = Interval_Sec + (Interval_Nsec / 1000000000)`.
 
-For example: `1.5s = 1s + 500000000ns`
+For example: `1.5s = 1s + 500000000ns`

pipeline/inputs/nginx.md

@@ -61,38 +61,82 @@ server {
 
 From the command line you can let Fluent Bit generate the checks with the following options:
 
-```bash
-fluent-bit -i nginx_metrics -p host=127.0.0.1 -p port=80 -p status_url=/status -p nginx_plus=off -o stdout
+```shell
+$ fluent-bit -i nginx_metrics -p host=127.0.0.1 -p port=80 -p status_url=/status -p nginx_plus=off -o stdout
 ```
 
 To gather metrics from the command line with the NGINX Plus REST API you need to turn on the
 `nginx_plus` property:
 
-```bash
-fluent-bit -i nginx_metrics -p host=127.0.0.1 -p port=80 -p nginx_plus=on -p status_url=/api -o stdout
+```shell
+$ fluent-bit -i nginx_metrics -p host=127.0.0.1 -p port=80 -p nginx_plus=on -p status_url=/api -o stdout
 ```
 
 ### Configuration File
 
-In your main configuration file append the following `Input` and `Output` sections:
+In your main configuration file append the following:
+
+{% tabs %}
+{% tab title="fluent-bit.yaml" %}
+
+```yaml
+pipeline:
+    inputs:
+        - name: nginx_metrics
+          nginx_plus: off
+          host: 127.0.0.1
+          port: 80
+          status_URL: /status
+           
+    outputs:
+        - name: stdout
+          match: '*'
+```
+
+{% endtab %}
+{% tab title="fluent-bit.conf" %}
 
-```python
+```text
 [INPUT]
     Name          nginx_metrics
     Nginx_Plus    off
     Host          127.0.0.1
     Port          80
     Status_URL    /status
-    Nginx_Plus    off
 
 [OUTPUT]
     Name   stdout
     Match  *
 ```
 
+{% endtab %}
+{% endtabs %}
+
 And for NGINX Plus API:
 
-```python
+{% tabs %}
+{% tab title="fluent-bit.yaml" %}
+
+```yaml
+pipeline:
+    inputs:
+        - name: nginx_metrics
+          nginx_plus: on
+          host: 127.0.0.1
+          port: 80
+          status_URL: /api
+          
+          
+    outputs:
+        - name: stdout
+          match: '*'
+```
+
+{% endtab %}
+{% tab title="fluent-bit.conf" %}
+
+```text
+[INPUT]
     Name          nginx_metrics
     Nginx_Plus    on
     Host          127.0.0.1
@@ -104,23 +148,43 @@ And for NGINX Plus API:
     Match  *
 ```
 
+{% endtab %}
+{% endtabs %}
+
 ## Test your configuration
 
 You can test against the NGINX server running on localhost by invoking it directly from the command line:
 
-```bash
-fluent-bit -i nginx_metrics -p host=127.0.0.1 -p nginx_plus=off -o stdout -p match=* -f 1
+```shell
+$ fluent-bit -i nginx_metrics -p host=127.0.0.1 -p nginx_plus=off -o stdout -p match=* -f 1
 ```
 
 Which should return something like the following:
 
 ```text
-Fluent Bit v2.x.x
-* Copyright (C) 2019-2020 The Fluent Bit Authors
-* Copyright (C) 2015-2018 Treasure Data
+Fluent Bit v4.0.3
+* Copyright (C) 2015-2025 The Fluent Bit Authors
 * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
 * https://fluentbit.io
 
+______ _                  _    ______ _ _             ___  _____
+|  ___| |                | |   | ___ (_) |           /   ||  _  |
+| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __/ /| || |/' |
+|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / /_| ||  /| |
+| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /\___  |\ |_/ /
+\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/     |_(_)___/
+
+
+[2025/07/01 14:44:47] [ info] [fluent bit] version=4.0.3, commit=f5f5f3c17d, pid=1
+[2025/07/01 14:44:47] [ info] [storage] ver=1.5.3, type=memory, sync=normal, checksum=off, max_chunks_up=128
+[2025/07/01 14:44:47] [ info] [simd    ] disabled
+[2025/07/01 14:44:47] [ info] [cmetrics] version=1.0.3
+[2025/07/01 14:44:47] [ info] [ctraces ] version=0.6.6
+[2025/07/01 14:44:47] [ info] [input:mem:mem.0] initializing
+[2025/07/01 14:44:47] [ info] [input:mem:mem.0] storage_strategy='memory' (memory only)
+[2025/07/01 14:44:47] [ info] [sp] stream processor started
+[2025/07/01 14:44:47] [ info] [engine] Shutdown Grace Period=5, Shutdown Input Grace Period=2
+[2025/07/01 14:44:47] [ info] [output:stdout:stdout.0] worker #0 started
 2021-10-14T19:37:37.228691854Z nginx_connections_accepted = 788253884
 2021-10-14T19:37:37.228691854Z nginx_connections_handled = 788253884
 2021-10-14T19:37:37.228691854Z nginx_http_requests_total = 42045501
@@ -133,4 +197,4 @@ Fluent Bit v2.x.x
 
 ## Exported metrics
 
-For a list of available metrics, see the [NGINX Prometheus Exporter metrics documentation](https://github.com/nginxinc/nginx-prometheus-exporter/blob/main/README.md) on GitHub.
+For a list of available metrics, see the [NGINX Prometheus Exporter metrics documentation](https://github.com/nginxinc/nginx-prometheus-exporter/blob/main/README.md) on GitHub.