Merge branch 'master' into lynettemiles/sc-136208/update-fluent-bit-docs-pipeline-inputs-thermal

Signed-off-by: Lynette  Miles <[email protected]>

Author: esmerel

pipeline/inputs/kafka.md

@@ -1,6 +1,8 @@
-# Kafka
+# Kafka Consumer
 
-The _Kafka_ input plugin subscribes to one or more Kafka topics to collect messages from an [Apache Kafka](https://kafka.apache.org/) service.
+The _Kafka_ input plugin enables Fluent Bit to consume messages directly from one or more [Apache Kafka](https://kafka.apache.org/) topics. By subscribing to specified topics, this plugin efficiently collects and forwards Kafka messages for further processing within your Fluent Bit pipeline.
+
+Starting with version 4.0.4, the Kafka input plugin supports authentication with AWS MSK IAM, enabling integration with Amazon MSK (Managed Streaming for Apache Kafka) clusters that require IAM-based access.
 
 This plugin uses the official [librdkafka C library](https://github.com/edenhill/librdkafka) as a built-in dependency.
 
@@ -30,7 +32,7 @@ The Kafka plugin can read parameters through the `-p` argument (property):
 $ fluent-bit -i kafka -o stdout -p brokers=192.168.1.3:9092 -p topics=some-topic
 ```
 
-### Configuration file
+### Configuration file (recommended)
 
 In your main configuration file append the following:
 
@@ -44,7 +46,7 @@ pipeline:
           brokers: 192.168.1.3:9092
           topics: some-topic
           poll_ms: 100
-          
+
     outputs:
         - name: stdout
           match: '*'
@@ -83,13 +85,13 @@ pipeline:
           topics: fb-source
           poll_ms: 100
           format: json
-          
+
     filters:
         - name: lua
           match: '*'
           script: kafka.lua
           call: modify_kafka_message
-          
+
     outputs:
         - name: kafka
           brokers: kafka-broker:9092
@@ -128,4 +130,80 @@ Since the payload will be in JSON format, the plugin is configured to parse the
 
 Every message received is then processed with `kafka.lua` and sent back to the `fb-sink` topic of the same broker.
 
-The example can be executed locally with `make start` in the `examples/kafka_filter` directory (`docker/compose` is used).
+The example can be executed locally with `make start` in the `examples/kafka_filter` directory (`docker/compose` is used).
+
+## AWS MSK IAM Authentication
+
+*Available since Fluent Bit v4.0.4*
+
+Fluent Bit supports authentication to Amazon MSK (Managed Streaming for Apache Kafka) clusters using AWS IAM. This allows you to securely connect to MSK brokers with AWS credentials, leveraging IAM roles and policies for access control.
+
+### Prerequisites
+
+**Build Requirements**
+
+If you are compiling Fluent Bit from source, ensure the following requirements are met to enable AWS MSK IAM support:
+
+- The packages `libsasl2` and `libsasl2-dev` must be installed on your build environment.
+
+**Runtime Requirements**
+- **Network Access:** Fluent Bit must be able to reach your MSK broker endpoints (AWS VPC setup).
+- **AWS Credentials:** Provide credentials using any supported AWS method:
+  - IAM roles (recommended for EC2, ECS, or EKS)
+  - Environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`)
+  - AWS credentials file (`~/.aws/credentials`)
+  - Instance metadata service (IMDS)
+
+  Note these credentials are discovery by default when `aws_msk_iam` flag is enabled.
+
+- **IAM Permissions:** The credentials must allow access to the target MSK cluster (see example policy below).
+
+### Configuration Parameters
+
+| Property                  | Description                                         | Type    | Required                      |
+|---------------------------|-----------------------------------------------------|---------|-------------------------------|
+| `aws_msk_iam`             | Enable AWS MSK IAM authentication                   | Boolean | No (default: false)           |
+| `aws_msk_iam_cluster_arn` | Full ARN of the MSK cluster for region extraction   | String  | Yes (if `aws_msk_iam` is true)|
+
+
+### Configuration Example
+
+```yaml
+pipeline:
+  inputs:
+    - name: kafka
+      brokers: my-cluster.abcdef.c1.kafka.us-east-1.amazonaws.com:9098
+      topics: my-topic
+      aws_msk_iam: true
+      aws_msk_iam_cluster_arn: arn:aws:kafka:us-east-1:123456789012:cluster/my-cluster/abcdef-1234-5678-9012-abcdefghijkl-s3
+
+  outputs:
+    - name: stdout
+      match: '*'
+```
+
+### Example AWS IAM Policy
+
+> **Note:** IAM policies and permissions can be complex and may vary depending on your organization's security requirements. If you are unsure about the correct permissions or best practices, please consult with your AWS administrator or an AWS expert who is familiar with MSK and IAM security.
+
+The AWS credentials used by Fluent Bit must have permission to connect to your MSK cluster. Here is a minimal example policy:
+
+```json
+{
+    "Version": "2012-10-17",
+    "Statement": [
+        {
+            "Sid": "VisualEditor0",
+            "Effect": "Allow",
+            "Action": [
+                "kafka-cluster:*",
+                "kafka-cluster:DescribeCluster",
+                "kafka-cluster:ReadData",
+                "kafka-cluster:DescribeTopic",
+                "kafka-cluster:Connect"
+            ],
+            "Resource": "*"
+        }
+    ]
+}
+```

pipeline/inputs/nginx.md

@@ -87,8 +87,7 @@ pipeline:
           host: 127.0.0.1
           port: 80
           status_URL: /status
-          
-          
+           
     outputs:
         - name: stdout
           match: '*'

pipeline/inputs/opentelemetry.md

@@ -63,7 +63,6 @@ The OpenTelemetry input plugin supports the following telemetry data types:
 A sample configuration file to get started will look something like the following:
 
 {% tabs %}
-
 {% tab title="fluent-bit.yaml" %}
 
 ```yaml
@@ -72,16 +71,16 @@ pipeline:
         - name: opentelemetry
           listen: 127.0.0.1
           port: 4318
+          
     outputs:
         - name: stdout
           match: '*'
 ```
 
 {% endtab %}
-
 {% tab title="fluent-bit.conf" %}
 
-```python
+```text
 [INPUT]
     name opentelemetry
     listen 127.0.0.1
@@ -93,13 +92,12 @@ pipeline:
 ```
 
 {% endtab %}
-
 {% endtabs %}
 
 With this configuration, Fluent Bit listens on port `4318` for data. You can now send telemetry data to the endpoints `/v1/metrics` for metrics, `/v1/traces` for traces, and `/v1/logs` for logs.
 
 A sample curl request to POST JSON encoded log data would be:
 
-```bash
-curl --header "Content-Type: application/json" --request POST --data '{"resourceLogs":[{"resource":{},"scopeLogs":[{"scope":{},"logRecords":[{"timeUnixNano":"1660296023390371588","body":{"stringValue":"{\"message\":\"dummy\"}"},"traceId":"","spanId":""}]}]}]}'   http://0.0.0.0:4318/v1/logs
-```
+```shell
+$ curl --header "Content-Type: application/json" --request POST --data '{"resourceLogs":[{"resource":{},"scopeLogs":[{"scope":{},"logRecords":[{"timeUnixNano":"1660296023390371588","body":{"stringValue":"{\"message\":\"dummy\"}"},"traceId":"","spanId":""}]}]}]}'   http://0.0.0.0:4318/v1/logs
+```

pipeline/inputs/podman-metrics.md

@@ -25,8 +25,8 @@ This plugin supports and automatically detects both `cgroups v1` and `v2`.
 
 You can run the following `curl` command:
 
-```bash
-curl 0.0.0.0:2021/metrics
+```shell
+$ curl 0.0.0.0:2021/metrics
 ```
 
 Which returns information like:
@@ -84,19 +84,41 @@ fluentbit_input_storage_chunks_busy_bytes{name="podman_metrics.0"} 0
 
 ### Configuration file
 
-```python
+{% tabs %}
+{% tab title="fluent-bit.yaml" %}
+
+```yaml
+pipeline:
+    inputs:
+        - name: podman_metrics
+          scrape_interval: 10
+          scrape_on_start: true
+          
+    outputs:
+        - name: prometheus_exporter
+```
+
+{% endtab %}
+{% tab title="fluent-bit.conf" %}
+
+
+```text
 [INPUT]
     name podman_metrics
     scrape_interval 10
     scrape_on_start true
+    
 [OUTPUT]
     name prometheus_exporter
 ```
 
+{% endtab %}
+{% endtabs %}
+
 ### Command line
 
-```bash
-fluent-bit -i podman_metrics -o prometheus_exporter
+```shell
+$ fluent-bit -i podman_metrics -o prometheus_exporter
 ```
 
 ### Exposed metrics
@@ -114,4 +136,4 @@ Currently supported counters are:
 - `container_network_transmit_bytes_total`
 - `container_network_transmit_errors_total`
 
-This plugin mimics the naming convention of Docker metrics exposed by [`cadvisor`](https://github.com/google/cadvisor).
+This plugin mimics the naming convention of Docker metrics exposed by [`cadvisor`](https://github.com/google/cadvisor).

pipeline/inputs/process-exporter-metrics.md

@@ -12,12 +12,12 @@ Fluent Bit 2.2 and later includes a process exporter plugin that builds off the
 The Process Exporter Metrics plugin implements collecting of the various metrics available from [the third party implementation of Prometheus Process Exporter](https://github.com/ncabatoff/process-exporter) and these will be expanded over time as needed.
 
 {% hint style="info" %}
- All metrics including those collected with this plugin flow through a separate pipeline from logs and current filters don't operate on top of metrics.
-{% endhint %}
 
-This plugin is only supported on Linux based operating systems as it uses the `proc` filesystem to access the relevant metrics.
+All metrics including those collected with this plugin flow through a separate pipeline from logs and current filters don't 
+operate on top of metrics. This plugin is only supported on Linux based operating systems as it uses the `proc` filesystem to 
+access the relevant metrics. MacOS doesn't have the `proc` filesystem so this plugin won't work for it.
 
-macOS doesn't have the `proc` filesystem so this plugin won't work for it.
+{% endhint %}
 
 ## Configuration
 
@@ -53,7 +53,40 @@ This input always runs in its own [thread](../../administration/multithreading.m
 
 In the following configuration file, the input plugin `process_exporter_metrics` collects metrics every 2 seconds and exposes them through the [Prometheus Exporter](../outputs/prometheus-exporter.md) output plugin on HTTP/TCP port 2021.
 
-```python
+{% tabs %}
+{% tab title="fluent-bit.yaml" %}
+
+```yaml
+# Process Exporter Metrics + Prometheus Exporter
+# -------------------------------------------
+# The following example collect host metrics on Linux and expose
+# them through a Prometheus HTTP end-point.
+#
+# After starting the service try it with:
+#
+# $ curl http://127.0.0.1:2021/metrics
+#
+service:
+    flush: 1
+    log_level: info
+    
+pipeline:
+    inputs:
+        - name: process_exporter_metrics
+          tag:  process_metrics
+          scrape_interval: 2
+          
+    outputs:
+        - name: prometheus_exporter
+          match: process_metrics
+          host: 0.0.0.0
+          port: 2021
+```
+
+{% endtab %}
+{% tab title="fluent-bit.conf" %}
+
+```text
 # Process Exporter Metrics + Prometheus Exporter
 # -------------------------------------------
 # The following example collect host metrics on Linux and expose
@@ -79,10 +112,13 @@ In the following configuration file, the input plugin `process_exporter_metrics`
     port            2021
 ```
 
+{% endtab %}
+{% endtabs %}
+
 You can see the metrics by using `curl`:
 
-```bash
-curl http://127.0.0.1:2021/metrics
+```shell
+$ curl http://127.0.0.1:2021/metrics
 ```
 
 ### Container to collect host metrics
@@ -93,18 +129,19 @@ The following `docker` command deploys Fluent Bit with a specific mount path for
 `procfs` and settings enabled to ensure that Fluent Bit can collect from the host.
 These are then exposed over port 2021.
 
-```bash
-docker run -ti -v /proc:/host/proc:ro \
-               -p 2021:2021        \
-               fluent/fluent-bit:2.2 \
-               /fluent-bit/bin/fluent-bit \
-                         -i process_exporter_metrics -p path.procfs=/host/proc  \
-                         -o prometheus_exporter \
-                         -f 1
+```shell
+$ docker run -ti -v /proc:/host/proc:ro \
+                 -p 2021:2021        \
+                 fluent/fluent-bit:2.2 \
+                 /fluent-bit/bin/fluent-bit \
+                 -i process_exporter_metrics \
+                 -p path.procfs=/host/proc  \
+                 -o prometheus_exporter \
+                 -f 1
 ```
 
 ## Enhancement requests
 
 Development prioritises a subset of the available collectors in the [third party implementation of Prometheus Process Exporter](https://github.com/ncabatoff/process-exporter). To request others, open a GitHub issue by using the following template:
 
-- [`in_process_exporter_metrics`](https://github.com/fluent/fluent-bit/issues/new?assignees=\&labels=\&template=feature_request.md\&title=in_process_exporter_metrics:%20add%20ABC%20collector)
+- [`in_process_exporter_metrics`](https://github.com/fluent/fluent-bit/issues/new?assignees=\&labels=\&template=feature_request.md\&title=in_process_exporter_metrics:%20add%20ABC%20collector)