diff --git a/.vale.ini b/.vale.ini index 73f30dec9..b9f4e951f 100755 --- a/.vale.ini +++ b/.vale.ini @@ -5,7 +5,7 @@ IgnoredScopes = tt, code [formats] mdx = md -[*.{md,markdown,txt,htm,html}] +[*.{md,markdown}] BasedOnStyles = FluentBit FluentBit.Terms = YES diff --git a/README.md b/README.md index b9cff22a1..070a46708 100644 --- a/README.md +++ b/README.md @@ -24,7 +24,7 @@ description: High Performance Telemetry Agent for Logs, Metrics and Traces - Connect nearly any source to nearly any destination using preexisting plugins - Extensibility: - Write input, filter, or output plugins in the C language - - WASM: [WASM Filter Plugins](development/wasm-filter-plugins.md) or [WASM Input Plugins](development/wasm-input-plugins.md) + - Wasm: [Wasm Filter Plugins](development/wasm-filter-plugins.md) or [Wasm Input Plugins](development/wasm-input-plugins.md) - Write [Filters in Lua](pipeline/filters/lua.md) or [Output plugins in Golang](development/golang-output-plugins.md) - [Monitoring](administration/monitoring.md): Expose internal metrics over HTTP in JSON and [Prometheus](https://prometheus.io/) format - [Stream Processing](stream-processing/introduction.md): Perform data selection and transformation using simple SQL queries diff --git a/SUMMARY.md b/SUMMARY.md index 7098f189f..7d775942b 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -98,7 +98,7 @@ * [Docker events](pipeline/inputs/docker-events.md) * [Docker metrics](pipeline/inputs/docker-metrics.md) * [Dummy](pipeline/inputs/dummy.md) - * [Ebpf](pipeline/inputs/ebpf.md) + * [eBPF](pipeline/inputs/ebpf.md) * [Elasticsearch](pipeline/inputs/elasticsearch.md) * [Exec WASI](pipeline/inputs/exec-wasi.md) * [Exec](pipeline/inputs/exec.md) diff --git a/pipeline/inputs/disk-io-metrics.md b/pipeline/inputs/disk-io-metrics.md index 4ed5cb5c7..11310cac9 100644 --- a/pipeline/inputs/disk-io-metrics.md +++ b/pipeline/inputs/disk-io-metrics.md @@ -78,4 +78,4 @@ pipeline: Total interval (sec) = `Interval_Sec` + `(Interval_Nsec` / 1000000000) -For example: `1.5s` = `1s` + `500000000ns` \ No newline at end of file +For example: `1.5s` = `1s` + `500000000ns` diff --git a/pipeline/inputs/ebpf.md b/pipeline/inputs/ebpf.md index a5b642e67..8ef357c17 100644 --- a/pipeline/inputs/ebpf.md +++ b/pipeline/inputs/ebpf.md @@ -1,4 +1,4 @@ -# Ebpf +# eBPF {% hint style="info" %} This plugin is experimental and might be unstable. Use it in development or testing environments only. Its features and behavior are subject to change. diff --git a/pipeline/inputs/elasticsearch.md b/pipeline/inputs/elasticsearch.md index a313efcce..8dfa7d3f6 100644 --- a/pipeline/inputs/elasticsearch.md +++ b/pipeline/inputs/elasticsearch.md @@ -11,12 +11,12 @@ The plugin supports the following configuration parameters: | `buffer_max_size` | Set the maximum size of buffer. | `4M` | | `buffer_chunk_size` | Set the buffer chunk size. | `512K` | | `tag_key` | Specify a key name for extracting as a tag. | `NULL` | -| `meta_key` | Specify a key name for meta information. | "@meta" | -| `hostname` | Specify hostname or fully qualified domain name. This parameter can be used for "sniffing" (auto-discovery of) cluster node information. | "localhost" | -| `version` | Specify Elasticsearch server version. This parameter is effective for checking a version of Elasticsearch/OpenSearch server version. | "8.0.0" | +| `meta_key` | Specify a key name for meta information. | `@meta` | +| `hostname` | Specify hostname or fully qualified domain name. This parameter can be used for "sniffing" (auto-discovery of) cluster node information. | `localhost` | +| `version` | Specify Elasticsearch server version. This parameter is effective for checking a version of Elasticsearch/OpenSearch server version. | `8.0.0` | | `threaded` | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` | -The Elasticsearch cluster uses "sniffing" to optimize the connections between its cluster and clients. Elasticsearch can build its cluster and dynamically generate a connection list which is called "sniffing". The `hostname` will be used for sniffing information and this is handled by the sniffing endpoint. +The Elasticsearch cluster uses "sniffing" to optimize the connections between its cluster and clients, which means it builds its cluster and dynamically generate a connection list. The `hostname` will be used for sniffing information and this is handled by the sniffing endpoint. ## Get started diff --git a/pipeline/inputs/exec-wasi.md b/pipeline/inputs/exec-wasi.md index 9ec5ec5af..d8e7992ee 100644 --- a/pipeline/inputs/exec-wasi.md +++ b/pipeline/inputs/exec-wasi.md @@ -10,7 +10,7 @@ The plugin supports the following configuration parameters: |:-------------------|:---------------------------------------------------------------------------------------------------------------------------------------------| | `WASI_Path` | The location of a Wasm program file. | | `Parser` | Specify the name of a parser to interpret the entry as a structured message. | -| `Accessible_Paths` | Specify the allowed list of paths to be able to access paths from WASM programs. | +| `Accessible_Paths` | Specify the allowed list of paths to be able to access paths from Wasm programs. | | `Interval_Sec` | Polling interval (seconds). | | `Interval_NSec` | Polling interval (nanosecond). | | `Wasm_Heap_Size` | Size of the heap size of Wasm execution. Review [unit sizes](../../administration/configuring-fluent-bit/unit-sizes.md) for allowed values. | @@ -23,11 +23,11 @@ The plugin supports the following configuration parameters: Here is a configuration example. -`in_exec_wasi` can handle parsers. To retrieve from structured data from a WASM program, you must create a `parser.conf`: +`in_exec_wasi` can handle parsers. To retrieve from structured data from a Wasm program, you must create a `parser.conf`: The `Time_Format` should be aligned for the format of your using timestamp. -This example assumes the WASM program writes JSON style strings to `stdout`. +This example assumes the Wasm program writes JSON style strings to `stdout`. {% tabs %} {% tab title="parsers.yaml" %} diff --git a/pipeline/inputs/exec.md b/pipeline/inputs/exec.md index 42500c15a..10896d6e4 100644 --- a/pipeline/inputs/exec.md +++ b/pipeline/inputs/exec.md @@ -194,4 +194,4 @@ The previous script would be safer if written with: -p command='echo '"$(printf '%q' "$@")" \ ``` -It's generally best to avoid dynamically generating the command or handling untrusted arguments. \ No newline at end of file +It's generally best to avoid dynamically generating the command or handling untrusted arguments. diff --git a/pipeline/inputs/forward.md b/pipeline/inputs/forward.md index 48a07cc58..581ccdc46 100644 --- a/pipeline/inputs/forward.md +++ b/pipeline/inputs/forward.md @@ -57,7 +57,7 @@ pipeline: port: 24224 buffer_chunk_size: 1M buffer_max_size: 6M - + outputs: - name: stdout match: '*' @@ -82,7 +82,7 @@ pipeline: {% endtab %} {% endtabs %} -## Fluent Bit and Secure Forward Setup +## Fluent Bit and secure forward setup In Fluent Bit v3 or later, `in_forward` can handle secure forward protocol. diff --git a/pipeline/inputs/http.md b/pipeline/inputs/http.md index 7256c34c8..ff5877279 100644 --- a/pipeline/inputs/http.md +++ b/pipeline/inputs/http.md @@ -51,7 +51,7 @@ pipeline: - name: http listen: 0.0.0.0 port: 8888 - + outputs: - name: stdout match: app.log @@ -171,7 +171,7 @@ The `success_header` parameter lets you set multiple HTTP headers on success. Th pipeline: inputs: - name: http - success_header: + success_header: - X-Custom custom-answer - X-Another another-answer ``` @@ -233,4 +233,4 @@ pipeline: ```shell fluent-bit -i http -p port=8888 -o stdout -``` \ No newline at end of file +``` diff --git a/pipeline/inputs/kafka.md b/pipeline/inputs/kafka.md index d59fa78fc..8aa5c4e67 100644 --- a/pipeline/inputs/kafka.md +++ b/pipeline/inputs/kafka.md @@ -1,4 +1,4 @@ -# Kafka Consumer +# Kafka The _Kafka_ input plugin enables Fluent Bit to consume messages directly from one or more [Apache Kafka](https://kafka.apache.org/) topics. By subscribing to specified topics, this plugin efficiently collects and forwards Kafka messages for further processing within your Fluent Bit pipeline. @@ -22,7 +22,7 @@ This plugin uses the official [librdkafka C library](https://github.com/edenhill ## Get started -To subscribe to or collect messages from Apache Kafka, run the plugin from the command line or through the configuration file as shown below. +To subscribe to or collect messages from Apache Kafka, run the plugin from the command line or through the configuration file as shown in the following examples. ### Command line @@ -132,41 +132,34 @@ Every message received is then processed with `kafka.lua` and sent back to the ` The example can be executed locally with `make start` in the `examples/kafka_filter` directory (`docker/compose` is used). -## AWS MSK IAM Authentication +## AWS MSK IAM authentication -*Available since Fluent Bit v4.0.4* +Fluent Bit v4.0.4 and later supports authentication to Amazon MSK (Managed Streaming for Apache Kafka) clusters using AWS IAM. This lets you securely connect to MSK brokers with AWS credentials, leveraging IAM roles and policies for access control. -Fluent Bit supports authentication to Amazon MSK (Managed Streaming for Apache Kafka) clusters using AWS IAM. This allows you to securely connect to MSK brokers with AWS credentials, leveraging IAM roles and policies for access control. - -### Prerequisites - -**Build Requirements** +### Build requirements If you are compiling Fluent Bit from source, ensure the following requirements are met to enable AWS MSK IAM support: - The packages `libsasl2` and `libsasl2-dev` must be installed on your build environment. -**Runtime Requirements** +### Runtime requirements + - **Network Access:** Fluent Bit must be able to reach your MSK broker endpoints (AWS VPC setup). -- **AWS Credentials:** Provide credentials using any supported AWS method: +- **AWS Credentials:** Provide these AWS credentials using any supported AWS method. These credentials are discovered by default when `aws_msk_iam` flag is enabled. - IAM roles (recommended for EC2, ECS, or EKS) - Environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`) - AWS credentials file (`~/.aws/credentials`) - Instance metadata service (IMDS) +- **IAM Permissions:** The credentials must allow access to the target MSK cluster, as shown in the following example policy. - Note these credentials are discovery by default when `aws_msk_iam` flag is enabled. - -- **IAM Permissions:** The credentials must allow access to the target MSK cluster (see example policy below). - -### Configuration Parameters +### Configuration parameters [#config-aws] -| Property | Description | Type | Required | -|---------------------------|---------------------------------------------------|---------|--------------------------------| -| `aws_msk_iam` | Enable AWS MSK IAM authentication | Boolean | No (default: false) | -| `aws_msk_iam_cluster_arn` | Full ARN of the MSK cluster for region extraction | String | Yes (if `aws_msk_iam` is true) | +| Property | Description | Required | +| -------- | ----------- | -------- | +| `aws_msk_iam` | If `true`, enables AWS MSK IAM authentication. Possible values: `true`, `false`. | `false` | +| `aws_msk_iam_cluster_arn` | Full ARN of the MSK cluster for region extraction. This value is required if `aws_msk_iam` is `true`. | _none_ | - -### Configuration Example +### Configuration example ```yaml pipeline: @@ -182,9 +175,13 @@ pipeline: match: '*' ``` -### Example AWS IAM Policy +### Example AWS IAM policy + +{% hint style="info" %} -> **Note:** IAM policies and permissions can be complex and may vary depending on your organization's security requirements. If you are unsure about the correct permissions or best practices, please consult with your AWS administrator or an AWS expert who is familiar with MSK and IAM security. +IAM policies and permissions can be complex and might vary depending on your organization's security requirements. If you are unsure about the correct permissions or best practices, consult your AWS administrator or an AWS expert who is familiar with MSK and IAM security. + +{% endhint %} The AWS credentials used by Fluent Bit must have permission to connect to your MSK cluster. Here is a minimal example policy: @@ -206,4 +203,4 @@ The AWS credentials used by Fluent Bit must have permission to connect to your M } ] } -``` \ No newline at end of file +``` diff --git a/pipeline/inputs/kernel-logs.md b/pipeline/inputs/kernel-logs.md index aba622fe3..accc1317d 100644 --- a/pipeline/inputs/kernel-logs.md +++ b/pipeline/inputs/kernel-logs.md @@ -1,6 +1,6 @@ # Kernel logs -The _kmsg_ input plugin reads the Linux Kernel log buffer from the beginning. It gets every record and parses fields as `priority`, `sequence`, `seconds`, `useconds`, and `message`. +The _Kernel logs_ (`kmsg`) input plugin reads the Linux Kernel log buffer from the beginning. It gets every record and parses fields as `priority`, `sequence`, `seconds`, `useconds`, and `message`. ## Configuration parameters @@ -64,4 +64,4 @@ pipeline: ``` {% endtab %} -{% endtabs %} \ No newline at end of file +{% endtabs %} diff --git a/pipeline/inputs/mqtt.md b/pipeline/inputs/mqtt.md index fd08f7384..e005ad2a6 100644 --- a/pipeline/inputs/mqtt.md +++ b/pipeline/inputs/mqtt.md @@ -55,7 +55,7 @@ pipeline: tag: data listen: 0.0.0.0 port: 1883 - + outputs: - name: stdout match: '*' @@ -77,4 +77,4 @@ pipeline: ``` {% endtab %} -{% endtabs %} \ No newline at end of file +{% endtabs %} diff --git a/pipeline/inputs/nginx.md b/pipeline/inputs/nginx.md index 323760792..ad826a0d9 100644 --- a/pipeline/inputs/nginx.md +++ b/pipeline/inputs/nginx.md @@ -71,7 +71,7 @@ To gather metrics from the command line with the NGINX Plus REST API you need to fluent-bit -i nginx_metrics -p host=127.0.0.1 -p port=80 -p nginx_plus=on -p status_url=/api -o stdout ``` -### Configuration File +### Configuration file In your main configuration file append the following: diff --git a/pipeline/inputs/node-exporter-metrics.md b/pipeline/inputs/node-exporter-metrics.md index c96f27f74..d4d122162 100644 --- a/pipeline/inputs/node-exporter-metrics.md +++ b/pipeline/inputs/node-exporter-metrics.md @@ -29,34 +29,34 @@ For example, if the global interval is set to `5` and an override interval of `6 This helps with down-sampling when collecting metrics. -| Key | Description | Default | -|-------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `scrape_interval` | The rate in seconds at which metrics are collected from the host operating system. | `5` | -| `path.procfs` | The mount point used to collect process information and metrics. | `/proc/` | -| `path.sysfs` | The path in the filesystem used to collect system metrics. | `/sys/` | -| `collector.cpu.scrape_interval` | The rate in seconds at which `cpu` metrics are collected from the host operating system. | `0` | -| `collector.cpufreq.scrape_interval` | The rate in seconds at which `cpufreq` metrics are collected from the host operating system. | `0` | -| `collector.meminfo.scrape_interval` | The rate in seconds at which `meminfo` metrics are collected from the host operating system. | `0` | -| `collector.diskstats.scrape_interval` | The rate in seconds at which `diskstats` metrics are collected from the host operating system. | `0` | -| `collector.filesystem.scrape_interval` | The rate in seconds at which `filesystem` metrics are collected from the host operating system. | `0` | -| `collector.uname.scrape_interval` | The rate in seconds at which `uname` metrics are collected from the host operating system. | `0` | -| `collector.stat.scrape_interval` | The rate in seconds at which `stat` metrics are collected from the host operating system. | `0` | -| `collector.time.scrape_interval` | The rate in seconds at which `time` metrics are collected from the host operating system. | `0` | -| `collector.loadavg.scrape_interval` | The rate in seconds at which `loadavg` metrics are collected from the host operating system. | `0` | -| `collector.vmstat.scrape_interval` | The rate in seconds at which `vmstat` metrics are collected from the host operating system. | `0` | -| `collector.thermal_zone.scrape_interval` | The rate in seconds at which `thermal_zone` metrics are collected from the host operating system. | `0` | -| `collector.filefd.scrape_interval` | The rate in seconds at which `filefd` metrics are collected from the host operating system. | `0` | -| `collector.nvme.scrape_interval` | The rate in seconds at which `nvme` metrics are collected from the host operating system. | `0` | -| `collector.processes.scrape_interval` | The rate in seconds at which system level `process` metrics are collected from the host operating system. | `0` | -| `metrics` | Specify which metrics are collected from the host operating system. These metrics depend on `/procfs` or `/sysfs`. The actual values of metrics will be read from `/proc` or `/sys` when needed. `cpu`, `cpufreq`, `meminfo`, `diskstats`, `filesystem`, `stat`, `loadavg`, `vmstat`, `netdev`, and `filefd` depend on `procfs`. `cpufreq` metrics depend on `sysfs`. | `"cpu,cpufreq,meminfo,diskstats,filesystem,uname,stat,time,loadavg,vmstat,netdev,filefd"` | -| `filesystem.ignore_mount_point_regex` | Specify the regular expression for the `mount` points to prevent collection of/ignore. | `^/(dev\|proc\|run/credentials/.+\|sys\|var/lib/docker/.+\|var/lib/containers/storage/.+)($\|/)` | -| `filesystem.ignore_filesystem_type_regex` | Specify the regular expression for the `filesystem` types to prevent collection of or ignore. | `^(autofs\|binfmt_misc\|bpf\|cgroup2?\|configfs\|debugfs\|devpts\|devtmpfs\|fusectl\|hugetlbfs\|iso9660\|mqueue\|nsfs\|overlay\|proc\|procfs\|pstore\|rpc_pipefs\|securityfs\|selinuxfs\|squashfs\|sysfs\|tracefs)$` | -| `diskstats.ignore_device_regex` | Specify the regular expression for the` diskstats` to prevent collection of/ignore. | `^(ram\|loop\|fd\|(h\|s\|v\|xv)d[a-z]\|nvme\\d+n\\d+p)\\d+$` | -| `systemd_service_restart_metrics` | Determines if the collector will include service restart metrics. | false | -| `systemd_unit_start_time_metrics` | Determines if the collector will include unit start time metrics. | false | -| `systemd_include_service_task_metrics` | Determines if the collector will include service task metrics. | false | -| `systemd_include_pattern` | Regular expression to determine which units are included in the metrics produced by the `systemd` collector. | Not applied unless explicitly set. | -| `systemd_exclude_pattern` | Regular expression to determine which units are excluded in the metrics produced by the `systemd` collector. | `.+\\.(automount\|device\|mount\|scope\|slice)"` | +| Key | Description | Default | +|---|---|---| +| `scrape_interval` | The rate in seconds at which metrics are collected from the host operating system. | `5` | +| `path.procfs` | The mount point used to collect process information and metrics. | `/proc/` | +| `path.sysfs` | The path in the filesystem used to collect system metrics. | `/sys/` | +| `collector.cpu.scrape_interval` | The rate in seconds at which `cpu` metrics are collected from the host operating system. | `0` | +| `collector.cpufreq.scrape_interval` | The rate in seconds at which `cpufreq` metrics are collected from the host operating system. | `0` | +| `collector.meminfo.scrape_interval` | The rate in seconds at which `meminfo` metrics are collected from the host operating system. | `0` | +| `collector.diskstats.scrape_interval` | The rate in seconds at which `diskstats` metrics are collected from the host operating system. | `0` | +| `collector.filesystem.scrape_interval` | The rate in seconds at which `filesystem` metrics are collected from the host operating system. | `0` | +| `collector.uname.scrape_interval` | The rate in seconds at which `uname` metrics are collected from the host operating system. | `0` | +| `collector.stat.scrape_interval` | The rate in seconds at which `stat` metrics are collected from the host operating system. | `0` | +| `collector.time.scrape_interval` | The rate in seconds at which `time` metrics are collected from the host operating system. | `0` | +| `collector.loadavg.scrape_interval` | The rate in seconds at which `loadavg` metrics are collected from the host operating system. | `0` | +| `collector.vmstat.scrape_interval` | The rate in seconds at which `vmstat` metrics are collected from the host operating system. | `0` | +| `collector.thermal_zone.scrape_interval` | The rate in seconds at which `thermal_zone` metrics are collected from the host operating system. | `0` | +| `collector.filefd.scrape_interval` | The rate in seconds at which `filefd` metrics are collected from the host operating system. | `0` | +| `collector.nvme.scrape_interval` | The rate in seconds at which `nvme` metrics are collected from the host operating system. | `0` | +| `collector.processes.scrape_interval` | The rate in seconds at which system level `process` metrics are collected from the host operating system. | `0` | +| `metrics` | Specify which metrics are collected from the host operating system. These metrics depend on `/procfs` or `/sysfs`. The actual values of metrics will be read from `/proc` or `/sys` when needed. `cpu`, `cpufreq`, `meminfo`, `diskstats`, `filesystem`, `stat`, `loadavg`, `vmstat`, `netdev`, and `filefd` depend on `procfs`. `cpufreq` metrics depend on `sysfs`. | `"cpu,cpufreq,meminfo,diskstats,filesystem,uname,stat,time,loadavg,vmstat,netdev,filefd"` | +| `filesystem.ignore_mount_point_regex` | Specify the regular expression for the `mount` points to prevent collection of/ignore. | `^/(dev\|proc\|run/credentials/.+\|sys\|var/lib/docker/.+\|var/lib/containers/storage/.+)($\|/)` | +| `filesystem.ignore_filesystem_type_regex` | Specify the regular expression for the `filesystem` types to prevent collection of or ignore. | `^(autofs\|binfmt_misc\|bpf\|cgroup2?\|configfs\|debugfs\|devpts\|devtmpfs\|fusectl\|hugetlbfs\|iso9660\|mqueue\|nsfs\|overlay\|proc\|procfs\|pstore\|rpc_pipefs\|securityfs\|selinuxfs\|squashfs\|sysfs\|tracefs)$` | +| `diskstats.ignore_device_regex` | Specify the regular expression for the` diskstats` to prevent collection of/ignore. | `^(ram\|loop\|fd\|(h\|s\|v\|xv)d[a-z]\|nvme\\d+n\\d+p)\\d+$` | +| `systemd_service_restart_metrics` | Determines if the collector will include service restart metrics. | `false` | +| `systemd_unit_start_time_metrics` | Determines if the collector will include unit start time metrics. | `false` | +| `systemd_include_service_task_metrics` | Determines if the collector will include service task metrics. | `false` | +| `systemd_include_pattern` | Regular expression to determine which units are included in the metrics produced by the `systemd` collector. | Not applied unless explicitly set. | +| `systemd_exclude_pattern` | Regular expression to determine which units are excluded in the metrics produced by the `systemd` collector. | `.+\\.(automount\|device\|mount\|scope\|slice)"` | ## Collectors available @@ -109,13 +109,13 @@ In the following configuration file, the input plugin `node_exporter_metrics` co service: flush: 1 log_level: info - + pipeline: inputs: - name: node_exporter_metrics tag: node_metrics scrape_interval: 2 - + outputs: - name: prometheus_exporter match: node_metrics @@ -190,7 +190,7 @@ If you use dashboards for monitoring, Grafana is one option. The Fluent Bit sour ```shell git clone https://github.com/fluent/fluent-bit - + cd fluent-bit/docker_compose/node-exporter-dashboard/ ``` @@ -219,4 +219,4 @@ docker-compose down The plugin implements a subset of the available collectors in the original Prometheus Node exporter. If you would like a specific collector prioritized, open a GitHub issue by using the following template: -- [`in_node_exporter_metrics`](https://github.com/fluent/fluent-bit/issues/new?assignees=\&labels=\&template=feature_request.md\&title=in_node_exporter_metrics:%20add%20ABC%20collector) \ No newline at end of file +- [`in_node_exporter_metrics`](https://github.com/fluent/fluent-bit/issues/new?assignees=\&labels=\&template=feature_request.md\&title=in_node_exporter_metrics:%20add%20ABC%20collector) diff --git a/pipeline/inputs/opentelemetry.md b/pipeline/inputs/opentelemetry.md index 86677ef04..60163d9ee 100644 --- a/pipeline/inputs/opentelemetry.md +++ b/pipeline/inputs/opentelemetry.md @@ -10,18 +10,18 @@ Fluent Bit has a compliant implementation which fully supports `OTLP/HTTP` and ` ## Configuration -| Key | Description | Default | -|----------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------| -| `listen` | The network address to listen on. | `0.0.0.0` | -| `port` | The port for Fluent Bit to listen for incoming connections. In Fluent Bit 3.0.2 or later, this port is used for both transport `OTLP/HTTP` and `OTLP/GRPC`. | `4318` | -| `tag` | Tag for all data ingested by this plugin. This will only be used if `tag_from_uri` is set to `false`. Otherwise, the tag will be created from the URI. | _none_ | -| `tag_key` | Specify the key name to overwrite a tag. If set, the tag will be overwritten by a value of the key. | _none_ | -| `raw_traces` | Route trace data as a log. | `false` | -| `buffer_max_size` | Specify the maximum buffer size in `KB`, `MB`, or `GB` to the HTTP payload. | `4M` | -| `buffer_chunk_size` | Initial size and allocation strategy to store the payload (advanced users only)` | `512K` | -| `successful_response_code` | Allows for setting a successful response code. Supported values: `200`, `201`, or `204`. | `201` | -| `tag_from_uri` | By default, the tag will be created from the URI. For example, `v1_metrics` from `/v1/metrics`. This must be set to false if using `tag`. | `true` | -| `threaded` | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` | +| Key | Description | Default | +|---|---|---| +| `listen` | The network address to listen on. | `0.0.0.0` | +| `port` | The port for Fluent Bit to listen for incoming connections. In Fluent Bit 3.0.2 or later, this port is used for both transport `OTLP/HTTP` and `OTLP/GRPC`. | `4318` | +| `tag` | Tag for all data ingested by this plugin. This will only be used if `tag_from_uri` is set to `false`. Otherwise, the tag will be created from the URI. | _none_ | +| `tag_key` | Specify the key name to overwrite a tag. If set, the tag will be overwritten by a value of the key. | _none_ | +| `raw_traces` | Route trace data as a log. | `false` | +| `buffer_max_size` | Specify the maximum buffer size in `KB`, `MB`, or `GB` to the HTTP payload. | `4M` | +| `buffer_chunk_size` | Initial size and allocation strategy to store the payload (advanced users only)` | `512K` | +| `successful_response_code` | Allows for setting a successful response code. Supported values: `200`, `201`, or `204`. | `201` | +| `tag_from_uri` | By default, the tag will be created from the URI. For example, `v1_metrics` from `/v1/metrics`. This must be set to false if using `tag`. | `true` | +| `threaded` | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` | Raw traces means that any data forwarded to the traces endpoint (`/v1/traces`) will be packed and forwarded as a log message, and won't be processed by Fluent Bit. The traces endpoint by default expects a valid `protobuf` encoded payload, but you can set the `raw_traces` option in case you want to get trace telemetry data to any of the Fluent Bit supported outputs. @@ -54,7 +54,7 @@ For `OTLP/GRPC`: The OpenTelemetry input plugin supports the following telemetry data types: -| Type | HTTP1/JSON | HTTP1/Protobuf | HTTP2/GRPC | +| Type | HTTP1/JSON | HTTP1/Protobuf | HTTP2/gRPC | |---------|---------------|----------------|------------| | Logs | Stable | Stable | Stable | | Metrics | Unimplemented | Stable | Stable | @@ -71,7 +71,7 @@ pipeline: - name: opentelemetry listen: 127.0.0.1 port: 4318 - + outputs: - name: stdout match: '*' @@ -100,4 +100,4 @@ A sample curl request to POST JSON encoded log data would be: ```shell curl --header "Content-Type: application/json" --request POST --data '{"resourceLogs":[{"resource":{},"scopeLogs":[{"scope":{},"logRecords":[{"timeUnixNano":"1660296023390371588","body":{"stringValue":"{\"message\":\"dummy\"}"},"traceId":"","spanId":""}]}]}]}' http://0.0.0.0:4318/v1/logs -``` \ No newline at end of file +``` diff --git a/pipeline/inputs/syslog.md b/pipeline/inputs/syslog.md index 8a42147fb..940f7ea1d 100644 --- a/pipeline/inputs/syslog.md +++ b/pipeline/inputs/syslog.md @@ -55,7 +55,7 @@ service: flush: 1 log_level: info parsers_file: parsers.yaml - + pipeline: inputs: - name: syslog @@ -63,7 +63,7 @@ pipeline: buffer_chunk_size: 32000 buffer_max_size: 64000 receive_buffer_size: 512000 - + outputs: - name: stdout match: '*' @@ -101,7 +101,7 @@ When Fluent Bit is running, you can send some messages using the logger tool: logger -u /tmp/in_syslog my_ident my_message ``` -Then run Fluent bit using the following command: +Then run Fluent Bit using the following command: ```shell # For YAML configuration. @@ -125,7 +125,7 @@ The following configuration examples cover different use cases to integrate Flue ### `rsyslog` to Fluent Bit: Network mode over TCP -#### Fluent Bit configuration +#### Fluent Bit configuration [#config-tcp] Put the following content in your configuration file: @@ -136,7 +136,7 @@ Put the following content in your configuration file: service: flush: 1 parsers_file: parsers.yaml - + pipeline: inputs: - name: syslog @@ -144,7 +144,7 @@ pipeline: listen: 0.0.0.0 port: 5140 mode: tcp - + outputs: - name: stdout match: '*' @@ -175,7 +175,7 @@ pipeline: Then, start Fluent Bit. -#### `rsyslog` configuration +#### `rsyslog` configuration [#rsyslog-tcp] Add a new file to your `rsyslog` configuration rules called `60-fluent-bit.conf` inside the directory `/etc/rsyslog.d/` and add the following content: @@ -191,7 +191,7 @@ sudo service rsyslog restart ### `rsyslog` to Fluent Bit: Unix socket mode over UDP -#### Fluent Bit configuration +#### Fluent Bit configuration [#config-udp] Put the following content in your Fluent Bit configuration: @@ -202,7 +202,7 @@ Put the following content in your Fluent Bit configuration: service: flush: 1 parsers_file: parsers.yaml - + pipeline: inputs: - name: syslog @@ -210,7 +210,7 @@ pipeline: path: /tmp/fluent-bit.sock mode: unix_udp unix_perm: 0644 - + outputs: - name: stdout match: '*' @@ -241,7 +241,7 @@ pipeline: Then, start Fluent Bit. -#### `rsyslog` configuration +#### `rsyslog` configuration [#rsyslog-udp] Add a new file to your `rsyslog` configuration rules called `60-fluent-bit.conf` inside the directory `/etc/rsyslog.d/` containing the following content: @@ -251,4 +251,4 @@ $OMUxSockSocket /tmp/fluent-bit.sock *.* :omuxsock: ``` -Make sure that the socket file is readable by `rsyslog` by modifying `Unix_Perm` key. \ No newline at end of file +Make sure that the socket file is readable by `rsyslog` by modifying `Unix_Perm` key. diff --git a/pipeline/inputs/tail.md b/pipeline/inputs/tail.md index 33670c4ec..25b86ec35 100644 --- a/pipeline/inputs/tail.md +++ b/pipeline/inputs/tail.md @@ -60,7 +60,7 @@ Although Fluent Bit has a soft limit of 2 MB for chunks, input plugins like ### Memory protection with `mem_buf_limit` -If Fluent Bit is not configured to use filesystem buffering, it needs mechanisms to protect against high memory consumption during backpressure scenarios (for example, when destination endpoints are down or network issues occur). The `mem_buf_limit` option restricts how much memory in chunks an input plugin can use. +If Fluent Bit isn't configured to use filesystem buffering, it needs mechanisms to protect against high memory consumption during backpressure scenarios (for example, when destination endpoints are down or network issues occur). The `mem_buf_limit` option restricts how much memory in chunks an input plugin can use. When filesystem buffering is enabled, memory management works differently. For more details, see [Buffering and Storage](../../administration/buffering-and-storage.md). @@ -400,11 +400,11 @@ By default, SQLite client tools don't format the columns in a human-readable way .width 5 32 12 12 10 ``` -## SQLite and Write Ahead Logging +## SQLite and write-ahead logging Fluent Bit keeps the state or checkpoint of each file through using a SQLite database file. If the service is restarted, it can continue consuming files from it last checkpoint position (offset). The default options set are enabled for high performance and corruption-safe. -The SQLite journal mode enabled is `Write Ahead Log` or `WAL`. This allows improved performance of read and write operations to disk. When enabled, you will see in your file system additional files being created, consider the following configuration statement: +The SQLite journal mode enabled is write-ahead logging or `WAL`. This allows improved performance of read and write operations to disk. When enabled, you will see in your file system additional files being created, consider the following configuration statement: {% tabs %} {% tab title="fluent-bit.yaml" %} @@ -437,9 +437,9 @@ The previous configuration enables a database file called `test.db` and in the s These two files support the `WAL` mechanism that helps improve performance and reduce the number system calls required. The `-wal` file refers to the file that stores the new changes to be committed. At some point the `WAL` file transactions are moved back to the real database file. The `-shm` file is a shared-memory type to allow concurrent-users to the `WAL` file. -### Write Ahead Logging (WAL) and memory usage +### Write-ahead logging (`WAL`) and memory usage -The `WAL` mechanism gives higher performance but also might increase the memory usage by Fluent Bit. Most of this usage comes from the memory mapped and cached pages. In some cases you might see that memory usage keeps a bit high giving the impression of a memory leak, but actually isn't relevant unless you want your memory metrics back to normal. Fluent Bit 1.7.3 introduced the new option `db.journal_mode` that sets the journal mode for databases. By default, this mode is set to `WAL (Write-Ahead Logging)`. Allowed configurations for `db.journal_mode` are `DELETE | TRUNCATE | PERSIST | MEMORY | WAL | OFF` . +The `WAL` mechanism gives higher performance but also might increase the memory usage by Fluent Bit. Most of this usage comes from the memory mapped and cached pages. In some cases you might see that memory usage keeps a bit high giving the impression of a memory leak, but actually isn't relevant unless you want your memory metrics back to normal. Fluent Bit 1.7.3 introduced the new option `db.journal_mode` that sets the journal mode for databases. By default, this mode is set to `WAL`. Allowed configurations for `db.journal_mode` are `DELETE | TRUNCATE | PERSIST | MEMORY | WAL | OFF` . ## File rotation @@ -454,4 +454,4 @@ While file rotation is handled, there are risks of potential log loss when using - See `logroate man page`: there is a very small-time slice between copying the file and truncating it, so some logging data might be lost. - Final note: the `Path` patterns can't match the rotated files. Otherwise, the rotated file would be read again and lead to duplicate records. -{% endhint %} \ No newline at end of file +{% endhint %} diff --git a/pipeline/inputs/tcp.md b/pipeline/inputs/tcp.md index 08cfccfe4..b2643ff34 100644 --- a/pipeline/inputs/tcp.md +++ b/pipeline/inputs/tcp.md @@ -2,7 +2,7 @@ The _TCP_ input plugin lets you retrieve structured JSON or raw messages over a TCP network interface (TCP port). -## Configuration Parameters +## Configuration parameters The plugin supports the following configuration parameters: @@ -11,7 +11,7 @@ The plugin supports the following configuration parameters: | `Listen` | Listener network interface. | `0.0.0.0` | | `Port` | TCP port to listen for connections. | `5170` | | `Buffer_Size` | Specify the maximum buffer size in KB to receive a JSON message. If not set, the default is the value of `Chunk_Size`. | `Chunk_Size` | -| `Chunk_Size` | The default buffer to store the incoming JSON messages. It doesn't allocate the maximum memory allowed; instead it allocates memory when required. The rounds of allocations are set by `Chunk_Size`. If not set, `Chunk_Size` is equal to 32 (32KB). | `32` | +| `Chunk_Size` | The default buffer to store the incoming JSON messages. It doesn't allocate the maximum memory allowed; instead it allocates memory when required. The rounds of allocations are set by `Chunk_Size`. If not set, `Chunk_Size` is equal to 32 (32 KB). | `32` | | `Format` | Specify the expected payload format. Supported values: `json` and `none`. When set to `json` it expects JSON maps. When set to `none`, every record splits using the defined `Separator`. | `json` | | `Separator` | When `Format` is set to `none`, Fluent Bit needs a separator string to split the records. | `LF` or `0x10` (break line) | | `Source_Address_Key` | Specify the key to inject the source address. | _none_ | @@ -105,4 +105,4 @@ You should see the following output: When receiving payloads in JSON format, there are high performance penalties. Parsing JSON is a very expensive task so you could expect your CPU usage increase under high load environments. -To get faster data ingestion, consider to use the option `Format none` to avoid JSON parsing if not needed. \ No newline at end of file +To get faster data ingestion, consider to use the option `Format none` to avoid JSON parsing if not needed. diff --git a/pipeline/inputs/windows-event-log-winevtlog.md b/pipeline/inputs/windows-event-log-winevtlog.md index 6c322b141..633d11afc 100644 --- a/pipeline/inputs/windows-event-log-winevtlog.md +++ b/pipeline/inputs/windows-event-log-winevtlog.md @@ -71,7 +71,7 @@ Some Windows Event Log channels, like `Security`, require administrative privile The default value of `Read_Limit_Per_Cycle` is `512KiB`. -512KiB(= 0x7ffff = 512 * 1024 * 1024) isn't equal to 512KB (= 512 * 1000 * 1000). To increase events per second on this plugin, specify larger value than 512KiB. +512 KiB(= 0x7ffff = 512 * 1024 * 1024) isn't equal to 512 KB (= 512 * 1000 * 1000). To increase events per second on this plugin, specify larger value than 512 KiB. #### Query languages for `Event_Query` parameter diff --git a/pipeline/inputs/windows-exporter-metrics.md b/pipeline/inputs/windows-exporter-metrics.md index e5ed7f5e2..bdf368a96 100644 --- a/pipeline/inputs/windows-exporter-metrics.md +++ b/pipeline/inputs/windows-exporter-metrics.md @@ -239,4 +239,4 @@ The WMI query will be called with the translated parameter as: The plugin implements a subset of the available collectors in the original Prometheus Windows Exporter. If you would like a specific collector prioritized, open a GitHub issue by using the following template: -- [`in_windows_exporter_metrics`](https://github.com/fluent/fluent-bit/issues/new?assignees=\&labels=\&template=feature_request.md\&title=in_windows_exporter_metrics:%20add%20ABC%20collector) \ No newline at end of file +- [`in_windows_exporter_metrics`](https://github.com/fluent/fluent-bit/issues/new?assignees=\&labels=\&template=feature_request.md\&title=in_windows_exporter_metrics:%20add%20ABC%20collector) diff --git a/vale-styles/FluentBit/Acronyms.yml b/vale-styles/FluentBit/Acronyms.yml index ee07184fb..c924ca5cc 100644 --- a/vale-styles/FluentBit/Acronyms.yml +++ b/vale-styles/FluentBit/Acronyms.yml @@ -20,6 +20,7 @@ exceptions: - BSD - CIDR - CLI + - CNCF - CORS - CPU - CRI diff --git a/vale-styles/FluentBit/FluentBitCasing.yml b/vale-styles/FluentBit/FluentBitCasing.yml index 3ebd581b6..b1282cfc4 100644 --- a/vale-styles/FluentBit/FluentBitCasing.yml +++ b/vale-styles/FluentBit/FluentBitCasing.yml @@ -5,12 +5,12 @@ ignorecase: false scope: text tokens: # Matches incorrect variants in plain prose but excludes filenames, links, quotes, and code - - '\bfluent bit\b(?![".\w:/\]])' - - '\bfluentbit\b(?![".\w:/\]])' - - '\bFluent bit\b(?![".\w:/\]])' - - '\bfluent Bit\b(?![".\w:/\]])' - - '\bfluent-bit\b(?![".\w:/\]])' - - '\bFluent-Bit\b(?![".\w:/\]])' - - '\bfluent\-Bit\b(?![".\w:/\]])' - - '\bFLUENT BIT\b(?![".\w:/\]])' - - '\bFLUENT-BIT\b(?![".\w:/\]])' + - '(?