diff --git a/administration/aws-credentials.md b/administration/aws-credentials.md index 9fbd5ea68..9ab87f568 100644 --- a/administration/aws-credentials.md +++ b/administration/aws-credentials.md @@ -23,11 +23,11 @@ The shared settings evaluate in the following order: | Setting | File | Description | |---|---|---| | `credential_process` | `config` | Linux only. See [Sourcing credentials with an external process in the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-sourcing-external.html). | -| `aws_access_key_id`
`aws_secret_access_key`
`aws_session_token` | `credentials` | Access key ID and secret key to use to authenticate. The session token must be set for temporary credentials. | +| `aws_access_key_id`, `aws_secret_access_key`, `aws_session_token` | `credentials` | Access key ID and secret key to use to authenticate. The session token must be set for temporary credentials. | No other settings are supported. -## EKS Web Identity Token (OIDC) +## EKS web identity token (OIDC) Credentials are fetched using a signed web identity token for a Kubernetes service account. See [IAM roles for service accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html). diff --git a/administration/backpressure.md b/administration/backpressure.md index 09d92a8ff..5e9308a02 100644 --- a/administration/backpressure.md +++ b/administration/backpressure.md @@ -68,4 +68,4 @@ With `storage.type filesystem` and `storage.max_chunks_up`, the following log me ```text [input] {input name or alias} paused (storage buf overlimit) [input] {input name or alias} resume (storage buf overlimit) -``` \ No newline at end of file +``` diff --git a/administration/buffering-and-storage.md b/administration/buffering-and-storage.md index 4b0230775..7d433e4a8 100644 --- a/administration/buffering-and-storage.md +++ b/administration/buffering-and-storage.md @@ -105,7 +105,7 @@ Fluent Bit controls the number of chunks that are `up` in memory by using the fi By default, the engine allows a total of 128 chunks `up` in memory in total, considering all chunks. This value is controlled by the service property `storage.max_chunks_up`. The active chunks that are `up` are ready for delivery and are still receiving records. Any other remaining chunk is in a `down` state, which means that it's only in the filesystem and won't be `up` in memory unless it's ready to be delivered. Chunks are never much larger than 2 MB, so with the default `storage.max_chunks_up` value of 128, each input is limited to roughly 256 MB of memory. -If the input plugin has enabled `storage.type` as `filesystem`, when reaching the `storage.max_chunks_up` threshold, instead of the plugin being paused, all new data will go to chunks that are `down` in the filesystem. This lets you control memory usage by the service and also provides a guarantee that the service won't lose any data. By default, the enforcement of the `storage.max_chunks_up` limit is best-effort. Fluent Bit can only append new data to chunks that are `up`. When the limit is reached chunks will be temporarily brought `up` in memory to ingest new data, and then put to a `down` state afterwards. In general, Fluent Bit works to keep the total number of `up` chunks at or below `storage.max_chunks_up`. +If the input plugin has enabled `storage.type` as `filesystem`, when reaching the `storage.max_chunks_up` threshold, instead of the plugin being paused, all new data will go to chunks that are `down` in the filesystem. This lets you control memory usage by the service and also provides a guarantee that the service won't lose any data. By default, the enforcement of the `storage.max_chunks_up` limit is best-effort. Fluent Bit can only append new data to chunks that are `up`. When the limit is reached chunks will be temporarily brought `up` in memory to ingest new data, and then put to a `down` state afterwards. In general, Fluent Bit works to keep the total number of `up` chunks at or under `storage.max_chunks_up`. If `storage.pause_on_chunks_overlimit` is enabled (default is off), the input plugin pauses upon exceeding `storage.max_chunks_up`. With this option, `storage.max_chunks_up` becomes a hard limit for the input. When the input is paused, records won't be ingested until the plugin resumes. For some inputs, such as TCP and tail, pausing the input will almost certainly lead to log loss. For the tail input, Fluent Bit can save its current offset in the current file it's reading, and pick back up when the input is resumed. @@ -122,7 +122,7 @@ Fluent Bit implements the concept of logical queues. Based on its tag, a chunk c It's common to find cases where multiple destinations with different response times exist for a chunk, or one of the destinations is generating backpressure. -To limit the amount of filesystem chunks logically queueing, Fluent Bit v1.6 and later includes the `storage.total_limit_size` configuration property for output This property limits the total size in bytes of chunks that can exist in the filesystem for a certain logical output destination. If one of the destinations reaches the configured `storage.total_limit_size`, the oldest chunk from its queue for that logical output destination will be discarded to make room for new data. +To limit the amount of filesystem chunks logically queueing, Fluent Bit v1.6 and later includes the `storage.total_limit_size` configuration property for output. This property limits the total size in bytes of chunks that can exist in the filesystem for a certain logical output destination. If one of the destinations reaches the configured `storage.total_limit_size`, the oldest chunk from its queue for that logical output destination will be discarded to make room for new data. ## Configuration @@ -145,9 +145,9 @@ The Service section refers to the section defined in the main [configuration fil | `storage.checksum` | Enable the data integrity check when writing and reading data from the filesystem. The storage layer uses the CRC32 algorithm. Accepted values: `Off`, `On`. | `Off` | | `storage.max_chunks_up` | If the input plugin has enabled `filesystem` storage type, this property sets the maximum number of chunks that can be `up` in memory. Use this setting to control memory usage when you enable `storage.type filesystem`. | `128` | | `storage.backlog.mem_limit` | If `storage.path` is set, Fluent Bit looks for data chunks that weren't delivered and are still in the storage layer. These are called _backlog_ data. _Backlog chunks_ are filesystem chunks that were left over from a previous Fluent Bit run; chunks that couldn't be sent before exit that Fluent Bit will pick up when restarted. Fluent Bit will check the `storage.backlog.mem_limit` value against the current memory usage from all `up` chunks for the input. If the `up` chunks currently consume less memory than the limit, it will bring the _backlog_ chunks up into memory so they can be sent by outputs. | `5M` | -| `storage.backlog.flush_on_shutdown` | When enabled, Fluent Bit will attempt to flush all backlog filesystem chunks to their destination(s) during the shutdown process. This can help ensure data delivery before Fluent Bit stops, but may increase shutdown time. Accepted values: `Off`, `On`. | `Off` | +| `storage.backlog.flush_on_shutdown` | When enabled, Fluent Bit will attempt to flush all backlog filesystem chunks to their destination during the shutdown process. This can help ensure data delivery before Fluent Bit stops, but can increase shutdown time. Accepted values: `Off`, `On`. | `Off` | | `storage.metrics` | If `http_server` option is enabled in the main `[SERVICE]` section, this option registers a new endpoint where internal metrics of the storage layer can be consumed. For more details refer to the [Monitoring](monitoring.md) section. | `off` | -| `storage.delete_irrecoverable_chunks` | When enabled, [irrecoverable chunks](./buffering-and-storage.md#irrecoverable-chunks) will be deleted during runtime, and any other irrecoverable chunk located in the configured storage path directory will be deleted when Fluent-Bit starts. Accepted values: 'Off`, 'On`. | `Off` | +| `storage.delete_irrecoverable_chunks` | When enabled, [irrecoverable chunks](./buffering-and-storage.md#irrecoverable-chunks) will be deleted during runtime, and any other irrecoverable chunk located in the configured storage path directory will be deleted when Fluent Bit starts. Accepted values: `Off`, 'On`. | `Off` | A Service section will look like this: @@ -184,7 +184,7 @@ service: This configuration sets an optional buffering mechanism where the route to the data is `/var/log/flb-storage/`. It uses `normal` synchronization mode, without running a checksum and up to a maximum of 5 MB of memory when processing backlog data. -### Input Section Configuration +### Input section configuration Optionally, any Input plugin can configure their storage preference. The following table describes the options available: @@ -242,7 +242,7 @@ pipeline: {% endtab %} {% endtabs %} -### Output Section Configuration +### Output section configuration If certain chunks are filesystem `storage.type` based, it's possible to control the size of the logical queue for an output plugin. The following table describes the options available: @@ -302,4 +302,5 @@ pipeline: {% endtab %} {% endtabs %} -If Fluent Bit is offline because of a network issue, it will continue buffering CPU samples, keeping a maximum of 5 MB of the newest data. \ No newline at end of file +If Fluent Bit is offline because of a network issue, it will continue buffering CPU +samples, keeping a maximum of 5 MB of the newest data. diff --git a/administration/configuring-fluent-bit/README.md b/administration/configuring-fluent-bit/README.md index f0fc8d99e..2ebce7678 100644 --- a/administration/configuring-fluent-bit/README.md +++ b/administration/configuring-fluent-bit/README.md @@ -133,4 +133,4 @@ Outputs prometheus_exporter Prometheus Exporter prometheus_remote_write Prometheus remote write s3 Send to S3 -``` \ No newline at end of file +``` diff --git a/administration/configuring-fluent-bit/classic-mode/commands.md b/administration/configuring-fluent-bit/classic-mode/commands.md index 1f1cad60f..6239eae52 100644 --- a/administration/configuring-fluent-bit/classic-mode/commands.md +++ b/administration/configuring-fluent-bit/classic-mode/commands.md @@ -6,10 +6,10 @@ Fluent Bit `Commands` extends a configuration file with specific built-in featur | Command | Prototype | Description | | :--- | :--- | :--- | -| [`@INCLUDE`](commands.md#cmd_include) | `@INCLUDE FILE` | Include a configuration file. | -| [`@SET`](commands.md#cmd_set) | `@SET KEY=VAL` | Set a configuration variable. | +| [`@INCLUDE`](#include) | `@INCLUDE FILE` | Include a configuration file. | +| [`@SET`](#set) | `@SET KEY=VAL` | Set a configuration variable. | -## `@INCLUDE` +## `@INCLUDE` Configuring a logging pipeline might lead to an extensive configuration file. In order to maintain a human-readable configuration, split the configuration in multiple files. @@ -47,7 +47,7 @@ The following is an example of an `inputs.conf` file, like the one called in the Tag varlog.* ``` -### outputs.conf +### `outputs.conf` The following is an example of an `outputs.conf` file, like the one called in the previous example. @@ -64,7 +64,7 @@ The following is an example of an `outputs.conf` file, like the one called in th Logstash_Format On ``` -## `@SET` +## `@SET` Fluent Bit supports [configuration variables](variables.md). One way to expose this variables to Fluent Bit is through setting a shell environment variable, the other is through the `@SET` command. diff --git a/administration/configuring-fluent-bit/classic-mode/configuration-file.md b/administration/configuring-fluent-bit/classic-mode/configuration-file.md index 9f0287a3e..a8f63d6e8 100644 --- a/administration/configuring-fluent-bit/classic-mode/configuration-file.md +++ b/administration/configuring-fluent-bit/classic-mode/configuration-file.md @@ -1,7 +1,3 @@ ---- -description: This page describes the main configuration file used by Fluent Bit. ---- - # Configuration file @@ -119,7 +115,7 @@ The following is an example of an `OUTPUT` section: Match my*cpu ``` -### Example: collecting CPU metrics +### Collecting `cpu` metrics example The following configuration file example demonstrates how to collect CPU metrics and flush the results every five seconds to the standard output: @@ -138,7 +134,7 @@ The following configuration file example demonstrates how to collect CPU metrics Match my*cpu ``` -## Config Include File +## Config include file To avoid complicated long configuration files is better to split specific parts in different files and call them (include) from one main file. The `@INCLUDE` can be used in the following way: diff --git a/administration/configuring-fluent-bit/classic-mode/format-schema.md b/administration/configuring-fluent-bit/classic-mode/format-schema.md index fde51d2e5..45ba6e549 100644 --- a/administration/configuring-fluent-bit/classic-mode/format-schema.md +++ b/administration/configuring-fluent-bit/classic-mode/format-schema.md @@ -27,13 +27,13 @@ A section is defined by a name or title inside brackets. Using the previous exam - Any commented line under a section must be indented too. - End-of-line comments aren't supported, only full-line comments. -## Entries: key/value +## Entries: `key`/`value` A section can contain entries. An entry is defined by a line of text that contains a `Key` and a `Value`. Using the previous example, the `[SERVICE]` section contains two entries: one is the key `Daemon` with value `off` and the other is the key `Log_Level` with the value `debug`. The following rules apply: - An entry is defined by a key and a value. - A key must be indented. -- A key must contain a value which ends in the breakline. +- A key must contain a value which ends in a line break. - Multiple keys with the same name can exist. Commented lines are set prefixing the `#` character. Commented lines aren't processed but they must be indented. diff --git a/administration/configuring-fluent-bit/multiline-parsing.md b/administration/configuring-fluent-bit/multiline-parsing.md index 7715a740e..264d0beee 100644 --- a/administration/configuring-fluent-bit/multiline-parsing.md +++ b/administration/configuring-fluent-bit/multiline-parsing.md @@ -59,7 +59,7 @@ A rule specifies how to match a multiline pattern and perform the concatenation. - regular expression pattern - next state -A rule might be defined as follows (comments added to simplify the definition) in corresponding YAML and classic configuration examples below: +A rule might be defined as follows (comments added to simplify the definition) in the corresponding YAML and classic configuration examples: {% tabs %} {% tab title="parsers_multiline.yaml" %} @@ -99,7 +99,7 @@ To simplify the configuration of regular expressions, you can use the [Rubular]( #### Configuration example -The following example provides a full Fluent Bit configuration file for multiline parsing by using the definition explained previously. It is provided in corresponding YAML and classic configuration examples below: +The following example provides a full Fluent Bit configuration file for multiline parsing by using the definition explained previously. It's provided in following YAML and classic configuration examples: {% tabs %} @@ -344,7 +344,7 @@ multiline_parsers: - state: start_state regex: '/([a-zA-Z]+ \d+ \d+\:\d+\:\d+)(.*)/' next_state: cont - + - state: cont regex: '/^\s+at.*/' next_state: cont @@ -367,9 +367,9 @@ This file defines a multiline parser for the classic example. flush_timeout 1000 # # Regex rules for multiline parsing - # --------------------------------- + # --------------------------------- # - # configuration hints: + # configuration hints: # # - first state always has the name: start_state # - every field in the rule must be inside double quotes @@ -425,4 +425,4 @@ $ ./fluent-bit --config fluent-bit.conf "}] [2] tail.0: [[1750333602.460998000, {}], {"log"=>"another line... "}] -``` \ No newline at end of file +``` diff --git a/administration/configuring-fluent-bit/yaml/README.md b/administration/configuring-fluent-bit/yaml/README.md index 0d2a33855..7d8c51566 100644 --- a/administration/configuring-fluent-bit/yaml/README.md +++ b/administration/configuring-fluent-bit/yaml/README.md @@ -1,28 +1,28 @@ # YAML configuration -## Before You Get Started +## Before you get started -Fluent Bit traditionally offered a `classic` configuration mode, a custom configuration format that we are gradually phasing out. While `classic` mode has served well for many years, it has several limitations. Its basic design only supports grouping sections with key-value pairs and lacks the ability to handle sub-sections or complex data structures like lists. +Fluent Bit traditionally offered a `classic` configuration mode, a custom configuration format that's phasing out. While `classic` mode has served well for many years, it has several limitations. Its basic design only supports grouping sections with key-value pairs and lacks the ability to handle sub-sections or complex data structures like lists. -YAML, now a mainstream configuration format, has become essential in a cloud ecosystem where everything is configured this way. To minimize friction and provide a more intuitive experience for creating data pipelines, we strongly encourage users to transition to YAML. The YAML format enables features, such as processors, that are not possible to configure in `classic` mode. +YAML has become essential in a cloud ecosystem. To minimize friction and provide a more intuitive experience for creating data pipelines, users are encouraged to transition to YAML. The YAML format enables features, such as processors, that aren't possible to configure in `classic` mode. As of Fluent Bit v3.2, you can configure everything in YAML. -## List of Available Sections +## List of available sections Configuring Fluent Bit with YAML introduces the following root-level sections: -| Section Name |Description | -|----------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------| -| `service` | Describes the global configuration for the Fluent Bit service. This section is optional; if not set, default values will apply. Only one `service` section can be defined. | -| `parsers` | Lists parsers to be used by components like inputs, processors, filters, or output plugins. You can define multiple `parsers` sections, which can also be loaded from external files included in the main YAML configuration. | -| `multiline_parsers` | Lists multiline parsers, functioning similarly to `parsers`. Multiple definitions can exist either in the root or in included files. | -| `pipeline` | Defines a pipeline composed of inputs, processors, filters, and output plugins. You can define multiple `pipeline` sections, but they will not operate independently. Instead, all components will be merged into a single pipeline internally. | -| `plugins` | Specifies the path to external plugins (.so files) to be loaded by Fluent Bit at runtime. | -| `upstream_servers` | Refers to a group of node endpoints that can be referenced by output plugins that support this feature. | -| `env` | Sets a list of environment variables for Fluent Bit. Note that system environment variables are available, while the ones defined in the configuration apply only to Fluent Bit. | - -## Section Documentation +| Section Name | Description | +|--------------|-------------| +| `service` | Describes the global configuration for the Fluent Bit service. Optional. If not set, default values will apply. Only one `service` section can be defined. | +| `parsers` | Lists parsers to be used by components like inputs, processors, filters, or output plugins. You can define multiple `parsers` sections, which can also be loaded from external files included in the main YAML configuration. | +| `multiline_parsers` | Lists multiline parsers, functioning similarly to `parsers`. Multiple definitions can exist either in the root or in included files. | +| `pipeline` | Defines a pipeline composed of inputs, processors, filters, and output plugins. You can define multiple `pipeline` sections, but they won't operate independently. Instead, all components will be merged into a single pipeline internally. | +| `plugins` | Specifies the path to external plugins (`.so` files) to be loaded by Fluent Bit at runtime. | +| `upstream_servers` | Refers to a group of node endpoints that can be referenced by output plugins that support this feature. | +| `env` | Sets a list of environment variables for Fluent Bit. System environment variables are available, while the ones defined in the configuration apply only to Fluent Bit. | + +## Section documentation To access detailed configuration guides for each section, use the following links: diff --git a/administration/configuring-fluent-bit/yaml/configuration-file.md b/administration/configuring-fluent-bit/yaml/configuration-file.md index 76d4c91c3..e314aa7e1 100644 --- a/administration/configuring-fluent-bit/yaml/configuration-file.md +++ b/administration/configuring-fluent-bit/yaml/configuration-file.md @@ -26,7 +26,7 @@ The YAML configuration file doesn't support the following sections: YAML configuration is used in the smoke tests for containers. An always-correct up-to-date example is here: . {% endhint %} -## Env +## `env` The `env` section allows the definition of configuration variables that will be used later in the configuration file. @@ -79,7 +79,7 @@ The `service` section defines the global properties of the service. The Service | `coro_stack_size` | Set the coroutines stack size in bytes. The value must be greater than the page size of the running system. Don't set too small a value (for example, `4096`), or coroutine threads can overrun the stack buffer. Don't change the default value of this parameter unless you know what you are doing. | `24576` | | `scheduler.cap` | Set a maximum retry time in seconds. Supported from v1.8.7. | `2000` | | `scheduler.base` | Sets the base of exponential backoff. Supported from v1.8.7. | `5` | -| `json.convert_nan_to_null` | If enabled, NaN is converted to null when fluent-bit converts msgpack to JSON. | `false` | +| `json.convert_nan_to_null` | If enabled, NaN is converted to null when Fluent Bit converts `msgpack` to JSON. | `false` | | `sp.convert_from_str_to_num` | If enabled, Stream processor converts from number string to number type. | `true | The following is an example of a `service` section: @@ -122,7 +122,7 @@ pipeline: port: 8080 ``` -This pipeline consists of two `inputs`: a tail plugin and an HTTP server plugin. Each plugin has its own map in the array of `inputs` consisting of simple properties. To use more advanced properties that consist of multiple values the property itself can be defined using an array, such as the `record` and `allowlist_key` properties for the `record_modifier` `filter`: +This pipeline consists of two `inputs`: a tail plugin and an HTTP server plugin. Each plugin has its own map in the array of `inputs` consisting of basic properties. To use more advanced properties that consist of multiple values the property itself can be defined using an array, such as the `record` and `allowlist_key` properties for the `record_modifier` `filter`: ```yaml pipeline: @@ -214,7 +214,7 @@ pipeline: match: 'my*cpu' ``` -#### Example: collecting CPU metrics +#### Collecting `cpu` metrics example The following configuration file example demonstrates how to collect CPU metrics and flush the results every five seconds to the standard output: @@ -235,7 +235,7 @@ pipeline: ## Processors -Fluent-Bit 2.1.2 and greater implements an interface called "processor" to extend the processing capabilities in input and output plugins directly without routing the data. The input and output plugins can run in separate threads. This interface allows users to apply data transformations and filtering to incoming data records before they're processed further in the pipeline. +Fluent Bit 2.1.2 and greater implements an interface called `processor` to extend the processing capabilities in input and output plugins directly without routing the data. The input and output plugins can run in separate threads. This interface allows users to apply data transformations and filtering to incoming data records before they're processed further in the pipeline. This capability is only exposed in YAML configuration and not in classic configuration mode due to the restriction of nested levels of configuration. diff --git a/administration/configuring-fluent-bit/yaml/environment-variables-section.md b/administration/configuring-fluent-bit/yaml/environment-variables-section.md index d1fa23715..b2f49e30d 100644 --- a/administration/configuring-fluent-bit/yaml/environment-variables-section.md +++ b/administration/configuring-fluent-bit/yaml/environment-variables-section.md @@ -2,7 +2,7 @@ The `env` section lets you define environment variables directly within the configuration file. These variables can then be used to dynamically replace values throughout your configuration using the `${VARIABLE_NAME}` syntax. -Variables set in this section cannot be overridden by system environment variables. +Variables set in this section can't be overridden by system environment variables. Values set in the `env` section are case-sensitive. However, as a best practice, Fluent Bit recommends using uppercase names for environment variables. The following example defines two variables, `FLUSH_INTERVAL` and `STDOUT_FMT`, which can be accessed in the configuration using `${FLUSH_INTERVAL}` and `${STDOUT_FMT}`: @@ -37,7 +37,7 @@ Fluent Bit provides a set of predefined environment variables that can be used i In addition to variables defined in the configuration file or the predefined ones, Fluent Bit can access system environment variables set in the user space. These external variables can be referenced in the configuration using the same `${VARIABLE_NAME}` pattern. -Variables set in the `env` section cannot be overridden by system environment variables. +Variables set in the `env` section can't be overridden by system environment variables. For example, to set the `FLUSH_INTERVAL` system environment variable to `2` and use it in your configuration: diff --git a/administration/configuring-fluent-bit/yaml/multiline-parsers-section.md b/administration/configuring-fluent-bit/yaml/multiline-parsers-section.md index ea0089347..570f3dc58 100644 --- a/administration/configuring-fluent-bit/yaml/multiline-parsers-section.md +++ b/administration/configuring-fluent-bit/yaml/multiline-parsers-section.md @@ -1,6 +1,6 @@ # Multiline parsers -Multiline parsers are used to combine logs that span multiple events into a single, cohesive message. This is particularly useful for handling stack traces, error logs, or any log entry that contains multiple lines of information. +Multiline parsers are used to combine logs that span multiple events into a single, cohesive message. Use this parser for handling stack traces, error logs, or any log entry that contains multiple lines of information. In YAML configuration, the syntax for defining multiline parsers differs slightly from the classic configuration format introducing minor breaking changes, specifically on how the rules are defined. diff --git a/administration/configuring-fluent-bit/yaml/pipeline-section.md b/administration/configuring-fluent-bit/yaml/pipeline-section.md index 7dcf5a656..21a0f7381 100644 --- a/administration/configuring-fluent-bit/yaml/pipeline-section.md +++ b/administration/configuring-fluent-bit/yaml/pipeline-section.md @@ -50,7 +50,7 @@ pipeline: Processors operate on specific signals such as logs, metrics, and traces. They're attached to an input plugin and must specify the signal type they will process. -### Example of a Processor +### Example of a processor In the following example, the `content_modifier` processor inserts or updates (upserts) the key `my_new_key` with the value `123` for all log records generated by the tail plugin. This processor is only applied to log signals: @@ -141,7 +141,7 @@ pipeline: Processors can be attached to inputs and outputs. -### How Processors are different from Filters +### How processors are different from filters While processors and filters are similar in that they can transform, enrich, or drop data from the pipeline, there is a significant difference in how they operate: @@ -149,11 +149,11 @@ While processors and filters are similar in that they can transform, enrich, or - Filters: Run in the main event loop. When multiple filters are used, they can introduce performance overhead, particularly under heavy workloads. -## Running Filters as Processors +## Running filters as processors You can configure existing [Filters](https://docs.fluentbit.io/manual/pipeline/filters) to run as processors. There are no specific changes needed; you use the filter name as if it were a native processor. -### Example of a Filter running as a Processor +### Example of a filter running as a processor In the following example, the `grep` filter is used as a processor to filter log events based on a pattern: diff --git a/administration/configuring-fluent-bit/yaml/plugins-section.md b/administration/configuring-fluent-bit/yaml/plugins-section.md index 9e2aca8f3..c446ada1d 100644 --- a/administration/configuring-fluent-bit/yaml/plugins-section.md +++ b/administration/configuring-fluent-bit/yaml/plugins-section.md @@ -1,6 +1,6 @@ # Plugins -Fluent Bit comes with a variety of built-in plugins, and also supports loading external plugins at runtime. This feature is especially useful for loading Go or WebAssembly (Wasm) plugins that are built as shared object files (.so). Fluent Bit YAML configuration provides the following ways to load these external plugins: +Fluent Bit comes with a variety of built-in plugins, and also supports loading external plugins at runtime. This feature is especially useful for loading Go or WebAssembly (Wasm) plugins that are built as shared object files (`.so`). Fluent Bit YAML configuration provides the following ways to load these external plugins: ## Inline YAML diff --git a/administration/configuring-fluent-bit/yaml/upstream-servers-section.md b/administration/configuring-fluent-bit/yaml/upstream-servers-section.md index 527b9fa0c..0301c3462 100644 --- a/administration/configuring-fluent-bit/yaml/upstream-servers-section.md +++ b/administration/configuring-fluent-bit/yaml/upstream-servers-section.md @@ -1,6 +1,6 @@ # Upstream servers -The `upstream_servers` section defines a group of endpoints, referred to as nodes. Nodes are used by output plugins to distribute data in a round-robin fashion. This is useful for plugins that require load balancing when sending data. Examples of plugins that support this capability include [Forward](https://docs.fluentbit.io/manual/pipeline/outputs/forward) and [Elasticsearch](https://docs.fluentbit.io/manual/pipeline/outputs/elasticsearch). +The `upstream_servers` section defines a group of endpoints, referred to as nodes. Nodes are used by output plugins to distribute data in a round-robin fashion. Use this section for plugins that require load balancing when sending data. Examples of plugins that support this capability include [Forward](https://docs.fluentbit.io/manual/pipeline/outputs/forward) and [Elasticsearch](https://docs.fluentbit.io/manual/pipeline/outputs/elasticsearch). The `upstream_servers` section require specifying a `name` for the group and a list of `nodes`. The following example defines two upstream server groups, `forward-balancing` and `forward-balancing-2`: diff --git a/administration/hot-reload.md b/administration/hot-reload.md index a6f83051d..77caf885c 100644 --- a/administration/hot-reload.md +++ b/administration/hot-reload.md @@ -54,18 +54,6 @@ For using curl to reload Fluent Bit, users must specify an empty request body as curl -X POST -d '{}' localhost:2020/api/v2/reload ``` -### Signal - -Hot reloading can be used with `SIGHUP`. - -`SIGHUP` signal isn't supported on Windows. - -## Confirm a reload - -Use one of the following methods to confirm the reload occurred. - -### HTTP - Obtain a count of hot reload using the HTTP endpoint: - `GET /api/v2/reload` @@ -77,3 +65,13 @@ The endpoint returns `hot_reload_count` as follows: ``` The default value of the counter is `0`. + +### Signal + +Hot reloading can be used with `SIGHUP`. + +`SIGHUP` signal isn't supported on Windows. + +## Confirm a reload + +Use one of the following methods to confirm the reload occurred. diff --git a/administration/memory-management.md b/administration/memory-management.md index 39951394e..b54e63783 100644 --- a/administration/memory-management.md +++ b/administration/memory-management.md @@ -16,7 +16,7 @@ When imposing a limit of `10MB` for the input plugins, and a worst case scenario In intensive environments where memory allocations happen in the orders of magnitude, the default memory allocator provided by Glibc could lead to high fragmentation, reporting a high memory usage by the service. -It's strongly suggested that in any production environment, Fluent Bit should be built with [jemalloc](http://jemalloc.net/) enabled (`-DFLB_JEMALLOC=On`). The jemalloc implementation of malloc is an alternative memory allocator that can reduce fragmentation, resulting in better performance. +It's strongly suggested that in any production environment, Fluent Bit should be built with [jemalloc](http://jemalloc.net/) enabled (`-DFLB_JEMALLOC=On`). The jemalloc implementation of `malloc` is an alternative memory allocator that can reduce fragmentation, resulting in better performance. Use the following command to determine if Fluent Bit has been built with jemalloc: @@ -33,4 +33,4 @@ FLB_HAVE_PROXY_GO FLB_HAVE_JEMALLOC JEMALLOC_MANGLE FLB_HAVE_REGEX FLB_HAVE_C_TLS FLB_HAVE_SETJMP FLB_HAVE_ACCEPT4 FLB_HAVE_INOTIFY ``` -If the `FLB_HAVE_JEMALLOC` option is listed in `Build Flags`, jemalloc is enabled. \ No newline at end of file +If the `FLB_HAVE_JEMALLOC` option is listed in `Build Flags`, jemalloc is enabled. diff --git a/administration/monitoring.md b/administration/monitoring.md index 999dec9b1..24051bf9b 100644 --- a/administration/monitoring.md +++ b/administration/monitoring.md @@ -61,7 +61,7 @@ pipeline: {% endtab %} {% endtabs %} -Start Fluent bit with the corresponding configuration chosen previously: +Start Fluent Bit with the corresponding configuration chosen previously: ```shell # For YAML configuration. @@ -122,7 +122,7 @@ Fluent Bit exposes the following endpoints for monitoring. | `/api/v1/health` | Display the Fluent Bit health check result. | String | | `/api/v2/metrics` | Display internal metrics per loaded plugin. | [cmetrics text format](https://github.com/fluent/cmetrics) | | `/api/v2/metrics/prometheus` | Display internal metrics per loaded plugin ready in Prometheus Server format. | Prometheus Text 0.0.4 | -| `/api/v2/reload | Execute hot reloading or get the status of hot reloading. See the [hot-reloading documentation](hot-reload.md). | JSON | +| `/api/v2/reload` | Execute hot reloading or get the status of hot reloading. See the [hot-reloading documentation](hot-reload.md). | JSON | ### v1 metrics @@ -137,7 +137,7 @@ The following terms are key to understanding how Fluent Bit processes metrics: - **Record**: a single message collected from a source, such as a single long line in a file. - **Chunk**: log records ingested and stored by Fluent Bit input plugin instances. A batch of records in a chunk are tracked together as a single unit. - The Fluent Bit engine attempts to fit records into chunks of at most `2 MB`, but the size can vary at runtime. Chunks are then sent to an output. An output plugin instance can either successfully send the full chunk to the destination and mark it as successful, or it can fail the chunk entirely if an unrecoverable error is encountered, or it can ask for the chunk to be retried. + The Fluent Bit engine attempts to fit records into chunks of at most `2 MB`, but the size can vary at runtime. Chunks are then sent to an output. An output plugin instance can successfully send the full chunk to the destination and mark it as successful, or it can fail the chunk entirely if an unrecoverable error is encountered, or it can ask for the chunk to be retried. | Metric name | Labels | Description | Type | Unit | | ----------- | ------ | ----------- | ---- | ---- | @@ -372,7 +372,7 @@ When querying the related metrics, the aliases are returned instead of the plugi -You can create Grafana dashboards and alerts using Fluent Bit's exposed Prometheus style metrics. +You can create Grafana dashboards and alerts using Fluent Bit exposed Prometheus style metrics. The provided [example dashboard](https://github.com/fluent/fluent-bit-docs/blob/master/monitoring/dashboard.json) is heavily inspired by [Banzai Cloud](https://github.com/banzaicloud)'s [logging operator dashboard](https://grafana.com/grafana/dashboards/7752) with a few key differences, such as the use of the `instance` label, stacked graphs, and a focus on Fluent Bit metrics. See [this blog post](https://www.robustperception.io/controlling-the-instance-label) for more information. @@ -384,7 +384,7 @@ Sample alerts [are available](https://github.com/fluent/fluent-bit-docs/blob/mas ## Health check for Fluent Bit -Fluent bit supports the following configurations to set up the health check. +Fluent Bit supports the following configurations to set up the health check. | Configuration name | Description | Default | | ---------------------- | ------------| ------------- | @@ -472,4 +472,6 @@ Health status = (HC_Errors_Count > 5) OR (HC_Retry_Failure_Count > 5) IN 5 secon ## Telemetry Pipeline -[Telemetry Pipeline](https://chronosphere.io/platform/telemetry-pipeline/) is a hosted service that lets you monitor your Fluent Bit agents including data flow, metrics, and configurations. \ No newline at end of file +[Telemetry Pipeline](https://chronosphere.io/platform/telemetry-pipeline/) is a +hosted service that lets you monitor your Fluent Bit agents including data flow, +metrics, and configurations. diff --git a/administration/multithreading.md b/administration/multithreading.md index d31d24775..2e68452de 100644 --- a/administration/multithreading.md +++ b/administration/multithreading.md @@ -37,6 +37,6 @@ Inputs aren't internally aware of multithreading. If an input runs in threaded m ## Outputs -When outputs flush data, they can either perform this operation inside Fluent Bit's main thread or inside a separate dedicated thread called a _worker_. Each output can have one or more workers running in parallel, and each worker can handle multiple concurrent flushes. You can configure this behavior by changing the value of the `workers` setting. +When outputs flush data, they can either perform this operation inside the main Fluent Bit thread or inside a separate dedicated thread called a _worker_. Each output can have one or more workers running in parallel, and each worker can handle multiple concurrent flushes. You can configure this behavior by changing the value of the `workers` setting. -All outputs are capable of running in multiple workers, and each output has a default value of `0`, `1`, or `2` workers. However, even if an output uses workers by default, you can safely reduce the number of workers below the default or disable workers entirely. +All outputs are capable of running in multiple workers, and each output has a default value of `0`, `1`, or `2` workers. However, even if an output uses workers by default, you can safely reduce the number of workers under the default or disable workers entirely. diff --git a/administration/performance.md b/administration/performance.md index 93e94f7af..90a22ddb5 100644 --- a/administration/performance.md +++ b/administration/performance.md @@ -2,9 +2,9 @@ Fluent Bit is designed for high performance and minimal resource usage. Depending on your use case, you can optimize further using specific configuration options to achieve faster performance or reduce resource consumption. -## Reading Files with Tail +## Reading files with `tail` -The `Tail` input plugin is used to read data from files on the filesystem. By default, it uses a small memory buffer of `32KB` per monitored file. While this is sufficient for most generic use cases and helps keep memory usage low when monitoring many files, there are scenarios where you may want to increase performance by using more memory. +The `Tail` input plugin is used to read data from files on the filesystem. By default, it uses a small memory buffer of `32KB` per monitored file. While this is sufficient for most generic use cases and helps keep memory usage low when monitoring many files, there are scenarios where you might want to increase performance by using more memory. If your files are typically larger than `32KB`, consider increasing the buffer size to speed up file reading. For example, you can experiment with a buffer size of `128KB`: @@ -19,17 +19,18 @@ pipeline: By increasing the buffer size, Fluent Bit will make fewer system calls (read(2)) to read the data, reducing CPU usage and improving performance. -## Fluent Bit and SIMD for JSON Encoding +## Fluent Bit and SIMD for JSON encoding -Starting in Fluent Bit v3.2, performance improvements have been introduced for JSON encoding. Plugins that convert logs from Fluent Bit's internal binary representation to JSON can now do so up to 30% faster using SIMD (Single Instruction, Multiple Data) optimizations. +Starting in Fluent Bit v3.2, performance improvements have been introduced for JSON encoding. Plugins that convert logs from the Fluent Bit internal binary representation +to JSON can now do so up to 30% faster using Single Instruction, Multiple Data (SIMD) optimizations. -### Enabling SIMD Support +### Enabling SIMD support Ensure that your Fluent Bit binary is built with SIMD support. This feature is available for architectures such as x86_64, amd64, aarch64, and arm64. As of now, SIMD is only enabled by default in Fluent Bit container images. You can check if SIMD is enabled by looking for the following log entry when Fluent Bit starts: -``` +```text [2024/11/10 22:25:53] [ info] [fluent bit] version=3.2.0, commit=12cb22e0e9, pid=74359 [2024/11/10 22:25:53] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128 [2024/11/10 22:25:53] [ info] [simd ] SSE2 @@ -38,15 +39,15 @@ You can check if SIMD is enabled by looking for the following log entry when Flu [2024/11/10 22:25:53] [ info] [sp] stream processor started ``` -Look for the simd entry, which will indicate the SIMD support type, such as SSE2, NEON, or none. +Look for the `simd` entry, which will indicate the SIMD support type, such as `SSE2`, `NEON`, or `none`. -If your Fluent Bit binary was not built with SIMD enabled and you are using a supported platform, you can build Fluent Bit from source using the CMake option `-DFLB_SIMD=On`. +If your Fluent Bit binary wasn't built with SIMD enabled and you are using a supported platform, you can build Fluent Bit from source using the CMake option `-DFLB_SIMD=On`. ## Run input plugins in threaded mode By default, most of input plugins runs in the same system thread than the main event loop, however by configuration you can instruct them to run in a separate thread which will allow you to take advantage of other CPU cores in your system. -To run an input plugin in threaded mode, just add `threaded: true` as in the example below: +To run an input plugin in threaded mode, add `threaded: true` as in the following example: ```yaml pipeline: diff --git a/administration/transport-security.md b/administration/transport-security.md index ca45f7e8f..f0c096d2e 100644 --- a/administration/transport-security.md +++ b/administration/transport-security.md @@ -22,7 +22,7 @@ To use TLS on input plugins, you must provide both a certificate and a private k The listed properties can be enabled in the configuration file, specifically in each output plugin section or directly through the command line. -The following **output** plugins can take advantage of the TLS feature: +The following output plugins can take advantage of the TLS feature: - [Amazon S3](../pipeline/outputs/s3.md) - [Apache SkyWalking](../pipeline/outputs/skywalking.md) @@ -51,11 +51,11 @@ The following **output** plugins can take advantage of the TLS feature: - [Splunk](../pipeline/outputs/splunk.md) - [Stackdriver](../pipeline/outputs/stackdriver.md) - [Syslog](../pipeline/outputs/syslog.md) -- [TCO and TLS](../pipeline/outputs/tcp-and-tls.md) +- [TCP and TLS](../pipeline/outputs/tcp-and-tls.md) - [Treasure Data](../pipeline/outputs/treasure-data.md) - [WebSocket](../pipeline/outputs/websocket.md) -The following **input** plugins can take advantage of the TLS feature: +The following input plugins can take advantage of the TLS feature: - [Docker Events](../pipeline/inputs/docker-events.md) - [Elasticsearch (Bulk API)](../pipeline/inputs/elasticsearch.md) @@ -93,7 +93,7 @@ fluent-bit -i http \ {% hint style="info" %} -See the Tips and Tricks section below for details on generating `self_signed.crt` and `self_signed.key` files shown in these examples. +See the Tips and Tricks section for details on generating `self_signed.crt` and `self_signed.key` files shown in these examples. {% endhint %} @@ -194,7 +194,7 @@ pipeline: {% endtab %} {% endtabs %} -## Tips and Tricks +## Tips and tricks ### Generate a self-signed certificates for testing purposes @@ -317,4 +317,4 @@ This outgoing connect will fail and disconnect: [2024/06/17 16:51:31] [error] [tls] error: unexpected EOF with reason: certificate verify failed [2024/06/17 16:51:31] [debug] [upstream] connection #50 failed to other.fluent-aggregator.net:24224 [2024/06/17 16:51:31] [error] [output:forward:forward.0] no upstream connections available -``` \ No newline at end of file +``` diff --git a/administration/troubleshooting.md b/administration/troubleshooting.md index 83272805f..c09c6f340 100644 --- a/administration/troubleshooting.md +++ b/administration/troubleshooting.md @@ -125,7 +125,7 @@ This response means Tap is active. The terminal with Fluent Bit running should n All the records that display are those emitted by the activities of the dummy plugin. -### Complex Tap example +### Tap example (complex) This example takes the same steps but demonstrates how the mechanism works with more complicated configurations. @@ -379,4 +379,4 @@ Fluent Bit relies on a custom storage layer interface designed for hybrid buffer | `mem chunks` | | Total number of Chunks memory-based. | | `fs chunks` | | Total number of Chunks filesystem based. | | | `up` | Total number of filesystem chunks up in memory. | -| | `down` | Total number of filesystem chunks down (not loaded in memory). | \ No newline at end of file +| | `down` | Total number of filesystem chunks down (not loaded in memory). | diff --git a/vale-styles/FluentBit/Acronyms.yml b/vale-styles/FluentBit/Acronyms.yml index 0d24b68b7..84fcf1954 100644 --- a/vale-styles/FluentBit/Acronyms.yml +++ b/vale-styles/FluentBit/Acronyms.yml @@ -16,6 +16,7 @@ exceptions: - ASC - ASP - AWS + - BSD - CIDR - CLI - CORS @@ -41,6 +42,7 @@ exceptions: - GCP - GCS - GDB + - GELF - GET - GNU - GPG @@ -78,6 +80,7 @@ exceptions: - NOTE - NVDA - OAP + - OOM - OSS - PATH - PEM @@ -99,6 +102,7 @@ exceptions: - SHA - SIEM - SLA + - SNI - SQL - SSH - SSL diff --git a/vale-styles/FluentBit/Headings.yml b/vale-styles/FluentBit/Headings.yml index a6ebec4ea..8951e4f31 100644 --- a/vale-styles/FluentBit/Headings.yml +++ b/vale-styles/FluentBit/Headings.yml @@ -44,6 +44,9 @@ exceptions: - Docker - Docker Hub - DogStatsD + - EC2 + - ECS + - EKS - Elastic Cloud - Emmet - EventBridge @@ -68,7 +71,7 @@ exceptions: - JSON - Kafka - Kafka Producer - - Kafka REST + - Kafka - Kinesis - KQL - Kubernetes @@ -86,6 +89,7 @@ exceptions: - NATS - New Relic - Observability Platform + - OIDC - Okta - OpenMetrics - OpenSearch @@ -94,13 +98,16 @@ exceptions: - Opsgenie - Oracle Cloud Infrastructure Logging Analytics - PagerDuty + - Pod Identity - PostgreSQL - Prometheus - PromQL - Raspberry Pi - REPL + - REST - ServiceMonitor - SignalFx + - SIMD - Slack - SSL - StatsD @@ -111,6 +118,7 @@ exceptions: - Telemetry Pipeline - Terraform - TLS + - Transport Layer Security - Treasure Data - TypeScript - URLs @@ -120,3 +128,4 @@ exceptions: - VS - WebSocket - Windows + - YAML