Merge branch 'fluent:master' into master

Author: Xata

.gitbook.yaml

@@ -82,3 +82,6 @@ redirects:
 
     # Processors
     processor/metrics_selector:   ./pipeline/processors/metrics_selector.md
+
+    # Other
+    concepts/buffering: ./pipeline/buffering.md

SUMMARY.md

@@ -13,7 +13,6 @@
 ## Concepts
 
 * [Key concepts](concepts/key-concepts.md)
-* [Buffering](concepts/buffering.md)
 * [Data pipeline](concepts/data-pipeline.md)
 
 ## Installation
@@ -26,16 +25,16 @@
     * [Build with static configuration](installation/downloads/source/build-with-static-configuration.md)
   * [Linux packages](installation/downloads/linux.md)
     * [Amazon Linux](installation/downloads/linux/amazon-linux.md)
-    * [Rocky Linux and Alma Linux ](installation/downloads/linux/alma-rocky.md)
-    * [Red Hat and CentOS](installation/downloads/linux/redhat-centos.md)
+    * [Buildroot embedded Linux](installation/downloads/linux/buildroot-embedded-linux.md)
     * [Debian](installation/downloads/linux/debian.md)
-    * [Ubuntu](installation/downloads/linux/ubuntu.md)
     * [Raspbian and Raspberry Pi](installation/downloads/linux/raspbian-raspberry-pi.md)
+    * [Red Hat and CentOS](installation/downloads/linux/redhat-centos.md)
+    * [Rocky Linux and Alma Linux ](installation/downloads/linux/alma-rocky.md)
+    * [Ubuntu](installation/downloads/linux/ubuntu.md)
     * [Yocto embedded Linux](installation/downloads/linux/yocto-embedded-linux.md)
-    * [Buildroot embedded Linux](installation/downloads/linux/buildroot-embedded-linux.md)
-  * [Docker](installation/downloads/docker.md)
-  * [Containers on AWS](installation/downloads/aws-container.md)
   * [Amazon EC2](installation/downloads/amazon-ec2.md)
+  * [Containers on AWS](installation/downloads/aws-container.md)
+  * [Docker](installation/downloads/docker.md)
   * [Kubernetes](installation/downloads/kubernetes.md)
   * [macOS](installation/downloads/macos.md)
   * [Windows](installation/downloads/windows.md)
@@ -45,36 +44,37 @@
 
 * [Configure Fluent Bit](administration/configuring-fluent-bit.md)
   * [YAML configuration](administration/configuring-fluent-bit/yaml.md)
+    * [Configuration file](administration/configuring-fluent-bit/yaml/configuration-file.md)
+    * [Environment variables](administration/configuring-fluent-bit/yaml/environment-variables-section.md)
+    * [Includes](administration/configuring-fluent-bit/yaml/includes-section.md) 
     * [Service](administration/configuring-fluent-bit/yaml/service-section.md)
     * [Parsers](administration/configuring-fluent-bit/yaml/parsers-section.md)
     * [Multiline parsers](administration/configuring-fluent-bit/yaml/multiline-parsers-section.md)
     * [Pipeline](administration/configuring-fluent-bit/yaml/pipeline-section.md)
     * [Plugins](administration/configuring-fluent-bit/yaml/plugins-section.md)
     * [Upstream servers](administration/configuring-fluent-bit/yaml/upstream-servers-section.md)
-    * [Environment variables](administration/configuring-fluent-bit/yaml/environment-variables-section.md)
-    * [Includes](administration/configuring-fluent-bit/yaml/includes-section.md)
   * [Classic mode](administration/configuring-fluent-bit/classic-mode.md)
-    * [Format and schema](administration/configuring-fluent-bit/classic-mode/format-schema.md)
-    * [Configuration file](administration/configuring-fluent-bit/classic-mode/configuration-file.md)
-    * [Variables](administration/configuring-fluent-bit/classic-mode/variables.md)
+    * [Configuration file](administration/configuring-fluent-bit/classic-mode/configuration-file.md) 
     * [Commands](administration/configuring-fluent-bit/classic-mode/commands.md)
+    * [Format and schema](administration/configuring-fluent-bit/classic-mode/format-schema.md)
+    * [Record accessor syntax](administration/configuring-fluent-bit/classic-mode/record-accessor.md) 
     * [Upstream servers](administration/configuring-fluent-bit/classic-mode/upstream-servers.md)
-    * [Record accessor syntax](administration/configuring-fluent-bit/classic-mode/record-accessor.md)
-  * [Unit Sizes](administration/configuring-fluent-bit/unit-sizes.md)
+    * [Variables](administration/configuring-fluent-bit/classic-mode/variables.md)
   * [Multiline parsing](administration/configuring-fluent-bit/multiline-parsing.md)
-* [TLS](administration/transport-security.md)
-* [Buffering and storage](administration/buffering-and-storage.md)
+  * [Unit Sizes](administration/configuring-fluent-bit/unit-sizes.md)
+* [AWS credentials](administration/aws-credentials.md) 
 * [Backpressure](administration/backpressure.md)
-* [Scheduling and retries](administration/scheduling-and-retries.md)
-* [Networking](administration/networking.md)
+* [Buffering and storage](administration/buffering-and-storage.md)
+* [Hot reload](administration/hot-reload.md)
+* [HTTP proxy](administration/http-proxy.md)
 * [Memory management](administration/memory-management.md)
 * [Monitoring](administration/monitoring.md)
 * [Multithreading](administration/multithreading.md)
-* [HTTP proxy](administration/http-proxy.md)
-* [Hot reload](administration/hot-reload.md)
-* [Troubleshooting](administration/troubleshooting.md)
+* [Networking](administration/networking.md)
 * [Performance tips](administration/performance.md)
-* [AWS credentials](administration/aws-credentials.md)
+* [Scheduling and retries](administration/scheduling-and-retries.md)
+* [TLS](administration/transport-security.md) 
+* [Troubleshooting](administration/troubleshooting.md)
 
 ## Local testing
 
@@ -83,7 +83,6 @@
 
 ## Data pipeline
 
-* [Pipeline monitoring](pipeline/pipeline-monitoring.md)
 * [Inputs](pipeline/inputs.md)
   * [Collectd](pipeline/inputs/collectd.md)
   * [CPU metrics](pipeline/inputs/cpu-metrics.md)
@@ -93,8 +92,8 @@
   * [Dummy](pipeline/inputs/dummy.md)
   * [eBPF](pipeline/inputs/ebpf.md)
   * [Elasticsearch](pipeline/inputs/elasticsearch.md)
-  * [Exec WASI](pipeline/inputs/exec-wasi.md)
   * [Exec](pipeline/inputs/exec.md)
+  * [Exec WASI](pipeline/inputs/exec-wasi.md)
   * [Fluent Bit metrics](pipeline/inputs/fluentbit-metrics.md)
   * [Forward](pipeline/inputs/forward.md)
   * [Head](pipeline/inputs/head.md)
@@ -126,25 +125,25 @@
   * [TCP](pipeline/inputs/tcp.md)
   * [Thermal](pipeline/inputs/thermal.md)
   * [UDP](pipeline/inputs/udp.md)
-  * [Windows Event logs (winevtlog)](pipeline/inputs/windows-event-log-winevtlog.md)
   * [Windows Event logs (winlog)](pipeline/inputs/windows-event-log.md)
+  * [Windows Event logs (winevtlog)](pipeline/inputs/windows-event-log-winevtlog.md)
   * [Windows exporter metrics](pipeline/inputs/windows-exporter-metrics.md)
 * [Parsers](pipeline/parsers.md)
   * [Configuring parsers](pipeline/parsers/configuring-parser.md)
+  * [Decoders](pipeline/parsers/decoders.md)
   * [JSON](pipeline/parsers/json.md)
-  * [Regular expression](pipeline/parsers/regular-expression.md)
-  * [LTSV](pipeline/parsers/ltsv.md)
   * [Logfmt](pipeline/parsers/logfmt.md)
-  * [Decoders](pipeline/parsers/decoders.md)
+  * [LTSV](pipeline/parsers/ltsv.md)
+  * [Regular expression](pipeline/parsers/regular-expression.md)
 * [Processors](pipeline/processors.md)
+  * [Conditional processing](pipeline/processors/conditional-processing.md)
   * [Content modifier](pipeline/processors/content-modifier.md)
+  * [Filters as processors](pipeline/processors/filters.md)
   * [Labels](pipeline/processors/labels.md)
   * [Metrics selector](pipeline/processors/metrics-selector.md)
   * [OpenTelemetry envelope](pipeline/processors/opentelemetry-envelope.md)
   * [Sampling](pipeline/processors/sampling.md)
   * [SQL](pipeline/processors/sql.md)
-  * [Filters as processors](pipeline/processors/filters.md)
-  * [Conditional processing](pipeline/processors/conditional-processing.md)
 * [Filters](pipeline/filters.md)
   * [AWS metadata](pipeline/filters/aws-metadata.md)
   * [CheckList](pipeline/filters/checklist.md)
@@ -168,22 +167,24 @@
   * [Throttle](pipeline/filters/throttle.md)
   * [Type converter](pipeline/filters/type-converter.md)
   * [Wasm](pipeline/filters/wasm.md)
-* [Router](pipeline/router.md)
+* [Buffering](pipeline/buffering.md)
+* [Routing](pipeline/router.md)
 * [Outputs](pipeline/outputs.md)
+  * [Azure Log Analytics](pipeline/outputs/azure.md)
+  * [Azure Blob](pipeline/outputs/azure_blob.md)
+  * [Azure Data Explorer](pipeline/outputs/azure_kusto.md)
+  * [Azure Logs Ingestion API](pipeline/outputs/azure_logs_ingestion.md)
   * [Amazon CloudWatch](pipeline/outputs/cloudwatch.md)
   * [Amazon Kinesis Data Firehose](pipeline/outputs/firehose.md)
   * [Amazon Kinesis Data Streams](pipeline/outputs/kinesis.md)
   * [Amazon S3](pipeline/outputs/s3.md)
   * [Apache SkyWalking](pipeline/outputs/skywalking.md)
-  * [Azure Blob](pipeline/outputs/azure_blob.md)
-  * [Azure Data Explorer](pipeline/outputs/azure_kusto.md)
-  * [Azure Log Analytics](pipeline/outputs/azure.md)
-  * [Azure Logs Ingestion API](pipeline/outputs/azure_logs_ingestion.md)
   * [Counter](pipeline/outputs/counter.md)
   * [Dash0](pipeline/outputs/dash0.md)
   * [Datadog](pipeline/outputs/datadog.md)
   * [Dynatrace](pipeline/outputs/dynatrace.md)
   * [Elasticsearch](pipeline/outputs/elasticsearch.md)
+  * [Exit](pipeline/outputs/exit.md)
   * [File](pipeline/outputs/file.md)
   * [Flow counter](pipeline/outputs/flowcounter.md)
   * [Forward](pipeline/outputs/forward.md)
@@ -200,10 +201,12 @@
   * [New Relic](pipeline/outputs/new-relic.md)
   * [Null](pipeline/outputs/null.md)
   * [Observe](pipeline/outputs/observe.md)
+  * [Oracle Cloud Infrastructure Logging Analytics](pipeline/outputs/oci-logging-analytics.md)
   * [OpenObserve](pipeline/outputs/openobserve.md)
   * [OpenSearch](pipeline/outputs/opensearch.md)
   * [OpenTelemetry](pipeline/outputs/opentelemetry.md)
-  * [Oracle Cloud Infrastructure Logging Analytics](pipeline/outputs/oci-logging-analytics.md)
+  * [Output Formats](pipeline/outputs/output_formats.md)
+  * [Parseable](pipeline/outputs/parseable.md)
   * [PostgreSQL](pipeline/outputs/postgresql.md)
   * [Prometheus exporter](pipeline/outputs/prometheus-exporter.md)
   * [Prometheus remote write](pipeline/outputs/prometheus-remote-write.md)
@@ -216,6 +219,7 @@
   * [Treasure Data](pipeline/outputs/treasure-data.md)
   * [Vivo Exporter](pipeline/outputs/vivo-exporter.md)
   * [WebSocket](pipeline/outputs/websocket.md)
+* [Pipeline monitoring](pipeline/pipeline-monitoring.md)
 
 ## Stream processing
 
@@ -232,6 +236,7 @@
 * [C library API](development/library_api.md)
 * [Ingest records manually](development/ingest-records-manually.md)
 * [Golang output plugins](development/golang-output-plugins.md)
+* [MessagePack format](development/msgpack-format.md)
 * [Wasm filter plugins](development/wasm-filter-plugins.md)
 * [Wasm input plugins](development/wasm-input-plugins.md)
 * [Contribution guide for beginners](development/developer-guide.md)

administration/backpressure.md

@@ -6,13 +6,13 @@ It's possible for logs or data to be ingested or created faster than the ability
 
 To avoid backpressure, Fluent Bit implements a mechanism in the engine that restricts the amount of data an input plugin can ingest. Restriction is done through the configuration parameters `Mem_Buf_Limit` and `storage.Max_Chunks_Up`.
 
-As described in the [Buffering](../concepts/buffering.md) concepts section, Fluent Bit offers two modes for data handling: in-memory only (default) and in-memory and filesystem (optional).
+As described in [Buffering and storage](../administration/buffering-and-storage.md) , Fluent Bit offers two modes for data handling: in-memory only (default) and in-memory and filesystem (optional).
 
 The default `storage.type memory` buffer can be restricted with `Mem_Buf_Limit`. If memory reaches this limit and you reach a backpressure scenario, you won't be able to ingest more data until the data chunks that are in memory can be flushed. The input pauses and Fluent Bit [emits](https://github.com/fluent/fluent-bit/blob/v2.0.0/src/flb_input_chunk.c#L1334) a `[warn] [input] {input name or alias} paused (mem buf overlimit)` log message.
 
 Depending on the input plugin in use, this might cause incoming data to be discarded (for example, TCP input plugin). The tail plugin can handle pauses without data loss, storing its current file offset and resuming reading later. When buffer memory is available, the input resumes accepting logs. Fluent Bit [emits](https://github.com/fluent/fluent-bit/blob/v2.0.0/src/flb_input_chunk.c#L1277) a `[info] [input] {input name or alias} resume (mem buf overlimit)` message.
 
-Mitigate the risk of data loss by configuring secondary storage on the filesystem using the `storage.type` of `filesystem` (as described in [Buffering and Storage](buffering-and-storage.md)). Initially, logs will be buffered to both memory and the filesystem. When the `storage.max_chunks_up` limit is reached, all new data will be stored in the filesystem. Fluent Bit stops queueing new data in memory and buffers only to the filesystem. When `storage.type filesystem` is set, the `Mem_Buf_Limit` setting no longer has any effect. Instead, the `[SERVICE]` level `storage.max_chunks_up` setting controls the size of the memory buffer.
+Mitigate the risk of data loss by configuring secondary storage on the filesystem using the `storage.type` of `filesystem` (as described in [Buffering and storage](../administration/buffering-and-storage.md)). Initially, logs will be buffered to both memory and the filesystem. When the `storage.max_chunks_up` limit is reached, all new data will be stored in the filesystem. Fluent Bit stops queueing new data in memory and buffers only to the filesystem. When `storage.type filesystem` is set, the `Mem_Buf_Limit` setting no longer has any effect. Instead, the `[SERVICE]` level `storage.max_chunks_up` setting controls the size of the memory buffer.
 
 ## `Mem_Buf_Limit`
 
@@ -70,4 +70,4 @@ With `storage.type filesystem` and `storage.max_chunks_up`, the following log me
 ```text
 [input] {input name or alias} paused (storage buf overlimit)
 [input] {input name or alias} resume (storage buf overlimit)
-```
+```

administration/configuring-fluent-bit.md

@@ -3,7 +3,7 @@
 Fluent Bit supports two configuration formats:
 
 - [YAML](./configuring-fluent-bit/yaml.md): Standard configuration format as of v3.2.
-- [Classic mode](./configuring-fluent-bit/classic-mode/README.md): To be deprecated at the end of 2026.
+- [Classic mode](./configuring-fluent-bit/classic-mode.md): To be deprecated at the end of 2026.
 
 ## Command line interface
 

administration/configuring-fluent-bit/classic-mode/README.md

@@ -36,6 +36,7 @@ The `Service` section defines global properties of the service. The following ke
 | `scheduler.base`  | Set a base of exponential backoff. Supported in v1.8.7 and greater. | `5` |
 | `json.convert_nan_to_null` | If enabled, `NaN` converts to `null` when Fluent Bit converts `msgpack` to `json`. | `false` |
 | `sp.convert_from_str_to_num` | If enabled, Stream processor converts from number string to number type. | `true` |
+| `windows.maxstdio` | If specified, the limit of stdio is adjusted. Only provided for Windows. From 512 to 2048 is allowed. | `512` |
 
 The following is an example of a `SERVICE` section:
 
@@ -156,4 +157,4 @@ Wildcard character (`*`) supports including multiple files. For example:
 @INCLUDE input_*.conf
 ```
 
-Files matching the wildcard character are included unsorted. If plugin ordering between files needs to be preserved, the files should be included explicitly.
+Files matching the wildcard character are included unsorted. If plugin ordering between files needs to be preserved, the files should be included explicitly.

administration/configuring-fluent-bit/classic-mode/configuration-file.md

@@ -81,7 +81,8 @@ The `service` section defines the global properties of the service. The Service
 | `scheduler.cap`   | Set a maximum retry time in seconds. Supported from v1.8.7. | `2000` |
 | `scheduler.base`  | Sets the base of exponential backoff. Supported from v1.8.7. | `5` |
 | `json.convert_nan_to_null` | If enabled, NaN is converted to null when Fluent Bit converts `msgpack` to JSON. | `false` |
-| `sp.convert_from_str_to_num` | If enabled, Stream processor converts from number string to number type. | `true |
+| `sp.convert_from_str_to_num` | If enabled, Stream processor converts from number string to number type. | `true` |
+| `windows.maxstdio` | If specified, the limit of stdio is adjusted. Only provided for Windows. From 512 to 2048 is allowed. | `512` |
 
 The following is an example of a `service` section:
 

administration/configuring-fluent-bit/yaml/configuration-file.md

@@ -2,26 +2,39 @@
 
 The Fluent Bit data pipeline incorporates several specific concepts. Data processing flows through the pipeline following these concepts in order.
 
-## Filters
+```mermaid
+graph LR
+    accTitle: Fluent Bit data pipeline
+    accDescr: A diagram of the Fluent Bit data pipeline, which includes input, a parser, a filter, a buffer, routing, and various outputs.
+    A[Input] --> B[Parser]
+    B --> C[Filter]
+    C --> D[Buffer]
+    D --> E((Routing))
+    E --> F[Output 1]
+    E --> G[Output 2]
+    E --> H[Output 3]
+```
 
-[Filters](../pipeline/filters.md) let you alter the collected data before delivering it to a destination. In production environments you need full control of the data you're collecting. Using filters lets you control data before processing.
+## Inputs
 
-## Buffer
+[Input plugins](../pipeline/inputs.md) gather information from different sources. Some plugins collect data from log files, and others gather metrics information from the operating system. There are many plugins to suit different needs.
 
-The [`buffer`](./buffering.md) phase in the pipeline aims to provide a unified and persistent mechanism to store your data, using the primary in-memory model or the file system-based mode.
+## Parser
 
-## Inputs
-
-Fluent Bit provides [input plugins](../pipeline/inputs.md) to gather information from different sources. Some plugins collect data from log files, and others gather metrics information from the operating system. There are many plugins to suit different needs.
+[Parsers](../pipeline/parsers.md) convert unstructured data to structured data. Use a parser to set a structure to the incoming data by using input plugins as data is collected.
 
-## Outputs
+## Filter
 
-[Output plugins](../pipeline/outputs.md) let you define destinations for your data. Common destinations are remote services, local file systems, or other standard interfaces.
+[Filters](../pipeline/filters.md) let you alter the collected data before delivering it to a destination. In production environments you need full control of the data you're collecting. Using filters lets you control data before processing.
 
-## Parsers
+## Buffer
 
-[Parsers](../pipeline/parsers.md) convert unstructured data to structured data. Use a parser to set a structure to the incoming data by using input plugins as data is collected.
+The [buffering](./buffering.md) phase in the pipeline aims to provide a unified and persistent mechanism to store your data, using the primary in-memory model or the file system-based mode.
 
-## Route
+## Routing
 
 [Routing](../pipeline/router.md) is a core feature that lets you route your data through filters, and then to one or multiple destinations. The router relies on the concept of [tags](./key-concepts.md#tag) and [matching](./key-concepts.md#match) rules.
+
+## Output
+
+[Output plugins](../pipeline/outputs.md) let you define destinations for your data. Common destinations are remote services, local file systems, or other standard interfaces.

concepts/data-pipeline.md

@@ -42,6 +42,8 @@ The following table describes the Linux container tags that are available on Doc
 
 | Tags       | Manifest Architectures    | Description                                                    |
 | ------------ | ------------------------- | -------------------------------------------------------------- |
+| 4.0.13-debug | x86_64, arm64v8, arm32v7, s390x | Debug images |
+| 4.0.13 | x86_64, arm64v8, arm32v7, s390x | Release [v4.0.13](https://fluentbit.io/announcements/v4.0.13/) |
 | 4.0.12-debug | x86_64, arm64v8, arm32v7, s390x | Debug images |
 | 4.0.12 | x86_64, arm64v8, arm32v7, s390x | Release [v4.0.12](https://fluentbit.io/announcements/v4.0.12/) |
 | 4.0.11-debug | x86_64, arm64v8, arm32v7, s390x | Debug images |