Skip to content

Commit 32929ff

Browse files
committed
filter_multiline: add docs for new buffered mode
Signed-off-by: Wesley Pettit <[email protected]>
1 parent b03dd71 commit 32929ff

File tree

1 file changed

+10
-3
lines changed

1 file changed

+10
-3
lines changed

pipeline/filters/multiline-stacktrace.md

Lines changed: 10 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -18,12 +18,13 @@ As part of the built-in functionality, without major configuration effort, you c
1818
Some comments about this filter:
1919

2020
* The usage of this filter depends on a previous configuration of a [Multiline Parser](../../administration/configuring-fluent-bit/multiline-parsing.md) definition.
21-
* If you aim to concatenate messages split originally by Docker or CRI container engines, we recommend doing the concatenation on [Tail plugin](https://docs.fluentbit.io/manual/pipeline/inputs/tail#multiline-support), this same functionality exists there.
21+
* If you wish to concatenate messages read from a log file, it is highly recommended to use the multiline support in the [Tail plugin](https://docs.fluentbit.io/manual/pipeline/inputs/tail#multiline-support) itself. This is because performing concatenation while reading the log file is more performant. Concatenating messages originally split by Docker or CRI container engines, is supported in the [Tail plugin](https://docs.fluentbit.io/manual/pipeline/inputs/tail#multiline-support).
2222

2323
{% hint style="warning" %}
24-
This filter does not perform buffering that persists across different Chunks. This filter **process one Chunk at a time** and is not suitable for sources that might send multiline messages in separated chunks.
24+
This filter only performs buffering that persists across different Chunks when `Buffer` is enabled. Otherwise, the filter will *process one Chunk at a time* and is not suitable for most inputs which might send multiline messages in separate chunks.
25+
26+
When buffering is enabled, the filter does not immediately emit messages it receives. It uses the in_emitter plugin, same as the [Rewrite Tag Filter](pipeline/filters/rewrite-tag.md), and emits messages once they are fully concatenated, or a timeout is reached.
2527

26-
For cases where Multiline mode is required and the source plugin does not support it, please file a Github Enhancement with such requirement and specific details of the use case.
2728
{% endhint %}
2829

2930
## Configuration Parameters
@@ -34,6 +35,12 @@ The plugin supports the following configuration parameters:
3435
| --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
3536
| multiline.parser | Specify one or multiple [Multiline Parser definitions](../../administration/configuring-fluent-bit/multiline-parsing.md) to apply to the content. You can specify multiple multiline parsers to detect different formats by separating them with a comma. |
3637
| multiline.key_content | Key name that holds the content to process. Note that a Multiline Parser definition can already specify the `key_content` to use, but this option allows to overwrite that value for the purpose of the filter. |
38+
| buffer | Enable buffered mode. In buffered mode, the filter can concatenate multilines from inputs that ingest records one by one (ex: Forward), rather than in chunks, re-emitting them into the beggining of the pipeline (with the same tag) using the in_emitter instance. With buffer off, this filter will not work with most inputs, except tail. |
39+
| flush_ms | Flush time for pending multiline records. Defaults to 2000. |
40+
| emitter_name | Name for the emitter input instance which re-emits the completed records at the beginning of the pipeline. |
41+
| emitter_storage.type | The storage type for the emitter input instance. This option supports the values `memory` \(default\) and `filesystem`. |
42+
| emitter\_mem\_buf\_limit | Set a limit on the amount of memory the emitter can consume if the outputs provide backpressure. The default for this limit is `10M`. The pipeline will pause once the buffer exceeds the value of this setting. For example, if the value is set to `10M` then the pipeline will pause if the buffer exceeds `10M`. The pipeline will remain paused until the output drains the buffer below the `10M` limit. |
43+
3744

3845
## Configuration Example
3946

0 commit comments

Comments
 (0)