Skip to content

Commit d75b9e4

Browse files
committed
filter_multiline: add partial_message docs
Signed-off-by: Wesley Pettit <[email protected]>
1 parent 5dfb071 commit d75b9e4

File tree

1 file changed

+22
-0
lines changed

1 file changed

+22
-0
lines changed

pipeline/filters/multiline-stacktrace.md

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,7 @@ The plugin supports the following configuration parameters:
4343
| --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
4444
| multiline.parser | Specify one or multiple [Multiline Parser definitions](../../administration/configuring-fluent-bit/multiline-parsing.md) to apply to the content. You can specify multiple multiline parsers to detect different formats by separating them with a comma. |
4545
| multiline.key_content | Key name that holds the content to process. Note that a Multiline Parser definition can already specify the `key_content` to use, but this option allows to overwrite that value for the purpose of the filter. |
46+
| mode | Mode can be `parser` for regex concat, or `partial_message` to concat split docker logs. |
4647
| buffer | Enable buffered mode. In buffered mode, the filter can concatenate multilines from inputs that ingest records one by one (ex: Forward), rather than in chunks, re-emitting them into the beggining of the pipeline (with the same tag) using the in_emitter instance. With buffer off, this filter will not work with most inputs, except tail. |
4748
| flush_ms | Flush time for pending multiline records. Defaults to 2000. |
4849
| emitter_name | Name for the emitter input instance which re-emits the completed records at the beginning of the pipeline. |
@@ -253,3 +254,24 @@ created by runtime.gcenable
253254
```
254255

255256
The lines that did not match a pattern are not considered as part of the multiline message, while the ones that matched the rules were concatenated properly.
257+
258+
259+
## Docker Partial Message Use Case
260+
261+
When Fluent Bit is consuming logs from a container runtime, such as docker, these logs will be split above a certain limit, usually 32KB. If your application emits a 500K log line, it will be split into 16 partial messages. If you are using the [Fluentd Docker Log Driver](https://docs.docker.com/config/containers/logging/fluentd/) to send the logs to Fluent Bit, they might look like this:
262+
263+
```
264+
{"source": "stdout", "log": "... omitted for brevity...", "partial_message": "true", "partial_id": "dc37eb08b4242c41757d4cd995d983d1cdda4589193755a22fcf47a638317da0", "partial_ordinal": "1", "partial_last": "false", "container_id": "a96998303938eab6087a7f8487ca40350f2c252559bc6047569a0b11b936f0f2", "container_name": "/hopeful_taussig"}]
265+
```
266+
267+
Fluent Bit can re-combine these logs that were split by the runtime and remove the partial message fields. The filter example below is for this use case.
268+
269+
```
270+
[FILTER]
271+
name multiline
272+
match *
273+
multiline.key_content log
274+
mode partial_message
275+
```
276+
277+
The two options for `mode` are mutually exclusive in the filter. If you set the `mode` to `partial_message` then the `multiline.parser` option is not allowed.

0 commit comments

Comments
 (0)