Skip to content

Clarify parsing multiline messages (#10212) #1927

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Aug 15, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions administration/configuring-fluent-bit/multiline-parsing.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,8 @@

| Parser | Description |
| ------ | ----------- |
| `docker` | Process a log entry generated by a Docker container engine. This parser supports the concatenation of log entries split by Docker. |
| `cri` | Process a log entry generated by CRI-O container engine. Like the `docker` parser, it supports concatenation of log entries. |
| `docker` | Process a log entry generated by a Docker container engine. This parser supports the concatenation of large log entries split by Docker. If you use this parser, and you also want to concatenate loglines like stacktraces, you can add the [multiline filter](../../pipeline/filters/multiline-stacktrace.md) to specify additional parsers. |

Check warning on line 20 in administration/configuring-fluent-bit/multiline-parsing.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [FluentBit.Spelling] Spelling check: 'stacktraces'? Raw Output: {"message": "[FluentBit.Spelling] Spelling check: 'stacktraces'?", "location": {"path": "administration/configuring-fluent-bit/multiline-parsing.md", "range": {"start": {"line": 20, "column": 222}}}, "severity": "INFO"}

Check warning on line 20 in administration/configuring-fluent-bit/multiline-parsing.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [FluentBit.Spelling] Spelling check: 'loglines'? Raw Output: {"message": "[FluentBit.Spelling] Spelling check: 'loglines'?", "location": {"path": "administration/configuring-fluent-bit/multiline-parsing.md", "range": {"start": {"line": 20, "column": 208}}}, "severity": "INFO"}
| `cri` | Process a log entry generated by CRI-O container engine. Like the `docker` parser, it supports concatenation of log entries. |
| `go` | Process log entries generated by a Go based language application and perform concatenation if multiline messages are detected. |
| `python` | Process log entries generated by a Python based language application and perform concatenation if multiline messages are detected. |
| `java` | Process log entries generated by a Google Cloud Java language application and perform concatenation if multiline messages are detected. |
Expand Down
5 changes: 3 additions & 2 deletions pipeline/filters/multiline-stacktrace.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
When using this filter:

- The usage of this filter depends on a previous configuration of a [multiline parser](../../administration/configuring-fluent-bit/multiline-parsing.md) definition.
- To concatenate messages read from a log file, it's highly recommended to use the multiline support in the [Tail plugin](https://docs.fluentbit.io/manual/pipeline/inputs/tail#multiline-support) itself. This is because performing concatenation while reading the log file is more performant. Concatenating messages originally split by Docker or CRI container engines, is supported in the [Tail plugin](https://docs.fluentbit.io/manual/pipeline/inputs/tail#multiline-support).
- To concatenate messages read from a log file, it's highly recommended to use the multiline support in the [Tail plugin](https://docs.fluentbit.io/manual/pipeline/inputs/tail#multiline-support) itself. This is because performing concatenation while reading the log file is more performant. Concatenating messages that were originally one line, but split by Docker or CRI container engines because of their size, is supported in the [Tail plugin](https://docs.fluentbit.io/manual/pipeline/inputs/tail#multiline-support) in combination with the `docker` or `cri` parser. To concatenate application logs like stacktraces on top of that, you can use this multiline filter.

Check warning on line 19 in pipeline/filters/multiline-stacktrace.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [FluentBit.Spelling] Spelling check: 'stacktraces'? Raw Output: {"message": "[FluentBit.Spelling] Spelling check: 'stacktraces'?", "location": {"path": "pipeline/filters/multiline-stacktrace.md", "range": {"start": {"line": 19, "column": 608}}}, "severity": "INFO"}

{% hint style="warning" %}

Expand Down Expand Up @@ -314,7 +314,8 @@

## Docker partial message use case

When Fluent Bit is consuming logs from a container runtime, such as Docker, these logs will be split when larger than a certain limit, usually 16&nbspKB. If your application emits a 100K log line, it will be split into seven partial messages. If you are using the [Fluentd Docker Log Driver](https://docs.docker.com/config/containers/logging/fluentd/) to send the logs to Fluent Bit, they might look like this:
When Fluent Bit is consuming logs from a container runtime, such as Docker, these logs will be split when larger than a certain limit, usually 16&nbspKB.
If your application emits a 100K log line, it will be split into seven partial messages. The docker parser will merge these back to one line. If instead you are using the [Fluentd Docker Log Driver](https://docs.docker.com/config/containers/logging/fluentd/) to send the logs to Fluent Bit, they might look like this:

```text
{"source": "stdout", "log": "... omitted for brevity...", "partial_message": "true", "partial_id": "dc37eb08b4242c41757d4cd995d983d1cdda4589193755a22fcf47a638317da0", "partial_ordinal": "1", "partial_last": "false", "container_id": "a96998303938eab6087a7f8487ca40350f2c252559bc6047569a0b11b936f0f2", "container_name": "/hopeful_taussig"}]
Expand Down
2 changes: 1 addition & 1 deletion pipeline/inputs/tail.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,7 @@ Multiline core is exposed by the following configuration:

### Multiline and containers

If you are running Fluent Bit to process logs coming from containers like Docker or CRI, you can use the built-in modes. This helps reassemble multiline messages originally split by Docker or CRI:
If you are running Fluent Bit to process logs coming from containers like Docker or CRI, you can use the built-in modes. This helps reassemble large messages originally split by Docker or CRI:

{% tabs %}
{% tab title="fluent-bit.yaml" %}
Expand Down