Skip to content

Commit de5fefd

Browse files
author
Martin Fleurke
committed
Clarify parsing multiline messages (#10212)
make more clear the distinction between docker/cri splitting and merging, and multiline application log (stacktraces) merging
1 parent 8353f70 commit de5fefd

File tree

3 files changed

+11
-10
lines changed

3 files changed

+11
-10
lines changed

administration/configuring-fluent-bit/multiline-parsing.md

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -19,13 +19,13 @@ The Multiline parser engine exposes two ways to configure and use the feature:
1919
Fluent Bit exposes certain pre-configured parsers (built-in) to solve specific
2020
multiline parser cases. For example:
2121

22-
| Parser | Description |
23-
| ------ | ----------- |
24-
| `docker` | Process a log entry generated by a Docker container engine. This parser supports the concatenation of log entries split by Docker. |
25-
| `cri` | Process a log entry generated by CRI-O container engine. Like the `docker` parser, it supports concatenation of log entries |
26-
| `go` | Process log entries generated by a Go based language application and perform concatenation if multiline messages are detected. |
27-
| `python` | Process log entries generated by a Python based language application and perform concatenation if multiline messages are detected. |
28-
| `java` | Process log entries generated by a Google Cloud Java language application and perform concatenation if multiline messages are detected. |
22+
| Parser | Description |
23+
|----------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
24+
| `docker` | Process a log entry generated by a Docker container engine. This This parser supports the concatenation of large log entries split by Docker. If you use this parser, and you also want to concatenate loglines like stacktraces, you can add the [multiline filter](../../pipeline/filters/multiline-stacktrace.md) to specify additional parsers |
25+
| `cri` | Process a log entry generated by CRI-O container engine. Like the `docker` parser, it supports concatenation of log entries |
26+
| `go` | Process log entries generated by a Go based language application and perform concatenation if multiline messages are detected. |
27+
| `python` | Process log entries generated by a Python based language application and perform concatenation if multiline messages are detected. |
28+
| `java` | Process log entries generated by a Google Cloud Java language application and perform concatenation if multiline messages are detected. |
2929

3030
### Configurable multiline parsers
3131

pipeline/filters/multiline-stacktrace.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ Along with multiline filters, you can enable one of the following built-in Fluen
1717
When using this filter:
1818

1919
- The usage of this filter depends on a previous configuration of a [multiline parser](../../administration/configuring-fluent-bit/multiline-parsing.md) definition.
20-
- To concatenate messages read from a log file, it's highly recommended to use the multiline support in the [Tail plugin](https://docs.fluentbit.io/manual/pipeline/inputs/tail#multiline-support) itself. This is because performing concatenation while reading the log file is more performant. Concatenating messages originally split by Docker or CRI container engines, is supported in the [Tail plugin](https://docs.fluentbit.io/manual/pipeline/inputs/tail#multiline-support).
20+
- To concatenate messages read from a log file, it's highly recommended to use the multiline support in the [Tail plugin](https://docs.fluentbit.io/manual/pipeline/inputs/tail#multiline-support) itself. This is because performing concatenation while reading the log file is more performant. Concatenating messages that were originally one line, but split by Docker or CRI container engines because of their size, is supported in the [Tail plugin](https://docs.fluentbit.io/manual/pipeline/inputs/tail#multiline-support) icw the `docker` or `cri` parser. to concatenate application logs like stacktraces on top of that, you can use this multiline filter.
2121

2222
{% hint style="warning" %}
2323

@@ -319,7 +319,8 @@ Lines that don't match a pattern aren't considered as part of the multiline mess
319319

320320
## Docker partial message use case
321321

322-
When Fluent Bit is consuming logs from a container runtime, such as Docker, these logs will be split when larger than a certain limit, usually 16KB. If your application emits a 100K log line, it will be split into seven partial messages. If you are using the [Fluentd Docker Log Driver](https://docs.docker.com/config/containers/logging/fluentd/) to send the logs to Fluent Bit, they might look like this:
322+
When Fluent Bit is consuming logs from a container runtime, such as Docker, these logs will be split when larger than a certain limit, usually 16KB.
323+
If your application emits a 100K log line, it will be split into seven partial messages. The docker parser will merge these back to one line. If instead you are using the [Fluentd Docker Log Driver](https://docs.docker.com/config/containers/logging/fluentd/) to send the logs to Fluent Bit, they might look like this:
323324

324325
```text
325326
{"source": "stdout", "log": "... omitted for brevity...", "partial_message": "true", "partial_id": "dc37eb08b4242c41757d4cd995d983d1cdda4589193755a22fcf47a638317da0", "partial_ordinal": "1", "partial_last": "false", "container_id": "a96998303938eab6087a7f8487ca40350f2c252559bc6047569a0b11b936f0f2", "container_name": "/hopeful_taussig"}]

pipeline/inputs/tail.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -150,7 +150,7 @@ The new multiline core is exposed by the following configuration:
150150

151151
### Multiline and containers
152152

153-
If you are running Fluent Bit to process logs coming from containers like Docker or CRI, you can use the built-in modes. This helps reassemble multiline messages originally split by Docker or CRI:
153+
If you are running Fluent Bit to process logs coming from containers like Docker or CRI, you can use the built-in modes. This helps reassemble large messages originally split by Docker or CRI:
154154

155155
{% tabs %}
156156
{% tab title="fluent-bit.yaml" %}

0 commit comments

Comments
 (0)