You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
| multiline.parser | Specify one or multiple [Multiline Parser definitions](../../administration/configuring-fluent-bit/multiline-parsing.md) to apply to the content. You can specify multiple multiline parsers to detect different formats by separating them with a comma. |
45
45
| multiline.key_content | Key name that holds the content to process. Note that a Multiline Parser definition can already specify the `key_content` to use, but this option allows to overwrite that value for the purpose of the filter. |
46
+
| mode | Mode can be `parser` for regex concat, or `partial_message` to concat split docker logs. |
46
47
| buffer | Enable buffered mode. In buffered mode, the filter can concatenate multilines from inputs that ingest records one by one (ex: Forward), rather than in chunks, re-emitting them into the beggining of the pipeline (with the same tag) using the in_emitter instance. With buffer off, this filter will not work with most inputs, except tail. |
47
48
| flush_ms | Flush time for pending multiline records. Defaults to 2000. |
48
49
| emitter_name | Name for the emitter input instance which re-emits the completed records at the beginning of the pipeline. |
@@ -253,3 +254,24 @@ created by runtime.gcenable
253
254
```
254
255
255
256
The lines that did not match a pattern are not considered as part of the multiline message, while the ones that matched the rules were concatenated properly.
257
+
258
+
259
+
## Docker Partial Message Use Case
260
+
261
+
When Fluent Bit is consuming logs from a container runtime, such as docker, these logs will be split above a certain limit, usually 32KB. If your application emits a 500K log line, it will be split into 16 partial messages. If you are using the [Fluentd Docker Log Driver](https://docs.docker.com/config/containers/logging/fluentd/) to send the logs to Fluent Bit, they might look like this:
Fluent Bit can re-combine these logs that were split by the runtime and remove the partial message fields. The filter example below is for this use case.
268
+
269
+
```
270
+
[FILTER]
271
+
name multiline
272
+
match *
273
+
multiline.key_content log
274
+
mode partial_message
275
+
```
276
+
277
+
The two options for `mode` are mutually exclusive in the filter. If you set the `mode` to `partial_message` then the `multiline.parser` option is not allowed.
0 commit comments