You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: pipeline/filters/multiline-stacktrace.md
+88-8Lines changed: 88 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -20,6 +20,7 @@ When using this filter:
20
20
- To concatenate messages read from a log file, it's highly recommended to use the multiline support in the [Tail plugin](https://docs.fluentbit.io/manual/pipeline/inputs/tail#multiline-support) itself. This is because performing concatenation while reading the log file is more performant. Concatenating messages originally split by Docker or CRI container engines, is supported in the [Tail plugin](https://docs.fluentbit.io/manual/pipeline/inputs/tail#multiline-support).
21
21
22
22
{% hint style="warning" %}
23
+
23
24
This filter only performs buffering that persists across different Chunks when `Buffer` is enabled. Otherwise, the filter processes one chunk at a time and isn't suitable for most inputs which might send multiline messages in separate chunks.
24
25
25
26
When buffering is enabled, the filter doesn't immediately emit messages it receives. It uses the `in_emitter` plugin, similar to the [Rewrite Tag filter](pipeline/filters/rewrite-tag.md), and emits messages once they're fully concatenated, or a timeout is reached.
@@ -58,10 +59,40 @@ The following example files can be located [in the Fluent Bit repository](https:
58
59
Example files content:
59
60
60
61
{% tabs %}
62
+
{% tab title="fluent-bit.yaml" %}
63
+
64
+
This is the primary Fluent Bit YAML configuration file. It includes the `parsers_multiline.yaml` and tails the file `test.log` by applying the multiline parsers `multiline-regex-test` and `go`. Then it sends the processing to the standard output.
65
+
66
+
67
+
```yaml
68
+
service:
69
+
flush: 1
70
+
log_level: info
71
+
parsers_file: parsers_multiline.yaml
72
+
73
+
pipeline:
74
+
inputs:
75
+
- name: tail
76
+
path: test.log
77
+
read_from_head: true
78
+
79
+
filters:
80
+
- name: multiline
81
+
match: '*'
82
+
multiline.key_content: log
83
+
multiline.parser: go,multiline-regex-test
84
+
85
+
outputs:
86
+
- name: stdout
87
+
match: '*'
88
+
```
89
+
90
+
{% endtab %}
61
91
{% tab title="fluent-bit.conf" %}
62
-
This is the primary Fluent Bit configuration file. It includes the `parsers_multiline.conf` and tails the file `test.log` by applying the multiline parsers `multiline-regex-test` and `go`. Then it sends the processing to the standard output.
63
92
64
-
```python
93
+
This is the primary Fluent Bit classic configuration file. It includes the `parsers_multiline.conf` and tails the file `test.log` by applying the multiline parsers `multiline-regex-test` and `go`. Then it sends the processing to the standard output.
94
+
95
+
```text
65
96
[SERVICE]
66
97
flush 1
67
98
log_level info
@@ -85,11 +116,41 @@ This is the primary Fluent Bit configuration file. It includes the `parsers_mult
85
116
```
86
117
87
118
{% endtab %}
119
+
{% tab title="parsers_multiline.yaml" %}
120
+
121
+
This file defines a multiline parser for the example. A second multiline parser called `go` is used in `fluent-bit.yaml`, but this one is a built-in parser.
122
+
123
+
```yaml
124
+
multiline_parsers:
125
+
- name: multiline-regex-test
126
+
type: regex
127
+
flush_timeout: 1000
128
+
#
129
+
# Regex rules for multiline parsing
130
+
# ---------------------------------
131
+
#
132
+
# configuration hints:
133
+
#
134
+
# - first state always has the name: start_state
135
+
# - every field in the rule must be inside double quotes
This second file defines a multiline parser for the example. A second multiline parser called `go` is used in `fluent-bit.conf`, but this one is a built-in parser.
91
150
92
-
```python
151
+
This file defines a multiline parser for the example. A second multiline parser called `go` is used in `fluent-bit.conf`, but this one is a built-in parser.
152
+
153
+
```text
93
154
[MULTILINE_PARSER]
94
155
name multiline-regex-test
95
156
type regex
@@ -111,8 +172,8 @@ This second file defines a multiline parser for the example. A second multiline
111
172
```
112
173
113
174
{% endtab %}
114
-
115
175
{% tab title="test.log" %}
176
+
116
177
An example file with multiline and multi-format content:
117
178
118
179
```text
@@ -185,7 +246,7 @@ one more line, no multiline
185
246
Running Fluent Bit with the given configuration file:
186
247
187
248
```shell
188
-
fluent-bit -c fluent-bit.conf
249
+
$ ./fluent-bit -c fluent-bit.conf
189
250
```
190
251
191
252
Should return something like the following:
@@ -266,12 +327,31 @@ When Fluent Bit is consuming logs from a container runtime, such as Docker, thes
266
327
267
328
Fluent Bit can re-combine these logs that were split by the runtime and remove the partial message fields. The following filter example is for this use case.
268
329
269
-
```python
330
+
{% tabs %}
331
+
{% tab title="fluent-bit.yaml" %}
332
+
333
+
```yaml
334
+
pipeline:
335
+
336
+
filters:
337
+
- name: multiline
338
+
match: '*'
339
+
multiline.key_content: log
340
+
mode: partial_message
341
+
```
342
+
343
+
{% endtab %}
344
+
{% tab title="fluent-bit.conf" %}
345
+
346
+
```text
270
347
[FILTER]
271
348
name multiline
272
349
match *
273
350
multiline.key_content log
274
351
mode partial_message
275
352
```
276
353
277
-
The two options for `mode` are mutually exclusive in the filter. If you set the `mode` to `partial_message` then the `multiline.parser` option isn't allowed.
354
+
{% endtab %}
355
+
{% endtabs %}
356
+
357
+
The two options for `mode` are mutually exclusive in the filter. If you set the `mode` to `partial_message` then the `multiline.parser` option isn't allowed.
0 commit comments