Skip to content

Commit d3208e5

Browse files
committed
Adding YAML examples for multiline stacktrace filter doc. Part of issue #1872.
Signed-off-by: Eric D. Schabell <[email protected]>
1 parent 574522f commit d3208e5

File tree

1 file changed

+88
-8
lines changed

1 file changed

+88
-8
lines changed

pipeline/filters/multiline-stacktrace.md

Lines changed: 88 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ When using this filter:
2020
- To concatenate messages read from a log file, it's highly recommended to use the multiline support in the [Tail plugin](https://docs.fluentbit.io/manual/pipeline/inputs/tail#multiline-support) itself. This is because performing concatenation while reading the log file is more performant. Concatenating messages originally split by Docker or CRI container engines, is supported in the [Tail plugin](https://docs.fluentbit.io/manual/pipeline/inputs/tail#multiline-support).
2121

2222
{% hint style="warning" %}
23+
2324
This filter only performs buffering that persists across different Chunks when `Buffer` is enabled. Otherwise, the filter processes one chunk at a time and isn't suitable for most inputs which might send multiline messages in separate chunks.
2425

2526
When buffering is enabled, the filter doesn't immediately emit messages it receives. It uses the `in_emitter` plugin, similar to the [Rewrite Tag filter](pipeline/filters/rewrite-tag.md), and emits messages once they're fully concatenated, or a timeout is reached.
@@ -58,10 +59,40 @@ The following example files can be located [in the Fluent Bit repository](https:
5859
Example files content:
5960

6061
{% tabs %}
62+
{% tab title="fluent-bit.yaml" %}
63+
64+
This is the primary Fluent Bit YAML configuration file. It includes the `parsers_multiline.yaml` and tails the file `test.log` by applying the multiline parsers `multiline-regex-test` and `go`. Then it sends the processing to the standard output.
65+
66+
67+
```yaml
68+
service:
69+
flush: 1
70+
log_level: info
71+
parsers_file: parsers_multiline.yaml
72+
73+
pipeline:
74+
inputs:
75+
- name: tail
76+
path: test.log
77+
read_from_head: true
78+
79+
filters:
80+
- name: multiline
81+
match: '*'
82+
multiline.key_content: log
83+
multiline.parser: go,multiline-regex-test
84+
85+
outputs:
86+
- name: stdout
87+
match: '*'
88+
```
89+
90+
{% endtab %}
6191
{% tab title="fluent-bit.conf" %}
62-
This is the primary Fluent Bit configuration file. It includes the `parsers_multiline.conf` and tails the file `test.log` by applying the multiline parsers `multiline-regex-test` and `go`. Then it sends the processing to the standard output.
6392
64-
```python
93+
This is the primary Fluent Bit classic configuration file. It includes the `parsers_multiline.conf` and tails the file `test.log` by applying the multiline parsers `multiline-regex-test` and `go`. Then it sends the processing to the standard output.
94+
95+
```text
6596
[SERVICE]
6697
flush 1
6798
log_level info
@@ -85,11 +116,41 @@ This is the primary Fluent Bit configuration file. It includes the `parsers_mult
85116
```
86117

87118
{% endtab %}
119+
{% tab title="parsers_multiline.yaml" %}
120+
121+
This file defines a multiline parser for the example. A second multiline parser called `go` is used in `fluent-bit.yaml`, but this one is a built-in parser.
122+
123+
```yaml
124+
multiline_parsers:
125+
- name: multiline-regex-test
126+
type: regex
127+
flush_timeout: 1000
128+
#
129+
# Regex rules for multiline parsing
130+
# ---------------------------------
131+
#
132+
# configuration hints:
133+
#
134+
# - first state always has the name: start_state
135+
# - every field in the rule must be inside double quotes
136+
#
137+
# rules | state name | regex pattern | next state
138+
# ------|---------------|--------------------------------------------
139+
rules:
140+
- state: start_state
141+
regex: '/([a-zA-Z]+ \d+ \d+\:\d+\:\d+)(.*)/'
142+
next_state: cont
143+
- state: cont
144+
regex: '/^\s+at.*/'
145+
next_state: cont
146+
```
88147

148+
{% endtab %}
89149
{% tab title="parsers_multiline.conf" %}
90-
This second file defines a multiline parser for the example. A second multiline parser called `go` is used in `fluent-bit.conf`, but this one is a built-in parser.
91150

92-
```python
151+
This file defines a multiline parser for the example. A second multiline parser called `go` is used in `fluent-bit.conf`, but this one is a built-in parser.
152+
153+
```text
93154
[MULTILINE_PARSER]
94155
name multiline-regex-test
95156
type regex
@@ -111,8 +172,8 @@ This second file defines a multiline parser for the example. A second multiline
111172
```
112173

113174
{% endtab %}
114-
115175
{% tab title="test.log" %}
176+
116177
An example file with multiline and multi-format content:
117178

118179
```text
@@ -185,7 +246,7 @@ one more line, no multiline
185246
Running Fluent Bit with the given configuration file:
186247

187248
```shell
188-
fluent-bit -c fluent-bit.conf
249+
$ ./fluent-bit -c fluent-bit.conf
189250
```
190251

191252
Should return something like the following:
@@ -266,12 +327,31 @@ When Fluent Bit is consuming logs from a container runtime, such as Docker, thes
266327

267328
Fluent Bit can re-combine these logs that were split by the runtime and remove the partial message fields. The following filter example is for this use case.
268329

269-
```python
330+
{% tabs %}
331+
{% tab title="fluent-bit.yaml" %}
332+
333+
```yaml
334+
pipeline:
335+
336+
filters:
337+
- name: multiline
338+
match: '*'
339+
multiline.key_content: log
340+
mode: partial_message
341+
```
342+
343+
{% endtab %}
344+
{% tab title="fluent-bit.conf" %}
345+
346+
```text
270347
[FILTER]
271348
name multiline
272349
match *
273350
multiline.key_content log
274351
mode partial_message
275352
```
276353

277-
The two options for `mode` are mutually exclusive in the filter. If you set the `mode` to `partial_message` then the `multiline.parser` option isn't allowed.
354+
{% endtab %}
355+
{% endtabs %}
356+
357+
The two options for `mode` are mutually exclusive in the filter. If you set the `mode` to `partial_message` then the `multiline.parser` option isn't allowed.

0 commit comments

Comments
 (0)