Skip to content

Commit 6c6d6e0

Browse files
eschabelleschabell
authored
Added YAML examples for admin configuring Fluent Bit multiline parsing docs. Fixes #1757. (#1758)
Signed-off-by: Eric D. Schabell <[email protected]>
1 parent 0254189 commit 6c6d6e0

File tree

1 file changed

+191
-32
lines changed

1 file changed

+191
-32
lines changed

‎administration/configuring-fluent-bit/multiline-parsing.md

Lines changed: 191 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,6 @@ reality applications generate multiple log messages that sometimes belong to the
55
context. Processing this information can be complex, like in application stack traces,
66
which always have multiple log lines.
77

8-
<img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=e19a4c14-a9e4-4163-8f3a-52196eb9a585" />
9-
108
Fluent Bit v1.8 implemented a unified Multiline core capability to solve corner cases.
119

1210
## Concepts
@@ -72,7 +70,27 @@ A rule specifies how to match a multiline pattern and perform the concatenation.
7270
- regular expression pattern
7371
- next state
7472

75-
A rule might be defined as follows (comments added to simplify the definition):
73+
A rule might be defined as follows (comments added to simplify the definition) in corresponding YAML and classic
74+
configuration examples below:
75+
76+
{% tabs %}
77+
{% tab title="parsers_multiline.yaml" %}
78+
79+
```yaml
80+
# rules | state name | regex pattern | next state
81+
# ------|---------------|--------------------------------------------
82+
rules:
83+
- state: start_state
84+
regex: '/([a-zA-Z]+ \d+ \d+\:\d+\:\d+)(.*)/'
85+
next_state: cont
86+
- state: cont
87+
regex: '/^\s+at.*/'
88+
next_state: cont
89+
```
90+
91+
{% endtab %}
92+
93+
{% tab title="parsers_multiline.conf" %}
7694
7795
```text
7896
# rules | state name | regex pattern | next state
@@ -81,6 +99,9 @@ rule "start_state" "/([a-zA-Z]+ \d+ \d+\:\d+\:\d+)(.*)/" "cont"
8199
rule "cont" "/^\s+at.*/" "cont"
82100
```
83101

102+
{% endtab %}
103+
{% endtabs %}
104+
84105
This example defines two rules. Each rule has its own state name, regex patterns, and the next state name. Every field that composes a rule must be inside double quotes.
85106

86107
The first rule of a state name must be `start_state`. The regex pattern must match the first line of a multiline message, and a next state must be set to specify what the possible continuation lines look like.
@@ -91,19 +112,42 @@ To simplify the configuration of regular expressions, you can use the [Rubular](
91112

92113
#### Configuration example
93114

94-
The following example provides a full Fluent Bit configuration file for multiline parsing by using the definition explained previously.
115+
The following example provides a full Fluent Bit configuration file for multiline parsing by using the definition
116+
explained previously. It is provided in corresponding YAML and classic configuration examples below:
95117

96-
{% hint style="info" %}
97-
The following example files can be located [at this link](https://github.com/fluent/fluent-bit/tree/master/documentation/examples/multiline/regex-001).
98-
{% endhint %}
118+
{% tabs %}
99119

100-
Example files content:
120+
{% tab title="fluent-bit.yaml" %}
121+
122+
This is the primary Fluent Bit YAML configuration file. It includes the `parsers_multiline.yaml` and tails the file `test.log`
123+
by applying the multiline parser `multiline-regex-test`. Then it sends the processing to the standard output.
124+
125+
```yaml
126+
service:
127+
flush: 1
128+
log_level: info
129+
parsers_file: parsers_multiline.yaml
130+
131+
pipeline:
132+
inputs:
133+
- name: tail
134+
path: test.log
135+
read_from_head: true
136+
multiline.parser: multiline-regex-test
137+
138+
outputs:
139+
- name: stdout
140+
match: '*'
141+
```
142+
143+
{% endtab %}
101144
102-
{% tabs %}
103145
{% tab title="fluent-bit.conf" %}
104-
This is the primary Fluent Bit configuration file. It includes the `parsers_multiline.conf` and tails the file `test.log` by applying the multiline parser `multiline-regex-test`. Then it sends the processing to the standard output.
105146
106-
```python
147+
This is the primary Fluent Bit classic configuration file. It includes the `parsers_multiline.conf` and tails the file `test.log`
148+
by applying the multiline parser `multiline-regex-test`. Then it sends the processing to the standard output.
149+
150+
```text
107151
[SERVICE]
108152
flush 1
109153
log_level info
@@ -120,12 +164,44 @@ This is the primary Fluent Bit configuration file. It includes the `parsers_mult
120164
match *
121165
```
122166

167+
{% endtab %}
168+
169+
{% tab title="parsers_multiline.yaml" %}
170+
171+
This file defines a multiline parser for the YAML configuration example.
172+
173+
```yaml
174+
multiline_parsers:
175+
- name: multiline-regex-test
176+
type: regex
177+
flush_timeout: 1000
178+
#
179+
# Regex rules for multiline parsing
180+
# ---------------------------------
181+
#
182+
# configuration hints:
183+
#
184+
# - first state always has the name: start_state
185+
# - every field in the rule must be inside double quotes
186+
#
187+
# rules | state name | regex pattern | next state
188+
# ------|---------------|--------------------------------------------
189+
rules:
190+
- state: start_state
191+
regex: '/([a-zA-Z]+ \d+ \d+\:\d+\:\d+)(.*)/'
192+
next_state: cont
193+
- state: cont
194+
regex: '/^\s+at.*/'
195+
```
196+
197+
123198
{% endtab %}
124199

125200
{% tab title="parsers_multiline.conf" %}
126-
This second file defines a multiline parser for the example.
127201

128-
```python
202+
This second file defines a multiline parser for the classic configuration example.
203+
204+
```text
129205
[MULTILINE_PARSER]
130206
name multiline-regex-test
131207
type regex
@@ -148,7 +224,8 @@ This second file defines a multiline parser for the example.
148224
{% endtab %}
149225

150226
{% tab title="test.log" %}
151-
An example file with multiline content:
227+
228+
The example log file with multiline content:
152229

153230
```text
154231
single line...
@@ -165,23 +242,27 @@ another line...
165242
{% endtab %}
166243
{% endtabs %}
167244

168-
By running Fluent Bit with the given configuration file you will obtain:
245+
By running Fluent Bit with the corresponding configuration file you will obtain the following output:
169246

170247
```text
171-
$ fluent-bit -c fluent-bit.conf
248+
# For YAML configuration.
249+
$ ./fluent-bit --config fluent-bit.yaml
250+
251+
# For classic configuration.
252+
$ ./fluent-bit --config fluent-bit.conf
172253
173-
[0] tail.0: [0.000000000, {"log"=>"single line...
254+
...
255+
[0] tail.0: [[1750332967.679671000, {}], {"log"=>"single line...
174256
"}]
175-
[1] tail.0: [1626634867.472226330, {"log"=>"Dec 14 06:41:08 Exception in thread "main" java.lang.RuntimeException: Something has gone wrong, aborting!
257+
[1] tail.0: [[1750332967.679677000, {}], {"log"=>"Dec 14 06:41:08 Exception in thread "main" java.lang.RuntimeException: Something has gone wrong, aborting!
176258
at com.myproject.module.MyProject.badMethod(MyProject.java:22)
177259
at com.myproject.module.MyProject.oneMoreMethod(MyProject.java:18)
178260
at com.myproject.module.MyProject.anotherMethod(MyProject.java:14)
179261
at com.myproject.module.MyProject.someMethod(MyProject.java:10)
180262
at com.myproject.module.MyProject.main(MyProject.java:6)
181263
"}]
182-
[2] tail.0: [1626634867.472226330, {"log"=>"another line...
264+
[2] tail.0: [[1750332967.679677000, {}], {"log"=>"another line...
183265
"}]
184-
185266
```
186267

187268
The lines that didn't match a pattern aren't considered as part of the multiline message, while the ones that matched the rules were concatenated properly.
@@ -201,11 +282,47 @@ The following example retrieves `date` and `message` from concatenated logs.
201282

202283
Example files content:
203284

204-
{% tabs %}
285+
% tabs %}
286+
287+
{% tab title="fluent-bit.yaml" %}
288+
289+
This is the primary Fluent Bit YAML configuration file. It includes the `parsers_multiline.conf` and tails the file `test.log`
290+
by applying the multiline parser `multiline-regex-test`. It also parses concatenated log by applying parser `named-capture-test`.
291+
Then it sends the processing to the standard output.
292+
293+
```yaml
294+
service:
295+
flush: 1
296+
log_level: info
297+
parsers_file: parsers_multiline.yaml
298+
299+
pipeline:
300+
inputs:
301+
- name: tail
302+
path: test.log
303+
read_from_head: true
304+
multiline.parser: multiline-regex-test
305+
306+
filters:
307+
- name: parser
308+
match: '*'
309+
key_name: log
310+
parser: named-capture-test
311+
312+
outputs:
313+
- name: stdout
314+
match: '*'
315+
```
316+
317+
{% endtab %}
318+
205319
{% tab title="fluent-bit.conf" %}
206-
This is the primary Fluent Bit configuration file. It includes the `parsers_multiline.conf` and tails the file `test.log` by applying the multiline parser `multiline-regex-test`. It also parses concatenated log by applying parser `named-capture-test`. Then it sends the processing to the standard output.
207320

208-
```python
321+
This is the primary Fluent Bit classic configuration file. It includes the `parsers_multiline.conf` and tails the file
322+
`test.log` by applying the multiline parser `multiline-regex-test`. It also parses concatenated log by applying parser
323+
`named-capture-test`. Then it sends the processing to the standard output.
324+
325+
```text
209326
[SERVICE]
210327
flush 1
211328
log_level info
@@ -230,10 +347,47 @@ This is the primary Fluent Bit configuration file. It includes the `parsers_mult
230347

231348
{% endtab %}
232349

350+
{% tab title="parsers_multiline.yaml" %}
351+
352+
This file defines a multiline parser for the YAML example.
353+
354+
```yaml
355+
multiline_parsers:
356+
- name: multiline-regex-test
357+
type: regex
358+
flush_timeout: 1000
359+
#
360+
# Regex rules for multiline parsing
361+
# ---------------------------------
362+
#
363+
# configuration hints:
364+
#
365+
# - first state always has the name: start_state
366+
# - every field in the rule must be inside double quotes
367+
#
368+
# rules | state name | regex pattern | next state
369+
# ------|---------------|--------------------------------------------
370+
rules:
371+
- state: start_state
372+
regex: '/([a-zA-Z]+ \d+ \d+\:\d+\:\d+)(.*)/'
373+
next_state: cont
374+
- state: cont
375+
regex: '/^\s+at.*/'
376+
next_state: cont
377+
378+
parsers:
379+
- name: named-capture-test
380+
format: regex
381+
regex: '/^(?<date>[a-zA-Z]+ \d+ \d+\:\d+\:\d+) (?<message>.*)/m'
382+
```
383+
384+
{% endtab %}
385+
233386
{% tab title="parsers_multiline.conf" %}
234-
This second file defines a multiline parser for the example.
235387

236-
```python
388+
This file defines a multiline parser for the classic example.
389+
390+
```text
237391
[MULTILINE_PARSER]
238392
name multiline-regex-test
239393
type regex
@@ -261,7 +415,8 @@ This second file defines a multiline parser for the example.
261415
{% endtab %}
262416

263417
{% tab title="test.log" %}
264-
An example file with multiline content:
418+
419+
The example log file with multiline content:
265420

266421
```text
267422
single line...
@@ -278,20 +433,24 @@ another line...
278433
{% endtab %}
279434
{% endtabs %}
280435

281-
By running Fluent Bit with the given configuration file you will obtain:
436+
By running Fluent Bit with the corresponding configuration file you will obtain:
282437

283438
```text
284-
$ fluent-bit -c fluent-bit.conf
439+
# For YAML configuration.
440+
$ ./fluent-bit --config fluent-bit.yaml
441+
442+
# For classic configuration
443+
$ ./fluent-bit --config fluent-bit.conf
285444
286-
[0] tail.0: [1669160706.737650473, {"log"=>"single line...
445+
[0] tail.0: [[1750333602.460984000, {}], {"log"=>"single line...
287446
"}]
288-
[1] tail.0: [1669160706.737657687, {"date"=>"Dec 14 06:41:08", "message"=>"Exception in thread "main" java.lang.RuntimeException: Something has gone wrong, aborting!
447+
[1] tail.0: [[1750333602.460998000, {}], {"date"=>"Dec 14 06:41:08", "message"=>"Exception in thread "main" java.lang.RuntimeException: Something has gone wrong, aborting!
289448
at com.myproject.module.MyProject.badMethod(MyProject.java:22)
290449
at com.myproject.module.MyProject.oneMoreMethod(MyProject.java:18)
291450
at com.myproject.module.MyProject.anotherMethod(MyProject.java:14)
292451
at com.myproject.module.MyProject.someMethod(MyProject.java:10)
293452
at com.myproject.module.MyProject.main(MyProject.java:6)
294453
"}]
295-
[2] tail.0: [1669160706.737657687, {"log"=>"another line...
454+
[2] tail.0: [[1750333602.460998000, {}], {"log"=>"another line...
296455
"}]
297-
```
456+
```

0 commit comments

Comments
 (0)