Skip to content

Commit 0254189

Browse files
eschabelleschabell
authored
Adding YAML examples to doc add fixed incorrect reserve_data key usage. Fixes #1709. (#1756)
Signed-off-by: Eric D. Schabell <[email protected]>
1 parent 4ea5e13 commit 0254189

File tree

1 file changed

+257
-41
lines changed

1 file changed

+257
-41
lines changed

‎pipeline/filters/parser.md

Lines changed: 257 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -21,16 +21,61 @@ The plugin needs a parser file which defines how to parse each field.
2121

2222
This is an example of parsing a record `{"data":"100 0.5 true This is example"}`.
2323

24-
```python
24+
{% tabs %}
25+
{% tab title="fluent-bit.yaml" %}
26+
27+
```yaml
28+
parsers:
29+
- name: dummy_test
30+
format: regex
31+
regex: '^(?<INT>[^ ]+) (?<FLOAT>[^ ]+) (?<BOOL>[^ ]+) (?<STRING>.+)$'
32+
```
33+
34+
{% endtab %}
35+
36+
{% tab title="fluent-bit.conf" %}
37+
38+
```text
2539
[PARSER]
2640
Name dummy_test
2741
Format regex
2842
Regex ^(?<INT>[^ ]+) (?<FLOAT>[^ ]+) (?<BOOL>[^ ]+) (?<STRING>.+)$
2943
```
3044

45+
{% endtab %}
46+
{% endtabs %}
47+
3148
The path of the parser file should be written in configuration file under the `[SERVICE]` section.
3249

33-
```python
50+
{% tabs %}
51+
{% tab title="fluent-bit.yaml" %}
52+
53+
```yaml
54+
service:
55+
parsers_file: /path/to/parsers.yaml
56+
57+
pipeline:
58+
inputs:
59+
- name: dummy
60+
tag: dummy.data
61+
dummy: '{"data":"100 0.5 true This is example"}'
62+
63+
filters:
64+
- name: parser
65+
match: 'dummy.*'
66+
key_name: data
67+
parser: dummy_test
68+
69+
outputs:
70+
- name: stdout
71+
match: '*'
72+
```
73+
74+
{% endtab %}
75+
76+
{% tab title="fluent-bit.conf" %}
77+
78+
```text
3479
[SERVICE]
3580
Parsers_File /path/to/parsers.conf
3681

@@ -50,21 +95,42 @@ The path of the parser file should be written in configuration file under the `[
5095
Match *
5196
```
5297

53-
The output is
98+
{% endtab %}
99+
{% endtabs %}
100+
101+
The output when running the corresponding configuration is as follows:
54102

55103
```text
56-
$ fluent-bit -c dummy.conf
57-
Fluent Bit v1.x.x
58-
* Copyright (C) 2019-2020 The Fluent Bit Authors
59-
* Copyright (C) 2015-2018 Treasure Data
104+
# For YAML configuration.
105+
$ ./fluent-bit --config fluent-bit.yaml
106+
107+
# For classic configuration.
108+
$ ./fluent-bit --config fluent-bit.conf
109+
110+
Fluent Bit v4.0.0
111+
* Copyright (C) 2015-2025 The Fluent Bit Authors
60112
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
61113
* https://fluentbit.io
62-
63-
[2017/07/06 22:33:12] [ info] [engine] started
64-
[0] dummy.data: [1499347993.001371317, {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}]
65-
[1] dummy.data: [1499347994.001303118, {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}]
66-
[2] dummy.data: [1499347995.001296133, {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}]
67-
[3] dummy.data: [1499347996.001320284, {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}]
114+
______ _ _ ______ _ _ ___ _____
115+
| ___| | | | | ___ (_) | / || _ |
116+
| |_ | |_ _ ___ _ __ | |_ | |_/ /_| |_ __ __/ /| || |/' |
117+
| _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / /_| || /| |
118+
| | | | |_| | __/ | | | |_ | |_/ / | |_ \ V /\___ |\ |_/ /
119+
\_| |_|\__,_|\___|_| |_|\__| \____/|_|\__| \_/ |_(_)___/
120+
121+
[2025/06/19 10:58:47] [ info] [fluent bit] version=4.0.0, commit=3a91b155d6, pid=76206
122+
[2025/06/19 10:58:47] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
123+
[2025/06/19 10:58:47] [ info] [simd ] disabled
124+
[2025/06/19 10:58:47] [ info] [cmetrics] version=0.9.9
125+
[2025/06/19 10:58:47] [ info] [ctraces ] version=0.6.2
126+
[2025/06/19 10:58:47] [ info] [input:dummy:dummy.0] initializing
127+
[2025/06/19 10:58:47] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
128+
[2025/06/19 10:58:47] [ info] [output:stdout:stdout.0] worker #0 started
129+
[2025/06/19 10:58:47] [ info] [sp] stream processor started
130+
[0] dummy.data: [[1750323528.603308000, {}], {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}]
131+
[0] dummy.data: [[1750323529.603788000, {}], {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}]
132+
[0] dummy.data: [[1750323530.604204000, {}], {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}]
133+
[0] dummy.data: [[1750323531.603961000, {}], {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}]
68134
```
69135

70136
You can see the records `{"data":"100 0.5 true This is example"}` are parsed.
@@ -73,16 +139,65 @@ You can see the records `{"data":"100 0.5 true This is example"}` are parsed.
73139

74140
By default, the parser plugin only keeps the parsed fields in its output.
75141

76-
If you enable `Reserve_Data`, all other fields are preserved:
142+
If you enable `Reserve_Data`, all other fields are preserved. First the contents of the corresponding parsers file,
143+
depending on the choice for YAML or classic configurations, would be as follows:
144+
145+
{% tabs %}
146+
{% tab title="parsers.yaml" %}
147+
148+
```yaml
149+
parsers:
150+
- name: dummy_test
151+
format: regex
152+
regex: '^(?<INT>[^ ]+) (?<FLOAT>[^ ]+) (?<BOOL>[^ ]+) (?<STRING>.+)$'
153+
```
77154
78-
```python
155+
{% endtab %}
156+
157+
{% tab title="parsers.conf" %}
158+
159+
```text
79160
[PARSER]
80161
Name dummy_test
81162
Format regex
82163
Regex ^(?<INT>[^ ]+) (?<FLOAT>[^ ]+) (?<BOOL>[^ ]+) (?<STRING>.+)$
83164
```
84165

85-
```python
166+
{% endtab %}
167+
{% endtabs %}
168+
169+
Now add `Reserve_Data` to the filter section of the corresponding configuration file as follows:
170+
171+
{% tabs %}
172+
{% tab title="fluent-bit.yaml" %}
173+
174+
```yaml
175+
service:
176+
parsers_file: /path/to/parsers.yaml
177+
178+
pipeline:
179+
inputs:
180+
- name: dummy
181+
tag: dummy.data
182+
dummy: '{"data":"100 0.5 true This is example", "key1":"value1", "key2":"value2"}'
183+
184+
filters:
185+
- name: parser
186+
match: 'dummy.*'
187+
key_name: data
188+
parser: dummy_test
189+
reserve_data: on
190+
191+
outputs:
192+
- name: stdout
193+
match: '*'
194+
```
195+
196+
{% endtab %}
197+
198+
{% tab title="fluent-bit.conf" %}
199+
200+
```text
86201
[SERVICE]
87202
Parsers_File /path/to/parsers.conf
88203

@@ -97,32 +212,109 @@ If you enable `Reserve_Data`, all other fields are preserved:
97212
Key_Name data
98213
Parser dummy_test
99214
Reserve_Data On
215+
216+
[OUTPUT]
217+
Name stdout
218+
Match *
100219
```
101220

102-
This will produce the output:
221+
{% endtab %}
222+
{% endtabs %}
223+
224+
The output when running the corresponding configuration is as follows:
103225

104226
```text
105-
$ fluent-bit -c dummy.conf
106-
Fluent-Bit v0.12.0
107-
Copyright (C) Treasure Data
108-
109-
[2017/07/06 22:33:12] [ info] [engine] started
110-
[0] dummy.data: [1499347993.001371317, {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}, "key1":"value1", "key2":"value2"]
111-
[1] dummy.data: [1499347994.001303118, {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}, "key1":"value1", "key2":"value2"]
112-
[2] dummy.data: [1499347995.001296133, {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}, "key1":"value1", "key2":"value2"]
113-
[3] dummy.data: [1499347996.001320284, {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}, "key1":"value1", "key2":"value2"]
227+
# For YAML configuration.
228+
$ ./fluent-bit --config fluent-bit.yaml
229+
230+
# For classic configuration.
231+
$ ./fluent-bit --config fluent-bit.conf
232+
233+
Fluent Bit v4.0.0
234+
* Copyright (C) 2015-2025 The Fluent Bit Authors
235+
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
236+
* https://fluentbit.io
237+
______ _ _ ______ _ _ ___ _____
238+
| ___| | | | | ___ (_) | / || _ |
239+
| |_ | |_ _ ___ _ __ | |_ | |_/ /_| |_ __ __/ /| || |/' |
240+
| _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / /_| || /| |
241+
| | | | |_| | __/ | | | |_ | |_/ / | |_ \ V /\___ |\ |_/ /
242+
\_| |_|\__,_|\___|_| |_|\__| \____/|_|\__| \_/ |_(_)___/
243+
244+
[2025/06/19 10:58:47] [ info] [fluent bit] version=4.0.0, commit=3a91b155d6, pid=76206
245+
[2025/06/19 10:58:47] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
246+
[2025/06/19 10:58:47] [ info] [simd ] disabled
247+
[2025/06/19 10:58:47] [ info] [cmetrics] version=0.9.9
248+
[2025/06/19 10:58:47] [ info] [ctraces ] version=0.6.2
249+
[2025/06/19 10:58:47] [ info] [input:dummy:dummy.0] initializing
250+
[2025/06/19 10:58:47] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
251+
[2025/06/19 10:58:47] [ info] [output:stdout:stdout.0] worker #0 started
252+
[2025/06/19 10:58:47] [ info] [sp] stream processor started
253+
[0] dummy.data: [[1750325238.681398000, {}], {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example", "key1"=>"value1", "key2"=>"value2"}]
254+
[0] dummy.data: [[1750325239.682090000, {}], {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example", "key1"=>"value1", "key2"=>"value2"}]
255+
[0] dummy.data: [[1750325240.682903000, {}], {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example", "key1"=>"value1", "key2"=>"value2"}]
114256
```
115257

116-
If you enable `Reserved_Data` and `Preserve_Key`, the original key field will also be preserved:
258+
If you enable `Reserve_Data` and `Preserve_Key`, the original key field will also be preserved. First the contents of
259+
the corresponding parsers file, depending on the choice for YAML or classic configurations, would be as follows:
117260

118-
```python
261+
{% tabs %}
262+
{% tab title="parsers.yaml" %}
263+
264+
```yaml
265+
parsers:
266+
- name: dummy_test
267+
format: regex
268+
regex: '^(?<INT>[^ ]+) (?<FLOAT>[^ ]+) (?<BOOL>[^ ]+) (?<STRING>.+)$'
269+
```
270+
271+
{% endtab %}
272+
273+
{% tab title="parsers.conf" %}
274+
275+
```text
119276
[PARSER]
120277
Name dummy_test
121278
Format regex
122279
Regex ^(?<INT>[^ ]+) (?<FLOAT>[^ ]+) (?<BOOL>[^ ]+) (?<STRING>.+)$
123280
```
124281

125-
```python
282+
{% endtab %}
283+
{% endtabs %}
284+
285+
Now add `Reserve_Data` and `Preserve_Key`to the filter section of the corresponding configuration file as follows:
286+
287+
{% tabs %}
288+
{% tab title="fluent-bit.yaml" %}
289+
290+
```yaml
291+
service:
292+
parsers_file: /path/to/parsers.yaml
293+
294+
pipeline:
295+
inputs:
296+
- name: dummy
297+
tag: dummy.data
298+
dummy: '{"data":"100 0.5 true This is example", "key1":"value1", "key2":"value2"}'
299+
300+
filters:
301+
- name: parser
302+
match: 'dummy.*'
303+
key_name: data
304+
parser: dummy_test
305+
reserve_data: on
306+
preserve_key: on
307+
308+
outputs:
309+
- name: stdout
310+
match: '*'
311+
```
312+
313+
{% endtab %}
314+
315+
{% tab title="fluent-bit.conf" %}
316+
317+
```text
126318
[SERVICE]
127319
Parsers_File /path/to/parsers.conf
128320

@@ -138,21 +330,45 @@ If you enable `Reserved_Data` and `Preserve_Key`, the original key field will al
138330
Parser dummy_test
139331
Reserve_Data On
140332
Preserve_Key On
141-
142-
[OUTPUT]
333+
334+
[OUTPUT]
143335
Name stdout
144336
Match *
145337
```
146338

147-
This will produce the following output:
339+
{% endtab %}
340+
{% endtabs %}
341+
342+
The output when running the corresponding configuration is as follows:
148343

149344
```text
150-
$ fluent-bit -c dummy.conf
151-
Fluent Bit v2.1.1
152-
* Copyright (C) 2015-2022 The Fluent Bit Authors
153-
...
154-
...
155-
[0] dummy.data: [[1687122778.299116136, {}], {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example", "data"=>"100 0.5 true This is example", "key1"=>"value1", "key2"=>"value2"}]
156-
[0] dummy.data: [[1687122779.296906553, {}], {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example", "data"=>"100 0.5 true This is example", "key1"=>"value1", "key2"=>"value2"}]
157-
[0] dummy.data: [[1687122780.297475803, {}], {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example", "data"=>"100 0.5 true This is example", "key1"=>"value1", "key2"=>"value2"}]
158-
```
345+
# For YAML configuration.
346+
$ ./fluent-bit --config fluent-bit.yaml
347+
348+
# For classic configuration.
349+
$ ./fluent-bit --config fluent-bit.conf
350+
351+
Fluent Bit v4.0.0
352+
* Copyright (C) 2015-2025 The Fluent Bit Authors
353+
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
354+
* https://fluentbit.io
355+
______ _ _ ______ _ _ ___ _____
356+
| ___| | | | | ___ (_) | / || _ |
357+
| |_ | |_ _ ___ _ __ | |_ | |_/ /_| |_ __ __/ /| || |/' |
358+
| _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / /_| || /| |
359+
| | | | |_| | __/ | | | |_ | |_/ / | |_ \ V /\___ |\ |_/ /
360+
\_| |_|\__,_|\___|_| |_|\__| \____/|_|\__| \_/ |_(_)___/
361+
362+
[2025/06/19 10:58:47] [ info] [fluent bit] version=4.0.0, commit=3a91b155d6, pid=76206
363+
[2025/06/19 10:58:47] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
364+
[2025/06/19 10:58:47] [ info] [simd ] disabled
365+
[2025/06/19 10:58:47] [ info] [cmetrics] version=0.9.9
366+
[2025/06/19 10:58:47] [ info] [ctraces ] version=0.6.2
367+
[2025/06/19 10:58:47] [ info] [input:dummy:dummy.0] initializing
368+
[2025/06/19 10:58:47] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
369+
[2025/06/19 10:58:47] [ info] [output:stdout:stdout.0] worker #0 started
370+
[2025/06/19 10:58:47] [ info] [sp] stream processor started
371+
[0] dummy.data: [[1750325678.572817000, {}], {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example", "data"=>"100 0.5 true This is example", "key1"=>"value1", "key2"=>"value2"}]
372+
[0] dummy.data: [[1750325679.574538000, {}], {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example", "data"=>"100 0.5 true This is example", "key1"=>"value1", "key2"=>"value2"}]
373+
[0] dummy.data: [[1750325680.569750000, {}], {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example", "data"=>"100 0.5 true This is example", "key1"=>"value1", "key2"=>"value2"}]
374+
```

0 commit comments

Comments
 (0)