You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: pipeline/parsers/decoders.md
+15-24Lines changed: 15 additions & 24 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,8 +1,6 @@
1
1
# Decoders
2
2
3
-
There are cases where the log messages being parsed contain encoded data. A typical
4
-
use case can be found in containerized environments with Docker. Docker logs its
5
-
data in JSON format, which uses escaped strings.
3
+
There are cases where the log messages that you want to parse contain encoded data. A typical use case can be found in containerized environments with Docker. Docker logs its data in JSON format, which uses escaped strings.
6
4
7
5
Consider the following message generated by the application:
8
6
@@ -16,20 +14,18 @@ The Docker log message encapsulates something like this:
16
14
{"log":"{\"status\": \"up and running\"}\r\n","stream":"stdout","time":"2018-03-09T01:01:44.851160855Z"}
17
15
```
18
16
19
-
The original message is handled as an escaped string. Fluent Bit wants to use the
20
-
original structured message and not a string.
17
+
The original message is handled as an escaped string. Fluent Bit wants to use the original structured message and not a string.
21
18
22
-
## Getting Started
19
+
## Get started
23
20
24
-
Decoders are a built-in feature available through the Parsers file. Each parser
25
-
definition can optionally set one or more decoders. There are two types of decoders:
21
+
Decoders are a built-in feature available through the `parsers.conf` file. Each parser definition can optionally set one or more decoders. There are two types of decoders:
26
22
27
23
-`Decode_Field`: If the content can be decoded in a structured message, append
28
24
the structured message (keys and values) to the original log message.
29
25
-`Decode_Field_As`: Any decoded content (unstructured or structured) will be
30
26
replaced in the same key/value, and no extra keys are added.
31
27
32
-
Our pre-defined Docker parser has the following definition:
28
+
For example, the predefined Docker parser has the following definition:
33
29
34
30
{% tabs %}
35
31
{% tab title="parsers.yaml" %}
@@ -64,34 +60,29 @@ parsers:
64
60
{% endtab %}
65
61
{% endtabs %}
66
62
67
-
Each line in the parser with a key `Decode_Field` instructs the parser to apply
68
-
a specific decoder on a given field. Optionally, it offers the option to take an
69
-
extra action if the decoder doesn't succeed.
63
+
Each line in the parser with a key `Decode_Field` instructs the parser to apply a specific decoder on a given field. Optionally, it offers the option to take an extra action if the decoder doesn't succeed.
70
64
71
65
### Decoder options
72
66
73
67
| Name | Description |
74
68
| -------------- | ----------- |
75
-
|`json`| Handle the field content as a JSON map. If it finds a JSON map, it replaces the content with a structured map. |
69
+
|`json`| Handle the field content as a JSON map. If the decoder finds a JSON map, it replaces the content with a structured map. |
76
70
|`escaped`| Decode an escaped string. |
77
71
|`escaped_utf8`| Decode a UTF8 escaped string. |
78
72
79
-
### Optional Actions
73
+
### Optional actions
80
74
81
-
If a decoder fails to decode the field or, you want to try another decoder, you can
82
-
define an optional action. Available actions are:
75
+
If a decoder fails to decode the field, or if you want to try another decoder, you can define an optional action. Available actions are:
83
76
84
77
| Name | Description |
85
78
| -----| ----------- |
86
-
|`try_next`|if the decoder failed, apply the next decoder in the list for the same field. |
87
-
|`do_next`|if the decoder succeeded or failed, apply the next decoder in the list for the same field. |
79
+
|`try_next`|If the decoder failed, apply the next decoder in the list for the same field. |
80
+
|`do_next`|If the decoder succeeded or failed, apply the next decoder in the list for the same field. |
88
81
89
82
Actions are affected by some restrictions:
90
83
91
-
-`Decode_Field_As`: If successful, another decoder of the same type and the same
92
-
field can be applied only if the data continues being an unstructured message (raw text).
93
-
-`Decode_Field`: If successful, can only be applied once for the same field.
94
-
`Decode`_Field` is intended to decode a structured message.
84
+
-`Decode_Field_As`: If successful, another decoder of the same type and the same field can be applied only if the data continues being an unstructured message (raw text).
85
+
-`Decode_Field`: If successful, can only be applied once for the same field. `Decode_Field` is intended to decode a structured message.
95
86
96
87
### Examples
97
88
@@ -124,7 +115,7 @@ Decoder example Fluent Bit configuration files:
0 commit comments