You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: pipeline/parsers/decoders.md
+40-30Lines changed: 40 additions & 30 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,29 +1,35 @@
1
1
# Decoders
2
2
3
-
There are certain cases where the log messages being parsed contains encoded data, a typical use case can be found in containerized environments with Docker: application logs it data in JSON format but becomes an escaped string, Consider the following example
3
+
There are cases where the log messages being parsed contain encoded data. A typical
4
+
use case can be found in containerized environments with Docker. Docker logs its
5
+
data in JSON format, which uses escaped strings.
4
6
5
-
Original message generated by the application:
7
+
Consider the following message generated by the application:
6
8
7
9
```text
8
10
{"status": "up and running"}
9
11
```
10
12
11
-
Then the Docker log message become encapsulated as follows:
13
+
The Docker log message encapsulates something like this:
12
14
13
15
```text
14
16
{"log":"{\"status\": \"up and running\"}\r\n","stream":"stdout","time":"2018-03-09T01:01:44.851160855Z"}
15
17
```
16
18
17
-
as you can see the original message is handled as an escaped string. Ideally in Fluent Bit we would like to keep having the original structured message and not a string.
19
+
The original message is handled as an escaped string. Fluent Bit wants to use the
20
+
original structured message and not a string.
18
21
19
22
## Getting Started
20
23
21
-
Decoders are a built-in feature available through the Parsers file, each Parser definition can optionally set one or multiple decoders. There are two type of decoders type:
24
+
Decoders are a built-in feature available through the Parsers file. Each parser
25
+
definition can optionally set one or more decoders. There are two types of decoders:
22
26
23
-
* Decode\_Field: if the content can be decoded in a structured message, append that structure message \(keys and values\) to the original log message.
24
-
* Decode\_Field\_As: any content decoded \(unstructured or structured\) will be replaced in the same key/value, no extra keys are added.
27
+
-`Decode_Field`: If the content can be decoded in a structured message, append
28
+
the structured message (keys and values) to the original log message.
29
+
-`Decode_Field_As`: Any decoded content (unstructured or structured) will be
30
+
replaced in the same key/value, and no extra keys are added.
25
31
26
-
Our pre-defined Docker Parser have the following definition:
32
+
Our pre-defined Docker parser has the following definition:
27
33
28
34
```text
29
35
[PARSER]
@@ -37,54 +43,59 @@ Our pre-defined Docker Parser have the following definition:
37
43
Decode_Field_As escaped log
38
44
```
39
45
40
-
Each line in the parser with a key _Decode\_Field_ instruct the parser to apply a specific decoder on a given field, optionally it offer the option to take an extra action if the decoder cannot succeed.
46
+
Each line in the parser with a key `Decode_Field` instructs the parser to apply
47
+
a specific decoder on a given field. Optionally, it offers the option to take an
48
+
extra action if the decoder doesn't succeed.
41
49
42
-
### Decoders
50
+
### Decoder options
43
51
44
-
| Name | Description |
45
-
|:--- |:--- |
46
-
| json| handle the field content as a JSON map. If it find a JSON map it will replace the content with a structured map. |
47
-
| escaped| decode an escaped string. |
48
-
|escaped\_utf8|decode a UTF8 escaped string. |
52
+
| Name | Description |
53
+
| --------------|----------- |
54
+
|`json`| Handle the field content as a JSON map. If it finds a JSON map, it replaces the content with a structured map. |
55
+
|`escaped`| Decode an escaped string. |
56
+
|`escaped_utf8`|Decode a UTF8 escaped string. |
49
57
50
58
### Optional Actions
51
59
52
-
By default if a decoder fails to decode the field or want to try a next decoder, is possible to define an optional action. Available actions are:
60
+
If a decoder fails to decode the field or, you want to try another decoder, you can
61
+
define an optional action. Available actions are:
53
62
54
63
| Name | Description |
55
-
|:---| :--- |
56
-
|try\_next| if the decoder failed, apply the next Decoder in the list for the same field. |
57
-
|do\_next| if the decoder succeeded or failed, apply the next Decoder in the list for the same field. |
64
+
| -----| ----------- |
65
+
|`try_next`| if the decoder failed, apply the next decoder in the list for the same field. |
66
+
|`do_next`| if the decoder succeeded or failed, apply the next decoder in the list for the same field. |
58
67
59
-
Note that actions are affected by some restrictions:
68
+
Actions are affected by some restrictions:
60
69
61
-
* on Decode\_Field\_As, if succeeded, another decoder of the same type in the same field can be applied only if the data continues being an unstructured message \(raw text\).
62
-
* on Decode\_Field, if succeeded, can only be applied once for the same field. By nature Decode\_Field aims to decode a structured message.
70
+
-`Decode_Field_As`: If successful, another decoder of the same type and the same
71
+
field can be applied only if the data continues being an unstructured message (raw text).
72
+
-`Decode_Field`: If successful, can only be applied once for the same field.
73
+
`Decode`_Field` is intended to decode a structured message.
63
74
64
75
### Examples
65
76
66
-
###escaped\_utf8
77
+
#### `escaped_utf8`
67
78
68
-
Example input \(from `/path/to/log.log` in configuration below\)
{"log":"\u0009\u0009Validated: _audit _internal _introspection _telemetry _thefishbucket history main snmp_data summary\n","stream":"stdout","time":"2018-02-19T23:25:29.1845536Z"}
0 commit comments