Skip to content

Commit 161fc32

Browse files
eschabelleschabell
authored
Indentation for Parsers section code examples in docs need to be 2 space standardized. Fixes #1967. (#1968)
* Fixing code example indent standards for configuring parser doc. Part of issue #1967. Signed-off-by: Eric D. Schabell <[email protected]> * Fixing code example indent standards for decoders parser doc. Part of issue #1967. Signed-off-by: Eric D. Schabell <[email protected]> * Fixing code example indent standards for json parser doc. Part of issue #1967. Signed-off-by: Eric D. Schabell <[email protected]> * Fixing code example indent standards for logfmt parser doc. Part of issue #1967. Signed-off-by: Eric D. Schabell <[email protected]> * Fixing code example indent standards for LTSV parser doc. Part of issue #1967. Signed-off-by: Eric D. Schabell <[email protected]> * Fixing code example indent standards for LTSV parser doc. Part of issue #1967. Signed-off-by: Eric D. Schabell <[email protected]> * Fixing code example indent standards for regular expression parser doc. Part of issue #1967. Signed-off-by: Eric D. Schabell <[email protected]> --------- Signed-off-by: Eric D. Schabell <[email protected]>
1 parent 6348339 commit 161fc32

File tree

6 files changed

+121
-115
lines changed

6 files changed

+121
-115
lines changed

‎pipeline/parsers/configuring-parser.md

Lines changed: 28 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -38,8 +38,8 @@ Multiple parsers can be defined and each section has it own properties. The foll
3838
| `Time_Keep` | If enabled, when a time key is recognized and parsed, the parser will keep the original time key. If disabled, the parser will drop the original time field. |
3939
| `Time_System_timezone` | If there is no time zone (`%z`) specified in the given `Time_Format`, enabling this option will make the parser detect and use the system's configured time zone. The configured time zone is detected from the [`TZ` environment variable](https://www.gnu.org/software/libc/manual/html_node/TZ-Variable.html). |
4040
| `Types` | Specifies the data type of parsed field. The syntax is `types <field_name_1>:<type_name_1> <field_name_2>:<type_name_2> ...`. The supported types are `string` (default), `integer`, `bool`, `float`, `hex`. The option is supported by `ltsv`, `logfmt` and `regex`. |
41-
| `Decode_Field` | If the content can be decoded in a structured message, append the structured message (keys and values) to the original log message. Decoder types: `json`, `escaped`, `escaped_utf8`. The syntax is: `Decode_Field <decoder_type> <field_name>`. See [Decoders](pipeline/parsers/decoders.md) for additional information. |
42-
| `Decode_Field_As` | Any decoded content (unstructured or structured) will be replaced in the same key/value, and no extra keys are added. Decoder types: `json`, `escaped`, `escaped_utf8`. The syntax is: `Decode_Field_As <decoder_type> <field_name>`. See [Decoders](pipeline/parsers/decoders.md) for additional information. |
41+
| `Decode_Field` | If the content can be decoded in a structured message, append the structured message (keys and values) to the original log message. Decoder types: `json`, `escaped`, `escaped_utf8`. The syntax is: `Decode_Field <decoder_type> <field_name>`. See [Decoders](decoders.md) for additional information. |
42+
| `Decode_Field_As` | Any decoded content (unstructured or structured) will be replaced in the same key/value, and no extra keys are added. Decoder types: `json`, `escaped`, `escaped_utf8`. The syntax is: `Decode_Field_As <decoder_type> <field_name>`. See [Decoders](decoders.md) for additional information. |
4343
| `Skip_Empty_Values` | Specifies a boolean which determines if the parser should skip empty values. The default is `true`. |
4444
| `Time_Strict` | The default value (`true`) tells the parser to be strict with the expected time format. With this option set to false, the parser will be permissive with the format of the time. You can use this when the format expects time fraction but the time to be parsed doesn't include it. |
4545

@@ -52,40 +52,40 @@ All parsers must be defined in a parsers file (see below for examples), not in t
5252

5353
```yaml
5454
parsers:
55-
- name: docker
56-
format: json
57-
time_key: time
58-
time_format: '%Y-%m-%dT%H:%M:%S.%L'
59-
time_keep: on
60-
61-
- name: syslog-rfc5424
62-
format: regex
63-
regex: '^\<(?<pri>[0-9]{1,5})\>1 (?<time>[^ ]+) (?<host>[^ ]+) (?<ident>[^ ]+) (?<pid>[-0-9]+) (?<msgid>[^ ]+) (?<extradata>(\[(.*)\]|-)) (?<message>.+)$'
64-
time_key: time
65-
time_format: '%Y-%m-%dT%H:%M:%S.%L'
66-
time_keep: on
67-
types: pid:integer
55+
- name: docker
56+
format: json
57+
time_key: time
58+
time_format: '%Y-%m-%dT%H:%M:%S.%L'
59+
time_keep: on
60+
61+
- name: syslog-rfc5424
62+
format: regex
63+
regex: '^\<(?<pri>[0-9]{1,5})\>1 (?<time>[^ ]+) (?<host>[^ ]+) (?<ident>[^ ]+) (?<pid>[-0-9]+) (?<msgid>[^ ]+) (?<extradata>(\[(.*)\]|-)) (?<message>.+)$'
64+
time_key: time
65+
time_format: '%Y-%m-%dT%H:%M:%S.%L'
66+
time_keep: on
67+
types: pid:integer
6868
```
6969
7070
{% endtab %}
7171
{% tab title="parsers.conf" %}
7272
7373
```text
7474
[PARSER]
75-
Name docker
76-
Format json
77-
Time_Key time
78-
Time_Format %Y-%m-%dT%H:%M:%S.%L
79-
Time_Keep On
75+
Name docker
76+
Format json
77+
Time_Key time
78+
Time_Format %Y-%m-%dT%H:%M:%S.%L
79+
Time_Keep On
8080

8181
[PARSER]
82-
Name syslog-rfc5424
83-
Format regex
84-
Regex ^\<(?<pri>[0-9]{1,5})\>1 (?<time>[^ ]+) (?<host>[^ ]+) (?<ident>[^ ]+) (?<pid>[-0-9]+) (?<msgid>[^ ]+) (?<extradata>(\[(.*)\]|-)) (?<message>.+)$
85-
Time_Key time
86-
Time_Format %Y-%m-%dT%H:%M:%S.%L
87-
Time_Keep On
88-
Types pid:integer
82+
Name syslog-rfc5424
83+
Format regex
84+
Regex ^\<(?<pri>[0-9]{1,5})\>1 (?<time>[^ ]+) (?<host>[^ ]+) (?<ident>[^ ]+) (?<pid>[-0-9]+) (?<msgid>[^ ]+) (?<extradata>(\[(.*)\]|-)) (?<message>.+)$
85+
Time_Key time
86+
Time_Format %Y-%m-%dT%H:%M:%S.%L
87+
Time_Keep On
88+
Types pid:integer
8989
```
9090

9191
{% endtab %}
@@ -161,7 +161,7 @@ The following time zone abbreviations are supported.
161161
| `CLT` | `-04:00` | `-14400` | no | Chile Standard Time |
162162
| `CLST` | `-03:00` | `-10800` | yes | Chile Summer Time |
163163

164-
### Australasian and Oceanian time zones
164+
### Australasian and Oceania time zones
165165

166166
| Abbreviation | UTC Offset (`HH:MM`) | Offset (seconds) | Is DST | Description |
167167
| ------------ | -------------------- | ---------------- | ------ | ---------------------------------- |

‎pipeline/parsers/decoders.md

Lines changed: 44 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -32,29 +32,29 @@ For example, the predefined Docker parser has the following definition:
3232

3333
```yaml
3434
parsers:
35-
- name: docker
36-
format: json
37-
time_key: time
38-
time_format: '%Y-%m-%dT%H:%M:%S.%L'
39-
time_keep: on
40-
# Command | Decoder | Field | Optional Action |
41-
# ==========|==========|=======|===================|
42-
decode_field_as: escaped log
35+
- name: docker
36+
format: json
37+
time_key: time
38+
time_format: '%Y-%m-%dT%H:%M:%S.%L'
39+
time_keep: on
40+
# Command | Decoder | Field | Optional Action |
41+
# ==========|==========|=======|===================|
42+
decode_field_as: escaped log
4343
```
4444
4545
{% endtab %}
4646
{% tab title="parsers.conf" %}
4747
4848
```text
4949
[PARSER]
50-
Name docker
51-
Format json
52-
Time_Key time
53-
Time_Format %Y-%m-%dT%H:%M:%S.%L
54-
Time_Keep On
55-
# Command | Decoder | Field | Optional Action |
56-
# ==============|===========|=======|===================|
57-
Decode_Field_As escaped log
50+
Name docker
51+
Format json
52+
Time_Key time
53+
Time_Format %Y-%m-%dT%H:%M:%S.%L
54+
Time_Keep On
55+
# Command | Decoder | Field | Optional Action |
56+
# ==============|===========|=======|===================|
57+
Decode_Field_As escaped log
5858
```
5959

6060
{% endtab %}
@@ -99,12 +99,14 @@ Example input from `/path/to/log.log`:
9999
Example output:
100100

101101
```text
102+
...
102103
[24] tail.0: [1519082729.184544400, {"log"=>" Checking indexes...
103104
", "stream"=>"stdout", "time"=>"2018-02-19T23:25:29.1845444Z"}]
104105
[25] tail.0: [1519082729.184553600, {"log"=>" Validated: _audit _internal _introspection _telemetry _thefishbucket history main snmp_data summary
105106
", "stream"=>"stdout", "time"=>"2018-02-19T23:25:29.1845536Z"}]
106107
[26] tail.0: [1519082729.184562200, {"log"=>" Done
107108
", "stream"=>"stdout", "time"=>"2018-02-19T23:25:29.1845622Z"}]
109+
...
108110
```
109111

110112
Decoder example Fluent Bit configuration files:
@@ -114,34 +116,34 @@ Decoder example Fluent Bit configuration files:
114116

115117
```yaml
116118
service:
117-
parsers_file: parsers.yaml
119+
parsers_file: parsers.yaml
118120

119121
pipeline:
120-
inputs:
121-
- name: tail
122-
parser: docker
123-
path: /path/to/log.log
124-
125-
outputs:
126-
- name: stdout
127-
match: '*'
122+
inputs:
123+
- name: tail
124+
parser: docker
125+
path: /path/to/log.log
126+
127+
outputs:
128+
- name: stdout
129+
match: '*'
128130
```
129131
130132
{% endtab %}
131133
{% tab title="fluent-bit.conf" %}
132134
133135
```text
134136
[SERVICE]
135-
Parsers_File parsers.conf
137+
Parsers_File parsers.conf
136138

137139
[INPUT]
138-
Name tail
139-
Parser docker
140-
Path /path/to/log.log
140+
Name tail
141+
Parser docker
142+
Path /path/to/log.log
141143

142144
[OUTPUT]
143-
Name stdout
144-
Match *
145+
Name stdout
146+
Match *
145147
```
146148

147149
{% endtab %}
@@ -154,24 +156,24 @@ The example parsers file:
154156

155157
```yaml
156158
parsers:
157-
- name: docker
158-
format: json
159-
time_key: time
160-
time_format: '%Y-%m-%dT%H:%M:%S %z'
161-
decode_field_as: escaped_utf8 log
159+
- name: docker
160+
format: json
161+
time_key: time
162+
time_format: '%Y-%m-%dT%H:%M:%S %z'
163+
decode_field_as: escaped_utf8 log
162164
```
163165
164166
{% endtab %}
165167
{% tab title="parsers.conf" %}
166168
167169
```text
168170
[PARSER]
169-
Name docker
170-
Format json
171-
Time_Key time
172-
Time_Format %Y-%m-%dT%H:%M:%S %z
173-
Decode_Field_as escaped_utf8 log
171+
Name docker
172+
Format json
173+
Time_Key time
174+
Time_Format %Y-%m-%dT%H:%M:%S %z
175+
Decode_Field_as escaped_utf8 log
174176
```
175177

176178
{% endtab %}
177-
{% endtabs %}
179+
{% endtabs %}

‎pipeline/parsers/json.md

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -9,21 +9,21 @@ For example, the default parsers configuration file includes a parser for parsin
99

1010
```yaml
1111
parsers:
12-
- name: docker
13-
format: json
14-
time_key: time
15-
time_format: '%Y-%m-%dT%H:%M:%S %z'
12+
- name: docker
13+
format: json
14+
time_key: time
15+
time_format: '%Y-%m-%dT%H:%M:%S %z'
1616
```
1717
1818
{% endtab %}
1919
{% tab title="parsers.conf" %}
2020
2121
```text
2222
[PARSER]
23-
Name docker
24-
Format json
25-
Time_Key time
26-
Time_Format %Y-%m-%dT%H:%M:%S %z
23+
Name docker
24+
Format json
25+
Time_Key time
26+
Time_Format %Y-%m-%dT%H:%M:%S %z
2727
```
2828

2929
{% endtab %}

‎pipeline/parsers/logfmt.md

Lines changed: 11 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -9,17 +9,17 @@ Here is an example parsers configuration:
99

1010
```yaml
1111
parsers:
12-
- name: logfmt
13-
format: logfmt
12+
- name: logfmt
13+
format: logfmt
1414
```
1515
1616
{% endtab %}
1717
{% tab title="parsers.conf" %}
1818
1919
```text
2020
[PARSER]
21-
Name logfmt
22-
Format logfmt
21+
Name logfmt
22+
Format logfmt
2323
```
2424

2525
{% endtab %}
@@ -46,20 +46,20 @@ If you want to be more strict than the logfmt standard and not parse lines where
4646

4747
```yaml
4848
parsers:
49-
- name: logfmt
50-
format: logfmt
51-
logfmt_no_bare_keys: true
49+
- name: logfmt
50+
format: logfmt
51+
logfmt_no_bare_keys: true
5252
```
5353
5454
{% endtab %}
5555
{% tab title="parsers.conf" %}
5656
5757
```text
5858
[PARSER]
59-
Name logfmt
60-
Format logfmt
61-
Logfmt_No_Bare_Keys true
59+
Name logfmt
60+
Format logfmt
61+
Logfmt_No_Bare_Keys true
6262
```
6363

6464
{% endtab %}
65-
{% endtabs %}
65+
{% endtabs %}

‎pipeline/parsers/ltsv.md

Lines changed: 16 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -20,23 +20,23 @@ The following is an example parsers configuration file:
2020

2121
```yaml
2222
parsers:
23-
- name: access_log_ltsv
24-
format: ltsv
25-
time_key: time
26-
time_format: '[%d/%b/%Y:%H:%M:%S %z]'
27-
types: status:integer size:integer
23+
- name: access_log_ltsv
24+
format: ltsv
25+
time_key: time
26+
time_format: '[%d/%b/%Y:%H:%M:%S %z]'
27+
types: status:integer size:integer
2828
```
2929
3030
{% endtab %}
3131
{% tab title="parsers.conf" %}
3232
3333
```text
3434
[PARSER]
35-
Name access_log_ltsv
36-
Format ltsv
37-
Time_Key time
38-
Time_Format [%d/%b/%Y:%H:%M:%S %z]
39-
Types status:integer size:integer
35+
Name access_log_ltsv
36+
Format ltsv
37+
Time_Key time
38+
Time_Format [%d/%b/%Y:%H:%M:%S %z]
39+
Types status:integer size:integer
4040
```
4141

4242
{% endtab %}
@@ -45,19 +45,23 @@ parsers:
4545
The following log entry is valid content for the previously defined parser:
4646

4747
```text
48+
...
4849
host:127.0.0.1 ident:- user:- time:[10/Jul/2018:13:27:05 +0200] req:GET / HTTP/1.1 status:200 size:16218 referer:http://127.0.0.1/ ua:Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0
4950
host:127.0.0.1 ident:- user:- time:[10/Jul/2018:13:27:05 +0200] req:GET /assets/plugins/bootstrap/css/bootstrap.min.css HTTP/1.1 status:200 size:121200 referer:http://127.0.0.1/ ua:Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0
5051
host:127.0.0.1 ident:- user:- time:[10/Jul/2018:13:27:05 +0200] req:GET /assets/css/headers/header-v6.css HTTP/1.1 status:200 size:37706 referer:http://127.0.0.1/ ua:Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0
5152
host:127.0.0.1 ident:- user:- time:[10/Jul/2018:13:27:05 +0200] req:GET /assets/css/style.css HTTP/1.1 status:200 size:1279 referer:http://127.0.0.1/ ua:Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0
53+
...
5254
```
5355

54-
After processing, it internal representation will be:
56+
After processing, its internal representation will be:
5557

5658
```text
59+
...
5760
[1531222025.000000000, {"host"=>"127.0.0.1", "ident"=>"-", "user"=>"-", "req"=>"GET / HTTP/1.1", "status"=>200, "size"=>16218, "referer"=>"http://127.0.0.1/", "ua"=>"Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0"}]
5861
[1531222025.000000000, {"host"=>"127.0.0.1", "ident"=>"-", "user"=>"-", "req"=>"GET /assets/plugins/bootstrap/css/bootstrap.min.css HTTP/1.1", "status"=>200, "size"=>121200, "referer"=>"http://127.0.0.1/", "ua"=>"Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0"}]
5962
[1531222025.000000000, {"host"=>"127.0.0.1", "ident"=>"-", "user"=>"-", "req"=>"GET /assets/css/headers/header-v6.css HTTP/1.1", "status"=>200, "size"=>37706, "referer"=>"http://127.0.0.1/", "ua"=>"Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0"}]
6063
[1531222025.000000000, {"host"=>"127.0.0.1", "ident"=>"-", "user"=>"-", "req"=>"GET /assets/css/style.css HTTP/1.1", "status"=>200, "size"=>1279, "referer"=>"http://127.0.0.1/", "ua"=>"Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0"}]
64+
...
6165
```
6266

63-
The time was converted to Unix timestamp (UTC).
67+
The time was converted to Unix timestamp (UTC).

0 commit comments

Comments
 (0)