Skip to content

Commit 125740b

Browse files
esmerelesmerel
committed
Pipeline: input: stdin: style
Signed-off-by: Lynette Miles <[email protected]>
1 parent 8c843cf commit 125740b

File tree

1 file changed

+96
-72
lines changed

1 file changed

+96
-72
lines changed

‎pipeline/inputs/standard-input.md

Lines changed: 96 additions & 72 deletions
Original file line numberDiff line numberDiff line change
@@ -1,58 +1,75 @@
1-
# Standard Input
1+
# Standard input
22

3-
The **stdin** plugin supports retrieving a message stream from the standard input interface \(stdin\) of the Fluent Bit process.
4-
In order to use it, specify the plugin name as the input, e.g:
3+
The _Standard input_ plugin supports retrieving a message stream from the standard input interface (`stdin`) of the Fluent Bit process.
4+
To use it, specify the plugin name as the input. For example:
55

66
```bash
7-
$ fluent-bit -i stdin -o stdout
7+
fluent-bit -i stdin -o stdout
88
```
99

10-
If the stdin stream is closed (end-of-file), the stdin plugin will instruct
11-
Fluent Bit to exit with success (0) after flushing any pending output.
10+
If the `stdin` stream is closed (`end-of-file`), the plugin instruct Fluent Bit to exit with success (`0`) after flushing any pending output.
11+
12+
## Configuration parameters
13+
14+
The plugin supports the following configuration parameters:
15+
16+
| Key | Description | Default |
17+
| :--- | :--- | :--- |
18+
| `Buffer_Size` | Set the buffer size to read data. This value is used to increase buffer size and must be set according to the [Unit Size](../../administration/configuring-fluent-bit/unit-sizes.md) specification. | `16k` |
19+
| `Parser` | The name of the parser to invoke instead of the default JSON input parser. | _none_ |
20+
| `Threaded` | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` |
1221

1322
## Input formats
1423

15-
If no parser is configured for the stdin plugin, it expects *valid JSON* input data in one of the following formats:
24+
If no parser is configured for the `stdin` plugin, it expects valid JSON input data in one of the following formats:
1625

17-
1. A JSON object with one or more key-value pairs: `{ "key": "value", "key2": "value2" }`
18-
3. A 2-element JSON array in [Fluent Bit Event](../../concepts/key-concepts.md#event-or-record) format, which may be:
19-
* `[TIMESTAMP, { "key": "value" }]` where TIMESTAMP is a floating point value representing a timestamp in seconds; or
20-
* from Fluent Bit v2.1.0, `[[TIMESTAMP, METADATA], { "key": "value" }]` where TIMESTAMP has the same meaning as above and METADATA is a JSON object.
26+
- A JSON object with one or more key-value pairs: `{ "key": "value", "key2": "value2" }`
27+
- A 2-element JSON array in [Fluent Bit Event](../../concepts/key-concepts.md#event-or-record) format, which can be:
28+
- `[TIMESTAMP, { "key": "value" }]` where TIMESTAMP is a floating point value representing a timestamp in seconds.
29+
- From Fluent Bit v2.1.0, `[[TIMESTAMP, METADATA], { "key": "value" }]` where _`TIMESTAMP`_ has the same meaning as previous and _`METADATA`_ is a JSON object.
2130

2231
Multi-line input JSON is supported.
2332

24-
Any input data that is *not* in one of the above formats will cause the plugin to log errors like:
33+
Any input data which isn't in one of the supported formats will cause the plugin to log errors like:
2534

26-
```
35+
```text
2736
[debug] [input:stdin:stdin.0] invalid JSON message, skipping
2837
[error] [input:stdin:stdin.0] invalid record found, it's not a JSON map or array
2938
```
3039

31-
To handle inputs in other formats, a parser must be explicitly specified in the configuration for the `stdin` plugin. See [parser input example](#parser-input-example) for sample configuration.
40+
To handle inputs in other formats, a parser must be explicitly specified in the configuration for the `stdin` plugin. See [parser input example](#parser-input) for sample configuration.
3241

3342
## Log event timestamps
3443

35-
The Fluent Bit event timestamp will be set from the input record if the 2-element event input is used or a custom parser configuration supplies a timestamp. Otherwise the event timestamp will be set to the timestamp at which the record is read by the stdin plugin.
44+
The Fluent Bit event timestamp will be set from the input record if the two-element event input is used or a custom parser configuration supplies a timestamp. Otherwise the event timestamp will be set to the timestamp at which the record is read by the `stdin` plugin.
3645

3746
## Examples
3847

39-
### Json input example
48+
### JSON input
4049

41-
A better example to demonstrate how it works will be through a _Bash_ script that generates messages and writes them to [Fluent Bit](http://fluentbit.io). Write the following content in a file named _test.sh_:
50+
To demonstrate how the plugin works, you can use a `bash` script that generates
51+
messages and writes them to [Fluent Bit](http://fluentbit.io).
4252

43-
```bash
44-
#!/bin/sh
53+
1. Write the following content in a file named `test.sh`:
4554

46-
for ((i=0; i<=5; i++)); do
47-
echo -n "{\"key\": \"some value\"}"
48-
sleep 1
49-
done
50-
```
55+
```bash
56+
#!/bin/sh
5157

52-
Now lets start the script and [Fluent Bit](http://fluentbit.io):
58+
for ((i=0; i<=5; i++)); do
59+
echo -n "{\"key\": \"some value\"}"
60+
sleep 1
61+
done
62+
```
5363

54-
```bash
55-
$ bash test.sh | fluent-bit -q -i stdin -o stdout
64+
1. Start the script and [Fluent Bit](http://fluentbit.io):
65+
66+
```bash
67+
bash test.sh | fluent-bit -q -i stdin -o stdout
68+
```
69+
70+
The command should return output like the following:
71+
72+
```text
5673
[0] stdin.0: [[1684196745.942883835, {}], {"key"=>"some value"}]
5774
[0] stdin.0: [[1684196746.938949056, {}], {"key"=>"some value"}]
5875
[0] stdin.0: [[1684196747.940162493, {}], {"key"=>"some value"}]
@@ -61,9 +78,9 @@ $ bash test.sh | fluent-bit -q -i stdin -o stdout
6178
[0] stdin.0: [[1684196750.943721442, {}], {"key"=>"some value"}]
6279
```
6380

64-
### Json input with timestamp example
81+
### JSON input with timestamp
6582

66-
An input event timestamp may also be supplied. Replace `test.sh` with:
83+
An input event timestamp can also be supplied. Replace `test.sh` with:
6784

6885
```bash
6986
#!/bin/sh
@@ -81,10 +98,15 @@ for ((i=0; i<=5; i++)); do
8198
done
8299
```
83100

84-
Re-run the sample command. Note that the timestamps output by Fluent Bit are now one day old because Fluent Bit used the input message timestamp.
101+
Re-run the sample command. Timestamps output by Fluent Bit are now one day old because Fluent Bit used the input message timestamp.
85102

86103
```bash
87-
$ bash test.sh | fluent-bit -q -i stdin -o stdout
104+
bash test.sh | fluent-bit -q -i stdin -o stdout
105+
```
106+
107+
Which returns the following:
108+
109+
```text
88110
[0] stdin.0: [[1684110480.028171300, {}], {"realtimestamp"=>1684196880.030070}]
89111
[0] stdin.0: [[1684110481.033753395, {}], {"realtimestamp"=>1684196881.034741}]
90112
[0] stdin.0: [[1684110482.036730051, {}], {"realtimestamp"=>1684196882.037704}]
@@ -93,10 +115,10 @@ $ bash test.sh | fluent-bit -q -i stdin -o stdout
93115
[0] stdin.0: [[1684110485.048710107, {}], {"realtimestamp"=>1684196885.049651}]
94116
```
95117

96-
### Json input with metadata example
118+
### JSON input with metadata
97119

98-
Additional metadata is also supported on Fluent Bit v2.1.0 and above by replacing the timestamp
99-
with a 2-element object, e.g.:
120+
Additional metadata is supported in Fluent Bit v2.1.0 and later by replacing the timestamp
121+
with a two-element object. For example:
100122

101123
```bash
102124
#!/bin/sh
@@ -116,8 +138,15 @@ for ((i=0; i<=5; i++)); do
116138
done
117139
```
118140

141+
Run test using the command:
142+
143+
```bash
144+
bash ./test.sh | fluent-bit -q -i stdin -o stdout
119145
```
120-
$ bash ./test.sh | fluent-bit -q -i stdin -o stdout
146+
147+
Which returns results like the following:
148+
149+
```text
121150
[0] stdin.0: [[1684110513.060139417, {"metakey"=>"metavalue"}], {"realtimestamp"=>1684196913.061017}]
122151
[0] stdin.0: [[1684110514.063085317, {"metakey"=>"metavalue"}], {"realtimestamp"=>1684196914.064145}]
123152
[0] stdin.0: [[1684110515.066210508, {"metakey"=>"metavalue"}], {"realtimestamp"=>1684196915.067155}]
@@ -126,47 +155,32 @@ $ bash ./test.sh | fluent-bit -q -i stdin -o stdout
126155
[0] stdin.0: [[1684110518.075428724, {"metakey"=>"metavalue"}], {"realtimestamp"=>1684196918.076292}]
127156
```
128157

129-
On older Fluent Bit versions records in this format will be discarded. Fluent Bit will log:
158+
On older Fluent Bit versions records in this format will be discarded. If the log level permits, Fluent Bit will log:
130159

131-
```
160+
```text
132161
[ warn] unknown time format 6
133162
```
134163

135-
if the log level permits.
136-
137-
### Parser input example <a id="parser-input-example"></a>
164+
### Parser input
138165

139-
To capture inputs in other formats, specify a parser configuration for the
140-
`stdin` plugin.
166+
To capture inputs in other formats, specify a parser configuration for the `stdin` plugin.
141167

142-
For example, if you want to read raw messages line-by-line and forward them you
143-
could use a `parser.conf` that captures the whole message line:
168+
For example, if you want to read raw messages line byline and forward them you could use a `parser.conf` that captures the whole message line:
144169

145-
```
170+
```text
146171
[PARSER]
147172
name stringify_message
148173
format regex
149174
Key_Name message
150175
regex ^(?<message>.*)
151176
```
152177

153-
then use that in the `parser` clause of the stdin plugin in the `fluent-bit.conf`:
178+
YOu can then use that in the `parser` clause of the `stdin` plugin in the `fluent-bit.conf`:
154179

155180
{% tabs %}
156-
{% tab title="fluent-bit.conf" %}
157-
```
158-
[INPUT]
159-
Name stdin
160-
Tag stdin
161-
Parser stringify_message
162-
163-
[OUTPUT]
164-
Name stdout
165-
Match *
166-
```
167-
{% endtab %}
168181

169182
{% tab title="fluent-bit.yaml" %}
183+
170184
```yaml
171185
pipeline:
172186
inputs:
@@ -177,14 +191,35 @@ pipeline:
177191
- name: stdout
178192
match: '*'
179193
```
194+
195+
{% endtab %}
196+
197+
{% tab title="fluent-bit.conf" %}
198+
199+
```text
200+
[INPUT]
201+
Name stdin
202+
Tag stdin
203+
Parser stringify_message
204+
205+
[OUTPUT]
206+
Name stdout
207+
Match *
208+
```
209+
180210
{% endtab %}
181211
{% endtabs %}
182212

183213
Fluent Bit will now read each line and emit a single message for each input
184-
line:
214+
line, using the following command:
185215

216+
```shell
217+
seq 1 5 | /opt/fluent-bit/bin/fluent-bit -c fluent-bit.conf -R parser.conf -q
186218
```
187-
$ seq 1 5 | /opt/fluent-bit/bin/fluent-bit -c fluent-bit.conf -R parser.conf -q
219+
220+
Which returns output similar to:
221+
222+
```text
188223
[0] stdin: [1681358780.517029169, {"message"=>"1"}]
189224
[1] stdin: [1681358780.517068334, {"message"=>"2"}]
190225
[2] stdin: [1681358780.517072116, {"message"=>"3"}]
@@ -193,15 +228,4 @@ $ seq 1 5 | /opt/fluent-bit/bin/fluent-bit -c fluent-bit.conf -R parser.conf -q
193228
$
194229
```
195230

196-
In real-world deployments it is best to use a more realistic parser that splits
197-
messages into real fields and adds appropriate tags.
198-
199-
## Configuration Parameters <a id="config"></a>
200-
201-
The plugin supports the following configuration parameters:
202-
203-
| Key | Description | Default |
204-
| :--- | :--- | :--- |
205-
| Buffer\_Size | Set the buffer size to read data. This value is used to increase buffer size. The value must be according to the [Unit Size](../../administration/configuring-fluent-bit/unit-sizes.md) specification. | 16k |
206-
| Parser | The name of the parser to invoke instead of the default JSON input parser | |
207-
| Threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` |
231+
In production deployments it's best to use a parser that splits messages into real fields and adds appropriate tags.

0 commit comments

Comments
 (0)