Skip to content

Commit 351849d

Browse files
esmerelesmerel
authored
Merge pull request #1824 from fluent/lynettemiles/sc-136202/update-fluent-bit-docs-pipeline-inputs-standard
2 parents f5efc15 + 42bd1a2 commit 351849d

File tree

1 file changed

+94
-73
lines changed

1 file changed

+94
-73
lines changed

‎pipeline/inputs/standard-input.md

Lines changed: 94 additions & 73 deletions
Original file line numberDiff line numberDiff line change
@@ -1,58 +1,74 @@
1-
# Standard Input
1+
# Standard input
22

3-
The **stdin** plugin supports retrieving a message stream from the standard input interface \(stdin\) of the Fluent Bit process.
4-
In order to use it, specify the plugin name as the input, e.g:
3+
The _Standard input_ plugin supports retrieving a message stream from the standard input interface (`stdin`) of the Fluent Bit process.
4+
To use it, specify the plugin name as the input. For example:
55

66
```bash
7-
$ fluent-bit -i stdin -o stdout
7+
fluent-bit -i stdin -o stdout
88
```
99

10-
If the stdin stream is closed (end-of-file), the stdin plugin will instruct
11-
Fluent Bit to exit with success (0) after flushing any pending output.
10+
If the `stdin` stream is closed (`end-of-file`), the plugin instructs Fluent Bit to exit with success (`0`) after flushing any pending output.
11+
12+
## Configuration parameters
13+
14+
The plugin supports the following configuration parameters:
15+
16+
| Key | Description | Default |
17+
| :--- | :--- | :--- |
18+
| `Buffer_Size` | Set the buffer size to read data. This value is used to increase buffer size and must be set according to the [Unit Size](../../administration/configuring-fluent-bit/unit-sizes.md) specification. | `16k` |
19+
| `Parser` | The name of the parser to invoke instead of the default JSON input parser. | _none_ |
20+
| `Threaded` | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` |
1221

1322
## Input formats
1423

15-
If no parser is configured for the stdin plugin, it expects *valid JSON* input data in one of the following formats:
24+
If no parser is configured for the `stdin` plugin, it expects valid JSON input data in one of the following formats:
1625

17-
1. A JSON object with one or more key-value pairs: `{ "key": "value", "key2": "value2" }`
18-
3. A 2-element JSON array in [Fluent Bit Event](../../concepts/key-concepts.md#event-or-record) format, which may be:
19-
* `[TIMESTAMP, { "key": "value" }]` where TIMESTAMP is a floating point value representing a timestamp in seconds; or
20-
* from Fluent Bit v2.1.0, `[[TIMESTAMP, METADATA], { "key": "value" }]` where TIMESTAMP has the same meaning as above and METADATA is a JSON object.
26+
- A JSON object with one or more key-value pairs: `{ "key": "value", "key2": "value2" }`
27+
- A 2-element JSON array in [Fluent Bit Event](../../concepts/key-concepts.md#event-or-record) format, which can be:
28+
- `[TIMESTAMP, { "key": "value" }]` where TIMESTAMP is a floating point value representing a timestamp in seconds.
29+
- From Fluent Bit v2.1.0, `[[TIMESTAMP, METADATA], { "key": "value" }]` where _`TIMESTAMP`_ has the same meaning as previous and _`METADATA`_ is a JSON object.
2130

2231
Multi-line input JSON is supported.
2332

24-
Any input data that is *not* in one of the above formats will cause the plugin to log errors like:
33+
Any input data which isn't in one of the supported formats will cause the plugin to log errors like:
2534

26-
```
35+
```text
2736
[debug] [input:stdin:stdin.0] invalid JSON message, skipping
2837
[error] [input:stdin:stdin.0] invalid record found, it's not a JSON map or array
2938
```
3039

31-
To handle inputs in other formats, a parser must be explicitly specified in the configuration for the `stdin` plugin. See [parser input example](#parser-input-example) for sample configuration.
40+
To handle inputs in other formats, a parser must be explicitly specified in the configuration for the `stdin` plugin. See [parser input example](#parser-input) for sample configuration.
3241

3342
## Log event timestamps
3443

35-
The Fluent Bit event timestamp will be set from the input record if the 2-element event input is used or a custom parser configuration supplies a timestamp. Otherwise the event timestamp will be set to the timestamp at which the record is read by the stdin plugin.
44+
The Fluent Bit event timestamp will be set from the input record if the two-element event input is used or a custom parser configuration supplies a timestamp. Otherwise the event timestamp will be set to the timestamp at which the record is read by the `stdin` plugin.
3645

3746
## Examples
3847

39-
### Json input example
48+
### JSON input
4049

41-
A better example to demonstrate how it works will be through a _Bash_ script that generates messages and writes them to [Fluent Bit](http://fluentbit.io). Write the following content in a file named _test.sh_:
50+
To demonstrate how the plugin works, you can use a `bash` script that generates messages and writes them to [Fluent Bit](http://fluentbit.io).
4251

43-
```bash
44-
#!/bin/sh
52+
1. Write the following content in a file named `test.sh`:
4553

46-
for ((i=0; i<=5; i++)); do
47-
echo -n "{\"key\": \"some value\"}"
48-
sleep 1
49-
done
50-
```
54+
```bash
55+
#!/bin/sh
5156

52-
Now lets start the script and [Fluent Bit](http://fluentbit.io):
57+
for ((i=0; i<=5; i++)); do
58+
echo -n "{\"key\": \"some value\"}"
59+
sleep 1
60+
done
61+
```
5362

54-
```bash
55-
$ bash test.sh | fluent-bit -q -i stdin -o stdout
63+
1. Start the script and [Fluent Bit](http://fluentbit.io):
64+
65+
```bash
66+
bash test.sh | fluent-bit -q -i stdin -o stdout
67+
```
68+
69+
The command should return output like the following:
70+
71+
```text
5672
[0] stdin.0: [[1684196745.942883835, {}], {"key"=>"some value"}]
5773
[0] stdin.0: [[1684196746.938949056, {}], {"key"=>"some value"}]
5874
[0] stdin.0: [[1684196747.940162493, {}], {"key"=>"some value"}]
@@ -61,9 +77,9 @@ $ bash test.sh | fluent-bit -q -i stdin -o stdout
6177
[0] stdin.0: [[1684196750.943721442, {}], {"key"=>"some value"}]
6278
```
6379

64-
### Json input with timestamp example
80+
### JSON input with timestamp
6581

66-
An input event timestamp may also be supplied. Replace `test.sh` with:
82+
An input event timestamp can also be supplied. Replace `test.sh` with:
6783

6884
```bash
6985
#!/bin/sh
@@ -81,10 +97,15 @@ for ((i=0; i<=5; i++)); do
8197
done
8298
```
8399

84-
Re-run the sample command. Note that the timestamps output by Fluent Bit are now one day old because Fluent Bit used the input message timestamp.
100+
Re-run the sample command. Timestamps output by Fluent Bit are now one day old because Fluent Bit used the input message timestamp.
85101

86102
```bash
87-
$ bash test.sh | fluent-bit -q -i stdin -o stdout
103+
bash test.sh | fluent-bit -q -i stdin -o stdout
104+
```
105+
106+
Which returns the following:
107+
108+
```text
88109
[0] stdin.0: [[1684110480.028171300, {}], {"realtimestamp"=>1684196880.030070}]
89110
[0] stdin.0: [[1684110481.033753395, {}], {"realtimestamp"=>1684196881.034741}]
90111
[0] stdin.0: [[1684110482.036730051, {}], {"realtimestamp"=>1684196882.037704}]
@@ -93,10 +114,9 @@ $ bash test.sh | fluent-bit -q -i stdin -o stdout
93114
[0] stdin.0: [[1684110485.048710107, {}], {"realtimestamp"=>1684196885.049651}]
94115
```
95116

96-
### Json input with metadata example
117+
### JSON input with metadata
97118

98-
Additional metadata is also supported on Fluent Bit v2.1.0 and above by replacing the timestamp
99-
with a 2-element object, e.g.:
119+
Additional metadata is supported in Fluent Bit v2.1.0 and later by replacing the timestamp with a two-element object. For example:
100120

101121
```bash
102122
#!/bin/sh
@@ -116,8 +136,15 @@ for ((i=0; i<=5; i++)); do
116136
done
117137
```
118138

139+
Run test using the command:
140+
141+
```bash
142+
bash ./test.sh | fluent-bit -q -i stdin -o stdout
119143
```
120-
$ bash ./test.sh | fluent-bit -q -i stdin -o stdout
144+
145+
Which returns results like the following:
146+
147+
```text
121148
[0] stdin.0: [[1684110513.060139417, {"metakey"=>"metavalue"}], {"realtimestamp"=>1684196913.061017}]
122149
[0] stdin.0: [[1684110514.063085317, {"metakey"=>"metavalue"}], {"realtimestamp"=>1684196914.064145}]
123150
[0] stdin.0: [[1684110515.066210508, {"metakey"=>"metavalue"}], {"realtimestamp"=>1684196915.067155}]
@@ -126,47 +153,32 @@ $ bash ./test.sh | fluent-bit -q -i stdin -o stdout
126153
[0] stdin.0: [[1684110518.075428724, {"metakey"=>"metavalue"}], {"realtimestamp"=>1684196918.076292}]
127154
```
128155

129-
On older Fluent Bit versions records in this format will be discarded. Fluent Bit will log:
156+
On older Fluent Bit versions records in this format will be discarded. If the log level permits, Fluent Bit will log:
130157

131-
```
158+
```text
132159
[ warn] unknown time format 6
133160
```
134161

135-
if the log level permits.
136-
137-
### Parser input example <a id="parser-input-example"></a>
162+
### Parser input
138163

139-
To capture inputs in other formats, specify a parser configuration for the
140-
`stdin` plugin.
164+
To capture inputs in other formats, specify a parser configuration for the `stdin` plugin.
141165

142-
For example, if you want to read raw messages line-by-line and forward them you
143-
could use a `parser.conf` that captures the whole message line:
166+
For example, if you want to read raw messages line by line and forward them, you could use a `parser.conf` that captures the whole message line:
144167

145-
```
168+
```text
146169
[PARSER]
147170
name stringify_message
148171
format regex
149172
Key_Name message
150173
regex ^(?<message>.*)
151174
```
152175

153-
then use that in the `parser` clause of the stdin plugin in the `fluent-bit.conf`:
176+
You can then use that in the `parser` clause of the `stdin` plugin in the `fluent-bit.conf` file:
154177

155178
{% tabs %}
156-
{% tab title="fluent-bit.conf" %}
157-
```
158-
[INPUT]
159-
Name stdin
160-
Tag stdin
161-
Parser stringify_message
162-
163-
[OUTPUT]
164-
Name stdout
165-
Match *
166-
```
167-
{% endtab %}
168179

169180
{% tab title="fluent-bit.yaml" %}
181+
170182
```yaml
171183
pipeline:
172184
inputs:
@@ -177,14 +189,34 @@ pipeline:
177189
- name: stdout
178190
match: '*'
179191
```
192+
193+
{% endtab %}
194+
195+
{% tab title="fluent-bit.conf" %}
196+
197+
```text
198+
[INPUT]
199+
Name stdin
200+
Tag stdin
201+
Parser stringify_message
202+
203+
[OUTPUT]
204+
Name stdout
205+
Match *
206+
```
207+
180208
{% endtab %}
181209
{% endtabs %}
182210

183-
Fluent Bit will now read each line and emit a single message for each input
184-
line:
211+
Fluent Bit will now read each line and emit a single message for each input line, using the following command:
185212

213+
```shell
214+
seq 1 5 | /opt/fluent-bit/bin/fluent-bit -c fluent-bit.conf -R parser.conf -q
186215
```
187-
$ seq 1 5 | /opt/fluent-bit/bin/fluent-bit -c fluent-bit.conf -R parser.conf -q
216+
217+
Which returns output similar to:
218+
219+
```text
188220
[0] stdin: [1681358780.517029169, {"message"=>"1"}]
189221
[1] stdin: [1681358780.517068334, {"message"=>"2"}]
190222
[2] stdin: [1681358780.517072116, {"message"=>"3"}]
@@ -193,15 +225,4 @@ $ seq 1 5 | /opt/fluent-bit/bin/fluent-bit -c fluent-bit.conf -R parser.conf -q
193225
$
194226
```
195227

196-
In real-world deployments it is best to use a more realistic parser that splits
197-
messages into real fields and adds appropriate tags.
198-
199-
## Configuration Parameters <a id="config"></a>
200-
201-
The plugin supports the following configuration parameters:
202-
203-
| Key | Description | Default |
204-
| :--- | :--- | :--- |
205-
| Buffer\_Size | Set the buffer size to read data. This value is used to increase buffer size. The value must be according to the [Unit Size](../../administration/configuring-fluent-bit/unit-sizes.md) specification. | 16k |
206-
| Parser | The name of the parser to invoke instead of the default JSON input parser | |
207-
| Threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` |
228+
In production deployments it's best to use a parser that splits messages into real fields and adds appropriate tags.

0 commit comments

Comments
 (0)