Skip to content

Pipeline: input: stdin: style #1824

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jul 1, 2025
Merged

Pipeline: input: stdin: style #1824

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
125740b
Pipeline: input: stdin: style
esmerel Jul 1, 2025
1d52661
Apply suggestions from code review
esmerel Jul 1, 2025
42bd1a2
Apply suggestions from code review
esmerel Jul 1, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
167 changes: 94 additions & 73 deletions pipeline/inputs/standard-input.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,58 +1,74 @@
# Standard Input
# Standard input

The **stdin** plugin supports retrieving a message stream from the standard input interface \(stdin\) of the Fluent Bit process.
In order to use it, specify the plugin name as the input, e.g:
The _Standard input_ plugin supports retrieving a message stream from the standard input interface (`stdin`) of the Fluent Bit process.
To use it, specify the plugin name as the input. For example:

```bash
$ fluent-bit -i stdin -o stdout
fluent-bit -i stdin -o stdout
```

If the stdin stream is closed (end-of-file), the stdin plugin will instruct
Fluent Bit to exit with success (0) after flushing any pending output.
If the `stdin` stream is closed (`end-of-file`), the plugin instructs Fluent Bit to exit with success (`0`) after flushing any pending output.

## Configuration parameters

The plugin supports the following configuration parameters:

| Key | Description | Default |
| :--- | :--- | :--- |
| `Buffer_Size` | Set the buffer size to read data. This value is used to increase buffer size and must be set according to the [Unit Size](../../administration/configuring-fluent-bit/unit-sizes.md) specification. | `16k` |
| `Parser` | The name of the parser to invoke instead of the default JSON input parser. | _none_ |
| `Threaded` | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` |

## Input formats

If no parser is configured for the stdin plugin, it expects *valid JSON* input data in one of the following formats:
If no parser is configured for the `stdin` plugin, it expects valid JSON input data in one of the following formats:

1. A JSON object with one or more key-value pairs: `{ "key": "value", "key2": "value2" }`
3. A 2-element JSON array in [Fluent Bit Event](../../concepts/key-concepts.md#event-or-record) format, which may be:
* `[TIMESTAMP, { "key": "value" }]` where TIMESTAMP is a floating point value representing a timestamp in seconds; or
* from Fluent Bit v2.1.0, `[[TIMESTAMP, METADATA], { "key": "value" }]` where TIMESTAMP has the same meaning as above and METADATA is a JSON object.
- A JSON object with one or more key-value pairs: `{ "key": "value", "key2": "value2" }`
- A 2-element JSON array in [Fluent Bit Event](../../concepts/key-concepts.md#event-or-record) format, which can be:
- `[TIMESTAMP, { "key": "value" }]` where TIMESTAMP is a floating point value representing a timestamp in seconds.
- From Fluent Bit v2.1.0, `[[TIMESTAMP, METADATA], { "key": "value" }]` where _`TIMESTAMP`_ has the same meaning as previous and _`METADATA`_ is a JSON object.

Multi-line input JSON is supported.

Any input data that is *not* in one of the above formats will cause the plugin to log errors like:
Any input data which isn't in one of the supported formats will cause the plugin to log errors like:

```
```text
[debug] [input:stdin:stdin.0] invalid JSON message, skipping
[error] [input:stdin:stdin.0] invalid record found, it's not a JSON map or array
```

To handle inputs in other formats, a parser must be explicitly specified in the configuration for the `stdin` plugin. See [parser input example](#parser-input-example) for sample configuration.
To handle inputs in other formats, a parser must be explicitly specified in the configuration for the `stdin` plugin. See [parser input example](#parser-input) for sample configuration.

## Log event timestamps

The Fluent Bit event timestamp will be set from the input record if the 2-element event input is used or a custom parser configuration supplies a timestamp. Otherwise the event timestamp will be set to the timestamp at which the record is read by the stdin plugin.
The Fluent Bit event timestamp will be set from the input record if the two-element event input is used or a custom parser configuration supplies a timestamp. Otherwise the event timestamp will be set to the timestamp at which the record is read by the `stdin` plugin.

## Examples

### Json input example
### JSON input

A better example to demonstrate how it works will be through a _Bash_ script that generates messages and writes them to [Fluent Bit](http://fluentbit.io). Write the following content in a file named _test.sh_:
To demonstrate how the plugin works, you can use a `bash` script that generates messages and writes them to [Fluent Bit](http://fluentbit.io).

```bash
#!/bin/sh
1. Write the following content in a file named `test.sh`:

for ((i=0; i<=5; i++)); do
echo -n "{\"key\": \"some value\"}"
sleep 1
done
```
```bash
#!/bin/sh

Now lets start the script and [Fluent Bit](http://fluentbit.io):
for ((i=0; i<=5; i++)); do
echo -n "{\"key\": \"some value\"}"
sleep 1
done
```

```bash
$ bash test.sh | fluent-bit -q -i stdin -o stdout
1. Start the script and [Fluent Bit](http://fluentbit.io):

```bash
bash test.sh | fluent-bit -q -i stdin -o stdout
```

The command should return output like the following:

```text
[0] stdin.0: [[1684196745.942883835, {}], {"key"=>"some value"}]
[0] stdin.0: [[1684196746.938949056, {}], {"key"=>"some value"}]
[0] stdin.0: [[1684196747.940162493, {}], {"key"=>"some value"}]
Expand All @@ -61,9 +77,9 @@ $ bash test.sh | fluent-bit -q -i stdin -o stdout
[0] stdin.0: [[1684196750.943721442, {}], {"key"=>"some value"}]
```

### Json input with timestamp example
### JSON input with timestamp

An input event timestamp may also be supplied. Replace `test.sh` with:
An input event timestamp can also be supplied. Replace `test.sh` with:

```bash
#!/bin/sh
Expand All @@ -81,10 +97,15 @@ for ((i=0; i<=5; i++)); do
done
```

Re-run the sample command. Note that the timestamps output by Fluent Bit are now one day old because Fluent Bit used the input message timestamp.
Re-run the sample command. Timestamps output by Fluent Bit are now one day old because Fluent Bit used the input message timestamp.

```bash
$ bash test.sh | fluent-bit -q -i stdin -o stdout
bash test.sh | fluent-bit -q -i stdin -o stdout
```

Which returns the following:

```text
[0] stdin.0: [[1684110480.028171300, {}], {"realtimestamp"=>1684196880.030070}]
[0] stdin.0: [[1684110481.033753395, {}], {"realtimestamp"=>1684196881.034741}]
[0] stdin.0: [[1684110482.036730051, {}], {"realtimestamp"=>1684196882.037704}]
Expand All @@ -93,10 +114,9 @@ $ bash test.sh | fluent-bit -q -i stdin -o stdout
[0] stdin.0: [[1684110485.048710107, {}], {"realtimestamp"=>1684196885.049651}]
```

### Json input with metadata example
### JSON input with metadata

Additional metadata is also supported on Fluent Bit v2.1.0 and above by replacing the timestamp
with a 2-element object, e.g.:
Additional metadata is supported in Fluent Bit v2.1.0 and later by replacing the timestamp with a two-element object. For example:

```bash
#!/bin/sh
Expand All @@ -116,8 +136,15 @@ for ((i=0; i<=5; i++)); do
done
```

Run test using the command:

```bash
bash ./test.sh | fluent-bit -q -i stdin -o stdout
```
$ bash ./test.sh | fluent-bit -q -i stdin -o stdout

Which returns results like the following:

```text
[0] stdin.0: [[1684110513.060139417, {"metakey"=>"metavalue"}], {"realtimestamp"=>1684196913.061017}]
[0] stdin.0: [[1684110514.063085317, {"metakey"=>"metavalue"}], {"realtimestamp"=>1684196914.064145}]
[0] stdin.0: [[1684110515.066210508, {"metakey"=>"metavalue"}], {"realtimestamp"=>1684196915.067155}]
Expand All @@ -126,47 +153,32 @@ $ bash ./test.sh | fluent-bit -q -i stdin -o stdout
[0] stdin.0: [[1684110518.075428724, {"metakey"=>"metavalue"}], {"realtimestamp"=>1684196918.076292}]
```

On older Fluent Bit versions records in this format will be discarded. Fluent Bit will log:
On older Fluent Bit versions records in this format will be discarded. If the log level permits, Fluent Bit will log:

```
```text
[ warn] unknown time format 6
```

if the log level permits.

### Parser input example <a id="parser-input-example"></a>
### Parser input

To capture inputs in other formats, specify a parser configuration for the
`stdin` plugin.
To capture inputs in other formats, specify a parser configuration for the `stdin` plugin.

For example, if you want to read raw messages line-by-line and forward them you
could use a `parser.conf` that captures the whole message line:
For example, if you want to read raw messages line by line and forward them, you could use a `parser.conf` that captures the whole message line:

```
```text
[PARSER]
name stringify_message
format regex
Key_Name message
regex ^(?<message>.*)
```

then use that in the `parser` clause of the stdin plugin in the `fluent-bit.conf`:
You can then use that in the `parser` clause of the `stdin` plugin in the `fluent-bit.conf` file:

{% tabs %}
{% tab title="fluent-bit.conf" %}
```
[INPUT]
Name stdin
Tag stdin
Parser stringify_message

[OUTPUT]
Name stdout
Match *
```
{% endtab %}

{% tab title="fluent-bit.yaml" %}

```yaml
pipeline:
inputs:
Expand All @@ -177,14 +189,34 @@ pipeline:
- name: stdout
match: '*'
```

{% endtab %}

{% tab title="fluent-bit.conf" %}

```text
[INPUT]
Name stdin
Tag stdin
Parser stringify_message

[OUTPUT]
Name stdout
Match *
```

{% endtab %}
{% endtabs %}

Fluent Bit will now read each line and emit a single message for each input
line:
Fluent Bit will now read each line and emit a single message for each input line, using the following command:

```shell
seq 1 5 | /opt/fluent-bit/bin/fluent-bit -c fluent-bit.conf -R parser.conf -q
```
$ seq 1 5 | /opt/fluent-bit/bin/fluent-bit -c fluent-bit.conf -R parser.conf -q

Which returns output similar to:

```text
[0] stdin: [1681358780.517029169, {"message"=>"1"}]
[1] stdin: [1681358780.517068334, {"message"=>"2"}]
[2] stdin: [1681358780.517072116, {"message"=>"3"}]
Expand All @@ -193,15 +225,4 @@ $ seq 1 5 | /opt/fluent-bit/bin/fluent-bit -c fluent-bit.conf -R parser.conf -q
$
```

In real-world deployments it is best to use a more realistic parser that splits
messages into real fields and adds appropriate tags.

## Configuration Parameters <a id="config"></a>

The plugin supports the following configuration parameters:

| Key | Description | Default |
| :--- | :--- | :--- |
| Buffer\_Size | Set the buffer size to read data. This value is used to increase buffer size. The value must be according to the [Unit Size](../../administration/configuring-fluent-bit/unit-sizes.md) specification. | 16k |
| Parser | The name of the parser to invoke instead of the default JSON input parser | |
| Threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` |
In production deployments it's best to use a parser that splits messages into real fields and adds appropriate tags.