Skip to content

Pipeline: input: tcp: style #1836

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jul 3, 2025
Merged

Pipeline: input: tcp: style #1836

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
bb58fbb
Pipeline: input: tcp: style
esmerel Jul 2, 2025
67960b0
Apply suggestions from code review
esmerel Jul 3, 2025
95bfdb1
Merge branch 'master' into lynettemiles/sc-136207/update-fluent-bit-d…
esmerel Jul 3, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 36 additions & 31 deletions pipeline/inputs/tcp.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,45 +1,45 @@
# TCP

The **tcp** input plugin allows to retrieve structured JSON or raw messages over a TCP network interface (TCP port).
The _TCP_ input plugin lets you retrieve structured JSON or raw messages over a TCP network interface (TCP port).

## Configuration Parameters

The plugin supports the following configuration parameters:

| Key | Description | Default |
| ------------ | ----------- | ------- |
| Listen | Listener network interface. | 0.0.0.0 |
| Port | TCP port where listening for connections | 5170 |
| Buffer\_Size | Specify the maximum buffer size in KB to receive a JSON message. If not set, the default size will be the value of _Chunk\_Size_. | |
| Chunk\_Size | By default the buffer to store the incoming JSON messages, do not allocate the maximum memory allowed, instead it allocate memory when is required. The rounds of allocations are set by _Chunk\_Size_ in KB. If not set, _Chunk\_Size_ is equal to 32 (32KB). | 32 |
| Format | Specify the expected payload format. It support the options _json_ and _none_. When using _json_, it expects JSON maps, when is set to _none_, it will split every record using the defined _Separator_ (option below). | json |
| Separator | When the expected _Format_ is set to _none_, Fluent Bit needs a separator string to split the records. By default it uses the breakline character (LF or 0x10). | |
| Source\_Address\_Key| Specify the key where the source address will be injected. | |
| Threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` |
| `Listen` | Listener network interface. | `0.0.0.0` |
| `Port` | TCP port to listen for connections. | `5170` |
| `Buffer_Size` | Specify the maximum buffer size in KB to receive a JSON message. If not set, the default is the value of `Chunk_Size`. | `Chunk_Size` |
| `Chunk_Size` | The default buffer to store the incoming JSON messages. It doesn't allocate the maximum memory allowed; instead it allocates memory when required. The rounds of allocations are set by `Chunk_Size`. If not set, `Chunk_Size` is equal to 32 (32KB). | `32` |
| `Format` | Specify the expected payload format. Supported values: `json` and `none`. When set to `json` it expects JSON maps. When set to `none`, every record splits using the defined `Separator`. | `json` |
| `Separator` | When `Format` is set to `none`, Fluent Bit needs a separator string to split the records. | `LF` or `0x10` (break line) |
| `Source_Address_Key`| Specify the key to inject the source address. | _none_ |
| `Threaded` | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` |

## Getting Started
## Get started

In order to receive JSON messages over TCP, you can run the plugin from the command line or through the configuration file:
To receive JSON messages over TCP, you can run the plugin from the command line or through the configuration file.

### Command Line
### Command line

From the command line you can let Fluent Bit listen for _JSON_ messages with the following options:
From the command line you can let Fluent Bit listen for JSON messages with the following options:

```shell
$ fluent-bit -i tcp -o stdout
```bash
fluent-bit -i tcp -o stdout
```

By default the service will listen an all interfaces (0.0.0.0) through TCP port 5170, optionally you can change this directly, e.g:
By default the service will listen an all interfaces (`0.0.0.0`) through TCP port `5170`. Optionally you can change this directly:

```shell
$ fluent-bit -i tcp://192.168.3.2:9090 -o stdout
```bash
fluent-bit -i tcp://192.168.3.2:9090 -o stdout
```

In the example the JSON messages will only arrive through network interface under 192.168.3.2 address and TCP Port 9090.
In the example the JSON messages will only arrive through the network interface at `192.168.3.2` address and TCP Port `9090`.

### Configuration File
### Configuration file

In your main configuration file append the following _Input_ & _Output_ sections:
In your main configuration file append the following sections:

{% tabs %}
{% tab title="fluent-bit.yaml" %}
Expand Down Expand Up @@ -79,21 +79,26 @@ pipeline:
{% endtab %}
{% endtabs %}

## Testing
## Test the configuration

Once Fluent Bit is running, you can send some messages using the _netcat_:
When Fluent Bit is running, you can send some messages using `netcat`:

```shell
$ echo '{"key 1": 123456789, "key 2": "abcdefg"}' | nc 127.0.0.1 5170
```bash
echo '{"key 1": 123456789, "key 2": "abcdefg"}' | nc 127.0.0.1 5170
```

In [Fluent Bit](http://fluentbit.io) we should see the following output:
Run Fluent Bit:

```shell
$ bin/fluent-bit -i tcp -o stdout -f 1
bin/fluent-bit -i tcp -o stdout -f 1
```

You should see the following output:

Fluent Bit v4.0.3
* Copyright (C) 2015-2025 The Fluent Bit Authors
```bash
Fluent Bit v1.x.x
* Copyright (C) 2019-2020 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

Expand All @@ -118,8 +123,8 @@ ______ _ _ ______ _ _ ___ _____
[0] tcp.0: [1570115975.581246030, {"key 1"=>123456789, "key 2"=>"abcdefg"}]
```

## Performance Considerations
## Performance considerations

When receiving payloads in JSON format, there are high performance penalties. Parsing JSON is a very expensive task so you could expect your CPU usage increase under high load environments.

To get faster data ingestion, consider to use the option `Format none` to avoid JSON parsing if not needed.
To get faster data ingestion, consider to use the option `Format none` to avoid JSON parsing if not needed.