Skip to content

Pipeline: output: forward: style #1940

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jul 17, 2025
Merged

Pipeline: output: forward: style #1940

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5c94448
Pipeline: output: forward: style
esmerel Jul 16, 2025
05fac9f
Apply suggestions from code review
esmerel Jul 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
147 changes: 76 additions & 71 deletions pipeline/outputs/forward.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,54 +1,54 @@
# Forward

_Forward_ is the protocol used by [Fluentd](http://www.fluentd.org) to route messages between peers. The **forward** output plugin provides interoperability between [Fluent Bit](http://fluentbit.io) and [Fluentd](http://fluentd.org).
_Forward_ is the protocol used by [Fluentd](http://www.fluentd.org) to route messages between peers. The `forward` output plugin provides interoperability between [Fluent Bit](http://fluentbit.io) and [Fluentd](http://fluentd.org).
There are no configuration steps required besides specifying where [Fluentd](http://fluentd.org) is located, which can be a local or a remote destination.

This plugin offers two different transports and modes:
This plugin offers the following transports and modes:

* Forward (TCP): It uses a plain TCP connection.
* Secure Forward (TLS): when TLS is enabled, the plugin switch to Secure Forward mode.
- Forward (TCP): It uses a plain TCP connection.
- Secure Forward (TLS): when TLS is enabled, the plugin switch to Secure Forward mode.

## Configuration Parameters
## Configuration parameters

The following parameters are mandatory for either Forward for Secure Forward modes:

| Key | Description | Default |
| -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- |
| Host | Target host where Fluent-Bit or Fluentd are listening for Forward messages. | 127.0.0.1 |
| Port | TCP Port of the target service. | 24224 |
| Time_as_Integer | Set timestamps in integer format, it enable compatibility mode for Fluentd v0.12 series. | False |
| Upstream | If Forward will connect to an _Upstream_ instead of a simple host, this property defines the absolute path for the Upstream configuration file, for more details about this refer to the [Upstream Servers ](../../administration/configuring-fluent-bit/classic-mode/upstream-servers.md)documentation section. | |
| Unix_Path | Specify the path to unix socket to send a Forward message. If set, `Upstream` is ignored. | |
| Tag | Overwrite the tag as we transmit. This allows the receiving pipeline start fresh, or to attribute source. | |
| Send_options | Always send options (with "size"=count of messages) | False |
| Require_ack_response | Send "chunk"-option and wait for "ack" response from server. Enables at-least-once and receiving server can control rate of traffic. (Requires Fluentd v0.14.0+ server) | False |
| Compress | Set to 'gzip' to enable gzip compression. Incompatible with `Time_as_Integer=True` and tags set dynamically using the [Rewrite Tag](../filters/rewrite-tag.md) filter. Requires Fluentd server v0.14.7 or later. | _none_ |
| Workers | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `2` |

## Secure Forward Mode Configuration Parameters

When using Secure Forward mode, the [TLS](../../administration/transport-security.md) mode requires to be enabled. The following additional configuration parameters are available:

| Key | Description | Default |
| ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | --------- |
| Shared_Key | A key string known by the remote Fluentd used for authorization. | |
| Empty_Shared_Key | Use this option to connect to Fluentd with a zero-length secret. | False |
| Username | Specify the username to present to a Fluentd server that enables `user_auth`. | |
| Password | Specify the password corresponding to the username. | |
| Self_Hostname | Default value of the auto-generated certificate common name (CN). | localhost |
| tls | Enable or disable TLS support | Off |
| tls.verify | Force certificate validation | On |
| tls.debug | Set TLS debug verbosity level. It accept the following values: 0 (No debug), 1 (Error), 2 (State change), 3 (Informational) and 4 Verbose | 1 |
| tls.ca_file | Absolute path to CA certificate file | |
| tls.crt_file | Absolute path to Certificate file. | |
| tls.key_file | Absolute path to private Key file. | |
| tls.key_passwd | Optional password for tls.key_file file. | |

## Forward Setup

Before proceeding, make sure that [Fluentd](http://fluentd.org) is installed, if it's not the case please refer to the following [Fluentd Installation](http://docs.fluentd.org/v0.12/categories/installation) document and go ahead with that.

Once [Fluentd](http://fluentd.org) is installed, create the following configuration file example that will allow us to stream data into it:
| Key | Description | Default |
| --- | ------------ | --------- |
| `Host` | Target host where Fluent Bit or Fluentd are listening for Forward messages. | `127.0.0.1` |
| `Port` | TCP Port of the target service. | `24224` |
| `Time_as_Integer` | Set timestamps in integer format, it enables compatibility mode for Fluentd v0.12 series. | `False` |
| `Upstream` | If Forward will connect to an `Upstream` instead of a basic host, this property defines the absolute path for the Upstream configuration file, for more details about this, see [Upstream Servers ](../../administration/configuring-fluent-bit/classic-mode/upstream-servers.md). | _none_ |
| `Unix_Path` | Specify the path to a Unix socket to send a Forward message. If set, `Upstream` is ignored. | _none_ |
| `Tag` | Overwrite the tag as Fluent Bit transmits. This allows the receiving pipeline start fresh, or to attribute a source. | _none_ |
| `Send_options` | Always send options (with `"size"=count of messages`) | `False` |
| `Require_ack_response` | Send `chunk` option and wait for an `ack` response from the server. Enables at-least-once and receiving server can control rate of traffic. Requires Fluentd v0.14.0+ or later | `False` |
| `Compress` | Set to `gzip` to enable gzip compression. Incompatible with `Time_as_Integer=True` and tags set dynamically using the [Rewrite Tag](../filters/rewrite-tag.md) filter. Requires Fluentd server v0.14.7 or later. | _none_ |
| `Workers` | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `2` |

## Secure Forward mode configuration parameters

When using Secure Forward mode, the [TLS](../../administration/transport-security.md) mode must be enabled. The following additional configuration parameters are available:

| Key | Description | Default |
| --- | ----------- | ------- |
| `Shared_Key` | A key string known by the remote Fluentd used for authorization. | _none_ |
| `Empty_Shared_Key` | Use this option to connect to Fluentd with a zero-length secret. | `False` |
| `Username` | Specify the username to present to a Fluentd server that enables `user_auth`. | _none_ |
| `Password` | Specify the password corresponding to the username. | _none_ |
| `Self_Hostname` | Default value of the auto-generated certificate common name (CN). | `localhost` |
| `tls` | Enable or disable TLS support. | `Off` |
| `tls.verify` | Force certificate validation. | `On` |
| `tls.debug` | Set TLS debug verbosity level. Allowed values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), and `4` (Verbose). | `1` |
| `tls.ca_file` | Absolute path to CA certificate file. | _none_ |
| `tls.crt_file` | Absolute path to Certificate file. | _none_ |
| `tls.key_file` | Absolute path to private Key file. | _none_ |
| `tls.key_passwd` | Optional password for `tls.key_file`. | _none_ |

## Forward setup

Before proceeding, ensure that [Fluentd](http://fluentd.org) is installed. If it's not, refer to the [Fluentd Installation](http://docs.fluentd.org/installation) document.

After installing Fluentd, create the following configuration file example which lets you to stream data into it:

```text
<source>
Expand All @@ -62,14 +62,19 @@ Once [Fluentd](http://fluentd.org) is installed, create the following configurat
</match>
```

That configuration file specifies that it will listen for _TCP_ connections on the port _24224_ through the **forward** input type.
Then for every message with a _fluent_bit_ **TAG**, will print the message to the standard output.
That configuration file specifies that it will listen for TCP connections on port `24224` through the `forward` input type.

In one terminal launch [Fluentd](http://fluentd.org) specifying the new configuration file created:
Every message with a `fluent_bit` tag will print a message to the standard output.

In one terminal, launch Fluentd while specifying the new configuration file created:

```shell
$ fluentd -c test.conf
fluentd -c test.conf
```

Which should return a response similar to the following:

```text
...
2017-03-23 11:50:43 -0600 [info]: reading config file path="test.conf"
...
Expand Down Expand Up @@ -103,24 +108,24 @@ $ fluentd -c test.conf
...
```

## Fluent Bit + Forward Setup <a href="forward_setup" id="forward_setup"></a>
## Fluent Bit and Forward setup

Now that [Fluentd](http://fluentd.org) is ready to receive messages, we need to specify where the **forward** output plugin will flush the information using the following format:
When Fluentd is ready to receive messages, specify where the `forward` output plugin will flush the information using the following format:

```shell
fluent-bit -i INPUT -o forward://HOST:PORT
```

If the **TAG** parameter is not set, the plugin will retain the tag.
Keep in mind that **TAG** is important for routing rules inside [Fluentd](http://fluentd.org).
If the `tag` parameter isn't set, the plugin will retain the tag.
The `tag` is important for routing rules inside Fluentd.

Using the [CPU](../inputs/cpu-metrics.md) input plugin as an example we will flush CPU metrics to [Fluentd](http://fluentd.org) with tag _fluent_bit_:
Using the [CPU](../inputs/cpu-metrics.md) input plugin as an example, you can flush `cpu` metrics with the `tag` `fluent_bit` to Fluentd:

```shell
fluent-bit -i cpu -t fluent_bit -o forward://127.0.0.1:24224
```

Now on the [Fluentd](http://fluentd.org) side, you will see the CPU metrics gathered in the last seconds:
In Fluentd, you will see the CPU metrics gathered in the last seconds:

```text
...
Expand All @@ -131,13 +136,13 @@ Now on the [Fluentd](http://fluentd.org) side, you will see the CPU metrics gath
...
```

So we gathered [CPU](../inputs/cpu-metrics.md) metrics and flushed them out to [Fluentd](http://fluentd.org) properly.
This shows that [CPU](../inputs/cpu-metrics.md) metrics were gathered and flushed out to Fluentd properly.

## Fluent Bit + Secure Forward Setup <a href="secure_forward_setup" id="secure_forward_setup"></a>
## Fluent Bit and Secure Forward setup

> DISCLAIMER: the following example does not consider the generation of certificates for best practice on production environments.
The following example doesn't consider the generation of certificates for best practice on production environments.

Secure Forward aims to provide a secure channel of communication with the remote Fluentd service using [TLS](../../administration/transport-security.md).
Secure Forward provides a secure channel of communication with the remote Fluentd service using [TLS](../../administration/transport-security.md).

### Fluent Bit

Expand All @@ -151,12 +156,12 @@ service:
flush: 5
daemon: off
log_level: info

pipeline:
inputs:
- name: cpu
tag: cpu_usage

outputs:
- name: forward
match: '*'
Expand Down Expand Up @@ -212,23 +217,23 @@ Paste this content in a file called `fld.conf`:
</match>
```

### Test Communication
### Test communication

Start Fluentd:
1. Start Fluentd:

```shell
fluentd -c fld.conf
```
```shell
fluentd -c fld.conf
```

Start Fluent Bit:
1. Start Fluent Bit:

```shell
# For YAML configuration.
fluent-bit --config flb.yaml
```shell
# For YAML configuration.
fluent-bit --config flb.yaml

# For classic configuration
fluent-bit --config flb.conf
```
# For classic configuration
fluent-bit --config flb.conf
```

After five seconds, Fluent Bit will write records to Fluentd.
In Fluentd output you will see a message like this:
Expand All @@ -251,4 +256,4 @@ In Fluentd output you will see a message like this:
2017-03-23 13:34:43 -0600 cpu_usage: {"cpu_p":1.75,"user_p":1.25,"system_p":0.5,"cpu0.p_cpu":3.0,"cpu0.p_user":3.0,"cpu0.p_system":0.0,"cpu1.p_cpu":2.0,"cpu1.p_user":2.0,"cpu1.p_system":0.0,"cpu2.p_cpu":0.0,"cpu2.p_user":0.0,"cpu2.p_system":0.0,"cpu3.p_cpu":1.0,"cpu3.p_user":0.0,"cpu3.p_system":1.0}
2017-03-23 13:34:44 -0600 cpu_usage: {"cpu_p":5.0,"user_p":3.25,"system_p":1.75,"cpu0.p_cpu":4.0,"cpu0.p_user":2.0,"cpu0.p_system":2.0,"cpu1.p_cpu":8.0,"cpu1.p_user":5.0,"cpu1.p_system":3.0,"cpu2.p_cpu":4.0,"cpu2.p_user":3.0,"cpu2.p_system":1.0,"cpu3.p_cpu":4.0,"cpu3.p_user":2.0,"cpu3.p_system":2.0}
...
```
```
1 change: 1 addition & 0 deletions vale-styles/FluentBit/Headings.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ exceptions:
- Emmet
- EventBridge
- Fluent Bit
- Forward
- GCP
- GitLab
- GitHub
Expand Down