Skip to content

Updating transport security for style and grammar #1545

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jan 15, 2025
Merged

Updating transport security for style and grammar #1545

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
44698c5
Updating transport security for style and gramamr
esmerel Jan 13, 2025
181a262
Apply suggestions from code review
esmerel Jan 15, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
195 changes: 107 additions & 88 deletions administration/transport-security.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,85 +1,93 @@

# Transport Security

Fluent Bit provides integrated support for _Transport Layer Security_ \(TLS\) and it predecessor _Secure Sockets Layer_ \(SSL\) respectively. In this section we will refer as TLS only for both implementations.
Fluent Bit provides integrated support for Transport Layer Security (TLS) and
it predecessors Secure Sockets Layer (SSL) respectively. This section refers only
to TLS for both implementations.

Both input and output plugins that perform Network I/O can optionally enable TLS and configure the behavior. The following table describes the properties available:
Both input and output plugins that perform Network I/O can optionally enable TLS and
configure the behavior. The following table describes the properties available:

| Property | Description | Default |
| :--- | :--- | :--- |
| tls | enable or disable TLS support | Off |
| tls.verify | force certificate validation | On |
| tls.verify\_hostname | force TLS verification of hostnames | Off |
| tls.debug | Set TLS debug verbosity level. It accept the following values: 0 \(No debug\), 1 \(Error\), 2 \(State change\), 3 \(Informational\) and 4 Verbose | 1 |
| tls.ca\_file | absolute path to CA certificate file | |
| tls.ca\_path | absolute path to scan for certificate files | |
| tls.crt\_file | absolute path to Certificate file | |
| tls.key\_file | absolute path to private Key file | |
| tls.key\_passwd | optional password for tls.key\_file file | |
| tls.vhost | hostname to be used for TLS SNI extension | |

*Note : in order to use TLS on input plugins the user is expected to provide both a certificate and private key*

The listed properties can be enabled in the configuration file, specifically on each output plugin section or directly through the command line.
| `tls` | Enable or disable TLS support | `Off` |
| `tls.verify` | Force certificate validation | `On` |
| `tls.verify_hostname` | Force TLS verification of host names | `Off` |
| `tls.debug` | Set TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational) and `4` (Verbose) | `1` |
| `tls.ca_file` | Absolute path to CA certificate file | _none_ |
| `tls.ca_path` | Absolute path to scan for certificate files | _none_ |
| `tls.crt_file` | Absolute path to Certificate file | _none_ |
| `tls.key_file` | Absolute path to private Key file | _none_ |
| `tls.key_passwd` | Optional password for `tls.key_file` file | _none_ |
| `tls.vhost` | Hostname to be used for TLS SNI extension | _none_ |

To use TLS on input plugins, the user is expected to provide both a certificate and
private key.

The listed properties can be enabled in the configuration file, specifically in each
output plugin section or directly through the command line.

The following **output** plugins can take advantage of the TLS feature:

* [Amazon S3](../pipeline/outputs/s3.md)
* [Apache SkyWalking](../pipeline/outputs/skywalking.md)
* [Azure](../pipeline/outputs/azure.md)
* [Azure Blob](../pipeline/outputs/azure_blob.md)
* [Azure Data Explorer (Kusto)](../pipeline/outputs/azure_kusto.md)
* [Azure Logs Ingestion API](../pipeline/outputs/azure_logs_ingestion.md)
* [BigQuery](../pipeline/outputs/bigquery.md)
* [Dash0](../pipeline/outputs/dash0.md)
* [Datadog](../pipeline/outputs/datadog.md)
* [Elasticsearch](../pipeline/outputs/elasticsearch.md)
* [Forward](../pipeline/outputs/forward.md)
* [GELF](../pipeline/outputs/gelf.md)
* [Google Chronicle](../pipeline/outputs/chronicle.md)
* [HTTP](../pipeline/outputs/http.md)
* [InfluxDB](../pipeline/outputs/influxdb.md)
* [Kafka REST Proxy](../pipeline/outputs/kafka-rest-proxy.md)
* [LogDNA](../pipeline/outputs/logdna.md)
* [Loki](../pipeline/outputs/loki.md)
* [New Relic](../pipeline/outputs/new-relic.md)
* [OpenSearch](../pipeline/outputs/opensearch.md)
* [OpenTelemetry](../pipeline/outputs/opentelemetry.md)
* [Oracle Cloud Infrastructure Logging Analytics](../pipeline/outputs/oci-logging-analytics.md)
* [Prometheus Remote Write](../pipeline/outputs/prometheus-remote-write.md)
* [Slack](../pipeline/outputs/slack.md)
* [Splunk](../pipeline/outputs/splunk.md)
* [Stackdriver](../pipeline/outputs/stackdriver.md)
* [Syslog](../pipeline/outputs/syslog.md)
* [TCP & TLS](../pipeline/outputs/tcp-and-tls.md)
* [Treasure Data](../pipeline/outputs/treasure-data.md)
* [WebSocket](../pipeline/outputs/websocket.md)
- [Amazon S3](../pipeline/outputs/s3.md)
- [Apache SkyWalking](../pipeline/outputs/skywalking.md)
- [Azure](../pipeline/outputs/azure.md)
- [Azure Blob](../pipeline/outputs/azure_blob.md)
- [Azure Data Explorer (Kusto)](../pipeline/outputs/azure_kusto.md)
- [Azure Logs Ingestion API](../pipeline/outputs/azure_logs_ingestion.md)
- [BigQuery](../pipeline/outputs/bigquery.md)
- [Dash0](../pipeline/outputs/dash0.md)
- [Datadog](../pipeline/outputs/datadog.md)
- [Elasticsearch](../pipeline/outputs/elasticsearch.md)
- [Forward](../pipeline/outputs/forward.md)
- [GELF](../pipeline/outputs/gelf.md)
- [Google Chronicle](../pipeline/outputs/chronicle.md)
- [HTTP](../pipeline/outputs/http.md)
- [InfluxDB](../pipeline/outputs/influxdb.md)
- [Kafka REST Proxy](../pipeline/outputs/kafka-rest-proxy.md)
- [LogDNA](../pipeline/outputs/logdna.md)
- [Loki](../pipeline/outputs/loki.md)
- [New Relic](../pipeline/outputs/new-relic.md)
- [OpenSearch](../pipeline/outputs/opensearch.md)
- [OpenTelemetry](../pipeline/outputs/opentelemetry.md)
- [Oracle Cloud Infrastructure Logging Analytics](../pipeline/outputs/oci-logging-analytics.md)
- [Prometheus Remote Write](../pipeline/outputs/prometheus-remote-write.md)
- [Slack](../pipeline/outputs/slack.md)
- [Splunk](../pipeline/outputs/splunk.md)
- [Stackdriver](../pipeline/outputs/stackdriver.md)
- [Syslog](../pipeline/outputs/syslog.md)
- [TCP & TLS](../pipeline/outputs/tcp-and-tls.md)
- [Treasure Data](../pipeline/outputs/treasure-data.md)
- [WebSocket](../pipeline/outputs/websocket.md)

The following **input** plugins can take advantage of the TLS feature:

* [Docker Events](../pipeline/inputs/docker-events.md)
* [Elasticsearch (Bulk API)](../pipeline/inputs/elasticsearch.md)
* [Forward](../pipeline/inputs/forward.md)
* [Health](../pipeline/inputs/health.md)
* [HTTP](../pipeline/inputs/http.md)
* [Kubernetes Events](../pipeline/inputs/kubernetes-events.md)
* [MQTT](../pipeline/inputs/mqtt.md)
* [NGINX Exporter Metrics](../pipeline/inputs/nginx.md)
* [OpenTelemetry](../pipeline/inputs/opentelemetry.md)
* [Prometheus Scrape Metrics](../pipeline/inputs/prometheus-scrape-metrics.md)
* [Prometheus Remote Write](../pipeline/inputs/prometheus-remote-write.md)
* [Splunk (HTTP HEC)](../pipeline/inputs/splunk.md)
* [Syslog](../pipeline/inputs/syslog.md)
* [TCP](../pipeline/inputs/tcp.md)

In addition, other plugins implements a sub-set of TLS support, meaning, with restricted configuration:

* [Kubernetes Filter](../pipeline/filters/kubernetes.md)
- [Docker Events](../pipeline/inputs/docker-events.md)
- [Elasticsearch (Bulk API)](../pipeline/inputs/elasticsearch.md)
- [Forward](../pipeline/inputs/forward.md)
- [Health](../pipeline/inputs/health.md)
- [HTTP](../pipeline/inputs/http.md)
- [Kubernetes Events](../pipeline/inputs/kubernetes-events.md)
- [MQTT](../pipeline/inputs/mqtt.md)
- [NGINX Exporter Metrics](../pipeline/inputs/nginx.md)
- [OpenTelemetry](../pipeline/inputs/opentelemetry.md)
- [Prometheus Scrape Metrics](../pipeline/inputs/prometheus-scrape-metrics.md)
- [Prometheus Remote Write](../pipeline/inputs/prometheus-remote-write.md)
- [Splunk (HTTP HEC)](../pipeline/inputs/splunk.md)
- [Syslog](../pipeline/inputs/syslog.md)
- [TCP](../pipeline/inputs/tcp.md)

In addition, other plugins implement a sub-set of TLS support, with
restricted configuration:

- [Kubernetes Filter](../pipeline/filters/kubernetes.md)

## Example: enable TLS on HTTP input
By default HTTP input plugin uses plain TCP, enabling TLS from the command line can be done with:

```text
By default HTTP input plugin uses plain TCP, enabling TLS from the command line can
be done using the following command:

```bash
./bin/fluent-bit -i http \
-p port=9999 \
-p tls=on \
Expand All @@ -90,11 +98,13 @@ By default HTTP input plugin uses plain TCP, enabling TLS from the command line
-m '*'
```

In the command line above, the two properties _tls_ and _tls.verify_ where enabled for demonstration purposes \(we strongly suggest always keep verification ON\).
In the previous command, the two properties `tls` and `tls.verify` are set
for demonstration purposes. It's strongly suggested to always keep verification
on.

The same behavior can be accomplished using a configuration file:

```text
```python
[INPUT]
name http
port 9999
Expand All @@ -110,20 +120,22 @@ The same behavior can be accomplished using a configuration file:

## Example: enable TLS on HTTP output

By default HTTP output plugin uses plain TCP, enabling TLS from the command line can be done with:
By default HTTP output plugin uses plain TCP, enabling TLS from the command line can
be done with:

```text
$ fluent-bit -i cpu -t cpu -o http://192.168.2.3:80/something \
```bash
fluent-bit -i cpu -t cpu -o http://192.168.2.3:80/something \
-p tls=on \
-p tls.verify=off \
-m '*'
```

In the command line above, the two properties _tls_ and _tls.verify_ where enabled for demonstration purposes \(we strongly suggest always keep verification ON\).
In the previous command, the properties `tls` and `tls.verify` are enabled
for demonstration purposes. Verification should always be on.

The same behavior can be accomplished using a configuration file:

```text
```python
[INPUT]
Name cpu
Tag cpu
Expand All @@ -140,11 +152,14 @@ The same behavior can be accomplished using a configuration file:

## Tips and Tricks

### Generate your own self signed certificates for testing purposes.
### Generate a self signed certificates for testing purposes

This will generate a 4096 bit RSA key pair and a certificate that is signed using SHA-256 with the expiration date set to 30 days in the future, `test.host.net` set as common name and since we opted out of `DES` the private key will be stored in plain text.
This will generate a 4096 bit RSA key pair and a certificate that's signed using
`SHA-256` with the expiration date set to 30 days in the future. In this example,
`test.host.net` set as common name. This example opts out of `DES`, so the private
key will be stored in plain text.

```
```bash
openssl req -x509 \
-newkey rsa:4096 \
-sha256 \
Expand All @@ -156,9 +171,12 @@ openssl req -x509 \

### Connect to virtual servers using TLS

Fluent Bit supports [TLS server name indication](https://en.wikipedia.org/wiki/Server_Name_Indication). If you are serving multiple hostnames on a single IP address \(a.k.a. virtual hosting\), you can make use of `tls.vhost` to connect to a specific hostname.
Fluent Bit supports
[TLS server name indication](https://en.wikipedia.org/wiki/Server_Name_Indication).
If you are serving multiple host names on a single IP address (for example, using
virtual hosting), you can make use of `tls.vhost` to connect to a specific hostname.

```text
```python
[INPUT]
Name cpu
Tag cpu
Expand All @@ -174,22 +192,23 @@ Fluent Bit supports [TLS server name indication](https://en.wikipedia.org/wiki/S
tls.vhost fluent.example.com
```

### Verify subjectAltName
### Verify `subjectAltName`

By default, TLS verification of hostnames is not done automatically.
As an example, we can extract the X509v3 Subject Alternative Name from a certificate:
By default, TLS verification of host names isn't done automatically.
As an example, you can extract the X509v3 Subject Alternative Name from a certificate:

```
```text
X509v3 Subject Alternative Name:
DNS:my.fluent-aggregator.net
```

As you can see, this certificate covers only `my.fluent-aggregator.net` so if we use a different hostname it should fail.

To fully verify the alternative name and demonstrate the failure we enable `tls.verify_hostname`:
This certificate covers only `my.fluent-aggregator.net` so if you use a different
hostname it should fail.

To fully verify the alternative name and demonstrate the failure, enable
`tls.verify_hostname`:

```text
```python
[INPUT]
Name cpu
Tag cpu
Expand All @@ -205,9 +224,9 @@ To fully verify the alternative name and demonstrate the failure we enable `tls.
tls.ca_file /path/to/fluent-x509v3-alt-name.crt
```

This outgoing connect will be failed and disconnected:
This outgoing connect will fail and disconnect:

```
```text
[2024/06/17 16:51:31] [error] [tls] error: unexpected EOF with reason: certificate verify failed
[2024/06/17 16:51:31] [debug] [upstream] connection #50 failed to other.fluent-aggregator.net:24224
[2024/06/17 16:51:31] [error] [output:forward:forward.0] no upstream connections available
Expand Down
Loading