Skip to content

Pipeline: output: posgresql: style #1963

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jul 22, 2025
Merged

Pipeline: output: posgresql: style #1963

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
13ec8cf
Pipeline: output: posgresql: style
esmerel Jul 21, 2025
6f53686
Pipeline: output: prom exporter: style
esmerel Jul 21, 2025
b260817
Discard changes to pipeline/outputs/prometheus-exporter.md
esmerel Jul 21, 2025
349577d
Apply suggestions from code review
esmerel Jul 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
120 changes: 61 additions & 59 deletions pipeline/outputs/postgresql.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,80 +1,82 @@
# PostgreSQL

[PostgreSQL](https://www.postgresql.org) is a very popular and versatile open source database management system that supports the SQL language and that is capable of storing both structured and unstructured data, such as JSON objects.
[PostgreSQL](https://www.postgresql.org) is an open source database management system that supports the SQL language and is capable of storing both structured and unstructured data, such as JSON objects.

Given that Fluent Bit is designed to work with JSON objects, the `pgsql` output plugin allows users to send their data to a PostgreSQL database and store it using the `JSONB` type.
Fluent Bit is designed to work with JSON objects, and the `pgsql` output plugin allows users to send their data to a PostgreSQL database and store it using the `JSONB` type.

PostgreSQL 9.4 or higher is required.

## Preliminary steps

According to the parameters you have set in the configuration file, the plugin will create the table defined by the `table` option in the database defined by the `database` option hosted on the server defined by the `host` option. It will use the PostgreSQL user defined by the `user` option, which needs to have the right privileges to create such a table in that database.
According to the parameters set in the configuration file, the plugin creates the table defined by the `table` option in the database defined by the `database` option hosted on the server defined by the `host` option. It will use the PostgreSQL user defined by the `user` option, which must have the correct privileges to create such a table in that database.

{% hint style="info" %}

If you are not familiar with how PostgreSQL's users and grants system works, you might find useful reading the recommended links in the "References" section at the bottom.
If you're not familiar with how PostgreSQL's users and grants system works, the links in the [References](#references) section at might be helpful.

{% endhint %}

A typical installation normally consists of a self-contained database for Fluent Bit in which you can store the output of one or more pipelines. Ultimately, it is your choice to store them in the same table, or in separate tables, or even in separate databases based on several factors, including workload, scalability, data protection and security.
A typical installation normally consists of a self-contained database for Fluent Bit in which you can store the output of one or more pipelines. You can store them in the same table, or in separate tables, or even in separate databases based on several factors, including workload, scalability, data protection and security.

In this example, for the sake of simplicity, we use a single table called `fluentbit` in a database called `fluentbit` that is owned by the user `fluentbit`. Feel free to use different names. Preferably, for security reasons, do not use the `postgres` user \(which has `SUPERUSER` privileges\).
This example uses a single table called `fluentbit` in a database called `fluentbit`, owned by the user `fluentbit`. In your environment, use names appropriate to your needs. For security reasons, don't use the `postgres` user which has `SUPERUSER` privileges.

### Create the `fluentbit` user

Generate a robust random password \(e.g. `pwgen 20 1`\) and store it safely. Then, as `postgres` system user on the server where PostgreSQL is installed, execute:
Generate a robust random password (for example, `pwgen 20 1`) and store it safely. Then, as the `postgres` system user on the server where PostgreSQL is installed, execute:

```shell
createuser -P fluentbit
```

At the prompt, please provide the password that you previously generated.
At the prompt, provide the password that you previously generated.

As a result, the user `fluentbit` without superuser privileges will be created.

If you prefer, instead of the `createuser` application, you can directly use the SQL command [`CREATE USER`](https://www.postgresql.org/docs/current/sql-createuser.html).

### Create the `fluentbit` database

As `postgres` system user, please run:
As `postgres` system user, run:

```shell
createdb -O fluentbit fluentbit
```

This will create a database called `fluentbit` owned by the `fluentbit` user. As a result, the `fluentbit` user will be able to safely create the data table.
This creates a database called `fluentbit` owned by the `fluentbit` user. As a result, the `fluentbit` user will be able to safely create the data table.

Alternatively, you can use the SQL command [`CREATE DATABASE`](https://www.postgresql.org/docs/current/sql-createdatabase.html).

### Connection

Make sure that the `fluentbit` user can connect to the `fluentbit` database on the specified target host. This might require you to properly configure the [`pg_hba.conf`](https://www.postgresql.org/docs/current/auth-pg-hba-conf.html) file.

## Configuration Parameters

| Key | Description | Default |
|:---------------------|:------------------------------------------------------------------------------------------------------------------------------------|:-------------------|
| `Host` | Hostname/IP address of the PostgreSQL instance | - \(127.0.0.1\) |
| `Port` | PostgreSQL port | - \(5432\) |
| `User` | PostgreSQL username | - \(current user\) |
| `Password` | Password of PostgreSQL username | - |
| `Database` | Database name to connect to | - \(current user\) |
| `Table` | Table name where to store data | - |
| `Connection_Options` | Specifies any valid [PostgreSQL connection options](https://www.postgresql.org/docs/devel/libpq-connect.html#LIBPQ-CONNECT-OPTIONS) | - |
| `Timestamp_Key` | Key in the JSON object containing the record timestamp | date |
| `Async` | Define if we will use async or sync connections | false |
| `min_pool_size` | Minimum number of connection in async mode | 1 |
| `max_pool_size` | Maximum amount of connections in async mode | 4 |
| `cockroachdb` | Set to `true` if you will connect the plugin with a CockroachDB | false |
| `workers` | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `0` |
Ensure that the `fluentbit` user can connect to the `fluentbit` database on the specified target host. This might require you to properly configure the [`pg_hba.conf`](https://www.postgresql.org/docs/current/auth-pg-hba-conf.html) file.

## Configuration parameters

This plugin supports the following parameters:

| Key | Description | Default |
|:----|:------------|:-------|
| `Host` | Hostname/IP address of the PostgreSQL instance. | `- (127.0.0.1)`|
| `Port` | PostgreSQL port. | `- (5432)` |
| `User` | PostgreSQL username. | `- (current user)` |
| `Password` | Password of PostgreSQL username. | `-` |
| `Database` | Database name to connect to. | `- (current user)` |
| `Table` | Table name where to store data. | `-` |
| `Connection_Options` | Specifies any valid [PostgreSQL connection options](https://www.postgresql.org/docs/devel/libpq-connect.html#LIBPQ-CONNECT-OPTIONS). | `-` |
| `Timestamp_Key` | Key in the JSON object containing the record timestamp. | `date` |
| `Async` | Define if the plugin will use asynchronous or synchronous connections. | `false` |
| `max_pool_size` | Maximum amount of connections in asynchronous mode. | `4`|
| `min_pool_size` | Minimum number of connection in asynchronous mode. | `1` |
| `cockroachdb` | Set to `true` if you will connect the plugin with a CockroachDB. | `false` |
| `workers` | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `0` |

### Libpq

Fluent Bit relies on [libpq](https://www.postgresql.org/docs/current/libpq.html), the PostgreSQL native client API, written in C language. For this reason, default values might be affected by [environment variables](https://www.postgresql.org/docs/current/libpq-envars.html) and compilation settings. The above table, in brackets, list the most common default values for each connection option.
Fluent Bit relies on [libpq](https://www.postgresql.org/docs/current/libpq.html), the PostgreSQL native client API. For this reason, default values might be affected by [environment variables](https://www.postgresql.org/docs/current/libpq-envars.html) and compilation settings. The previous table lists the most common default values for each connection option.

For security reasons, it is advised to follow the directives included in the [password file](https://www.postgresql.org/docs/current/libpq-pgpass.html) section.
For security reasons, you're advised to follow the directives included in the [password file](https://www.postgresql.org/docs/current/libpq-pgpass.html) section.

## Configuration Example
## Configuration example

In your main configuration file add the following section:

Expand All @@ -83,7 +85,7 @@ In your main configuration file add the following section:

```yaml
pipeline:

outputs:
- name: pgsql
match: '*'
Expand All @@ -94,7 +96,7 @@ pipeline:
database: fluentbit
table: fluentbit
connection_options: '-c statement_timeout=0'
timestamp_key: ts
timestamp_key: ts
```

{% endtab %}
Expand All @@ -121,41 +123,41 @@ pipeline:

The output plugin automatically creates a table with the name specified by the `table` configuration option and made up of the following fields:

* `tag TEXT`
* `time TIMESTAMP WITHOUT TIMEZONE`
* `data JSONB`
- `tag TEXT`
- `time TIMESTAMP WITHOUT TIMEZONE`
- `data JSONB`

As you can see, the timestamp does not contain any information about the time zone, and it is therefore referred to the time zone used by the connection to PostgreSQL \(`timezone` setting\).
The timestamp doesn't contain any information about the time zone, and it's therefore referred to the time zone used by the connection to PostgreSQL (`timezone` setting).

For more information on the `JSONB` data type in PostgreSQL, please refer to the [JSON types](https://www.postgresql.org/docs/current/datatype-json.html) page in the official documentation, where you can find instructions on how to index or query the objects \(including `jsonpath` introduced in PostgreSQL 12\).
For more information on the `JSONB` data type in PostgreSQL, refer to the [JSON types](https://www.postgresql.org/docs/current/datatype-json.html) page in the official documentation. You can find instructions on how to index or query the objects (including `jsonpath` introduced in PostgreSQL 12).

## Scalability

PostgreSQL 10 introduces support for declarative partitioning. In order to improve vertical scalability of the database, you can decide to partition your tables on _time ranges_ \(for example on a monthly basis\). PostgreSQL supports also subpartitions, allowing you to even partition by _hash_ your records \(version 11+\), and default partitions \(version 11+\).
PostgreSQL includes support for declarative partitioning. To improve vertical scalability of the database, you can partition your tables on time ranges, such as on a monthly basis. PostgreSQL supports also sub-partitions, allowing you to partition your records by hash and default partitions.

For more information on horizontal partitioning in PostgreSQL, please refer to the [Table partitioning](https://www.postgresql.org/docs/current/ddl-partitioning.html) page in the official documentation.
For more information on horizontal partitioning in PostgreSQL, refer to the [Table partitioning](https://www.postgresql.org/docs/current/ddl-partitioning.html) page in the official documentation.

If you are starting now, our recommendation at the moment is to choose the latest major version of PostgreSQL.
Choose the latest major version of PostgreSQL if you're starting now.

## There's more ...
## Additional information

PostgreSQL is a really powerful and extensible database engine. More expert users can indeed take advantage of `BEFORE INSERT` triggers on the main table and re-route records on normalised tables, depending on tags and content of the actual JSON objects.
PostgreSQL is a powerful and extensible database engine. More expert users can take advantage of `BEFORE INSERT` triggers on the main table and reroute records on normalised tables, depending on tags and content of the actual JSON objects.

For example, you can use Fluent Bit to send HTTP log records to the _landing_ table defined in the configuration file. This table contains a `BEFORE INSERT` trigger \(a function in `plpgsql` language\) that _normalises_ the content of the JSON object and that inserts the record in another table \(with its own structure and partitioning model\). This kind of triggers allow you to _discard_ the record from the landing table by returning `NULL`.
For example, you can use Fluent Bit to send HTTP log records to the `landing` table defined in the configuration file. This table contains a `BEFORE INSERT` trigger  (a function in `plpgsql` language) that normalizes the content of the JSON object and that inserts the record in another table (with its own structure and partitioning model). This kind of trigger lets you discard the record from the landing table by returning `NULL`.

## References

Here follows a list of useful resources from the PostgreSQL documentation:

* [Database Roles](https://www.postgresql.org/docs/current/user-manag.html)
* [`GRANT`](https://www.postgresql.org/docs/current/sql-grant.html)
* [`CREATE USER`](https://www.postgresql.org/docs/current/sql-createuser.html)
* [`CREATE DATABASE`](https://www.postgresql.org/docs/current/sql-createdatabase.html)
* [The `pg_hba.conf` file](https://www.postgresql.org/docs/current/auth-pg-hba-conf.html)
* [JSON types](https://www.postgresql.org/docs/current/datatype-json.html)
* [Date/Time functions and operators](https://www.postgresql.org/docs/current/functions-datetime.html)
* [Table partitioning](https://www.postgresql.org/docs/current/ddl-partitioning.html)
* [libpq - C API for PostgreSQL](https://www.postgresql.org/docs/current/libpq.html)
* [libpq - Environment variables](https://www.postgresql.org/docs/current/libpq-envars.html)
* [libpq - password file](https://www.postgresql.org/docs/current/libpq-pgpass.html)
* [Trigger functions](https://www.postgresql.org/docs/current/plpgsql-trigger.html)
The following a list of resources from the PostgreSQL documentation:

- [Database Roles](https://www.postgresql.org/docs/current/user-manag.html)
- [`GRANT`](https://www.postgresql.org/docs/current/sql-grant.html)
- [`CREATE USER`](https://www.postgresql.org/docs/current/sql-createuser.html)
- [`CREATE DATABASE`](https://www.postgresql.org/docs/current/sql-createdatabase.html)
- [The `pg_hba.conf` file](https://www.postgresql.org/docs/current/auth-pg-hba-conf.html)
- [JSON types](https://www.postgresql.org/docs/current/datatype-json.html)
- [Date/Time functions and operators](https://www.postgresql.org/docs/current/functions-datetime.html)
- [Table partitioning](https://www.postgresql.org/docs/current/ddl-partitioning.html)
- [libpq - C API for PostgreSQL](https://www.postgresql.org/docs/current/libpq.html)
- [libpq - Environment variables](https://www.postgresql.org/docs/current/libpq-envars.html)
- [libpq - password file](https://www.postgresql.org/docs/current/libpq-pgpass.html)
- [Trigger functions](https://www.postgresql.org/docs/current/plpgsql-trigger.html)
2 changes: 2 additions & 0 deletions vale-styles/FluentBit/Headings.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ exceptions:
- Kinesis
- Kubernetes
- LaunchDarkly
- Libpq
- Linux
- LTSV
- macOS
Expand All @@ -72,6 +73,7 @@ exceptions:
- OpenTelemetry
- Opsgenie
- PagerDuty
- PostgreSQL
- Prometheus
- PromQL
- REPL
Expand Down
2 changes: 2 additions & 0 deletions vale-styles/FluentBit/Spelling-exceptions.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,7 @@ Kube
Kubernetes
Kusto
labelset
libpq
librdkafka
loadgenerator
logfmt
Expand Down Expand Up @@ -134,6 +135,7 @@ performant
persistable
Podman
Postgres
PostgreSQL
PowerShell
prepopulate
preprocessor
Expand Down