You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Missing YAML examples for the remaining output plugins. Fixes#1941. (#1942)
* Adding YAML examples and cleaning up shell usage for NATS output plugin. Part of issue #1941.
Signed-off-by: Eric D. Schabell <[email protected]>
* Adding YAML examples, formatting table, and cleaning up shell usage for PostgreSQL output plugin. Part of issue #1941.
Signed-off-by: Eric D. Schabell <[email protected]>
* Adding YAML examples, formatting table, and cleaning up shell usage for Prometheus Remote Write output plugin. Part of issue #1941.
Signed-off-by: Eric D. Schabell <[email protected]>
* Adding YAML examples, formatting table, and cleaning up shell usage for Skywalking output plugin. Part of issue #1941.
Signed-off-by: Eric D. Schabell <[email protected]>
* Adding YAML examples, formatting table, and cleaning up shell usage for Prometheus Exporter output plugin. Part of issue #1941.
Signed-off-by: Eric D. Schabell <[email protected]>
* Adding YAML examples, formatting table, and cleaning up shell usage for Slack output plugin. Part of issue #1941.
Signed-off-by: Eric D. Schabell <[email protected]>
* Adding YAML examples, formatting table, and cleaning up shell usage for Splunk output plugin. Part of issue #1941.
Signed-off-by: Eric D. Schabell <[email protected]>
* Adding YAML examples, formatting table, and cleaning up shell usage for Stackdriver output plugin. Part of issue #1941.
Signed-off-by: Eric D. Schabell <[email protected]>
* Adding YAML examples, formatting table, and cleaning up shell usage for Stackdriver Special Fields doc. Part of issue #1941.
Signed-off-by: Eric D. Schabell <[email protected]>
* Adding YAML examples, formatting table, and cleaning up shell usage for Standard Output output plugin doc. Part of issue #1941.
Signed-off-by: Eric D. Schabell <[email protected]>
* Adding YAML examples, formatting table, and cleaning up shell usage for Syslog output plugin doc. Part of issue #1941.
Signed-off-by: Eric D. Schabell <[email protected]>
* Adding YAML examples, formatting table, and cleaning up shell usage for TCP and TLS output plugin doc. Part of issue #1941.
Signed-off-by: Eric D. Schabell <[email protected]>
* Adding YAML examples, formatting table, and cleaning up shell usage for Treasure Data output plugin doc. Part of issue #1941.
Signed-off-by: Eric D. Schabell <[email protected]>
* Adding YAML examples, formatting table, and cleaning up shell usage for Vivo Exporter output plugin doc. Part of issue #1941.
Signed-off-by: Eric D. Schabell <[email protected]>
* Adding YAML examples, formatting table, and cleaning up shell usage for WebSocket output plugin doc. Part of issue #1941.
Signed-off-by: Eric D. Schabell <[email protected]>
---------
Signed-off-by: Eric D. Schabell <[email protected]>
| host | IP address or hostname of the NATS Server| 127.0.0.1 |
10
+
| port | TCP port of the target NATS Server | 4222|
11
+
| workers | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. |`0`|
12
12
13
13
In order to override the default configuration values, the plugin uses the optional Fluent Bit network address format, e.g:
14
14
@@ -20,14 +20,10 @@ nats://host:port
20
20
21
21
[Fluent Bit](http://fluentbit.io) only requires to know that it needs to use the **nats** output plugin, if no extra information is given, it will use the default values specified in the above table.
22
22
23
-
```bash
24
-
$ bin/fluent-bit -i cpu -o nats -V -f 5
25
-
Fluent Bit v1.x.x
26
-
* Copyright (C) 2019-2020 The Fluent Bit Authors
27
-
* Copyright (C) 2015-2018 Treasure Data
28
-
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
29
-
* https://fluentbit.io
23
+
```shell
24
+
$ fluent-bit -i cpu -o nats -V -f 5
30
25
26
+
...
31
27
[2016/03/04 10:17:33] [ info] Configuration
32
28
flush time: 5 seconds
33
29
input plugins : cpu
@@ -46,22 +42,22 @@ As described above, the target service and storage point can be changed, e.g:
46
42
47
43
## Data format
48
44
49
-
For every set of records flushed to a NATS Server, Fluent Bit uses the following JSON format:
45
+
For every set of records flushed to a NATS Server, Fluent Bit uses the following format:
50
46
51
-
```javascript
47
+
```text
52
48
[
53
49
[UNIX_TIMESTAMP, JSON_MAP_1],
54
50
[UNIX_TIMESTAMP, JSON_MAP_2],
55
51
[UNIX_TIMESTAMP, JSON_MAP_N],
56
52
]
57
53
```
58
54
59
-
Each record is an individual entity represented in a JSON array that contains a UNIX\_TIMESTAMP and a JSON map with a set of key/values. A summarized output of the CPU input plugin will looks as this:
55
+
Each record is an individual entity represented in a JSON array that contains a UNIX\_TIMESTAMP and a JSON map with a set of key/values. A summarized output of the CPU input plugin will look as this:
Copy file name to clipboardExpand all lines: pipeline/outputs/postgresql.md
+59-30Lines changed: 59 additions & 30 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -10,7 +10,11 @@ PostgreSQL 9.4 or higher is required.
10
10
11
11
According to the parameters you have set in the configuration file, the plugin will create the table defined by the `table` option in the database defined by the `database` option hosted on the server defined by the `host` option. It will use the PostgreSQL user defined by the `user` option, which needs to have the right privileges to create such a table in that database.
12
12
13
-
> **NOTE:** If you are not familiar with how PostgreSQL's users and grants system works, you might find useful reading the recommended links in the "References" section at the bottom.
13
+
{% hint style="info" %}
14
+
15
+
If you are not familiar with how PostgreSQL's users and grants system works, you might find useful reading the recommended links in the "References" section at the bottom.
16
+
17
+
{% endhint %}
14
18
15
19
A typical installation normally consists of a self-contained database for Fluent Bit in which you can store the output of one or more pipelines. Ultimately, it is your choice to store them in the same table, or in separate tables, or even in separate databases based on several factors, including workload, scalability, data protection and security.
16
20
@@ -20,7 +24,7 @@ In this example, for the sake of simplicity, we use a single table called `fluen
20
24
21
25
Generate a robust random password \(e.g. `pwgen 20 1`\) and store it safely. Then, as `postgres` system user on the server where PostgreSQL is installed, execute:
22
26
23
-
```bash
27
+
```shell
24
28
createuser -P fluentbit
25
29
```
26
30
@@ -34,7 +38,7 @@ If you prefer, instead of the `createuser` application, you can directly use the
34
38
35
39
As `postgres` system user, please run:
36
40
37
-
```bash
41
+
```shell
38
42
createdb -O fluentbit fluentbit
39
43
```
40
44
@@ -48,21 +52,21 @@ Make sure that the `fluentbit` user can connect to the `fluentbit` database on t
48
52
49
53
## Configuration Parameters
50
54
51
-
| Key | Description | Default |
52
-
|:---|:---|:---|
53
-
|`Host`| Hostname/IP address of the PostgreSQL instance | - \(127.0.0.1\)|
|`Host`| Hostname/IP address of the PostgreSQL instance | - \(127.0.0.1\)|
58
+
|`Port`| PostgreSQL port | - \(5432\)|
59
+
|`User`| PostgreSQL username| - \(current user\)|
60
+
|`Password`| Password of PostgreSQL username | -|
61
+
|`Database`| Database name to connect to| - \(current user\)|
62
+
|`Table`| Table name where to store data | -|
63
+
|`Connection_Options`| Specifies any valid [PostgreSQL connection options](https://www.postgresql.org/docs/devel/libpq-connect.html#LIBPQ-CONNECT-OPTIONS)| - |
64
+
|`Timestamp_Key`| Key in the JSON object containing the record timestamp | date|
65
+
|`Async`| Define if we will use async or sync connections | false|
66
+
|`min_pool_size`| Minimum number of connection in async mode | 1|
67
+
|`max_pool_size`| Maximum amount of connections in async mode | 4|
68
+
|`cockroachdb`| Set to `true` if you will connect the plugin with a CockroachDB | false|
69
+
|`workers`| The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. |`0`|
66
70
67
71
### Libpq
68
72
@@ -74,20 +78,45 @@ For security reasons, it is advised to follow the directives included in the [pa
74
78
75
79
In your main configuration file add the following section:
76
80
81
+
{% tabs %}
82
+
{% tab title="fluent-bit.yaml" %}
83
+
84
+
```yaml
85
+
pipeline:
86
+
87
+
outputs:
88
+
- name: pgsql
89
+
match: '*'
90
+
host: 172.17.0.2
91
+
port: 5432
92
+
user: fluentbit
93
+
password: YourCrazySecurePassword
94
+
database: fluentbit
95
+
table: fluentbit
96
+
connection_options: '-c statement_timeout=0'
97
+
timestamp_key: ts
98
+
```
99
+
100
+
{% endtab %}
101
+
{% tab title="fluent-bit.conf" %}
102
+
77
103
```text
78
104
[OUTPUT]
79
-
Name pgsql
80
-
Match *
81
-
Host 172.17.0.2
82
-
Port 5432
83
-
User fluentbit
84
-
Password YourCrazySecurePassword
85
-
Database fluentbit
86
-
Table fluentbit
87
-
Connection_Options -c statement_timeout=0
88
-
Timestamp_Key ts
105
+
Name pgsql
106
+
Match *
107
+
Host 172.17.0.2
108
+
Port 5432
109
+
User fluentbit
110
+
Password YourCrazySecurePassword
111
+
Database fluentbit
112
+
Table fluentbit
113
+
Connection_Options -c statement_timeout=0
114
+
Timestamp_Key ts
89
115
```
90
116
117
+
{% endtab %}
118
+
{% endtabs %}
119
+
91
120
## The output table
92
121
93
122
The output plugin automatically creates a table with the name specified by the `table` configuration option and made up of the following fields:
@@ -96,7 +125,7 @@ The output plugin automatically creates a table with the name specified by the `
96
125
*`time TIMESTAMP WITHOUT TIMEZONE`
97
126
*`data JSONB`
98
127
99
-
As you can see, the timestamp does not contain any information about the time zone and it is therefore referred to the time zone used by the connection to PostgreSQL \(`timezone` setting\).
128
+
As you can see, the timestamp does not contain any information about the time zone, and it is therefore referred to the time zone used by the connection to PostgreSQL \(`timezone` setting\).
100
129
101
130
For more information on the `JSONB` data type in PostgreSQL, please refer to the [JSON types](https://www.postgresql.org/docs/current/datatype-json.html) page in the official documentation, where you can find instructions on how to index or query the objects \(including `jsonpath` introduced in PostgreSQL 12\).
102
131
@@ -129,4 +158,4 @@ Here follows a list of useful resources from the PostgreSQL documentation:
129
158
*[libpq - C API for PostgreSQL](https://www.postgresql.org/docs/current/libpq.html)
| host | This is address Fluent Bit will bind to when hosting prometheus metrics. Note: `listen` parameter is deprecated from v1.9.0.| 0.0.0.0 |
14
+
| port | This is the port Fluent Bit will bind to when hosting prometheus metrics | 2021|
15
+
| add\_label | This allows you to add custom labels to all metrics exposed through the prometheus exporter. You may have multiple of these fields ||
16
+
| workers | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. |`1`|
17
17
18
18
## Getting Started
19
19
20
20
The Prometheus exporter only works with metrics captured from metric plugins. In the following example, host metrics are captured by the node exporter metrics plugin and then are routed to prometheus exporter. Within the output plugin two labels are added `app="fluent-bit"`and `color="blue"`
21
21
22
22
{% tabs %}
23
-
{% tab title="fluent-bit.conf" %}
24
-
```text
23
+
{% tab title="fluent-bit.yaml" %}
24
+
25
+
```yaml
25
26
# Node Exporter Metrics + Prometheus Exporter
26
27
# -------------------------------------------
27
28
# The following example collect host metrics on Linux and expose
@@ -31,28 +32,31 @@ The Prometheus exporter only works with metrics captured from metric plugins. In
31
32
#
32
33
# $ curl http://127.0.0.1:2021/metrics
33
34
#
34
-
[SERVICE]
35
-
flush 1
36
-
log_level info
35
+
service:
36
+
flush:1
37
+
log_level:info
37
38
38
-
[INPUT]
39
-
name node_exporter_metrics
40
-
tag node_metrics
41
-
scrape_interval 2
39
+
pipeline:
40
+
inputs:
41
+
- name: node_exporter_metrics
42
+
tag: node_metrics
43
+
scrape_interval: 2
42
44
43
-
[OUTPUT]
44
-
name prometheus_exporter
45
-
match node_metrics
46
-
host 0.0.0.0
47
-
port 2021
48
-
# add user-defined labels
49
-
add_label app fluent-bit
50
-
add_label color blue
45
+
outputs:
46
+
- name: prometheus_exporter
47
+
match: node_metrics
48
+
host: 0.0.0.0
49
+
port: 2021
50
+
# add user-defined labels
51
+
add_label:
52
+
- app fluent-bit
53
+
- color blue
51
54
```
55
+
52
56
{% endtab %}
57
+
{% tab title="fluent-bit.conf" %}
53
58
54
-
{% tab title="fluent-bit.yaml" %}
55
-
```yaml
59
+
```text
56
60
# Node Exporter Metrics + Prometheus Exporter
57
61
# -------------------------------------------
58
62
# The following example collect host metrics on Linux and expose
@@ -62,23 +66,24 @@ The Prometheus exporter only works with metrics captured from metric plugins. In
0 commit comments