You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: pipeline/outputs/kafka.md
+69-70Lines changed: 69 additions & 70 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,42 +1,44 @@
1
1
# Kafka Producer
2
2
3
-
Kafka output plugin, producer, allows to ingest your records into an [Apache Kafka](https://kafka.apache.org/) service. This plugin use the official [librdkafka C library](https://github.com/edenhill/librdkafka)\(built-in dependency\)
3
+
The _Kafka Producer_output plugin lets you ingest your records into an [Apache Kafka](https://kafka.apache.org/) service. This plugin uses the official [librdkafka C library](https://github.com/edenhill/librdkafka).
4
4
5
-
Starting with version 4.0.4, the Kafka input plugin supports authentication with AWS MSK IAM, enabling integration with Amazon MSK (Managed Streaming for Apache Kafka) clusters that require IAM-based access.
5
+
In Fluent Bit 4.0.4 and later, the Kafka input plugin supports authentication with AWS MSK IAM, enabling integration with Amazon MSK (Managed Streaming for Apache Kafka) clusters that require IAM-based access.
6
6
7
-
## Configuration Parameters
7
+
## Configuration parameters
8
8
9
-
| Key | Description | default |
9
+
This plugin supports the following parameters:
10
+
11
+
| Key | Description | Default |
10
12
| :--- | :--- | :--- |
11
-
|'format'| Specify data format, options available: json, msgpack, raw. | json |
12
-
|'message_key'| Optional key to store the message ||
13
-
|'message_key_field'| If set, the value of Message_Key_Field in the record will indicate the message key. If not set nor found in the record, Message_Key will be used (if set). ||
14
-
|'timestamp_key'| Set the key to store the record timestamp |@timestamp|
15
-
|'timestamp_format'| Specify timestamp format, should be 'double', '[iso8601](https://en.wikipedia.org/wiki/ISO_8601)' (seconds precision) or 'iso8601_ns' (fractional seconds precision) | double |
16
-
|'brokers'| Single or multiple list of Kafka Brokers, e.g: 192.168.1.3:9092, 192.168.1.4:9092. ||
17
-
|'topics'| Single entry or list of topics separated by comma (,) that Fluent Bit will use to send messages to Kafka. If only one topic is set, that one will be used for all records. Instead if multiple topics exists, the one set in the record by Topic_Key will be used. | fluent-bit |
18
-
|'topic_key'| If multiple Topics exists, the value of Topic_Key in the record will indicate the topic to use. E.g: if Topic_Key is _router_ and the record is {"key1": 123, "router": "route_2"}, Fluent Bit will use topic _route_2_. Note that if the value of Topic_Key is not present in Topics, then by default the first topic in the Topics list will indicate the topic to be used. ||
19
-
|'dynamic_topic'|adds unknown topics (found in Topic_Key) to Topics. So in Topics only a default topic needs to be configured | Off |
20
-
|'queue_full_retries'| Fluent Bit queues data into rdkafka library, if for some reason the underlying library cannot flush the records the queue might fills up blocking new addition of records. The `queue_full_retries`option set the number of local retries to enqueue the data. The default value is 10 times, the interval between each retry is 1 second. Setting the `queue_full_retries` value to `0`set's an unlimited number of retries. |10|
21
-
|'rdkafka.{property}'|'{property}' can be any [librdkafka properties](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md)||
22
-
|'raw_log_key'| When using the raw format and set, the value of raw_log_key in the record will be send to kafka as the payload. ||
23
-
|'workers'| The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. |`0`|
13
+
|`format`| Specify data format. Available formats: `json`, `msgpack`, `raw`. |`json`|
14
+
|`message_key`| Optional key to store the message |_none_|
15
+
|`message_key_field`| If set, the value of `message_key_field` in the record will indicate the message key. If not set nor found in the record, `message_key` will be used if set. |_none_|
16
+
|`timestamp_key`| Set the key to store the record timestamp |`@timestamp`|
|`brokers`| Single or multiple list of Kafka Brokers. For example, `192.168.1.3:9092`, `192.168.1.4:9092`. |_none_|
19
+
|`topics`| Single entry or list of topics separated by comma (,) that Fluent Bit will use to send messages to Kafka. If only one topic is set, that one will be used for all records. Instead if multiple topics exists, the one set in the record by `Topic_Key` will be used. | fluent-bit |
20
+
|`topic_key`| If multiple `topics` exist, the value of `Topic_Key` in the record will indicate the topic to use. For example, if `Topic_Key` is `router` and the record is `{"key1": 123, "router": "route_2"}`, Fluent Bit will use `topic _route_2_`. If the value of `Topic_Key` isn't present in `topics`, then the first topic in the `topics` list will indicate the topic to be used. |_none_|
21
+
|`dynamic_topic`|Adds unknown topics (found in `Topic_Key`) to `topics`. In `topics`, only a default topic needs to be configured.|`Off`|
22
+
|`queue_full_retries`| Fluent Bit queues data into `rdkafka` library. If the underlying library can't flush the records the queue might fill up, blocking new addition of records. `queue_full_retries`sets the number of local retries to enqueue the data. The interval between retries is 1 second. Setting the `queue_full_retries` value to `0`sets an unlimited number of retries. |`10`|
23
+
|`rdkafka.{property}`|`{property}` can be any [librdkafka properties](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md)||
24
+
|`raw_log_key`| When using the raw format and set, the value of `raw_log_key` in the record will be send to Kafka as the payload. |_none_|
25
+
|`workers`| The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. |`0`|
24
26
25
-
> Setting `rdkafka.log.connection.close` to `false` and `rdkafka.request.required.acks` to 1 are examples of recommended settings of librdfkafka properties.
27
+
Setting `rdkafka.log.connection.close` to `false` and `rdkafka.request.required.acks` to `1` are examples of recommended settings of `librdfkafka` properties.
26
28
27
-
## Getting Started
29
+
## Get started
28
30
29
-
In order to insert records into Apache Kafka, you can run the plugin from the command line or through the configuration file:
31
+
To insert records into Apache Kafka, you can run the plugin from the command line or through the configuration file.
30
32
31
-
### Command Line
33
+
### Command line
32
34
33
-
The **kafka** plugin can read parameters through the **-p** argument \(property\), e.g:
35
+
The Kafka plugin can read parameters through the `-p` argument (property):
34
36
35
37
```shell
36
38
fluent-bit -i cpu -o kafka -p brokers=192.168.1.3:9092 -p topics=test
37
39
```
38
40
39
-
### Configuration File
41
+
### Configuration file
40
42
41
43
In your main configuration file append the following:
42
44
@@ -47,7 +49,7 @@ In your main configuration file append the following:
47
49
pipeline:
48
50
inputs:
49
51
- name: cpu
50
-
52
+
51
53
outputs:
52
54
- name: kafka
53
55
match: '*'
@@ -72,29 +74,26 @@ pipeline:
72
74
{% endtab %}
73
75
{% endtabs %}
74
76
75
-
### Avro Support
77
+
### Avro support
76
78
77
-
Fluent-bit comes with support for avro encoding for the out_kafka plugin.
78
-
Avro support is optional and must be activated at build-time by using a
79
-
build def with cmake: `-DFLB_AVRO_ENCODER=On` such as in the following
79
+
Fluent Bit comes with support for Avro encoding for the `out_kafka` plugin.
80
+
Avro support is optional and must be activated at buildtime by using a
81
+
build def with `cmake`: `-DFLB_AVRO_ENCODER=On` such as in the following
This is example fluent-bit config tails kubernetes logs, decorates the
95
-
log lines with kubernetes metadata via the kubernetes filter, and then
96
-
sends the fully decorated log lines to a kafka broker encoded with a
97
-
specific avro schema.
96
+
In this example, the Fluent Bit configuration tails Kubernetes logs, updates the log lines with Kubernetes metadata using the Kubernetes filter. It then sends the updated log lines to a Kafka broker encoded with a specific Avro schema.
This example Fluent Bit configuration file creates example records with the
198
-
_payloadkey_ and _msgkey_ keys. The _msgkey_ value is used as the Kafka message
199
-
key, and the _payloadkey_ value as the payload.
196
+
This example Fluent Bit configuration file creates example records with the `payloadkey` and `msgkey` keys. The `msgkey` value is used as the Kafka message key, and the `payloadkey` value as the payload.
200
197
201
198
{% tabs %}
202
199
{% tab title="fluent-bit.yaml" %}
@@ -207,7 +204,7 @@ pipeline:
207
204
- name: dummy
208
205
tag: example.data
209
206
dummy: '{"payloadkey":"Data to send to kafka", "msgkey": "Key to use in the message"}'
210
-
207
+
211
208
outputs:
212
209
- name: kafka
213
210
match: '*'
@@ -241,39 +238,41 @@ pipeline:
241
238
{% endtab %}
242
239
{% endtabs %}
243
240
244
-
## AWS MSK IAM Authentication
241
+
## AWS MSK IAM authentication
245
242
246
-
*Available since Fluent Bit v4.0.4*
247
-
248
-
Fluent Bit supports authentication to Amazon MSK (Managed Streaming for Apache Kafka) clusters using AWS IAM for the Kafka output plugin. This allows you to securely send data to MSK brokers with AWS credentials, leveraging IAM roles and policies for access control.
243
+
Fluent Bit 4.0.4 and later supports authentication to Amazon MSK (Managed Streaming for Apache Kafka) clusters using AWS IAM for the Kafka output plugin. This lets you securely send data to MSK brokers with AWS credentials, leveraging IAM roles and policies for access control.
249
244
250
245
### Prerequisites
251
246
252
-
**Build Requirements**
253
-
If you are compiling Fluent Bit from source, ensure the following requirements are met to enable AWS MSK IAM support:
247
+
If you are compiling Fluent Bit from source, ensure the following requirements are met to enable AWS MSK IAM support:
248
+
249
+
- Build Requirements
250
+
251
+
The packages `libsasl2` and `libsasl2-dev` must be installed on your build environment.
254
252
255
-
-The packages `libsasl2` and `libsasl2-dev` must be installed on your build environment.
253
+
-Runtime Requirements:
256
254
257
-
**Runtime Requirements**
258
-
-**Network Access:** Fluent Bit must be able to reach your MSK broker endpoints (AWS VPC setup).
259
-
-**AWS Credentials:** Provide credentials using any supported AWS method:
|`aws_msk_iam_cluster_arn`| Full ARN of the MSK cluster for region extraction| String | Yes (if `aws_msk_iam` is true)|
272
+
|`aws_msk_iam`|Optional. Enable AWS MSK IAM authentication.| Boolean |`false`|
273
+
|`aws_msk_iam_cluster_arn`| Full ARN of the MSK cluster for region extraction. Required if `aws_msk_iam` is set. | String |_none_|
275
274
276
-
### Configuration Example
275
+
### Configuration example
277
276
278
277
279
278
{% tabs %}
@@ -296,9 +295,9 @@ pipeline:
296
295
{% endtab %}
297
296
{% endtabs %}
298
297
299
-
### Example AWS IAM Policy
298
+
### AWS IAM policy
300
299
301
-
> **Note:** IAM policies and permissions can be complex and may vary depending on your organization's security requirements. If you are unsure about the correct permissions or best practices, please consult with your AWS administrator or an AWS expert who is familiar with MSK and IAM security.
300
+
IAM policies and permissions can be complex and can vary depending on your organization's security requirements. If you are unsure about the correct permissions or best practices, consult with your AWS administrator or an AWS expert who is familiar with MSK and IAM security.
302
301
303
302
The AWS credentials used by Fluent Bit must have permission to connect to your MSK cluster. Here is a minimal example policy:
304
303
@@ -320,4 +319,4 @@ The AWS credentials used by Fluent Bit must have permission to connect to your M
0 commit comments