You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: pipeline/outputs/kafka.md
+66-70Lines changed: 66 additions & 70 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,42 +1,44 @@
1
1
# Kafka Producer
2
2
3
-
Kafka output plugin, producer, allows to ingest your records into an [Apache Kafka](https://kafka.apache.org/) service. This plugin use the official [librdkafka C library](https://github.com/edenhill/librdkafka)\(built-in dependency\)
3
+
The _Kafka Producer_output plugin lets you ingest your records into an [Apache Kafka](https://kafka.apache.org/) service. This plugin uses the official [librdkafka C library](https://github.com/edenhill/librdkafka).
4
4
5
-
Starting with version 4.0.4, the Kafka input plugin supports authentication with AWS MSK IAM, enabling integration with Amazon MSK (Managed Streaming for Apache Kafka) clusters that require IAM-based access.
5
+
In Fluent Bit 4.0.4 and later, the Kafka input plugin supports authentication with AWS MSK IAM, enabling integration with Amazon MSK (Managed Streaming for Apache Kafka) clusters that require IAM-based access.
6
6
7
-
## Configuration Parameters
7
+
## Configuration parameters
8
8
9
-
| Key | Description | default |
9
+
This plugin supports the following parameters:
10
+
11
+
| Key | Description | Default |
10
12
| :--- | :--- | :--- |
11
-
|'format'| Specify data format, options available: json, msgpack, raw. | json |
12
-
|'message_key'| Optional key to store the message ||
13
-
|'message_key_field'| If set, the value of Message_Key_Field in the record will indicate the message key. If not set nor found in the record, Message_Key will be used (if set). ||
14
-
|'timestamp_key'| Set the key to store the record timestamp |@timestamp|
15
-
|'timestamp_format'| Specify timestamp format, should be 'double', '[iso8601](https://en.wikipedia.org/wiki/ISO_8601)' (seconds precision) or 'iso8601_ns' (fractional seconds precision) | double |
16
-
|'brokers'| Single or multiple list of Kafka Brokers, e.g: 192.168.1.3:9092, 192.168.1.4:9092. ||
17
-
|'topics'| Single entry or list of topics separated by comma (,) that Fluent Bit will use to send messages to Kafka. If only one topic is set, that one will be used for all records. Instead if multiple topics exists, the one set in the record by Topic_Key will be used. | fluent-bit |
18
-
|'topic_key'| If multiple Topics exists, the value of Topic_Key in the record will indicate the topic to use. E.g: if Topic_Key is _router_ and the record is {"key1": 123, "router": "route_2"}, Fluent Bit will use topic _route_2_. Note that if the value of Topic_Key is not present in Topics, then by default the first topic in the Topics list will indicate the topic to be used. ||
19
-
|'dynamic_topic'|adds unknown topics (found in Topic_Key) to Topics. So in Topics only a default topic needs to be configured | Off |
20
-
|'queue_full_retries'| Fluent Bit queues data into rdkafka library, if for some reason the underlying library cannot flush the records the queue might fills up blocking new addition of records. The `queue_full_retries`option set the number of local retries to enqueue the data. The default value is 10 times, the interval between each retry is 1 second. Setting the `queue_full_retries` value to `0`set's an unlimited number of retries. |10|
21
-
|'rdkafka.{property}'|'{property}' can be any [librdkafka properties](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md)||
22
-
|'raw_log_key'| When using the raw format and set, the value of raw_log_key in the record will be send to kafka as the payload. ||
23
-
|'workers'| The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. |`0`|
13
+
|`format`| Specify data format. Available formats: `json`, `msgpack`, `raw`. |`json`|
14
+
|`message_key`| Optional key to store the message.|_none_|
15
+
|`message_key_field`| If set, the value of `message_key_field` in the record will indicate the message key. If not set nor found in the record, `message_key` will be used if set. |_none_|
16
+
|`timestamp_key`| Set the key to store the record timestamp |`@timestamp`|
|`brokers`| Single or multiple list of Kafka Brokers. For example, `192.168.1.3:9092`, `192.168.1.4:9092`. |_none_|
19
+
|`topics`| Single entry or list of topics separated by comma (,) that Fluent Bit will use to send messages to Kafka. If only one topic is set, that one will be used for all records. Instead if multiple topics exists, the one set in the record by `Topic_Key` will be used. |`fluent-bit`|
20
+
|`topic_key`| If multiple `topics` exist, the value of `Topic_Key` in the record will indicate the topic to use. For example, if `Topic_Key` is `router` and the record is `{"key1": 123, "router": "route_2"}`, Fluent Bit will use `topic _route_2_`. If the value of `Topic_Key` isn't present in `topics`, then the first topic in the `topics` list will indicate the topic to be used. |_none_|
21
+
|`dynamic_topic`|Adds unknown topics (found in `Topic_Key`) to `topics`. In `topics`, only a default topic needs to be configured.|`Off`|
22
+
|`queue_full_retries`| Fluent Bit queues data into `rdkafka` library. If the underlying library can't flush the records the queue might fill up, blocking new addition of records. `queue_full_retries`sets the number of local retries to enqueue the data. The interval between retries is 1 second. Setting the `queue_full_retries` value to `0`sets an unlimited number of retries. |`10`|
23
+
|`rdkafka.{property}`|`{property}` can be any [librdkafka properties](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md)||
24
+
|`raw_log_key`| When using the raw format and set, the value of `raw_log_key` in the record will be send to Kafka as the payload. |_none_|
25
+
|`workers`| The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. |`0`|
24
26
25
-
> Setting `rdkafka.log.connection.close` to `false` and `rdkafka.request.required.acks` to 1 are examples of recommended settings of librdfkafka properties.
27
+
Setting `rdkafka.log.connection.close` to `false` and `rdkafka.request.required.acks` to `1` are examples of recommended settings of `librdfkafka` properties.
26
28
27
-
## Getting Started
29
+
## Get started
28
30
29
-
In order to insert records into Apache Kafka, you can run the plugin from the command line or through the configuration file:
31
+
To insert records into Apache Kafka, you can run the plugin from the command line or through the configuration file.
30
32
31
-
### Command Line
33
+
### Command line
32
34
33
-
The **kafka** plugin can read parameters through the **-p** argument \(property\), e.g:
35
+
The Kafka plugin can read parameters through the `-p` argument (property):
34
36
35
37
```shell
36
38
fluent-bit -i cpu -o kafka -p brokers=192.168.1.3:9092 -p topics=test
37
39
```
38
40
39
-
### Configuration File
41
+
### Configuration file
40
42
41
43
In your main configuration file append the following:
42
44
@@ -47,7 +49,7 @@ In your main configuration file append the following:
47
49
pipeline:
48
50
inputs:
49
51
- name: cpu
50
-
52
+
51
53
outputs:
52
54
- name: kafka
53
55
match: '*'
@@ -72,29 +74,23 @@ pipeline:
72
74
{% endtab %}
73
75
{% endtabs %}
74
76
75
-
### Avro Support
77
+
### Avro support
76
78
77
-
Fluent-bit comes with support for avro encoding for the out_kafka plugin.
78
-
Avro support is optional and must be activated at build-time by using a
79
-
build def with cmake: `-DFLB_AVRO_ENCODER=On` such as in the following
80
-
example which activates:
79
+
Fluent Bit comes with support for Avro encoding for the `out_kafka` plugin. Avro support is optional and must be activated at build time by using a build def with `cmake`: `-DFLB_AVRO_ENCODER=On` such as in the following example which activates:
This is example fluent-bit config tails kubernetes logs, decorates the
95
-
log lines with kubernetes metadata via the kubernetes filter, and then
96
-
sends the fully decorated log lines to a kafka broker encoded with a
97
-
specific avro schema.
93
+
In this example, the Fluent Bit configuration tails Kubernetes logs, updates the log lines with Kubernetes metadata using the Kubernetes filter. It then sends the updated log lines to a Kafka broker encoded with a specific Avro schema.
This example Fluent Bit configuration file creates example records with the
198
-
_payloadkey_ and _msgkey_ keys. The _msgkey_ value is used as the Kafka message
199
-
key, and the _payloadkey_ value as the payload.
193
+
This example Fluent Bit configuration file creates example records with the `payloadkey` and `msgkey` keys. The `msgkey` value is used as the Kafka message key, and the `payloadkey` value as the payload.
200
194
201
195
{% tabs %}
202
196
{% tab title="fluent-bit.yaml" %}
@@ -207,7 +201,7 @@ pipeline:
207
201
- name: dummy
208
202
tag: example.data
209
203
dummy: '{"payloadkey":"Data to send to kafka", "msgkey": "Key to use in the message"}'
210
-
204
+
211
205
outputs:
212
206
- name: kafka
213
207
match: '*'
@@ -241,39 +235,41 @@ pipeline:
241
235
{% endtab %}
242
236
{% endtabs %}
243
237
244
-
## AWS MSK IAM Authentication
238
+
## AWS MSK IAM authentication
245
239
246
-
*Available since Fluent Bit v4.0.4*
247
-
248
-
Fluent Bit supports authentication to Amazon MSK (Managed Streaming for Apache Kafka) clusters using AWS IAM for the Kafka output plugin. This allows you to securely send data to MSK brokers with AWS credentials, leveraging IAM roles and policies for access control.
240
+
Fluent Bit 4.0.4 and later supports authentication to Amazon MSK (Managed Streaming for Apache Kafka) clusters using AWS IAM for the Kafka output plugin. This lets you securely send data to MSK brokers with AWS credentials, leveraging IAM roles and policies for access control.
249
241
250
242
### Prerequisites
251
243
252
-
**Build Requirements**
253
244
If you are compiling Fluent Bit from source, ensure the following requirements are met to enable AWS MSK IAM support:
254
245
255
-
- The packages `libsasl2` and `libsasl2-dev` must be installed on your build environment.
246
+
- Build Requirements
247
+
248
+
The packages `libsasl2` and `libsasl2-dev` must be installed on your build environment.
249
+
250
+
- Runtime Requirements:
256
251
257
-
**Runtime Requirements**
258
-
-**Network Access:** Fluent Bit must be able to reach your MSK broker endpoints (AWS VPC setup).
259
-
-**AWS Credentials:** Provide credentials using any supported AWS method:
|`aws_msk_iam_cluster_arn`| Full ARN of the MSK cluster for region extraction| String | Yes (if `aws_msk_iam` is true)|
269
+
|`aws_msk_iam`|Optional. Enable AWS MSK IAM authentication.| Boolean |`false`|
270
+
|`aws_msk_iam_cluster_arn`| Full ARN of the MSK cluster for region extraction. Required if `aws_msk_iam` is set. | String |_none_|
275
271
276
-
### Configuration Example
272
+
### Configuration example
277
273
278
274
279
275
{% tabs %}
@@ -296,9 +292,9 @@ pipeline:
296
292
{% endtab %}
297
293
{% endtabs %}
298
294
299
-
### Example AWS IAM Policy
295
+
### AWS IAM policy
300
296
301
-
> **Note:** IAM policies and permissions can be complex and may vary depending on your organization's security requirements. If you are unsure about the correct permissions or best practices, please consult with your AWS administrator or an AWS expert who is familiar with MSK and IAM security.
297
+
IAM policies and permissions can be complex and can vary depending on your organization's security requirements. If you are unsure about the correct permissions or best practices, consult with your AWS administrator or an AWS expert who is familiar with MSK and IAM security.
302
298
303
299
The AWS credentials used by Fluent Bit must have permission to connect to your MSK cluster. Here is a minimal example policy:
304
300
@@ -320,4 +316,4 @@ The AWS credentials used by Fluent Bit must have permission to connect to your M
0 commit comments