You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: reference/fleet/kafka-output-settings.md
+15-2Lines changed: 15 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -131,14 +131,27 @@ Use this option to set the Kafka topic for each {{agent}} event.
131
131
132
132
You can set a static topic, for example `elastic-agent`, or you can choose to set a topic dynamically based on an [Elastic Common Schema (ECS)](ecs://reference/index.md) field. Available fields include:
133
133
134
-
* `data_stream_type`
134
+
* `data_stream.type`
135
135
* `data_stream.dataset`
136
136
* `data_stream.namespace`
137
137
* `@timestamp`
138
138
* `event-dataset`
139
139
140
-
You can also set a custom field. This is useful if you’re using the [`add_fields` processor](/reference/fleet/add_fields-processor.md) as part of your {{agent}} input. Otherwise, setting a custom field is not recommended.
140
+
You can also set a custom field. This is useful if you need to construct a more complex or structured topic name.
141
+
142
+
To set a dynamic topic value for outputting {{agent}} data to Kafka, you can add the [`add_fields` processor](/reference/fleet/add_fields-processor.md) to any integration policies on your Fleet-managed {{agents}}.
143
+
144
+
For example, the following processor creates a dynamic topic value by interpolating multiple [data stream fields](ecs://reference/ecs-data_stream.md):
1. Depending on the values of the data stream fields, this generates topic names such as `logs-system.auth-production` or `metrics-nginx-default` as the value of the custom `kafka_topic` field.
153
+
154
+
For more information, refer to [](/reference/fleet/agent-processors.md).
Copy file name to clipboardExpand all lines: reference/fleet/kafka-output.md
+28-4Lines changed: 28 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -69,10 +69,10 @@ inputs {
69
69
70
70
The `kafka` output supports the following settings, grouped by category. Many of these settings have sensible defaults that allow you to run {{agent}} with minimal configuration.
@@ -157,12 +157,36 @@ This sample configuration forwards events to the output when there are enough ev
157
157
**Default:** `10s`
158
158
159
159
160
-
## Topics settings [output-kafka-topics-settings]
160
+
## Topic settings [output-kafka-topic-settings]
161
161
162
162
Use these options to set the Kafka topic for each {{agent}} event.
163
163
164
164
`topic`$$$kafka-topic-setting$$$
165
-
: The default Kafka topic used for produced events.
165
+
: Set a default topic to use for events sent by {{agent}} to the Kafka output.
166
+
167
+
You can set a static topic, for example `elastic-agent`, or you can choose to set a topic dynamically based on an [Elastic Common Schema (ECS)](ecs://reference/index.md) field. Available fields include:
168
+
169
+
* `data_stream.type`
170
+
* `data_stream.dataset`
171
+
* `data_stream.namespace`
172
+
* `@timestamp`
173
+
* `event-dataset`
174
+
175
+
You can also set a custom field. This is useful if you need to construct a more complex or structured topic name.
176
+
177
+
To set a dynamic topic value for outputting {{agent}} data to Kafka, you can add the [`add_fields` processor](/reference/fleet/add_fields-processor.md) to the input configuration settings of your standalone {{agent}}.
178
+
179
+
For example, the following processor creates a dynamic topic value by interpolating multiple [data stream fields](ecs://reference/ecs-data_stream.md):
1. Depending on the values of the data stream fields, this generates topic names such as `logs-system.auth-production` or `metrics-nginx-default` as the value of the custom `kafka_topic` field.
188
+
189
+
For more information, refer to [](/reference/fleet/agent-processors.md).
0 commit comments