diff --git a/reference/fleet/kafka-output-settings.md b/reference/fleet/kafka-output-settings.md index 9b5685b0c3..20140366cd 100644 --- a/reference/fleet/kafka-output-settings.md +++ b/reference/fleet/kafka-output-settings.md @@ -127,7 +127,7 @@ The number of partitions created is set automatically by the Kafka broker based Use this option to set the Kafka topic for each {{agent}} event. **Default topic** $$$kafka-output-topics-default$$$ -: Set a default topic to use for events sent by {{agent}} to the Kafka output. +: Set the default Kafka topic used for events sent by {{agent}}. You can set a static topic, for example `elastic-agent`, or you can choose to set a topic dynamically based on an [Elastic Common Schema (ECS)](ecs://reference/index.md) field. Available fields include: @@ -135,19 +135,19 @@ Use this option to set the Kafka topic for each {{agent}} event. * `data_stream.dataset` * `data_stream.namespace` * `@timestamp` - * `event-dataset` + * `event.dataset` - You can also set a custom field. This is useful if you need to construct a more complex or structured topic name. + You can also set a custom field. This is useful if you need to construct a more complex or structured topic name. For example, you can use the `fields.kafka_topic` custom field to set a dynamic topic for each event. To set a dynamic topic value for outputting {{agent}} data to Kafka, you can add the [`add_fields` processor](/reference/fleet/add_fields-processor.md) to any integration policies on your {{fleet}}-managed {{agents}}. - For example, the following `add_fields` processor creates a dynamic topic value by interpolating multiple [data stream fields](ecs://reference/ecs-data_stream.md): + For example, the following `add_fields` processor creates a dynamic topic value for the `fields.kafka_topic` field by interpolating multiple [data stream fields](ecs://reference/ecs-data_stream.md): ```yaml - add_fields: - target: '' - fields: - kafka_topic: '${data_stream.type}-${data_stream.dataset}-${data_stream.namespace}' <1> + target: '' + fields: + kafka_topic: '${data_stream.type}-${data_stream.dataset}-${data_stream.namespace}' <1> ``` 1. Depending on the values of the data stream fields, this generates topic names such as `logs-nginx.access-production` or `metrics-system.cpu-staging` as the value of the custom `kafka_topic` field. diff --git a/reference/fleet/kafka-output.md b/reference/fleet/kafka-output.md index fac4c73f59..bb48366fc9 100644 --- a/reference/fleet/kafka-output.md +++ b/reference/fleet/kafka-output.md @@ -164,25 +164,35 @@ Use these options to set the Kafka topic for each {{agent}} event. `topic` $$$kafka-topic-setting$$$ : The default Kafka topic used for produced events. - You can set a static topic, for example `elastic-agent`, or you can choose to set a topic dynamically based on an [Elastic Common Schema (ECS)](ecs://reference/index.md) field. Available fields include: + You can set a static topic, for example `elastic-agent`, or you can use a format string to set a topic dynamically based on an [Elastic Common Schema (ECS)](ecs://reference/index.md) field. Available fields include: * `data_stream.type` * `data_stream.dataset` * `data_stream.namespace` * `@timestamp` - * `event-dataset` + * `event.dataset` + + For example: + + ```yaml + topic: '${data_stream.type}' + ``` - You can also set a custom field. This is useful if you need to construct a more complex or structured topic name. + You can also set a custom field. This is useful if you need to construct a more complex or structured topic name. For example, this configuration uses the `fields.kafka_topic` custom field to set the topic for each event: + + ```yaml + topic: '${fields.kafka_topic}' + ``` - To set a dynamic topic value for outputting {{agent}} data to Kafka, you can add the [`add_fields` processor](/reference/fleet/add_fields-processor.md) to the input configuration settings of your standalone {{agent}}. + To set a dynamic topic value for outputting {{agent}} data to Kafka, you can add the [`add_fields` processor](/reference/fleet/add_fields-processor.md) to the input configuration settings of your standalone {{agent}}. - For example, the following `add_fields` processor creates a dynamic topic value by interpolating multiple [data stream fields](ecs://reference/ecs-data_stream.md): + For example, the following `add_fields` processor creates a dynamic topic value for the `fields.kafka_topic` field by interpolating multiple [data stream fields](ecs://reference/ecs-data_stream.md): ```yaml - add_fields: - target: '' - fields: - kafka_topic: '${data_stream.type}-${data_stream.dataset}-${data_stream.namespace}' <1> + target: '' + fields: + kafka_topic: '${data_stream.type}-${data_stream.dataset}-${data_stream.namespace}' <1> ``` 1. Depending on the values of the data stream fields, this generates topic names such as `logs-nginx.access-production` or `metrics-system.cpu-staging` as the value of the custom `kafka_topic` field.