Skip to content

Update the topic description of the Kafka output docs to fix some errors #3356

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
+25 −15
Open

Update the topic description of the Kafka output docs to fix some errors #3356

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6a3da73
Update the topic description of the Kafka output docs
vishaangelova Oct 6, 2025
8a88e76
Merge branch 'main' into fix-kafka-output-doc-fleet
vishaangelova Oct 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions reference/fleet/kafka-output-settings.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,27 +127,27 @@ The number of partitions created is set automatically by the Kafka broker based
Use this option to set the Kafka topic for each {{agent}} event.

**Default topic** $$$kafka-output-topics-default$$$
: Set a default topic to use for events sent by {{agent}} to the Kafka output.
: Set the default Kafka topic used for events sent by {{agent}}.

You can set a static topic, for example `elastic-agent`, or you can choose to set a topic dynamically based on an [Elastic Common Schema (ECS)](ecs://reference/index.md) field. Available fields include:

* `data_stream.type`
* `data_stream.dataset`
* `data_stream.namespace`
* `@timestamp`
* `event-dataset`
* `event.dataset`

You can also set a custom field. This is useful if you need to construct a more complex or structured topic name.
You can also set a custom field. This is useful if you need to construct a more complex or structured topic name. For example, you can use the `fields.kafka_topic` custom field to set a dynamic topic for each event.
Copy link
Contributor Author

@vishaangelova vishaangelova Oct 7, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the Kafka output doc under “Manage Elastic Agents in Fleet”. The description here needs to match what the user enters in the field on the UI, under Kafka output > Topics > Default topic > Dynamic Topic > Topic from field(s)

I wasn’t sure whether the user needs to wrap the custom field in ${} or %{[]} on the UI, but seeing that the dropdown list contained the ECS fields without these characters, I used the plain fields.kafka_topic here. Is that a correct assumption?


To set a dynamic topic value for outputting {{agent}} data to Kafka, you can add the [`add_fields` processor](/reference/fleet/add_fields-processor.md) to any integration policies on your {{fleet}}-managed {{agents}}.

For example, the following `add_fields` processor creates a dynamic topic value by interpolating multiple [data stream fields](ecs://reference/ecs-data_stream.md):
For example, the following `add_fields` processor creates a dynamic topic value for the `fields.kafka_topic` field by interpolating multiple [data stream fields](ecs://reference/ecs-data_stream.md):

```yaml
- add_fields:
target: ''
fields:
kafka_topic: '${data_stream.type}-${data_stream.dataset}-${data_stream.namespace}' <1>
target: ''
fields:
kafka_topic: '${data_stream.type}-${data_stream.dataset}-${data_stream.namespace}' <1>
```
1. Depending on the values of the data stream fields, this generates topic names such as `logs-nginx.access-production` or `metrics-system.cpu-staging` as the value of the custom `kafka_topic` field.

Expand Down
26 changes: 18 additions & 8 deletions reference/fleet/kafka-output.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -164,25 +164,35 @@ Use these options to set the Kafka topic for each {{agent}} event.
`topic` $$$kafka-topic-setting$$$
: The default Kafka topic used for produced events.

You can set a static topic, for example `elastic-agent`, or you can choose to set a topic dynamically based on an [Elastic Common Schema (ECS)](ecs://reference/index.md) field. Available fields include:
You can set a static topic, for example `elastic-agent`, or you can use a format string to set a topic dynamically based on an [Elastic Common Schema (ECS)](ecs://reference/index.md) field. Available fields include:

* `data_stream.type`
* `data_stream.dataset`
* `data_stream.namespace`
* `@timestamp`
* `event-dataset`
* `event.dataset`

For example:

```yaml
topic: '${data_stream.type}'
Copy link
Contributor Author

@vishaangelova vishaangelova Oct 7, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the Kafka output doc under “Configure standalone Elastic Agents”. I assume the topic settings need to be added in the elastic-agent.yml. Is it correct to provide the substitution variable wrapped in ${} (and not %{[]})?

```

You can also set a custom field. This is useful if you need to construct a more complex or structured topic name.
You can also set a custom field. This is useful if you need to construct a more complex or structured topic name. For example, this configuration uses the `fields.kafka_topic` custom field to set the topic for each event:

```yaml
topic: '${fields.kafka_topic}'
```

To set a dynamic topic value for outputting {{agent}} data to Kafka, you can add the [`add_fields` processor](/reference/fleet/add_fields-processor.md) to the input configuration settings of your standalone {{agent}}.
To set a dynamic topic value for outputting {{agent}} data to Kafka, you can add the [`add_fields` processor](/reference/fleet/add_fields-processor.md) to the input configuration settings of your standalone {{agent}}.

For example, the following `add_fields` processor creates a dynamic topic value by interpolating multiple [data stream fields](ecs://reference/ecs-data_stream.md):
For example, the following `add_fields` processor creates a dynamic topic value for the `fields.kafka_topic` field by interpolating multiple [data stream fields](ecs://reference/ecs-data_stream.md):

```yaml
- add_fields:
target: ''
fields:
kafka_topic: '${data_stream.type}-${data_stream.dataset}-${data_stream.namespace}' <1>
target: ''
fields:
kafka_topic: '${data_stream.type}-${data_stream.dataset}-${data_stream.namespace}' <1>
```
1. Depending on the values of the data stream fields, this generates topic names such as `logs-nginx.access-production` or `metrics-system.cpu-staging` as the value of the custom `kafka_topic` field.

Expand Down
Loading