milvus external kafka

[rere950303]

• Milvus version: v2.5.27
• Mode: Cluster

We are planning to deploy Milvus using the Milvus Operator. For the message queue, we intend to use an external Kafka cluster.

In our production environment, Kafka topic auto-creation is disabled, so we need to manually create all Milvus-related topics. We have the following questions:

1. What is the full list of topics that need to be created for Milvus?
   • Topic names
   • Required configurations (e.g., number of partitions, replication factor, cleanup policy, etc.)
2. How should we configure the Milvus Operator YAML to connect to an external Kafka cluster where the topics are created manually?

We would appreciate your guidance on this.

Thank you!

[xiaofan-luan]

here is my suggestion:

1. Full List of Kafka Topics

With default settings (cluster prefix by-dev, dmlChannelNum=16), you need to pre-create 17 topics:

DML Channels (16 topics)

These are the primary data channels carrying insert/delete/upsert operations and time-tick messages.

by-dev-rootcoord-dml_0
by-dev-rootcoord-dml_1
by-dev-rootcoord-dml_2
by-dev-rootcoord-dml_3
by-dev-rootcoord-dml_4
by-dev-rootcoord-dml_5
by-dev-rootcoord-dml_6
by-dev-rootcoord-dml_7
by-dev-rootcoord-dml_8
by-dev-rootcoord-dml_9
by-dev-rootcoord-dml_10
by-dev-rootcoord-dml_11
by-dev-rootcoord-dml_12
by-dev-rootcoord-dml_13
by-dev-rootcoord-dml_14
by-dev-rootcoord-dml_15

TimeTick Channel (1 topic)

by-dev-datacoord-timetick-channel

Topic Naming Convention

The format is: {cluster-prefix}-{channel-suffix}[_{index}]

• Cluster prefix: Controlled by msgChannel.chanNamePrefix.cluster (default: by-dev). Change this for production (e.g., prod-milvus).
• DML suffix: rootcoord-dml (controlled by msgChannel.chanNamePrefix.rootCoordDml)
• DML count: rootCoord.dmlChannelNum (default: 16, immutable after first startup)

---

2. Recommended Topic Configuration

Milvus itself does not set any topic-level configurations — it relies on Kafka broker defaults. Here are the recommended settings:

┌─────────────────────────────────┬─────────────────────────┬─────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Parameter │ Recommended Value │ Notes │
├─────────────────────────────────┼─────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Partitions │ 1 │ Milvus requires exactly 1 partition per topic. Multiple partitions will break message ordering. │
├─────────────────────────────────┼─────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Replication factor │ 3 (production) │ Match your Kafka cluster's min.insync.replicas for durability │
├─────────────────────────────────┼─────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Cleanup policy │ delete │ Default Kafka behavior. Do NOT use compact. │
├─────────────────────────────────┼─────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Retention │ 168h (7 days) or longer │ Milvus manages its own offsets; 7 days gives buffer for recovery │
├─────────────────────────────────┼─────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────┤
│ message.max.bytes │ 10485760 (10MB) │ Milvus can send large batch messages │
├─────────────────────────────────┼─────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────┤
│ max.message.bytes (topic-level) │ 10485760 (10MB) │ Must match or exceed producer setting

[jfwm2]

Hello 👋

I am jumping into the conversation because we also have an external kafka with Milvus v2.5.26 and v2.6.10 (we are in the middle of the version upgrade). For all our clusters we have the following topics (see below), which does not include the "TimeTick Channel" (topic <cluster_name>-datacoord-timetick-channel) mentioned above by @xiaofan-luan. Is it something we should worry about?

Thanks

<cluster_name>-replicate-msg
<cluster_name>-rootcoord-dml_0
<cluster_name>-rootcoord-dml_1
<cluster_name>-rootcoord-dml_2
<cluster_name>-rootcoord-dml_3
<cluster_name>-rootcoord-dml_4
<cluster_name>-rootcoord-dml_5
<cluster_name>-rootcoord-dml_6
<cluster_name>-rootcoord-dml_7
<cluster_name>-rootcoord-dml_8
<cluster_name>-rootcoord-dml_9
<cluster_name>-rootcoord-dml_10
<cluster_name>-rootcoord-dml_11
<cluster_name>-rootcoord-dml_12
<cluster_name>-rootcoord-dml_13
<cluster_name>-rootcoord-dml_14
<cluster_name>-rootcoord-dml_15

[xiaofan-luan]

Timetick channle is only for backward compatibility, it is not used any more

[jfwm2]

Thank you for the information @xiaofan-luan

[rere950303]

@xiaofan-luan
Then conversely, in version v2.5.27, is the <cluster_name>-replicate-msg topic not required?

[xiaofan-luan]

only cluster with primary standby will use this topic