Potentially redundant heartbeat topics?

[vladimir-tsvetkov-nexo]

I'm testing Mirror Maker 2 in a local setup with docker-compose, where I'm mirroring some topics from a staging environment into local development environment.

docker-compose.yaml

version: '3.8'

services:
    zookeeper:
        image: docker.io/bitnami/zookeeper
        container_name: zoo
        hostname: zookeeper
        ports:
            - '2181:2181'
        environment:
            - ALLOW_ANONYMOUS_LOGIN=yes

    kafka:
        image: docker.io/bitnami/kafka
        container_name: kafka
        ports:
            - '9092:9092'
        environment:
            - KAFKA_ENABLE_KRAFT=no
            - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
            - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=EXTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT
            - KAFKA_CFG_LISTENERS=EXTERNAL://:9092,PLAINTEXT://:29092
            - KAFKA_CFG_ADVERTISED_LISTENERS=EXTERNAL://localhost:9092,PLAINTEXT://kafka:29092
            - KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE=true
            - KAFKA_INTER_BROKER_LISTENER_NAME=PLAINTEXT
            - ALLOW_PLAINTEXT_LISTENER=yes
        depends_on:
            - zookeeper

    kafka-ui:
        image: provectuslabs/kafka-ui
        container_name: kafka-ui
        ports:
            - '8090:8080'
        restart: always
        environment:
            - KAFKA_CLUSTERS_0_NAME=local
            - KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=kafka:29092
            - KAFKA_CLUSTERS_0_ZOOKEEPER=localhost:2181
        depends_on:
            - kafka

    mirror-maker:
        image: confluentinc/cp-kafka
        volumes:
            - ./mirror-maker/mm2.properties:/etc/mm2.properties
            - ./mirror-maker/log4j.properties:/etc/kafka/log4j.properties
            - ./mirror-maker/connect-log4j.properties:/usr/bin/../config/connect-log4j.properties
        command: bash -c "cub kafka-ready -z zookeeper:2181 1 40 && connect-mirror-maker /etc/mm2.properties"
        depends_on:
            - kafka
            - zookeeper

mm2.properties

clusters = STG, LOC

STG.bootstrap.servers=<BOOTSTRAP_SERVERS>
STG.exclude.internal.topics=true
STG.client.id=test_mirror_maker_consumer
STG.group.id=test_mirror_maker_consumer
STG.security.protocol=SASL_SSL
STG.sasl.mechanism=SCRAM-SHA-512
STG.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
  username="<USERNAME>" \
  password="<PASSWORD>";

LOC.bootstrap.servers=kafka:29092
LOC.client.id=test_mirror_maker_producer
LOC.max.in.flight.requests.per.connection=1

STG->LOC.enabled=true

STG->LOC.topics=<SOME_TOPIC>

# Setting replication factor of newly created remote topics
replication.factor=1

offset-syncs.topic.replication.factor=1
checkpoints.topic.replication.factor=1
heartbeats.topic.replication.factor=1

offset.storage.replication.factor=1
status.storage.replication.factor=1
config.storage.replication.factor=1

When I start the setup, I notice that in the target Kafka cluster, there are a lot of additional topics:

$ kafka-topics.sh --bootstrap-server=localhost:29092 --list
STG.checkpoints.internal
STG.heartbeats
STG.kafka-strimzi-mm2.heartbeats
STG.kafka-strimzi-mm2.kafka-strimzi.heartbeats
STG.kafka-strimzi-mm2.kafka-strimzi.kafka-strimzi.heartbeats
STG.kafka-strimzi-mm2.kafka-strimzi.kafka-strimzi.kafka-strimzi.heartbeats
STG.kafka-strimzi-mm2.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.heartbeats
STG.kafka-strimzi-mm2.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.heartbeats
STG.kafka-strimzi-mm2.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.heartbeats
STG.kafka-strimzi-mm2.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.heartbeats
STG.kafka-strimzi-mm2.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.heartbeats
STG.kafka-strimzi-mm2.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.heartbeats
STG.kafka-strimzi.heartbeats
STG.kafka-strimzi.kafka-strimzi.heartbeats
STG.kafka-strimzi.kafka-strimzi.kafka-strimzi.heartbeats
STG.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.heartbeats
STG.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.heartbeats
STG.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.heartbeats
STG.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.heartbeats
STG.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.heartbeats
STG.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.heartbeats
STG.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.heartbeats
STG.mirror.heartbeats
STG.mirror.kafka-strimzi.heartbeats
STG.mirror.kafka-strimzi.kafka-strimzi.heartbeats
STG.mirror.kafka-strimzi.kafka-strimzi.kafka-strimzi.heartbeats
STG.mirror.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.heartbeats
STG.mirror.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.kafka-strimzi.heartbeats
STG.<SOME MIRRORED TOPIC>
__consumer_offsets
heartbeats
mm2-configs.STG.internal
mm2-offsets.STG.internal
mm2-status.STG.internal

Where are this additional topics coming from?
They keep being empty, but recreate upon restart if I delete them.

[scholzj]

I'm not sure where Strimzi fits into this. We deploy and configure Mirror Maker differently as a set of Kafka Connect connectors.

In Strimzi-based deployments, a situation like this happens when you are trying to do active-active mirroring with 2 MM2 instances, but you badly name the aliases of the Kafka clusters in the MM2 configuration. In particular, when the aliases are nto consistent.

For example

• When you have cluster A and cluster B you want to have active-active mirroring between
• In one instance of MM2, you call cluster A source and cluster B target
• In second MM2 instance you call cluster A target and cluster B source

This leads to similar issues where the heartbeat topic is mirrored again and again. As I do not understand your MM2 configuration, I'm not sure what it does, but it could be something similar.

Two more thoughts:

1. I also saw a case where these topics were once some time ago created by MM2 misconfigured as described above but nobody deleted them. And later, a correctly configured MM2 just mirrored them as any other topics. So this is worth checking as well if it doesn't just mirror topics that already existed in the source cluster.
2. Unless you actually use it for something, the heartbeat connector has no value for MM2 itself. So if you don't consume the messages, you can also simply disable it.

[vladimir-tsvetkov-nexo]

Thank you!
This was a really good lead!
I tested with "unmirrored" Kafka, and the overhead of Mirror Maker 2 topics does not include such kafka-strimzi... topics.

I'm gonna look into the strimzi setup where the problematic mirror maker is running.