mirror maker 2 cant replicate topic between namespaces

[michaelcourcy]

Hi

I was using mirrormaker1 to replicate a blogs topic between 2 namespaces and that worked fine

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker
metadata:
  name: blogs-mirror-maker
  namespace: kafka-mirror
spec:
  consumer:
    bootstrapServers: my-cluster-kafka-bootstrap.kafka.svc.cluster.local:9092
    groupId: blogs-group
  producer:
    bootstrapServers: my-cluster-kafka-bootstrap.kafka-mirror.svc.cluster.local:9092
  replicas: 3
  version: 2.8.0
  whitelist: blogs

Now I try the same thing with mirror-maker-2

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker2
metadata:
  namespace: kafka-mirror
  name: blogs-mirror-maker-2
spec:
  version: 2.8.0
  replicas: 3
  connectCluster: "my-target-cluster"
  clusters:
  - alias: "my-source-cluster"
    bootstrapServers: my-cluster-kafka-bootstrap.kafka.svc.cluster.local:9092
  - alias: "my-target-cluster"
    bootstrapServers: my-cluster-kafka-bootstrap.kafka-mirror.svc.cluster.local:9092
    config:
      # -1 means it will use the default replication factor configured in the broker
      config.storage.replication.factor: -1
      offset.storage.replication.factor: -1
      status.storage.replication.factor: -1
  mirrors:
  - sourceCluster: "my-source-cluster"
    targetCluster: "my-target-cluster"
    sourceConnector:
      config:
        replication.factor: 1
        offset-syncs.topic.replication.factor: 1
        sync.topic.acls.enabled: "false"
    heartbeatConnector:
      config:
        heartbeats.topic.replication.factor: 1
    checkpointConnector:
      config:
        checkpoints.topic.replication.factor: 1
    topicsPattern: "blogs"
    groupsPattern: ".*"

But in the target cluster no blogs are created and no error message that I can read in the mirror-maker-2 pod. I guess I do something wrong but I can't see it.

[scholzj]

Mirror Maker 2 works differently from Mirror Maker one. So if your topic is named blogs in the source cluster, it will be named my-source-cluster.blogs in the target cluster. Can you see such topic? TBH I never used the topicsPattern, so I'm not sure what format it expects - but I guess that could be other issue?

[michaelcourcy]

So finally destroy and recreate my env and was not able to have the topic synchronized. On the kafka-mirror ns It automatically create my-source-cluster.blogs topic with exactly the same parameters than the blogs topic on the kafka ns. But the my-source-cluster.blogs remains empty. The 3 mirror-maker-2 pod are up and running, with no error message, Still searching ...

[michaelcourcy]

Apology, my creation script scale down the strimzi opeartor right after the mirror maker creation, the deployment was there but the sync did not start.

[michaelcourcy]

Note sure I understand what's going on but the number of replicas in very important to start the replication if for instance i create this MM2 with 20 replicas the replication does not start.

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker2
metadata:
  namespace: kafka-mirror
  name: blogs-mirror-maker
spec:
  version: 2.8.0
  replicas: 20
  connectCluster: "my-target-cluster"
  clusters:
  - alias: "my-source-cluster"
    bootstrapServers: my-cluster-kafka-bootstrap.kafka.svc.cluster.local:9092
  - alias: "my-target-cluster"
    bootstrapServers: my-cluster-kafka-bootstrap.kafka-mirror.svc.cluster.local:9092
    config:
      # -1 means it will use the default replication factor configured in the broker
      config.storage.replication.factor: -1
      offset.storage.replication.factor: -1
      status.storage.replication.factor: -1
  mirrors:
  - sourceCluster: "my-source-cluster"
    targetCluster: "my-target-cluster"
    sourceConnector:
      config:
        replication.factor: 1
        offset-syncs.topic.replication.factor: 1
        sync.topic.acls.enabled: "false"
    heartbeatConnector:
      config:
        heartbeats.topic.replication.factor: 1
    checkpointConnector:
      config:
        checkpoints.topic.replication.factor: 1
    topicsPattern: "blogs"
    groupsPattern: ".*"

In this case I have to reduce to replicas=1 , delete the last pod to have the deployment controller recreate it and then the sync starts.

At this point I can re-increase the number of replicas without any issue.

I see to possible cause :
First: there is a relation between replicas and other parameters like replication.factor: 1 or offset-syncs.topic.replication.factor: 1 that I don't understand but that is blocking the whole thing

Second: by adding so many replicas I create an interblocking situation that prevent sync from starting

If I get more insights I will share them here.

[scholzj]

I never tried to run it with so many replicas. I know there was some bug in Kafka which had some relationship between number of tasks and number of partitions but I do not remember exactly how it worked or which Kafka version was impacted.

I also think that 20 replicas are useless unless you configure the number of tasks for the individual connectors. The pods will be just running doing nothing.

First: there is a relation between replicas and other parameters like replication.factor: 1 or offset-syncs.topic.replication.factor: 1 that I don't understand but that is blocking the whole thing

The replicas field defines number of pods. The two options you mentioned should cover how many replicas will the topics storing the data have in Kafka. These seem completely unrelated to me apart from the name.

[michaelcourcy]

ok thanks for this precisions