MirrorMaker2 not replicating consumer groups

[RomanTheLegend]

When setting up a migration from Kafka2 to Kafka3 using operator, MirrorMaker2 correctly replicates all topics, but none of consumer groups are created

operator:0.32.0

MirrorMaker2 is installed from manifest (which used to work for replicating between Kafka2 clusters using MM2 version: 2.7.0):

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker2
metadata:
  name: mm2-kafka3
  namespace: kafka3
spec:
  version: 3.3.1
  replicas: 1
  connectCluster: "target"
  template:
    pod:
      tolerations:
        - key: "workloadsType"
          operator: "Exists"
          effect: "NoSchedule"
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                - key: workloadsType
                  operator: In
                  values:
                    - mm2
      metadata:
        annotations:
          sidecar.istio.io/inject: "false"
  jvmOptions:
    -Xms: 7G
    -Xmx: 7G
  resources:
      limits:
        cpu: 6
        memory: 9G
      requests:
        cpu: 1
        memory: 7G
  clusters:
  - alias: "source"
    bootstrapServers: kafka2-cluster-kafka-bootstrap.kafka2:9092
  - alias: "target"
    bootstrapServers: kafka3-cluster-kafka-bootstrap:9092
    config:
      producer.max.request.size: 52428800
      producer.compression.type: gzip
      config.storage.replication.factor: 2
      offset.storage.replication.factor: 2
      status.storage.replication.factor: 2
  mirrors:
  - sourceCluster: "source"
    targetCluster: "target"
    topicsPattern: "topic1|mastertopic1.*|mastertopic2.*|sometopic.*"
    groupsPattern: ".*"
    sourceConnector:
      tasksMax: 40
      config:
        producer.enable.idempotence: true
        producer.max.in.flight.requests.per.connection: 1
        producer.acks: all
        producer.retries: 2147483647
        offset-syncs.topic.replication.factor: 2
        offset-syncs.topic.location: "target"
        sync.topic.acls.enabled: "false"
        replication.policy.separator: ""
        replication.policy.class: "io.strimzi.kafka.connect.mirror.IdentityReplicationPolicy"
    heartbeatConnector:
      config:
        heartbeats.topic.replication.factor: 1
    checkpointConnector:
      config:
        checkpoints.topic.replication.factor: 2
        replication.policy.separator: ""
        replication.policy.class: "io.strimzi.kafka.connect.mirror.IdentityReplicationPolicy"
        sync.group.offsets.enabled: "true"

To Reproduce

1. Install Kafka3 from operator:0.32.0
2. Install MM2 from aforementioned manifest

In logs it says it tries to sync idle consumer group offset from source to target but in reality none of the groups gets created

Environment:

• Strimzi version: [e.g. main, 0.32.0]
• Installation method: Helm chart via terraform
• Kubernetes cluster: Kubernetes 1.24
• Infrastructure: GKE

[scholzj]

Maybe sharing the log would help. I never tried the offset sync with offset-syncs.topic.location set to target - I think it defaults to source. So I wonder if that could cause some problems.

[RomanTheLegend]

Logs are rather big, is there any section you're interested in particular?

offset-sync location was set to target because I'm testing MM2 data migration to Kafka3, and because target cluster gets frequently recreated it's easier to do a cleanup when everything is on target, otherwise I'd have to manually clean internal topics on source

[RomanTheLegend]

I've removed offset-syncs.topic.location: "target" from sourceConnector section and it created the groups.
Should I raise the bug that consumer groups are not created when mirroring state is kept on Target?

[scholzj]

TBH, I have no idea how that is supposed to work. If it is expected to work differently, if you are missing some more configuration, or anything like that. I only suggested to try it without it because in the past when I used it it was always without it, so I knew that worked for me. So I do not know if this is a bug or not.

@mimaison might have an idea.

[mimaison]

If you change offset-syncs.topic.location, you need to set it on both sourceConnector and checkpointConnector.

[lenin-joseph]

Hi @mimaison ,
The consumer group replication doesn't work in strimzi-0.36.1(kafka-3.5.1) when offset-syncs.topic.location=target is set only in sourceConnector.
But, it works fine in strimzi-0.31.1(kafka-3.2.3).
Any changes between these two versions w.r.t the config "offset-syncs.topic.location" ?

[mimaison]

If you want to store offset-syncs on the target cluster, setting offset-syncs.topic.location=target on both connectors has always been required. I'd suggest double checking your environment with strimzi-0.31.1(kafka-3.2.3) as it may just appear to work, for example due to a previously existing offset-syncs topic. It should not work.