Offset translation isn't working for Multi-Region Strimzi cluster.

[AAR256]

Hi,

We have Multi-Region Strimzi cluster deployed in Production environment running with Kafka version - 3.4.0. The MM2 config used for the setup in down below -

---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker2
metadata:
  name: pp-kafka-west-2
  namespace: kafka
spec:
  version: 3.4.0
  replicas: 3
  template:
    pod:
      metadata:
        annotations:
          prometheus.io/scrape: "true"
          prometheus.io/port: "9404"
      tolerations:
      - key: "kubernetes.azure.com/scalesetpriority"
        operator: "Equal"
        value: "spot"
        effect: "NoSchedule"
  connectCluster: pp-north-1 #"my-cluster-target"
  clusters:
  - alias: pp-west-2 #"my-cluster-source"
    authentication:
      type: scram-sha-512
      username: pp-shared-admin
      passwordSecret:
        secretName: pp-shared-admin
        password: password
    bootstrapServers: pp-west-2-bootstrap.net:443
    tls:
      trustedCertificates:
        - secretName: pp-west-2-cert
          certificate: ca.crt

  - alias: pp-north-1 #"my-cluster-target"
    authentication:
      type: scram-sha-512
      username: pp-shared-admin
      passwordSecret:
        secretName: pp-shared-admin
        password: password
    bootstrapServers: pp-north-1-bootstrap.net:443
    tls:
      trustedCertificates:
        - secretName: pp-north-1-cert
          certificate: tls.crt
    config:
      # -1 means it will use the default replication factor configured in the broker
      config.storage.replication.factor: 3
      offset.storage.replication.factor: 3
      status.storage.replication.factor: 3
  mirrors:
  - sourceCluster: pp-west-2 #"my-cluster-source"
    targetCluster: pp-north-1 #"my-cluster-target"
    sourceConnector:
      config:
        replication.factor: 3
        offset-syncs.topic.replication.factor: 3
        tasks.max: 6
    heartbeatConnector:
      config:
        heartbeats.topic.replication.factor: 3
    checkpointConnector:
      config:
        checkpoints.topic.replication.factor: 3
        sync.group.offsets.enabled: true
    topicsPattern: ".*"
    groupsPattern: ".*"
  resources:
    requests:
      memory: 4Gi
      cpu: "1"
    limits:
      memory: 8Gi
      cpu: "2"
  metricsConfig:
    type: jmxPrometheusExporter
    valueFrom:
      configMapKeyRef:
        name: mirror-maker-2-metrics
        key: metrics-config.yml
---

The scenario I'm trying to figure out is that I have 2 regions - West and North. If a topic is created in West region with few messages consumed in the same region. Through MM2 I should be able to consume the subsequent offsets in North region which has the replicated events using the same consumer group. But that doesn't seem to be the case. During consumption with the same consumer group, it's starting over from the initial offset. Please let me know if I'm missing something here.

[scholzj]

There have been many changes to Kafka's offset synchronization since 3.4.0. So you should probably consider upgrading to not solve something that will work differently anyway. You should also make sure you give it enough time to discover the new groups and topics (by default up to 10 minutes) etc. Also, keep in mind that the synchronization cannot happen when the consumer group is active on the target cluster.

[AAR256]

@scholzj Sure, we'll give upgrading a thought. Right now after going through the docs I found out 3 topics were handling the offset translation which I've now added to the checkpoint connector config -

sync.group.offsets.interval.seconds: 5
refresh.groups.interval.seconds: 5
emit.checkpoints.interval.seconds: 5

Even after applying those in, it doesn't seem to be working. I have couple of questions though -

1. In Confluent replicators, they have a condition that only java consumer will be able to use offset translation. Do we have a similar requirement for MM2 because at the moment, I'm testing using a python producer and consumer application.
2. How can I figure out if consumer group is active on the target cluster. In my application, after consumption of available events I'm shutting it down. Since it's a test consumer group no other application is using it in the target/source cluster.

[scholzj]

Re 1) I do not know anything about Confluent replicator. The offset mirroring in MM2 works in two levels:

• It mirrors and translates the offsets into a special topic and you have to use a special library to decode and recover the data. This is available only as a Java library, but is not part of the Kafka consumer it self (not even the Java one)
• When you enable the sync.group.offsets.enabled as you did, the offsets are synced directly into the __consumer_offsets topic, and any consumer using the consumer group protocol should be able to recover it.

Re 2) You should be able to list the consumer groups and see if it has some active clients.

I'm not sure how well this works inside MM2. But I'm not sure the general expectation is to start the consumer on the target cluster, consume and commit something, then stop it and consume on the source cluster again and then on the target cluster again etc. and have the offsets synced in between. Normally, it is used more for active-passive scenarios where you switch to the target cluster only after some disaster and losing the source cluster.

[AAR256]

@scholzj Thanks for the info. After some debugging, we came to know that the schema mirror present in the same MM2 CR was the issue. After removing that, the offset translation is happening.

So, a follow up question would be for multi-region setup we have 2 mirrors -

1. For normal topic replication with alias prefixed.
2. For _schemas topic specifically without alias using identity replication policy.

Both the mirrors are from same source Strimzi cluster. Is this the right approach to handle multi-region schema registry i.e - replicate the _schemas topic from primary to secondary kafka cluster or is there a better approach we can follow?

Another question is since _schemas is an internal topic, is it alright to whitelist that for replication? Please let me know your thoughts.

[scholzj]

Sorry, I do not think I have any answers for this. I have no idea what multi-region schema registry is or how it works. I think you need to ask the authors of your Schema Registry for advice, as they know how it works.

Another question is since _schemas is an internal topic, is it alright to whitelist that for replication? Please let me know your thoughts.

Apache Kafka does not have any support for schemas or anything like that. So for MM2 it is just a topic as any other.