Kafka MirrorMaker2 not replicating consumer group offset

[zt4123]

Hi team, I'm having troubles with kafka mirrormaker in GKE.
I'm doing replication from source cluster to target cluster, kmm2 is deployed in target cluster.

Issue description:
When I create a new topic in source cluster with few messages, the new topic will be replicated successfully to target cluster in few seconds. Then I create a consumer group in source, consume 1 message in source, this consumer group will also be replicated successfully to target cluster. Current offset in this new consumer group in both cluster will be 1 since I consumed 1 message.
Then I consume next message in source, the offset in consumer group in source cluster will be 2, but the offset in target cluster is still 1.

Detailed steps:

1. In source cluster
   ./bin/kafka-console-producer.sh --bootstrap-server xxx:xxx --topic cnx-test --producer.config /tmp/consumer.properties
   topic cnx-test with data will be appear in both source and target.

2. In source cluster
   ./bin/kafka-console-consumer.sh --bootstrap-server xxx:xxx --topic cnx-test --group cnx-test-group --consumer.config /tmp/consumer.properties --max-messages 1
   group cnx-test-group will appear in both source and target, with offset 1

3. In source cluster
   ./bin/kafka-console-consumer.sh --bootstrap-server connectcx-dev-kafka-external-bootstrap.connectcx-dev:9094 --topic cnx-test --group cnx-test-group --consumer.config /tmp/consumer.properties --max-messages 1

The output shows I consumed the second message. cnx-test-group in source cluster shows offset = 2, but cnx-test-group in target cluster will still be with offset = 1.

I've tried add offset.lag.max: 1 in sourceConnector, but it turns out that the consumer group cannot be replicated to the target at the beginning (when i firstly create consumer group and consume first message)

here's my kmm2 config:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker2
metadata:
name: mirror-maker
spec:
version: 3.5.1
replicas: 3
connectCluster: "my-target-cluster"
clusters:
- alias: "my-source-cluster"
authentication:
type: scram-sha-512
username: developer
passwordSecret:
secretName: developer
password: password
bootstrapServers: 10.xxx.xxx.xx:9094
- alias: "my-target-cluster"
authentication:
type: scram-sha-512
username: developer
passwordSecret:
secretName: developer
password: password
bootstrapServers: 10.xxx.xxx.xx:9094
config:
config.storage.replication.factor: -1
config.storage.topic: mirrormaker2-cluster-configs
offset.storage.replication.factor: -1
offset.storage.topic: mirrormaker2-cluster-offsets
status.storage.replication.factor: -1
status.storage.topic: mirrormaker2-cluster-status

mirrors:
- sourceCluster: "my-source-cluster"
targetCluster: "my-target-cluster"
sourceConnector:
tasksMax: 12
config:
replication.factor: 3
offset-syncs.topic.replication.factor: 3
sync.topic.acls.enabled: "false"
replication.policy.class: "io.strimzi.kafka.connect.mirror.IdentityReplicationPolicy"
refresh.topics.interval.seconds: 60
# offset.lag.max: 1
heartbeatConnector:
config:
heartbeats.topic.replication.factor: 3
checkpointConnector:
config:
checkpoints.topic.replication.factor: 3
replication.policy.class: "io.strimzi.kafka.connect.mirror.IdentityReplicationPolicy"
sync.group.offsets.enabled: true
sync.group.offsets.interval.seconds: 20
emit.checkpoints.interval.seconds: 20
refresh.groups.interval.seconds: 20
topicsPattern: "."
groupsPattern: "."
logging:
type: inline
loggers:
connect.root.logger.level: "INFO"
readinessProbe:
initialDelaySeconds: 45
timeoutSeconds: 5
livenessProbe:
initialDelaySeconds: 45
timeoutSeconds: 5

Thanks.

[scholzj]

You should format your post properly to make it readable. Also, MM2 by default does not mirror the offset everytime it changes but only in some chunks. So you need to be actually consuming a stream to see the numbers moving.