mirrormaker2 duplicated data

[elvis-cai]

Hi guys

we are migrating our kafka clusters to strimzi operator, while testing https://github.com/rmarting/strimzi-migration-demo, when we stop hello-world-consumer in the source cluster and start it again in the dest cluster, we can see it reconsumes 2-3 msg again, you can find details in the logs below, just wondering is that expected due to mirorrmaker2 or kafka connect limitaiton according to https://cwiki.apache.org/confluence/display/KAFKA/KIP-656%3A+MirrorMaker2+Exactly-once+Semantics?
thanks.

	Time	Event
	19/06/202221:49:32.793	2022-06-19 11:49:32 INFO KafkaConsumerExample:50 - value: "436 - dlrow olleH"host = gke-test-f591e6a4-32z7.c-preprod-c7cbb7f9.internalsource = /var/log/pods/infra_hello-world-consumer-68fc6c6dc8-52gjr_5a08d1df-5281-4ed8-9dd7-fc5b81792565/hello-world-consumer/0.logsourcetype = kube:container:hello-world-consumer
	19/06/202221:47:45.364	2022-06-19 11:47:45 INFO KafkaConsumerExample:50 - value: "436 - dlrow olleH"host = gke-test-f591e6a4-32z7.c-preprod-c7cbb7f9.internalsource = /var/log/pods/infra_hello-world-consumer-85d6c76697-ngzcb_522d5b0e-0e30-43f6-a018-342de1a16485/hello-world-consumer/0.logsourcetype = kube:container:hello-world-consumer

[scholzj]

You mean that the old consumer on the old cluster consumed the message and then the new consumer on the new cluster after you switched consumed it as well?

I think that can be caused by multiple things:

• Did the consumer on the old cluster actually commit the offsets (e.g. with auto-commit and some forced termination, it might not have been committed and then the message would be consumed again)
• If it did commit the offsets, did you give mmě enough time to mirror the offsets? The offsets are not mirrored immediately, but in some periodical checks. So if you don't give it the time, you have a some window for duplicates.
• There is no exactly once semantic in MM2. So that can of course be related as well. But unless you had some restarts of MM2 or the Kafka brokers or some network outages, the first two are normally more likely.

[elvis-cai]

thanks @scholzj for the quick reply and detailed answers.

• we are using https://github.com/rmarting/strimzi-migration-demo/tree/main/01-source-cluster/05-sample-apps, so consumer will commit the offsets
• this is actually the root cause for me, it looks like we need to wait for some time, one min or more, and I can see consumer in the target cluster could continue to consume the msg from the stop point in the source cluster. 👍
  in that case, is there any metrics available in the mm2, so we could monitor that and start the consumer in the target cluster without waiting in the blind 😂.
  Thanks.

[scholzj]

in that case, is there any metrics available in the mm2, so we could monitor that and start the consumer in the target cluster without waiting in the blind

I don't know I'm afraid. I think it shows the lag only for mirroring of messages. I'm not aware of any easy way to do it for offsets.