Kafka mirror maker 2 consumer group failover test

[ervikrant06]

Setup details.

source kafka cluster: kafka-test
target kafka cluster: kafka-test1
Using kafka operator: 0.32.0
MM2 setup and working fine, it's replicating the kafkatopic from source to destination cluster with same name without any prefix.

1. Create test-app-topic4 (KafkaTopic) and sample-app-user4 (KafkaUser) in source cluster. User is also created in target cluster.
2. hello-world-producer4 (with 100 MESSAGE_COUNT) and hello-world-consumer4 (with 10 MESSAGE_COUNT) on source cluster.
   In stdout logs of consumer POD I see it was able to consume some messages not exactly 10 something like 15 (it's fine)
3. Create kafkauser on target cluster.
   First issue: I was expecting that it should honour the consumer group offset reflected from source to target cluster but it doesn't seem to be doing it.
   Second issue: consumer POD on doesn't show anything consuming from topic. (Does this mean data is also not replicated from source to target cluster)?

Logs from target consumer:

2022-11-28 13:42:34 INFO  KafkaConsumer:965 - [Consumer clientId=consumer-test-consumer-group4-1, groupId=test-consumer-group4] Subscribed to topic(s): test-app-topic4
2022-11-28T13:42:34.758+0000: [Full GC (Metadata GC Threshold) 2022-11-28T13:42:34.758+0000: [Tenured: 0K->12445K(1398144K), 0.0330395 secs] 78292K->12445K(2027264K), [Metaspace: 20777K->20777K(1069056K)], 0.0331143 secs] [Times: user=0.04 sys=0.00, real=0.03 secs]
2022-11-28 13:42:34 WARN  NetworkClient:1073 - [Consumer clientId=consumer-test-consumer-group4-1, groupId=test-consumer-group4] Error while fetching metadata with correlation id 2 : {test-app-topic4=LEADER_NOT_AVAILABLE}
2022-11-28 13:42:35 INFO  Metadata:279 - [Consumer clientId=consumer-test-consumer-group4-1, groupId=test-consumer-group4] Cluster ID: GegP8EHOQ4KUCdQURfUDWQ
2022-11-28 13:42:35 INFO  AbstractCoordinator:815 - [Consumer clientId=consumer-test-consumer-group4-1, groupId=test-consumer-group4] Discovered group coordinator test-kafka1-kafka-0.test-kafka1-kafka-brokers.ns-strimzi-poc1.svc:9093 (id: 2147483647 rack: null)
2022-11-28 13:42:35 INFO  AbstractCoordinator:553 - [Consumer clientId=consumer-test-consumer-group4-1, groupId=test-consumer-group4] (Re-)joining group
2022-11-28 13:42:35 INFO  AbstractCoordinator:456 - [Consumer clientId=consumer-test-consumer-group4-1, groupId=test-consumer-group4] Join group failed with org.apache.kafka.common.errors.MemberIdRequiredException: The group member needs to have a valid member id before actually entering a consumer group.
2022-11-28 13:42:35 INFO  AbstractCoordinator:553 - [Consumer clientId=consumer-test-consumer-group4-1, groupId=test-consumer-group4] (Re-)joining group
2022-11-28 13:42:38 INFO  ConsumerCoordinator:627 - [Consumer clientId=consumer-test-consumer-group4-1, groupId=test-consumer-group4] Finished assignment for group at generation 1: {consumer-test-consumer-group4-1-bf00fa2d-7021-4f4a-b3d6-65780a329e73=Assignment(partitions=[test-app-topic4-0, test-app-topic4-1, test-app-topic4-2, test-app-topic4-3])}
2022-11-28 13:42:38 INFO  AbstractCoordinator:504 - [Consumer clientId=consumer-test-consumer-group4-1, groupId=test-consumer-group4] Successfully joined group with generation 1
2022-11-28 13:42:38 INFO  ConsumerCoordinator:276 - [Consumer clientId=consumer-test-consumer-group4-1, groupId=test-consumer-group4] Notifying assignor about the new Assignment(partitions=[test-app-topic4-0, test-app-topic4-1, test-app-topic4-2, test-app-topic4-3])
2022-11-28 13:42:38 INFO  ConsumerCoordinator:288 - [Consumer clientId=consumer-test-consumer-group4-1, groupId=test-consumer-group4] Adding newly assigned partitions: test-app-topic4-1, test-app-topic4-0, test-app-topic4-3, test-app-topic4-2
2022-11-28 13:42:38 INFO  ConsumerCoordinator:1349 - [Consumer clientId=consumer-test-consumer-group4-1, groupId=test-consumer-group4] Found no committed offset for partition test-app-topic4-1
2022-11-28 13:42:38 INFO  ConsumerCoordinator:1349 - [Consumer clientId=consumer-test-consumer-group4-1, groupId=test-consumer-group4] Found no committed offset for partition test-app-topic4-0
2022-11-28 13:42:38 INFO  ConsumerCoordinator:1349 - [Consumer clientId=consumer-test-consumer-group4-1, groupId=test-consumer-group4] Found no committed offset for partition test-app-topic4-3
2022-11-28 13:42:38 INFO  ConsumerCoordinator:1349 - [Consumer clientId=consumer-test-consumer-group4-1, groupId=test-consumer-group4] Found no committed offset for partition test-app-topic4-2
2022-11-28 13:42:38 INFO  SubscriptionState:397 - [Consumer clientId=consumer-test-consumer-group4-1, groupId=test-consumer-group4] Resetting offset for partition test-app-topic4-2 to offset 0.
2022-11-28 13:42:38 INFO  SubscriptionState:397 - [Consumer clientId=consumer-test-consumer-group4-1, groupId=test-consumer-group4] Resetting offset for partition test-app-topic4-1 to offset 0.
2022-11-28 13:42:38 INFO  SubscriptionState:397 - [Consumer clientId=consumer-test-consumer-group4-1, groupId=test-consumer-group4] Resetting offset for partition test-app-topic4-0 to offset 0.
2022-11-28 13:42:38 INFO  SubscriptionState:397 - [Consumer clientId=consumer-test-consumer-group4-1, groupId=test-consumer-group4] Resetting offset for partition test-app-topic4-3 to offset 0.

[scholzj]

Without the configurations, logs etc., I don't think anyone will be able to tell you what is going on or what have you configured incorrectly.