Kafka Connector tasks not running after Cluster upgrade

[kinihun]

Hi,
I observed that after a kubernetes upgrade that involved rolling update of the nodes, Kafka Connect consumers have issues with the Group coordinators because both the Kafka Connector and the Broker pods have all moved around during the draining and rescheduling process.
How can I prevent this from happening? I read about using static memberships to reduce rebalances, but there is no way to set this for individual Kafka Connect pods since they are spawned by the the a Replicaset.
The impact of this issue is that my connectors are not running as scheduled.

"2022-10-11 09:29:32,756 INFO [Producer clientId=producer-1] Resetting the last seen epoch of partition prod-kafka-connect-offsets-21 to 100 since the associated topicId changed from null to 7XOm2ghlQACkM5TYq_BvOQ (org.apache.kafka.clients.Metadata) [kafka-producer-network-thread | producer-1]" "Oct 11, 2022 9:28:48 AM com.denodo.util.logging.JULDenodoLogImpl logMessage" SEVERE: Unable to connect to the server "2022-10-11 09:28:48,995 ERROR [denodo-jdbc-connector-prod|task-0] SQL exception while running query for table: BulkTableQuerier{table='""capacity_prod"".""capacity""', query='null', topicPrefix='rdtech.csi.qed.capacitydb.hrdata.prod.hierarchical.immutable.'}, java.sql.SQLException: unexpected exception; nested exception is: " " com.denodo.internal.o.a.r.netty.KeepAliveException: Channel time out exception. Channel not available at least since 2022-10-11T09:26:17.892Z. Attempting retry 1 of -1 attempts. (io.confluent.connect.jdbc.source.JdbcSourceTask) [task-thread-denodo-jdbc-connector-prod-0]" "Oct 11, 2022 9:28:48 AM com.denodo.util.logging.JULDenodoLogImpl logMessage" SEVERE: Unable to connect to the server "Oct 11, 2022 9:28:48 AM com.denodo.util.logging.JULDenodoLogImpl logMessage" SEVERE: Timeout checking channel [338ff9c3-/10.160.224.55:9999] state. Keep alive check sent 2022-10-11T09:26:17.892Z but no response received. Closing channel "Oct 11, 2022 9:28:48 AM com.denodo.util.logging.JULDenodoLogImpl logMessage" SEVERE: Timeout checking channel [338ff9c3-/10.160.224.55:9999] state. Keep alive check sent 2022-10-11T09:26:17.892Z but no response received. Closing channel "2022-10-11 09:28:28,738 INFO [Consumer clientId=consumer-prod-kafka-1, groupId=prod-kafka] Discovered group coordinator prod-kafka-kafka-1.prod-kafka-kafka-brokers.kafka.svc:9093 (id: 2147483646 rack: null) (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator) [KafkaBasedLog Work Thread - prod-kafka-connect-offsets]" "2022-10-11 09:28:28,675 INFO [Consumer clientId=consumer-prod-kafka-2, groupId=prod-kafka] Discovered group coordinator prod-kafka-kafka-1.prod-kafka-kafka-brokers.kafka.svc:9093 (id: 2147483646 rack: null) (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator) [KafkaBasedLog Work Thread - prod-kafka-connect-status]" "2022-10-11 09:28:28,637 INFO [Consumer clientId=consumer-prod-kafka-1, groupId=prod-kafka] Discovered group coordinator prod-kafka-kafka-1.prod-kafka-kafka-brokers.kafka.svc:9093 (id: 2147483646 rack: null) (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator) [KafkaBasedLog Work Thread - prod-kafka-connect-offsets]" "2022-10-11 09:28:28,637 INFO [Consumer clientId=consumer-prod-kafka-1, groupId=prod-kafka] Group coordinator prod-kafka-kafka-1.prod-kafka-kafka-brokers.kafka.svc:9093 (id: 2147483646 rack: null) is unavailable or invalid due to cause: coordinator unavailable.isDisconnected: false. Rediscovery will be attempted. (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator) [KafkaBasedLog Work Thread - prod-kafka-connect-offsets]" "2022-10-11 09:28:28,637 INFO [Consumer clientId=consumer-prod-kafka-1, groupId=prod-kafka] Requesting disconnect from last known coordinator prod-kafka-kafka-1.prod-kafka-kafka-brokers.kafka.svc:9093 (id: 2147483646 rack: null) (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator) [KafkaBasedLog Work Thread - prod-kafka-connect-offsets]" "2022-10-11 09:28:28,636 INFO [Consumer clientId=consumer-prod-kafka-1, groupId=prod-kafka] Node 2147483646 disconnected. (org.apache.kafka.clients.NetworkClient) [KafkaBasedLog Work Thread - prod-kafka-connect-offsets]" "2022-10-11 09:28:28,636 INFO [Consumer clientId=consumer-prod-kafka-1, groupId=prod-kafka] Group coordinator prod-kafka-kafka-1.prod-kafka-kafka-brokers.kafka.svc:9093 (id: 2147483646 rack: null) is unavailable or invalid due to cause: coordinator unavailable.isDisconnected: true. Rediscovery will be attempted. (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator) [KafkaBasedLog Work Thread - prod-kafka-connect-offsets]" "2022-10-11 09:28:28,373 INFO [Consumer clientId=consumer-prod-kafka-2, groupId=prod-kafka] Node 2147483646 disconnected. (org.apache.kafka.clients.NetworkClient) [KafkaBasedLog Work Thread - prod-kafka-connect-status]" "2022-10-11 09:28:28,373 INFO [Consumer clientId=consumer-prod-kafka-2, groupId=prod-kafka] Group coordinator prod-kafka-kafka-1.prod-kafka-kafka-brokers.kafka.svc:9093 (id: 2147483646 rack: null) is unavailable or invalid due to cause: coordinator unavailable.isDisconnected: true. Rediscovery will be attempted. (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator) [KafkaBasedLog Work Thread - prod-kafka-connect-status]"

[scholzj]

The logs are impossible to read without proper formatting. But basically every rolling update of the Kafka cluster will cause the group coordinator to change. Connect should be able to find the new coordinator and use it. So if it doesn't, or if the new coordinator doesn't exist, you need to find out why.

[kinihun]

@scholzj ,

I am attaching the logs instead, I am really out of my depth here so I really appreciate your time in responding so far.
I run multiple Kafka clusters and this is becoming a recurring problem with all of them. The KafkaConnect pods run on the same nodes as the Kafka Broker pods so I don't have a connectivity issue, all the communication is internal to the cluster. The problem is the constant discovery and disconnection going on. I wish I understood the process a bit more so I am studying it but at this point I still can't figure it.

I also attached the KafkaConnect configuration. Any idea what I need to change or be looking into.

connect-log.txt
kafkaconnect.txt

[scholzj]

I'm not an expert on this area. But I think that ...

• It does not matter here if you run your Connect on the same node as some Kafka broker -> it does not necessarily connect to the broker on the same node.
• It is complaining about the group coordinator. So you need to figure out if the problem is that your Connect cannot find it / connect to it. Or if the problem is that it is changing in your Kafka cluster. That tells you what your problem is. I think your problem suggests the issue is in your Kafka cluster not in your Connect.

It is also a bit confusing when does it happen. You said during a Kubernetes upgrade on the beginning. That rolls the broker pods, so regardless whether you want to or not, the coordinator will change during such event. You cannot avoid it. The log messages in your log just say that the coordinator changed - that happens, that is why it is an INFO message.

Now you say that it is happening constantly. That is confusing as Kubernetes upgrades do not happen constantly. If it happens constantly wirthout any rolling updates, maybe your Kafka cluster has some networking issues which cause the roles to change.

[kinihun]

Thanks for your comment, of course, cluster upgrades don't happen all the time, I meant across multiple clusters.
In the time being I have put the following workaround in place while I research the root cause by studying the related kafka consumer source code.

• restart the affected connector tasks daily
• increase the readiness probe timer to give connect and broker pods more time to run and sync during a rolling upgrade.