KafkaConnect not running

[vipul0805]

Hey All,

I am trying to deploy KafkaConnect cluster in my EKS cluster and it is giving the same log repetedly and not registering any Kafka Connectors.

I checked that the Kafka and Zookeeper pods are running but could not find the actual issue.

The KafkaConnect config is as follows:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: strimzi-kafka-cluster
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  version: 3.3.2
  replicas: 1
  bootstrapServers: strimzi-kafka-kafka-bootstrap:9092
  config:
    config.providers: file
    config.providers.file.class: org.apache.kafka.common.config.provider.FileConfigProvider
    # -1 means it will use the default replication factor configured in the broker
    config.storage.replication.factor: -1
    offset.storage.replication.factor: -1
    status.storage.replication.factor: -1
  externalConfiguration:
    # env:
    #   - name: AWS_ACCESS_KEY_ID
    #     valueFrom:
    #       secretKeyRef:
    #         name: aws-creds
    #         key: awsAccessKey
    #   - name: AWS_SECRET_ACCESS_KEY
    #     valueFrom:
    #       secretKeyRef:
    #         name: aws-creds
    #         key: awsSecretAccessKey  
    volumes:
      - name: dev-psql-credentials
        secret:
          secretName: dev-psql-credentials
  build:
    output:
      type: docker
      image: 128945561656.dkr.ecr.us-east-1.amazonaws.com/debezium-connect-mysql:latest
      pushSecret: ecr-secret
    plugins:
      - name: debezium-postgres-connector
        artifacts:
          - type: maven
            group: io.debezium
            artifact: debezium-connector-postgres
            version: 2.1.1.Final
  template:
    pod:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
                - key: errors.ea.com/linux-distro
                  operator: In
                  values:
                    - bottlerocket
    buildPod:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
                - key: errors.ea.com/linux-distro
                  operator: In
                  values:
                    - bottlerocket

The Log is as follows:

2023-02-03 13:00:08,675 INFO [Worker clientId=connect-1, groupId=connect-cluster] Client requested disconnect from node 2147483646 (org.apache.kafka.clients.NetworkClient) [DistributedHerder-connect-1-1]
2023-02-03 13:00:08,677 INFO [Worker clientId=connect-1, groupId=connect-cluster] Discovered group coordinator strimzi-kafka-kafka-1.strimzi-kafka-kafka-brokers.strimzi-kafka.svc.cluster.local:9092 (id: 2147483646 rack: null) (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-02-03 13:00:08,677 INFO [Worker clientId=connect-1, groupId=connect-cluster] Group coordinator strimzi-kafka-kafka-1.strimzi-kafka-kafka-brokers.strimzi-kafka.svc.cluster.local:9092 (id: 2147483646 rack: null) is unavailable or invalid due to cause: coordinator unavailable. isDisconnected: false. Rediscovery will be attempted. (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-02-03 13:00:08,677 INFO [Worker clientId=connect-1, groupId=connect-cluster] Requesting disconnect from last known coordinator strimzi-kafka-kafka-1.strimzi-kafka-kafka-brokers.strimzi-kafka.svc.cluster.local:9092 (id: 2147483646 rack: null) (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-02-03 13:00:08,778 INFO [Worker clientId=connect-1, groupId=connect-cluster] Discovered group coordinator strimzi-kafka-kafka-1.strimzi-kafka-kafka-brokers.strimzi-kafka.svc.cluster.local:9092 (id: 2147483646 rack: null) (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-02-03 13:00:08,778 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-02-03 13:00:08,781 INFO [Worker clientId=connect-1, groupId=connect-cluster] Group coordinator strimzi-kafka-kafka-1.strimzi-kafka-kafka-brokers.strimzi-kafka.svc.cluster.local:9092 (id: 2147483646 rack: null) is unavailable or invalid due to cause: error response NOT_COORDINATOR. isDisconnected: false. Rediscovery will be attempted. (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-02-03 13:00:08,781 INFO [Worker clientId=connect-1, groupId=connect-cluster] Requesting disconnect from last known coordinator strimzi-kafka-kafka-1.strimzi-kafka-kafka-brokers.strimzi-kafka.svc.cluster.local:9092 (id: 2147483646 rack: null) (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-02-03 13:00:08,781 INFO [Worker clientId=connect-1, groupId=connect-cluster] JoinGroup failed: This is not the correct coordinator. Marking coordinator unknown. Sent generation was Generation{generationId=-1, memberId='', protocol='null'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-02-03 13:00:08,781 INFO [Worker clientId=connect-1, groupId=connect-cluster] Request joining group due to: rebalance failed due to 'This is not the correct coordinator.' (NotCoordinatorException) (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-02-03 13:00:08,881 INFO [Worker clientId=connect-1, groupId=connect-cluster] Client requested disconnect from node 2147483646 (org.apache.kafka.clients.NetworkClient) [DistributedHerder-connect-1-1]
2023-02-03 13:00:08,882 INFO [Worker clientId=connect-1, groupId=connect-cluster] Discovered group coordinator strimzi-kafka-kafka-1.strimzi-kafka-kafka-brokers.strimzi-kafka.svc.cluster.local:9092 (id: 2147483646 rack: null) (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-02-03 13:00:08,882 INFO [Worker clientId=connect-1, groupId=connect-cluster] Group coordinator strimzi-kafka-kafka-1.strimzi-kafka-kafka-brokers.strimzi-kafka.svc.cluster.local:9092 (id: 2147483646 rack: null) is unavailable or invalid due to cause: coordinator unavailable. isDisconnected: false. Rediscovery will be attempted. (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-02-03 13:00:08,882 INFO [Worker clientId=connect-1, groupId=connect-cluster] Requesting disconnect from last known coordinator strimzi-kafka-kafka-1.strimzi-kafka-kafka-brokers.strimzi-kafka.svc.cluster.local:9092 (id: 2147483646 rack: null) (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]

Any suggestions are appreciated.

Thanks.

[scholzj]

I guess it is failing to connect to your Kafka cluster?

[vipul0805]

Ah okay. But the log is not saying anything about why it is not able to connect.

Is it required for kafkaConnect to use TLS or authentication to communicate with Kafka borkers?

[scholzj]

Well, strictly speaking, it can connect. But it seems to have trouble finding the coordinator for some reason. But TBH I do not know why. The broker logs might tell you something more, so you should check them out. And as always ... restart sometimes helps :-o. But I'm not really an expert on this area.

You do not need TLS or authentication to use Connect (obviously you need to have the not enabled in the broker as well).

[vipul0805]

Yes, I have not configured any TLS listener in the broker config and I checked the broker logs too and they do not show any ERROR or exception. As for the restart, I removed all the deployed configs and redeployed the Kafka cluster and the connector.

I will try to analyze the broker logs more carefully.

Thanks.

[vipul0805]

Sorry to ask this in the same discussion but the connector started running now. I am trying to deploy the Postgres connector but after applying the KaqfkaConnector resource its failing with below exception

023-02-03 13:30:43,860 ERROR Uncaught exception in REST call to /connectors/debezium-connector-psql/config (org.apache.kafka.connect.runtime.rest.errors.ConnectExceptionMapper) [qtp876669389-67]
org.apache.kafka.connect.errors.ConnectException: Failed to find any class that implements Connector and which name matches io.debezium.connector.postgresql.PostgresConnector, available connectors are: 

Why is not showing up in the Available connectors?

Only these 3 are showing up:

[
{
"class": "org.apache.kafka.connect.mirror.MirrorCheckpointConnector",
"type": "source",
"version": "3.3.2"
},
{
"class": "org.apache.kafka.connect.mirror.MirrorHeartbeatConnector",
"type": "source",
"version": "3.3.2"
},
{
"class": "org.apache.kafka.connect.mirror.MirrorSourceConnector",
"type": "source",
"version": "3.3.2"
}
]

I checked the build too and its copying the plugin in the correct path

COPY --from=downloadArtifacts '/tmp/artifacts/debezium-postgres-connector/5ea0ca0c' '/opt/kafka/plugins/debezium-postgres-connector/5ea0ca0c'

Any suggestions would be helpful.

Thanks

[scholzj]

Do you still use the same custom resource as above? I wonder if you should download the connector archive instead of the Maven artifact - are you sure it contains all the required dependencies etc.?

[vipul0805]

Yes, I am using the same resource as above. I am not sure if it contains all the required dependencies.I was under the impression that the maven artifact will have all the required dependencies. Is it not the case?

…

On Fri, 3 Feb, 2023, 20:09 Jakub Scholz, ***@***.***> wrote: Do you still use the same custom resource as above? I wonder if you should download the connector archive instead of the Maven artifact - are you sure it contains all the required dependencies etc.? — Reply to this email directly, view it on GitHub <#8038 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AF2QZXUGVOK2OTNP7JZC7I3WVUKCLANCNFSM6AAAAAAUQHZF6M> . You are receiving this because you authored the thread.Message ID: <strimzi/strimzi-kafka-operator/repo-discussions/8038/comments/4862807@ github.com>

[scholzj]

That will depend on the connector. But Debezium docs seem to use the archive file: https://debezium.io/documentation/reference/2.2/operations/kubernetes.html

[ns-jhu]

I encountered the same dead loop situation with identical log message. currently we use connections.max.idle.ms=-1 on both client side and server side to keep socket connection up forever as tentative work around in development environment. However, we don't know if this can be used in production as we don't know what will happen if the long socket connection breaks. Given that in production , long socket will break sooner or later, we will be facing an uncertain consequence.

On the other hand, when I used command

/opt/ns/kafka/bin/kafka-consumer-groups.sh --bootstrap-server $kafka_brokers --list

the debezium consumer group doesn't show up, while consumer group for other consumers showed up. this leads me to believe that the consumer group for debezium was never established properly, which explains that when consumer disconnect and connect back, the kafka broker reports that the consumer group coordinator is not available.

This is definitely a serious debezium bug with regard to improper kafka client usage or kafka bug with regard to debezium consumer connection. The bottom line is, the debezium consumer group should show up in the list produced by

/opt/ns/kafka/bin/kafka-consumer-groups.sh --bootstrap-server $kafka_brokers --list

by the way, https://groups.google.com/g/debezium/c/k7NBXCyMSHI is also about this issue.

[ns-jhu]

https://groups.google.com/g/debezium/c/k7NBXCyMSHI is also discussing this issue.

[ns-jhu]

Just an update:

I noticed a new kafkaconnect image http://quay.io/strimzi/kafka:0.34.0-kafka-3.3.2 released recently.
So I used that one and the consumer group coordinator unavailable issue is gone.
but we still do not know the cause of this, and how it is fixed.

While doing this, I also noticed that the kafka needs to be upgraded to version 3.0 or above to avoid
A deadlock issue in the kafka.

[scholzj]

Please keep in mind that you should always use container images belonging to a particular Strimzi release. The kafka:0.34.0-kafka-3.3.2 image didn't just appear like that. It was released as part of Strimzi 0.34.0 and you should use it if you use Strimzi 0.34.0. If you use some other Strimzi version, you should either upgrade Strimzi or use one of the Kafka images/versions belonging to and supported in the given Strimzi version.