Kafka Connect timed out connecting to service

[LinzeSu]

Hi,
I'm new to kafka connect and want to try using it. The yaml file used to create the cluster looks like below:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: sftp-source-connect
  namespace: boc
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  replicas: 1
  version: 3.2.3
  image: quay.io/strimzi/kafka:0.32.0-kafka-3.2.3
  bootstrapServers: boc-kafka-cluster-kafka-bootstrap:9092,boc-kafka-cluster-kafka-bootstrap:9091
  resources:
    requests:
      cpu: 250m
      memory: 512Mi
    limits:
      cpu: 500m
      memory: 1Gi
  jvmOptions:
    gcLoggingEnabled: false
  config:
    group.id: sftp-source-connect
    key.converter: org.apache.kafka.connect.storage.StringConverter
    value.converter: org.apache.kafka.connect.storage.StringConverter
    offset.storage.topic: sftp-source-connect-offsets
    config.storage.topic: sftp-source-connect-configs
    status.storage.topic: sftp-source-connect-status
    config.storage.replication.factor: 2
    offset.storage.replication.factor: 2
    status.storage.replication.factor: 2

But it won't get ready. The describe command shows:

Status:
  Conditions:
    Last Transition Time:  2023-01-18T07:17:43.398056Z
    Message:               connection timed out: sftp-source-connect-connect-api.boc.svc.cluster.local/10.100.232.0:8083
    Reason:                ConnectTimeoutException
    Status:                True
    Type:                  NotReady

The relevant service goes:

apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2023-01-18T07:15:41Z"
  labels:
    app.kubernetes.io/instance: sftp-source-connect
    app.kubernetes.io/managed-by: strimzi-cluster-operator
    app.kubernetes.io/name: kafka-connect
    app.kubernetes.io/part-of: strimzi-sftp-source-connect
    strimzi.io/cluster: sftp-source-connect
    strimzi.io/kind: KafkaConnect
    strimzi.io/name: sftp-source-connect-connect-api
  name: sftp-source-connect-connect-api
  namespace: boc
  ownerReferences:
  - apiVersion: kafka.strimzi.io/v1beta2
    blockOwnerDeletion: false
    controller: false
    kind: KafkaConnect
    name: sftp-source-connect
    uid: c42ec068-7048-4e43-acb3-f28749e78c46
  resourceVersion: "12002114"
  uid: 39637111-1e2b-43df-a2c0-5f17baf47219
spec:
  clusterIP: 10.100.232.0
  clusterIPs:
  - 10.100.232.0
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: rest-api
    port: 8083
    protocol: TCP
    targetPort: 8083
  selector:
    strimzi.io/cluster: sftp-source-connect
    strimzi.io/kind: KafkaConnect
    strimzi.io/name: sftp-source-connect-connect
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

I did some search of similar problem but got nothing. What could be the problem? A little bit extra explanation about this service would be nice. Thanks in advance.
Strimzi operator version: 0.32
Kafka version: 3.2.3
Kubernetes version: 1.25.4

[scholzj]

Are you sure your bootstrap server configuration is correct? If the Kafka cluster you want it to connect to is powered by Strimzi, then the address with port 9091 should definitely not be there. Apart from that, sharing the logs would be a good start.

[scholzj]

And did you disable the network policy creation in the operator? By default, Strimzi creates its own network policy for this. So that should not be a problem unless you have some special SDN. Something what I saw in the past was that this was caused by an issue in communication between worker nodes. So that might be something to check as well 🤷

[LinzeSu]

Thanks for the suggestions. I'll try and report back if I get any progress.

[LinzeSu]

Sorry for late reply. But I found after I delete the auto-created networkpolicy, the service can be accessed and the kafka-connect status stays Ready for a little while before the networkpolicy is created again. The networkpolicy yaml below looks pretty normal to me:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  creationTimestamp: "2023-01-28T10:15:51Z"
  generation: 1
  labels:
    app.kubernetes.io/instance: sftp-source-connect
    app.kubernetes.io/managed-by: strimzi-cluster-operator
    app.kubernetes.io/name: kafka-connect
    app.kubernetes.io/part-of: strimzi-sftp-source-connect
    strimzi.io/cluster: sftp-source-connect
    strimzi.io/kind: KafkaConnect
    strimzi.io/name: strimzi
  name: sftp-source-connect-connect
  namespace: boc
  ownerReferences:
  - apiVersion: kafka.strimzi.io/v1beta2
    blockOwnerDeletion: false
    controller: false
    kind: KafkaConnect
    name: sftp-source-connect
    uid: 5058ce23-cd04-45b3-834e-7fd3069608b6
  resourceVersion: "15071721"
  uid: 61fdd57d-8998-47f5-bf22-2205a797828f
spec:
  ingress:
  - from:
    - podSelector:
        matchLabels:
          strimzi.io/cluster: sftp-source-connect
          strimzi.io/kind: KafkaConnect
          strimzi.io/name: sftp-source-connect-connect
    - podSelector:
        matchLabels:
          strimzi.io/kind: cluster-operator
    ports:
    - port: 8083
      protocol: TCP
  podSelector:
    matchLabels:
      strimzi.io/cluster: sftp-source-connect
      strimzi.io/kind: KafkaConnect
      strimzi.io/name: sftp-source-connect-connect
  policyTypes:
  - Ingress
status: {}

the operator pod label:

strimzi-cluster-operator-56d64c8584-l7dbq            1/1     Running   1 (15d ago)   50d    name=strimzi-cluster-operator,pod-template-hash=56d64c8584,strimzi.io/kind=cluster-operator

the kafka connect pod label:

sftp-source-connect-connect-84b7679dd-lq7c4          1/1     Running   0             5m5s   app.kubernetes.io/instance=sftp-source-connect,app.kubernetes.io/managed-by=strimzi-cluster-operator,app.kubernetes.io/name=kafka-connect,app.kubernetes.io/part-of=strimzi-sftp-source-connect,pod-template-hash=84b7679dd,strimzi.io/cluster=sftp-source-connect,strimzi.io/kind=KafkaConnect,strimzi.io/name=sftp-source-connect-connect

Is there anything I could be missing?

[scholzj]

That is weird as this network policy should ensure the Connect nodes and the operator can talk with each other :-/. The labels seem correct. So I'm afraid I'm not sure what is wrong here. Maybe something with your SDN? Do you know what does your cluster use?

You can use the STRIMZI_NETWORK_POLICY_GENERATION environment variable in the Cluster Operator deployment to tell the operator to stop managing the network policies. That way, after you delete it it should not be recreated again. But that is of course more workaround than a proper solution.

[LinzeSu]

By saying SDN I assume that's another name of CNI? I'm using weave as CNI and not an expert of it. I'll try looking into the problem and report back. And thanks for the workaround.

[scholzj]

Yes, exactly. That is what I meant. I'm not an expert on it either TBH - so no idea what exactly to look for. But that is what interprets the network policies.

Maybe one more question ... does the operator run in the same namespace as the Connect?

[LinzeSu]

Yes the operator run in the same namespace as the Connect.
The weave CNI seems to be the problem(went through some similar issue on weave github page). I'm considering replacing it with cilium in the future. Now I'm using the workaround and it works well. Will report back when the CNI is replaced.

[LinzeSu]

Confirmed it's a CNI problem.