Couple of issues with kafka and zookeeper

[stuartm21]

Hi @scholzj ,

We are facing a couple of issues while connecting to the zookeeper and Kafka. Please note that we are using MTLS for our kafka cluster I am pasting the kafka cluster config here:

Kafka.yaml

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: strimzi
  namespace: kafka
spec:
  clusterCa:
    renewalDays: 30
    validityDays: 3650
    generateCertificateAuthority: true
  clientsCa:
    renewalDays: 30
    validityDays: 3650
    generateCertificateAuthority: true
  kafka:
    version: 3.1.0
    replicas: 3
    listeners:
      - name: external
        port: 9094
        type: ingress
        tls: true
        authentication:
          type: tls
        configuration:
          class: kafka
          bootstrap:
            host: prod-kafka-bootstrap.myhost.com
          brokers:
          - broker: 0
            host: broker-0.myhost.com
          - broker: 1
            host: broker-1.myhost.com
          - broker: 2
            host: broker-2.myhost.com
    authorization:
      type: simple
    livenessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    readinessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    jvmOptions:
      -Xms: 65536m
      -Xmx: 65536m
    resources: 
      requests:
        memory: 64Gi
        cpu: "8"
      limits:
        memory: 72Gi
        cpu: "12"
    config:
      auto.create.topics.enable: "false"
      log.message.format.version: "3.1"
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      inter.broker.protocol.version: "3.1"
      log.retention.hours: 48
    storage:
      type: persistent-claim
      size: 3000Gi
      class: xfs-1-replica-local
    #metricsConfig:
      #type: jmxPrometheusExporter
      #valueFrom:
        #configMapKeyRef:
          #name: connect-metrics
          #key: kafka-metrics-config.yml
    template:
      pod:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                  - key: app
                    operator: In
                    values:
                    - kafka
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                - key: app.kubernetes.io/name
                  operator: In
                  values:
                  - kafka
              topologyKey: "kubernetes.io/hostname"
  zookeeper:
    replicas: 3
    livenessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    readinessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    storage:
      type: persistent-claim
      size: 5Gi
      class: xfs-2-replicas
    template:
      pod:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                  - key: app
                    operator: In
                    values:
                    - kafka
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                - key: app.kubernetes.io/name
                  operator: In
                  values:
                  - zookeeper
              topologyKey: "kubernetes.io/hostname"
  entityOperator:
    topicOperator: {}
    userOperator: {}

We are using ingress in our infra, so the bootstrap and bockers are pointed to same Ingress IP address

List of errors:

[kafka@strimzi-zookeeper-0 kafka]$ bin/zookeeper-shell.sh localhost:2181
Connecting to localhost:2181
Welcome to ZooKeeper!
JLine support is disabled
[2022-03-31 08:52:24,019] WARN Session 0x0 for sever localhost/127.0.0.1:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. (org.apache.zookeeper.ClientCnxn)
EndOfStreamException: Unable to read additional data from server sessionid 0x0, likely server has closed socket
at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290)

[kafka@strimzi-zookeeper-0 kafka]$ netstat -tuna
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 127.0.0.1:60192 127.0.0.1:12181 TIME_WAIT
tcp 0 0 127.0.0.1:60276 127.0.0.1:12181 TIME_WAIT
tcp 0 0 127.0.0.1:60334 127.0.0.1:12181 TIME_WAIT
tcp 0 0 127.0.0.1:60274 127.0.0.1:12181 TIME_WAIT
tcp 0 0 127.0.0.1:60190 127.0.0.1:12181 TIME_WAIT
tcp 0 0 127.0.0.1:60220 127.0.0.1:12181 TIME_WAIT
tcp 0 0 127.0.0.1:60218 127.0.0.1:12181 TIME_WAIT
tcp 0 0 127.0.0.1:60336 127.0.0.1:12181 TIME_WAIT
tcp 0 0 127.0.0.1:60248 127.0.0.1:12181 TIME_WAIT
tcp 0 0 127.0.0.1:60308 127.0.0.1:12181 TIME_WAIT
tcp 0 0 127.0.0.1:60250 127.0.0.1:12181 TIME_WAIT
tcp 0 0 127.0.0.1:60306 127.0.0.1:12181 TIME_WAIT
tcp6 0 0 :::42313 :::* LISTEN
tcp6 0 0 10.244.9.250:3888 :::* LISTEN
tcp6 0 0 :::8080 :::* LISTEN
tcp6 0 0 127.0.0.1:12181 :::* LISTEN
tcp6 0 0 :::2181 :::* LISTEN
tcp6 0 0 10.244.9.250:3888 10.244.11.177:53390 ESTABLISHED
tcp6 0 0 10.244.9.250:2181 10.244.11.179:55368 ESTABLISHED
tcp6 0 0 127.0.0.1:2181 127.0.0.1:53196 TIME_WAIT
tcp6 0 0 10.244.9.250:32848 10.244.8.243:2888 ESTABLISHED
tcp6 0 0 10.244.9.250:2181 10.244.8.244:37852 ESTABLISHED

[kafka@strimzi-zookeeper-0 kafka]$ bin/zookeeper-shell.sh strimzi-zookeeper-nodes:2181
Connecting to strimzi-zookeeper-nodes:2181
Welcome to ZooKeeper!
JLine support is disabled
[2022-03-31 08:53:48,161] WARN Session 0x0 for sever strimzi-zookeeper-nodes/10.244.11.177:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. (org.apache.zookeeper.ClientCnxn)
EndOfStreamException: Unable to read additional data from server sessionid 0x0, likely server has closed socket
at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290)
[2022-03-31 08:53:49,557] WARN Session 0x0 for sever strimzi-zookeeper-nodes/10.244.8.243:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. (org.apache.zookeeper.ClientCnxn)
EndOfStreamException: Unable to read additional data from server sessionid 0x0, likely server has closed socket
at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290)

From zookeeper pod:

2022-03-31 08:52:24,017 ERROR Unsuccessful handshake with session 0x0 (org.apache.zookeeper.server.NettyServerCnxnFactory) [nioEventLoopGroup-7-2]
2022-03-31 08:52:24,018 WARN Exception caught (org.apache.zookeeper.server.NettyServerCnxnFactory) [nioEventLoopGroup-7-2]
io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 0000002d000000000000000000000000000075300000000000000000000000100000000000000000000000000000000000

[kafka@strimzi-kafka-0 kafka]$ ./bin/kafka-topics.sh --bootstrap-server=prod-kafka-bootstrap.myhost:443 --topic kafka-topic --describe

Error while executing topic command : Timed out waiting for a node assignment. Call: listTopics
[2022-03-31 09:02:25,673] ERROR org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listTopics
(kafka.admin.TopicCommand$)

[kafka@strimzi-kafka-0 kafka]$ bin/kafka-topics.sh --create --bootstrap-server prod-kafka-bootstrap.myhost.com:443 --replication-factor 1 --partitions 1 --topic test-topic
Error while executing topic command : Timed out waiting for a node assignment. Call: createTopics
[2022-03-31 09:18:18,449] ERROR org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: createTopics
(kafka.admin.TopicCommand$)

Can you please let me know, what is the exact issue here and how can I resolve this?

[scholzj]

In general, ZooKeeper is not user-accessible in Strimzi. So you want be able to connect to it that way. I cannot say much for the KAfka errors apart from what the error message says.