Kafka broker unable to connect to Zookeeper in latest 0.22.1 version

[hari819]

Hello ,

i am trying to create kafka cluster in k8s environment and expose the kafka service using ingress and the documentation link ,https://strimzi.io/blog/2019/05/23/accessing-kafka-part-5/

following is my yaml file ,

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
name: kafka
spec:
kafka:
version: 2.7.0
replicas: 3
listeners:
- name: plain
port: 9092
type: internal
tls: false
- name: tls
port: 9093
type: internal
tls: true
- name: external
port: 9094
type: ingress
tls: true
configuration:
bootstrap:
host: kafka-bootstrap.XXX.XXXX
annotations:
external-dns.alpha.kubernetes.io/hostname: kafka-bootstrap.XXX.XXXX
external-dns.alpha.kubernetes.io/ttl: "60"
brokers:
- broker: 0
host: kafka-broker-0.XXX.XXXX
annotations:
external-dns.alpha.kubernetes.io/hostname: kafka-broker-0.XXX.XXXX
external-dns.alpha.kubernetes.io/ttl: "60"
- broker: 1
host: kafka-broker-1.XXX.XXXX
annotations:
external-dns.alpha.kubernetes.io/hostname: kafka-broker-1.XXX.XXXX
external-dns.alpha.kubernetes.io/ttl: "60"
- broker: 2
host: kafka-broker-2.XXX.XXXX
annotations:
external-dns.alpha.kubernetes.io/hostname: kafka-broker-2.XXX.XXXX
external-dns.alpha.kubernetes.io/ttl: "60"
config:
offsets.topic.replication.factor: 3
transaction.state.log.replication.factor: 3
transaction.state.log.min.isr: 2
log.message.format.version: "2.7"
inter.broker.protocol.version: "2.7"
zookeeper.connection.timeout.ms: 90000
zookeeper.session.timeout.ms: 30000
readinessProbe:
initialDelaySeconds: 300
timeoutSeconds: 10
livenessProbe:
initialDelaySeconds: 100
timeoutSeconds: 10
periodSeconds: 2
failureThreshold: 3
logging:
type: inline
loggers:
kafka.root.logger.level: "DEBUG"
storage:
type: jbod
volumes:
- id: 0
type: persistent-claim
class: nfs-local-storage-hdd-2
size: 10Gi
deleteClaim: true
zookeeper:
replicas: 3
storage:
type: persistent-claim
class: nfs-local-storage-hdd-2
size: 10Gi
deleteClaim: true
jvmOptions:
javaSystemProperties:
- name: zookeeper.ssl.hostnameVerification
value: "false"
- name: zookeeper.ssl.quorum.hostnameVerification
value: "false"
- name: javax.net.debug
value: "ssl:handshake:verbose"
logging:
type: inline
loggers:
zookeeper.root.logger: "DEBUG"

ZK cluster is coming up fine , but it is not allowing Kafka to connect to it , so kafka brokers keep crashlooping ,

the error i see in brokers is the timeout error ,
2021-05-05 06:41:02,336 WARN Client session timed out, have not heard from server in 30356ms for sessionid 0x0 (org.apache.zookeeper.ClientCnxn) [main-SendThread(kafka-zookeeper-client:2181)] 2021-05-05 06:41:02,337 INFO Client session timed out, have not heard from server in 30356ms for sessionid 0x0, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn) [main-SendThread(kafka-zookeeper-client:2181)] 2021-05-05 06:41:02,689 INFO future isn't success, cause: (org.apache.zookeeper.ClientCnxnSocketNetty) [nioEventLoopGroup-2-1] **io.netty.channel.ConnectTimeoutException: connection timed out: kafka-zookeeper-client/10.223.180.129:2181** at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:261) at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:834) 2021-05-05 06:41:03,350 INFO Opening socket connection to server kafka-zookeeper-client/10.223.180.129:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn) [main-SendThread(kafka-zookeeper-client:2181)] 2021-05-05 06:41:03,460 INFO SSL handler added for channel: [id: 0xccb54676] (org.apache.zookeeper.ClientCnxnSocketNetty) [nioEventLoopGroup-2-1]

i can see ZK cluster is green always ,

Thanks,

[scholzj]

Well, you need to start by checking the ZooKeeper logs and the networking etc.

[hari819]

also i am expecting the same port which I have overridden in cluster.yaml to be set in the advertiesed.listeners=

EXTERNAL-9094://xxx.xx.xx.xx:port

, as well ,

even this is also set as a different port , am i missing something basic here?

[hari819]

@scholzj

this is the log from the operator ,

AbstractOperator:144 - Kafka resource kafka in namespace *: Contains object at path spec.kafka.listeners.genericKafkaListeners with an unknown property: overrides

so what is the way I can set a pre-defined nodeport?

[scholzj]

You are mixing the old configuration and new configuration. With the array based listeners it should be like this:

  - name: external
    port: 9094
    type: nodeport
    tls: true
    configuration:
      bootstrap:
        nodePort: 32269

Also, to affect the advertised address, you need to set this for the brokers since that are the advertised things. The bootstrap is not advertised by the brokers.

[hari819]

thankyou @scholzj ,

i think i will explain the requirement for clarity , i am trying to setup a kafka cluster in k8s environment ,along with this i will need kafka to be accessible outside the k8s environment where the developers can use it to create producers , consumers for message transfers,

for that i tried ingress first , but it looked complex with the setup we have like ingress +externalDNS ,
so i switched to nodeport type and trying to address the same issue of accessing kafka from outside the cluster ,

so as per your reply , i will have to set three different nodeport values to brokers along with bootstrap , in the end i will be able to access the kafka connection using the value set for "bootstrap" nodePort ?

is this correct , am i in the right direction towards this ?

Thanks,

[scholzj]

The client uses the bootstrap address for the initial connection used to obtain the metadata from the Kafka cluster. The metadata will tell it the addresses for the individual brokers (these are the advertised addresses) and the client will use these to open additional connections directly to the brokers to produce, consume messages etc. So at the end the client will usually use all of them.

[hari819]

thankyou @scholzj , i will try with setting ports for brokers

[hari819]

@scholzj , well it was the side effect of using NFS storage for zookeeper , problem solved now,

Thanks,

[scholzj]

Interesting ... never heard about anything like that. But glad it works now and thanks for the update.

[hari819]

@scholzj , yes it is interesting , it works now which is good ,

thanks,

[hari819]

@scholzj , here is the interesting link , https://stackoverflow.com/questions/60900481/kafka-doesnt-work-with-external-nfs-volume

thanks,

[scholzj]

Well, I know Kafka does not work with NFS. That is also covered in the docs. But I never saw this show up with an error like this.