kafka connect zk timeout

[lvyanru8200]

kafka Operator version: 0.28

[lvyanru8200]

log:

io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection timed out: fage-kafka-yaml-zookeeper-client/172.31.244.212:2181
  | Caused by: java.net.ConnectException: Connection timed out
  | at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
  | at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)
  | at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
  | at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
  | at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:707)
  | at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
  | at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
  | at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
  | at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
  | at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
  | at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
  | at java.base/java.lang.Thread.run(Thread.java:829)
  | 2022-12-07 08:04:00,944 INFO Opening socket connection to server fage-kafka-yaml-zookeeper-client/172.31.244.212:2181. (org.apache.zookeeper.ClientCnxn) [main-SendThread(fage-kafka-yaml-zookeeper-client:2181)]
  | 2022-12-07 08:04:00,944 INFO SASL config status: Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn) [main-SendThread(fage-kafka-yaml-zookeeper-client:2181)]

[lvyanru8200]

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
annotations:
zhujie: '5'
generation: 9
labels:
name: fage
name: fage-kafka-yaml
namespace: tankaihua-test-01
spec:
entityOperator:
tlsSidecar:
livenessProbe:
initialDelaySeconds: 30
timeoutSeconds: 50
readinessProbe:
initialDelaySeconds: 30
timeoutSeconds: 50
topicOperator:
livenessProbe:
initialDelaySeconds: 30
timeoutSeconds: 50
logging:
type: inline
userOperator:
livenessProbe:
initialDelaySeconds: 30
timeoutSeconds: 50
logging:
type: inline
kafka:
authorization:
type: simple
config:
default.replication.factor: 3
inter.broker.protocol.version: '3.1'
log.message.format.version: '3.1'
min.insync.replicas: 3
offsets.topic.replication.factor: 3
transaction.state.log.min.isr: 3
transaction.state.log.replication.factor: 3
zookeeper.connection.timeout.ms: 400000
listeners:
- name: plain
port: 9092
tls: false
type: internal
- authentication:
type: tls
name: tls
port: 9093
tls: true
type: internal
livenessProbe:
initialDelaySeconds: 100
timeoutSeconds: 100
logging:
type: inline
rack:
topologyKey: kubernetes.io/os
readinessProbe:
initialDelaySeconds: 100
timeoutSeconds: 100
replicas: 3
resources:
limits:
cpu: '1'
ephemeral-storage: 1Gi
memory: 1Gi
requests:
cpu: '1'
ephemeral-storage: 1Gi
memory: 1Gi
storage:
class: managed-nfs-storage
deleteClaim: true
size: 1Gi
type: persistent-claim
version: 3.1.0
zookeeper:
jvmOptions:
javaSystemProperties:
- name: zookeeper.ssl.hostnameVerification
value: ”false“
- name: zookeeper.ssl.quorum.hostnameVerification
value: “false”
livenessProbe:
initialDelaySeconds: 30
timeoutSeconds: 50
logging:
type: inline
readinessProbe:
initialDelaySeconds: 30
timeoutSeconds: 50
replicas: 3
resources:
limits:
cpu: '1'
ephemeral-storage: 1Gi
memory: 1Gi
requests:
cpu: '1'
ephemeral-storage: 1Gi
memory: 1Gi
storage:
class: managed-nfs-storage
deleteClaim: true
size: 1Gi
type: persistent-claim

[lvyanru8200]

Only two of the three kafka nodes can connect to zk properly

[scholzj]

That probably suggests some networking issue affecting only some worker nodes maybe?

[lvyanru8200]

这可能表明某些网络问题可能只影响某些工作节点？

Very strange, I changed the storageclass to local-storage and it works, but the storageclass provided by nfs does not work

[scholzj]

NFS does not work well with Kafka. But that does not seem related as it would for sure not cause timeout issue like this. Maybe the local storage just affected scheduling and you were not scheduled onto the node with networking issues? I don't know. TBH, Connection timed out error can mean many different things. So it is really hard to say.

[lvyanru8200]

cc @scholzj thanks