[SocketServer listenerType=ZK_BROKER, nodeId=0] Failed authentication with /10.x.x.x (channelId=10.x.x.x:9090-10.x.x.x:54428-1) (SSL handshake failed) (org.apache.kafka.common.network.Selector) [control-plane-kafka-network-thread-0-ListenerName(CONTROLPLANE-9090)-SSL-0]

[raghunadh999]

Dears,

Could some help me please on this issue with Kafka. We have been struggling to find RC and resolution for this issue since weeks and couldn't get luck.
My application is deployed in Openshift and kafka as well with Strimzi operator..
This issue is present in all environments for my application.

Log:
2023-04-26 08:29:00,295 INFO [UnifiedLog partition=__strimzi-topic-operator-kstreams-topic-store-changelog-0, dir=/var/lib/kafka/data/kafka-log0] Truncating to 0 has no effect as the largest offset in the log is -1 (kafka.log.UnifiedLog) [ReplicaFetcherThread-0-2]
Apr 27, 2023 5:47:29 AM sun.rmi.transport.tcp.TCPTransport$AcceptLoop run
WARNING: RMI TCP Accept-0: accept loop for ServerSocket[addr=0.0.0.0/0.0.0.0,localport=35403] throws
java.io.IOException: The server sockets created using the LocalRMIServerSocketFactory only accept connections from clients running on the host where the RMI remote objects have been exported.
at jdk.management.agent/sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:114)
at java.rmi/sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:394)
at java.rmi/sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:366)
at java.base/java.lang.Thread.run(Thread.java:829)
Apr 27, 2023 5:47:29 AM sun.rmi.transport.tcp.TCPTransport$AcceptLoop run
WARNING: RMI TCP Accept-0: accept loop for ServerSocket[addr=0.0.0.0/0.0.0.0,localport=35403] throws
java.io.IOException: The server sockets created using the LocalRMIServerSocketFactory only accept connections from clients running on the host where the RMI remote objects have been exported.
at jdk.management.agent/sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:114)
at java.rmi/sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:394)
at java.rmi/sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:366)
at java.base/java.lang.Thread.run(Thread.java:829)
2023-04-27 05:47:31,810 INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Failed authentication with /10.x.x.x (channelId=10.x.xx.xxx:9090-10.x.x.x:54428-1) (SSL handshake failed) (org.apache.kafka.common.network.Selector) [control-plane-kafka-network-thread-0-ListenerName(CONTROLPLANE-9090)-SSL-0]
2023-04-27 05:47:31,819 INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Failed authentication with /10.x.x.x (channelId=10.x.xx.xxx:9090-10.x.x.x:54438-2) (SSL handshake failed) (org.apache.kafka.common.network.Selector) [control-plane-kafka-network-thread-0-ListenerName(CONTROLPLANE-9090)-SSL-0]
2023-04-27 05:47:33,824 INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Failed authentication with /10.x.x.x (channelId=10.x.xx.xxx:9091-10.x.x.x:53350-461) (SSL handshake failed) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-0-ListenerName(REPLICATION-9091)-SSL-3]
2023-04-27 05:47:33,834 INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Failed authentication with /10.x.x.x (channelId=10.x.xx.xxx:9091-10.x.x.x:53352-462) (SSL handshake failed) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-0-ListenerName(REPLICATION-9091)-SSL-1]
2023-04-27 05:47:35,846 WARN [SocketServer listenerType=ZK_BROKER, nodeId=0] Unexpected error from /10.x.x.x (channelId=10.x.xx.xxx:9092-10.x.x.x:55178-0); closing connection (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-0-ListenerName(PLAINTEXT-9092)-PLAINTEXT-4]
org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 1330664521 larger than 104857600)
at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:105)
at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:452)
at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:402)
at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:674)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:576)
at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
at kafka.network.Processor.poll(SocketServer.scala:1144)
at kafka.network.Processor.run(SocketServer.scala:1047)
at java.base/java.lang.Thread.run(Thread.java:829)
2023-04-27 05:47:35,852 WARN [SocketServer listenerType=ZK_BROKER, nodeId=0] Unexpected error from /10.x.x.x (channelId=10.x.xx.xxx:9092-10.x.x.x:55182-0); closing connection (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-0-ListenerName(PLAINTEXT-9092)-PLAINTEXT-5]
org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 369295616 larger than 104857600)
at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:105)
at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:452)
at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:402)
at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:674)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:576)
at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
at kafka.network.Processor.poll(SocketServer.scala:1144)
at kafka.network.Processor.run(SocketServer.scala:1047)
at java.base/java.lang.Thread.run(Thread.java:829)

Kafka Config:
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
spec:
entityOperator:
topicOperator: {}
userOperator: {}
kafka:
config:
inter.broker.protocol.version: '3.2'
transaction.state.log.replication.factor: 3
min.insync.replicas: 2
num.recovery.threads.per.data.dir: 2
num.partitions: 10
log.message.format.version: '3.2'
transaction.state.log.min.isr: 2
offsets.topic.replication.factor: 3
default.replication.factor: 3
resources:
limits:
cpu: 500m
memory: 1Gi
readinessProbe:
failureThreshold: 3
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 15
livenessProbe:
failureThreshold: 3
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 30
version: 3.2.3
storage:
type: ephemeral
replicas: 3
jvmOptions:
'-Xms': 700m
'-Xmx': 700m
listeners:
- name: plaintext
port: 9092
tls: false
type: internal
logging:
loggers:
kafka.root.logger.level: INFO
type: inline
zookeeper:
jvmOptions:
'-Xms': 750m
'-Xmx': 750m
livenessProbe:
failureThreshold: 3
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 30
logging:
loggers:
zookeeper.root.logger: INFO
type: inline
readinessProbe:
failureThreshold: 3
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 15
replicas: 3
resources:
limits:
memory: 1Gi
storage:
type: ephemeral
status:
clusterId: uwJOHA8mQ-6D0EWvc
conditions:
- lastTransitionTime: '2023-04-26T08:29:10.879Z'
status: 'True'
type: Ready
listeners:
- addresses:
- host: xxx-kafka-cluster-kafka-bootstrap.psu-xxx.svc
port: 9092
bootstrapServers: 'xxx-kafka-cluster-kafka-bootstrap.psu-xxx.svc:9092'
name: plaintext
type: plaintext
observedGeneration: 2

[scholzj]

All the errors suggest that something is trying to connect to all the Kafka broker ports without properly configured TLS. It seems to try all of them -> even the 9090 and 9091 which should normally be protected by the network policies. You should try to find out what IP address it is connecting from.