SSL Handshake error with strimzi kafka 0.23.0

[rprincer]

We have setup the Strimzi kafka operator, client examples with these command.
Did not do any manual certificate generation.

helm repo add strimzi https://strimzi.io/charts/ -n strimzi-kafka
helm install kafka-strimzi strimzi/strimzi-kafka-operator --version 0.23.0 -f ./values_strimzi_org.yaml -n strimzi-kafka --set watchAnyNamespace=true
kubectl apply -f ./kafka.yml -n strimzi-kafka
kubectl apply -f ./topic.yml -n strimzi-kafka
kubectl apply -f ./prod-con-users.yml -n strimzi-kafka
kubectl apply -f ./strimzi-client-example.yml -n strimzi-kafka

Attached the commands.

Operator comes up fine without any issues along with kafka and zookeper.
however produser fails with the below SSL handshake error.

javax.net.ssl|ALL|4A|data-plane-kafka-network-thread-0-ListenerName(SASLTLS-9093)-SSL-10|2021-06-17 11:35:05.946 UTC|X509Authentication.java:295|No X.509 cert selected for EC
javax.net.ssl|DEBUG|4A|data-plane-kafka-network-thread-0-ListenerName(SASLTLS-9093)-SSL-10|2021-06-17 11:35:05.955 UTC|SSLCipher.java:1994|KeyLimit write side: algorithm = AES/GCM/NOPADDING:KEYUPDATE
countdown value = 137438953472
javax.net.ssl|ERROR|4A|data-plane-kafka-network-thread-0-ListenerName(SASLTLS-9093)-SSL-10|2021-06-17 11:35:05.961 UTC|TransportContext.java:341|Fatal (UNEXPECTED_MESSAGE): Unexpected content: 20 (
"throwable" : {
javax.net.ssl.SSLProtocolException: Unexpected content: 20
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:129)
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:336)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:292)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:283)
at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:192)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:171)
at java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:681)
at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:636)
at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:454)
at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:433)
at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:637)
at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:509)
at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:368)
at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:291)
at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543)
at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
2021-06-17 11:35:05,962 INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Failed authentication with /10.42.5.87 (SSL handshake failed) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-0-ListenerName(SASLTLS-9093)-SSL-10]
at kafka.network.Processor.poll(SocketServer.scala:989)
at kafka.network.Processor.run(SocketServer.scala:892)
at java.base/java.lang.Thread.run(Thread.java:829)}
)
javax.net.ssl|WARNING|4A|data-plane-kafka-network-thread-0-ListenerName(SASLTLS-9093)-SSL-10|2021-06-17 11:35:05.962 UTC|SSLEngineOutputRecord.java:168|outbound has closed, ignore outbound application data
javax.net.ssl|DEBUG|4B|data-plane-kafka-network-thread-0-ListenerName(SASLTLS-9093)-SSL-11|2021-06-17 11:35:06.136 UTC|SSLCipher.java:1840|KeyLimit read side: algorithm = [AES/GCM/NOPADDING:KEYUPDATE]([url](url
configs.zip
))

[scholzj]

I never saw this kind of error. Do you have something intercepting the traffic such as Istio etc.?

[rprincer]

Thanks scholzj for your response.

Yes, we have ISTIO as a service mesh on this cluster.
However we did not inject ISTIO in the namsespace where strimzi operator, kafka, zookeeper are running.

We verified the strimzi app related pods and dont see istio-init, so we can confirm istio injection is not enabled.

[scholzj]

Actually, your setup is a bit weird. It looks like your client is running inside the same Kubernetes cluster as your Kafka cluster. But is connecting through the node port. So one thing to try would be to use an internal listener for the connection and not the external one (type: internal instead of type: nodeport in the Kafka cluster).

[rprincer]

thanks .. will give it a try and get back

[rprincer]

Thanks a lot scholzj.. changing to internal port did the trick.

[scholzj]

I guess that means there is something wrong with the node ports ... wrong in a sense that they are not accessible or something like that. I guess you might run into this again once you need to connect an client running outside the Kubernetes :-/