How to fix "Unexpected handshake message: server_hello"

[NathanAvx]

Hi Team,

Tried to access Kafka using external Ingress listener but SSL handshake failed and got the below exception.

/kafka_2.13-2.7.0/bin$ ./kafka-console-consumer.sh --bootstrap-server bootstrap.192.168.49.2.nip.io:443 --consumer-property security.protocol=SSL --consumer-property ssl.truststore.password=xxxxxx --consumer-property ssl.truststore.location=/path/truststore.jks --topic my-topic


javax.net.ssl|FINE|01|main|2021-02-10 13:19:14.157 IST|Logger.java:765|jdk.tls.keyLimits:  entry = AES/GCM/NoPadding KeyUpdate 2^37. AES/GCM/NOPADDING:KEYUPDATE = 137438953472
javax.net.ssl|ALL|01|main|2021-02-10 13:19:15.147 IST|Logger.java:765|No X.509 cert selected for EC
javax.net.ssl|ALL|01|main|2021-02-10 13:19:15.147 IST|Logger.java:765|No X.509 cert selected for EC
javax.net.ssl|ALL|01|main|2021-02-10 13:19:15.147 IST|Logger.java:765|No X.509 cert selected for EC
javax.net.ssl|ALL|01|main|2021-02-10 13:19:15.147 IST|Logger.java:765|No X.509 cert selected for RSA
javax.net.ssl|ALL|01|main|2021-02-10 13:19:15.148 IST|Logger.java:765|No X.509 cert selected for RSA
javax.net.ssl|ALL|01|main|2021-02-10 13:19:15.148 IST|Logger.java:765|No X.509 cert selected for RSA
javax.net.ssl|ALL|01|main|2021-02-10 13:19:15.148 IST|Logger.java:765|No X.509 cert selected for RSASSA-PSS
javax.net.ssl|ALL|01|main|2021-02-10 13:19:15.148 IST|Logger.java:765|No X.509 cert selected for RSASSA-PSS
javax.net.ssl|ALL|01|main|2021-02-10 13:19:15.149 IST|Logger.java:765|No X.509 cert selected for RSASSA-PSS
javax.net.ssl|ALL|01|main|2021-02-10 13:19:15.149 IST|Logger.java:765|No X.509 cert selected for RSA
javax.net.ssl|ALL|01|main|2021-02-10 13:19:15.149 IST|Logger.java:765|No X.509 cert selected for RSA
javax.net.ssl|ALL|01|main|2021-02-10 13:19:15.149 IST|Logger.java:765|No X.509 cert selected for RSA
javax.net.ssl|ALL|01|main|2021-02-10 13:19:15.149 IST|Logger.java:765|No X.509 cert selected for DSA
javax.net.ssl|ALL|01|main|2021-02-10 13:19:15.150 IST|Logger.java:765|No X.509 cert selected for EC
javax.net.ssl|ALL|01|main|2021-02-10 13:19:15.150 IST|Logger.java:765|No X.509 cert selected for RSA
javax.net.ssl|ALL|01|main|2021-02-10 13:19:15.150 IST|Logger.java:765|No X.509 cert selected for DSA
javax.net.ssl|ALL|01|main|2021-02-10 13:19:15.150 IST|Logger.java:765|No X.509 cert selected for EC
javax.net.ssl|ALL|01|main|2021-02-10 13:19:15.150 IST|Logger.java:765|No X.509 cert selected for RSA
javax.net.ssl|ALL|01|main|2021-02-10 13:19:15.151 IST|Logger.java:765|No X.509 cert selected for DSA
javax.net.ssl|SEVERE|01|main|2021-02-10 13:19:15.242 IST|Logger.java:765|Fatal (UNEXPECTED_MESSAGE): **Unexpected handshake message: server_hello** (
"throwable" : {
  javax.net.ssl.SSLProtocolException: Unexpected handshake message: server_hello
  	at sun.security.ssl.Alert.createSSLException(Alert.java:129)
  	at sun.security.ssl.Alert.createSSLException(Alert.java:117)
  	at sun.security.ssl.TransportContext.fatal(TransportContext.java:311)
  	at sun.security.ssl.TransportContext.fatal(TransportContext.java:267)
  	at sun.security.ssl.TransportContext.fatal(TransportContext.java:258)
  	at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:438)
  	at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:968)
  	at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:955)
  	at java.security.AccessController.doPrivileged(Native Method)
  	at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:902)
  	at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:430)
  	at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:514)
  	at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:368)
  	at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:291)
  	at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:173)
  	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543)
  	at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
  	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:561)
  	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265)
  	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236)
  	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:215)
  	at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:244)
  	at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:480)
  	at org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1257)
  	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1226)
  	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1206)
  	at kafka.tools.ConsoleConsumer$ConsumerWrapper.receive(ConsoleConsumer.scala:444)
  	at kafka.tools.ConsoleConsumer$.process(ConsoleConsumer.scala:103)
  	at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:77)
  	at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:54)
  	at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)}

)
[2021-02-10 13:19:15,246] ERROR [Consumer clientId=consumer-console-consumer-74880-1, groupId=console-consumer-74880] Connection to node -1 (bootstrap.192.168.49.2.nip.io/192.168.49.2:443) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2021-02-10 13:19:15,247] WARN [Consumer clientId=consumer-console-consumer-74880-1, groupId=console-consumer-74880] Bootstrap broker bootstrap.192.168.49.2.nip.io:443 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2021-02-10 13:19:15,263] ERROR Error processing message, terminating consumer process:  (kafka.tools.ConsoleConsumer$)
**org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed**
**Caused by: javax.net.ssl.SSLProtocolException: Unexpected handshake message: server_hello**
	at sun.security.ssl.Alert.createSSLException(Alert.java:129)
	at sun.security.ssl.Alert.createSSLException(Alert.java:117)
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:311)
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:267)
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:258)
	at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:438)
	at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:968)
	at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:955)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:902)
	at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:430)
	at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:514)
	at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:368)
	at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:291)
	at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:173)
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543)
	at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:561)
	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265)
	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236)
	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:215)
	at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:244)
	at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:480)
	at org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1257)
	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1226)
	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1206)
	at kafka.tools.ConsoleConsumer$ConsumerWrapper.receive(ConsoleConsumer.scala:444)
	at kafka.tools.ConsoleConsumer$.process(ConsoleConsumer.scala:103)
	at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:77)
	at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:54)
	at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)
Processed a total of 0 messages

Config as follows :

> apiVersion: kafka.strimzi.io/v1beta1
> 
> kind: Kafka
> metadata:
>   name: my-cluster
> spec:
>   kafka:
>     version: 2.7.0
>     replicas: 1
>     listeners:
>       - name: plain
>         port: 9092
>         type: internal
>         tls: false
>       - name: tls
>         port: 9093
>         type: internal
>         tls: true
>       - name: external
>         port: 9094
>         type: ingress
>         tls: true
>         authentication:
>           type: tls
>         configuration:
>           bootstrap:
>             host: bootstrap.192.168.49.2.nip.io
>           brokers:
>           - broker: 0
>             host: broker-0.192.168.49.2.nip.io
>     config:
>       offsets.topic.replication.factor: 1
>       transaction.state.log.replication.factor: 1
>       transaction.state.log.min.isr: 1
>       log.message.format.version: "2.7"
>       inter.broker.protocol.version: "2.7"
>     storage:
>       type: jbod
>       volumes:
>       - id: 0
>         type: persistent-claim
>         size: 10Gi
>         deleteClaim: false
>   zookeeper:
>     replicas: 1
>     storage:
>       type: persistent-claim
>       size: 10Gi
>       deleteClaim: false
>   entityOperator:
>     topicOperator: {}
>     userOperator: {}
> 

SSL pass through is enable in Nginx:

What am I missing ? Please help me solve this SSL handshake failure.

[scholzj]

You Ingress listener is configured with authentication:

        authentication:
          type: tls

But the client configures only the truststore. So you have two options:

1. Remove the authentication from the Kafka CR
2. Create a KafkaUser resource with TLS authentication, get the certificate it generates and use it in your client.

I think that one of these should fix this.

[NathanAvx]

Thank you very much for replying. Let me try it.

[NathanAvx]

Tried both the options it worked. Thanks @scholzj

[CsBigDataHub]

@scholzj I have been getting the same error, I have used custom certs generated by Venafi for external listener(using load balancer).

Everything work fine if I use the my-user tls certs generated by strimzi but If I try to delete the my-user secret to replace it with custom tls certs , I have been getting the same error. Unexpected handshake message: server_hello

Is it possible configure custom certs for user by deleting the secret and recreating it with same name but different certs?

[scholzj]

No, that is not how it works. The secret is for clients to use the certs from it. But it has nothing to do with the broker trusting the certificate. That is achieved through it being signed by the Clients CA. So what you need is configuring your own Clients CA instead.

[CsBigDataHub]

got it, Thanks. Essentially follow this example?

https://github.com/scholzj/strimzi-custom-ca-test

If I replace strimizi generated my-cluster-clients-ca and my-cluster-clients-ca-cert with the ones that I create from Vanefi, I will be able to issue Venafi certs for the user aswell?

[scholzj]

Yeah. It is also described in the docs: https://strimzi.io/docs/operators/latest/full/using.html#installing-your-own-ca-certificates-str ... Just keep in mind that that repository has custom cluster and clients CA. For your use case as you described it, you probably need only the clients CA.

Also, if you generate the certificates on your own, you might need to decide what to do with the User Operator. In 0.25 (hopefully released some time next week), we add a new authentication type tls-external to the KafkaUser resource for these cases. Until then, the User Operator actually needs the full CA and it always generates the user certs as well. That would duplicate your own user cert. So until 0.25, you might need to disable or not use it.