Ingress controller not working with custom certificates. #7150

sk2991 · 2022-08-03T17:41:03Z

sk2991
Aug 3, 2022

Describe the bug
We have deployed Strimzi Kafka on Azure AKS with External loadbalancer. Whenever we try to connect to Ingress controller using Kafka console producer, we see below issue. I have enabled -Djavax.net.debug=ssl to see the exact issue

javax.net.ssl|SEVERE|16|kafka-producer-network-thread | console-producer|2022-08-03 12:29:15.408 CDT|null:-1|Fatal (UNEXPECTED_MESSAGE): Unexpected handshake message: server_hello (
"throwable" : {
  javax.net.ssl.SSLProtocolException: Unexpected handshake message: server_hello
        at sun.security.ssl.Alert.createSSLException(Unknown Source)
        at sun.security.ssl.Alert.createSSLException(Unknown Source)
        at sun.security.ssl.TransportContext.fatal(Unknown Source)
        at sun.security.ssl.TransportContext.fatal(Unknown Source)
        at sun.security.ssl.TransportContext.fatal(Unknown Source)
        at sun.security.ssl.HandshakeContext.dispatch(Unknown Source)
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(Unknown Source)
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(Unknown Source)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(Unknown Source)
        at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:430)
        at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:514)
        at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:368)
        at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:291)
        at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178)
        at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543)
        at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
        at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:560)
        at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:328)
        at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:243)
        at java.lang.Thread.run(Unknown Source)}

)

Kafka.yaml

    listeners: 
      - name: tls
        port: 9093
        type: internal
        tls: true
        authentication:
          type: tls    
      - name: external
        port: 9094
        type: ingress
        tls: true
        authentication:
          type: tls
        configuration:
          brokerCertChainAndKey:
            secretName:host-secret
            certificate: host.crt
            key: host.key
          bootstrap:
            annotations:
              kubernetes.io/ingress.class: nginx                        
            host: strimzi-kafka.internal.com
          brokers:
          - broker: 0
            host: kafka-broker-0.internal.com
            annotations:
              kubernetes.io/ingress.class: nginx                        
          - broker: 1
            host: kafka-broker-1.internal.com
            annotations:
              kubernetes.io/ingress.class: nginx                           
          - broker: 2
            host: kafka-broker-2.internal.com
            annotations:
              kubernetes.io/ingress.class: nginx         
    authorization:
      type: simple

scholzj · 2022-08-03T18:36:18Z

scholzj
Aug 3, 2022
Maintainer

It is really hard to say what the issue is without you sharing the full log. It could be for example that your ingress does not have TLS passthrough enabled.

15 replies

sk2991 Aug 3, 2022
Author

I have added new ca certificates, still wondering on why it is displaying strimzi certs. Any idea how can i use custom certs here?

Acceptable client certificate CA names
O = io.strimzi, CN = clients-ca v0
Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA1:RSA+SHA1
Shared Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits

sk2991 Aug 3, 2022
Author

I'm using console producer to connect to Kafka cluster - Let me know if you need any other logs

bin/kafka-console-producer.sh --bootstrap-server strimzi-kafka.internal.com:443 --producer-property security.protocol=SSL --producer-property ssl.truststore.password=password --producer-property ssl.truststore.location=./truststore.p12 --producer-property ssl.truststore.type=PKCS12 --producer-property ssl.keystore.password=password --producer-property ssl.keystore.location=./kafkauser-strimzi-user.internal.keystore.p12 --producer-property ssl.keystore.type=PKCS12 --topic my-topic

Log:

javax.net.ssl|FINE|01|main|2022-08-03 15:16:10.031 CDT|null:-1|jdk.tls.keyLimits:  entry = AES/GCM/NoPadding KeyUpdate 2^37. AES/GCM/NOPADDING:KEYUPDATE = 137438953472
javax.net.ssl|ALL|16|kafka-producer-network-thread | console-producer|2022-08-03 15:16:10.329 CDT|null:-1|No X.509 cert selected for EC
javax.net.ssl|ALL|16|kafka-producer-network-thread | console-producer|2022-08-03 15:16:10.329 CDT|null:-1|No X.509 cert selected for EC
javax.net.ssl|ALL|16|kafka-producer-network-thread | console-producer|2022-08-03 15:16:10.329 CDT|null:-1|No X.509 cert selected for EC
javax.net.ssl|ALL|16|kafka-producer-network-thread | console-producer|2022-08-03 15:16:10.329 CDT|null:-1|No X.509 cert selected for RSA
javax.net.ssl|ALL|16|kafka-producer-network-thread | console-producer|2022-08-03 15:16:10.330 CDT|null:-1|No X.509 cert selected for RSA
javax.net.ssl|ALL|16|kafka-producer-network-thread | console-producer|2022-08-03 15:16:10.330 CDT|null:-1|No X.509 cert selected for RSA
javax.net.ssl|ALL|16|kafka-producer-network-thread | console-producer|2022-08-03 15:16:10.330 CDT|null:-1|No X.509 cert selected for RSASSA-PSS
javax.net.ssl|ALL|16|kafka-producer-network-thread | console-producer|2022-08-03 15:16:10.330 CDT|null:-1|No X.509 cert selected for RSASSA-PSS
javax.net.ssl|ALL|16|kafka-producer-network-thread | console-producer|2022-08-03 15:16:10.330 CDT|null:-1|No X.509 cert selected for RSASSA-PSS
javax.net.ssl|ALL|16|kafka-producer-network-thread | console-producer|2022-08-03 15:16:10.330 CDT|null:-1|No X.509 cert selected for RSA
javax.net.ssl|ALL|16|kafka-producer-network-thread | console-producer|2022-08-03 15:16:10.330 CDT|null:-1|No X.509 cert selected for RSA
javax.net.ssl|ALL|16|kafka-producer-network-thread | console-producer|2022-08-03 15:16:10.330 CDT|null:-1|No X.509 cert selected for RSA
javax.net.ssl|ALL|16|kafka-producer-network-thread | console-producer|2022-08-03 15:16:10.330 CDT|null:-1|No X.509 cert selected for EC
javax.net.ssl|ALL|16|kafka-producer-network-thread | console-producer|2022-08-03 15:16:10.330 CDT|null:-1|No X.509 cert selected for RSA
javax.net.ssl|ALL|16|kafka-producer-network-thread | console-producer|2022-08-03 15:16:10.330 CDT|null:-1|No X.509 cert selected for EC
javax.net.ssl|ALL|16|kafka-producer-network-thread | console-producer|2022-08-03 15:16:10.331 CDT|null:-1|No X.509 cert selected for RSA
javax.net.ssl|SEVERE|16|kafka-producer-network-thread | console-producer|2022-08-03 15:16:10.404 CDT|null:-1|Fatal (UNEXPECTED_MESSAGE): Unexpected handshake message: server_hello (
"throwable" : {
  javax.net.ssl.SSLProtocolException: Unexpected handshake message: server_hello
        at sun.security.ssl.Alert.createSSLException(Unknown Source)
        at sun.security.ssl.Alert.createSSLException(Unknown Source)
        at sun.security.ssl.TransportContext.fatal(Unknown Source)
        at sun.security.ssl.TransportContext.fatal(Unknown Source)
        at sun.security.ssl.TransportContext.fatal(Unknown Source)
        at sun.security.ssl.HandshakeContext.dispatch(Unknown Source)
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(Unknown Source)
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(Unknown Source)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(Unknown Source)
        at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:430)
        at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:514)
        at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:368)
        at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:291)
        at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178)
        at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543)
        at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
        at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:560)
        at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:328)
        at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:243)
        at java.lang.Thread.run(Unknown Source)}

)
javax.net.ssl|WARNING|16|kafka-producer-network-thread | console-producer|2022-08-03 15:16:10.406 CDT|null:-1|outbound has closed, ignore outbound application data
[2022-08-03 15:16:10,408] ERROR [Producer clientId=console-producer] Connection to node -1 (strimzi-kafka.internal.com:443) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2022-08-03 15:16:10,408] WARN [Producer clientId=console-producer] Bootstrap broker strimzi-kafka.internal.com:443 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
javax.net.ssl|ALL|16|kafka-producer-network-thread | console-producer|2022-08-03 15:16:10.737 CDT|null:-1|No X.509 cert selected for EC
javax.net.ssl|ALL|16|kafka-producer-network-thread | console-producer|2022-08-03 15:16:10.737 CDT|null:-1|No X.509 cert selected for EC
javax.net.ssl|ALL|16|kafka-producer-network-thread | console-producer|2022-08-03 15:16:10.738 CDT|null:-1|No X.509 cert selected for EC
javax.net.ssl|ALL|16|kafka-producer-network-thread | console-producer|2022-08-03 15:16:10.738 CDT|null:-1|No X.509 cert selected for RSA
javax.net.ssl|ALL|16|kafka-producer-network-thread | console-producer|2022-08-03 15:16:10.738 CDT|null:-1|No X.509 cert selected for RSA
javax.net.ssl|ALL|16|kafka-producer-network-thread | console-producer|2022-08-03 15:16:10.738 CDT|null:-1|No X.509 cert selected for RSA
javax.net.ssl|ALL|16|kafka-producer-network-thread | console-producer|2022-08-03 15:16:10.738 CDT|null:-1|No X.509 cert selected for RSASSA-PSS
javax.net.ssl|ALL|16|kafka-producer-network-thread | console-producer|2022-08-03 15:16:10.738 CDT|null:-1|No X.509 cert selected for RSASSA-PSS
javax.net.ssl|ALL|16|kafka-producer-network-thread | console-producer|2022-08-03 15:16:10.738 CDT|null:-1|No X.509 cert selected for RSASSA-PSS
javax.net.ssl|ALL|16|kafka-producer-network-thread | console-producer|2022-08-03 15:16:10.738 CDT|null:-1|No X.509 cert selected for RSA
javax.net.ssl|ALL|16|kafka-producer-network-thread | console-producer|2022-08-03 15:16:10.739 CDT|null:-1|No X.509 cert selected for RSA
javax.net.ssl|ALL|16|kafka-producer-network-thread | console-producer|2022-08-03 15:16:10.739 CDT|null:-1|No X.509 cert selected for RSA
javax.net.ssl|ALL|16|kafka-producer-network-thread | console-producer|2022-08-03 15:16:10.739 CDT|null:-1|No X.509 cert selected for EC
javax.net.ssl|ALL|16|kafka-producer-network-thread | console-producer|2022-08-03 15:16:10.739 CDT|null:-1|No X.509 cert selected for RSA
javax.net.ssl|ALL|16|kafka-producer-network-thread | console-producer|2022-08-03 15:16:10.739 CDT|null:-1|No X.509 cert selected for EC
javax.net.ssl|ALL|16|kafka-producer-network-thread | console-producer|2022-08-03 15:16:10.739 CDT|null:-1|No X.509 cert selected for RSA
javax.net.ssl|SEVERE|16|kafka-producer-network-thread | console-producer|2022-08-03 15:16:10.803 CDT|null:-1|Fatal (UNEXPECTED_MESSAGE): Unexpected

scholzj Aug 3, 2022
Maintainer

That is for the TLS client authentication. The server certificate you configured in the listener is configured only as a server certificate. If you want to use your own CA for the client authentication, you will need to configure the custom Clients CA: https://strimzi.io/docs/operators/latest/full/configuring.html#installing-your-own-ca-certificates-str

sk2991 Aug 4, 2022
Author

Thanks for sharing the documentation. In the documentation, there is a step to create ca key secret. I was not able to find any ca.key and i don't see CA-KEY-SECRET generated for the cluster.

kubectl create secret generic CA-KEY-SECRET --from-file=ca.key=ca.key
kubectl label secret CA-CERTIFICATE-SECRET strimzi.io/kind=Kafka strimzi.io/cluster=CLUSTER-NAME
kubectl label secret CA-KEY-SECRET strimzi.io/kind=Kafka strimzi.io/cluster=CLUSTER-NAME

scholzj Aug 4, 2022
Maintainer

These names are placeholders. You have to replace them with the real names. The naming is explained in the beginning of the chapter I linked above.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strimzi

Ingress controller not working with custom certificates. #7150

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment 15 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Strimzi

Ingress controller not working with custom certificates. #7150

Uh oh!

sk2991 Aug 3, 2022

Replies: 1 comment · 15 replies

Uh oh!

scholzj Aug 3, 2022 Maintainer

Uh oh!

sk2991 Aug 3, 2022 Author

Uh oh!

sk2991 Aug 3, 2022 Author

Uh oh!

scholzj Aug 3, 2022 Maintainer

Uh oh!

sk2991 Aug 4, 2022 Author

Uh oh!

scholzj Aug 4, 2022 Maintainer

sk2991
Aug 3, 2022

Replies: 1 comment 15 replies

scholzj
Aug 3, 2022
Maintainer

sk2991 Aug 3, 2022
Author

sk2991 Aug 3, 2022
Author

scholzj Aug 3, 2022
Maintainer

sk2991 Aug 4, 2022
Author

scholzj Aug 4, 2022
Maintainer