SSL handshake:- Failed to process post-handshake messages

[subinmt]

Hi Team,

I am running Kafka cluster with ingress external listener. But when I am trying to test the producer/consumer I am getting the following error message.

nginx version: nginx/1.19.9
Strimzi Kafka Operator version: 0.27.1

$ bin/kafka-console-producer.sh --broker-list dev-kafka-bootstrap.mydomain.com:443 --producer-property security.protocol=SSL --producer-property ssl.truststore.password=password --producer-property ssl.truststore.location=/tmp/truststore.jks --topic kafka-topic

[2022-02-15 06:09:48,699] WARN [Producer clientId=console-producer] Bootstrap broker dev-kafka-bootstrap.mydomain.com:443 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2022-02-15 06:09:48,869] ERROR [Producer clientId=console-producer] Connection to node -1 (dev-kafka-bootstrap.mydomain.com/30.xx.xx.xx:443) failed authentication due to: Failed to process post-handshake messages (org.apache.kafka.clients.NetworkClient)
[2022-02-15 06:09:48,869] WARN [Producer clientId=console-producer] Bootstrap broker dev-kafka-bootstrap.mydomain.com:443 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2022-02-15 06:09:48,998] ERROR [Producer clientId=console-producer] Connection to node -1 (dev-kafka-bootstrap.mydomain.com/30.xx.xx.xx:443) failed authentication due to: Failed to process post-handshake messages (org.apache.kafka.clients.NetworkClient)
[2022-02-15 06:09:48,998] WARN [Producer clientId=console-producer] Bootstrap broker dev-kafka-bootstrap.mydomain.com:443 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient

Ingress Controller logs:

2022/02/15 01:41:24 http: TLS handshake error from 102.xx.xx.xx:53074: EOF
E0215 02:51:10.305192       7 server.go:75] "Failed to decode request body" err="couldn't get version/kind; json parse error: unexpected end of JSON input"
2022/02/15 02:51:10 http: TLS handshake error from 167.xx.xx.xx:53220: tls: client requested unsupported application protocols ([h2])
E0215 03:51:48.779847       7 server.go:75] "Failed to decode request body" err="couldn't get version/kind; json parse error: unexpected end of JSON input"
E0215 06:03:24.303060       7 server.go:75] "Failed to decode request body" err="couldn't get version/kind; json parse error: unexpected end of JSON input

Nginx ingress controller installed with SSL Passthrough:

    Args:
      /nginx-ingress-controller
      --publish-service=$(POD_NAMESPACE)/ingress-kafka-ingress-nginx-controller
      --election-id=ingress-controller-leader
      --controller-class=k8s.io/kafka
      --ingress-class=kafka
      --configmap=$(POD_NAMESPACE)/ingress-kafka-ingress-nginx-controller
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
      --enable-ssl-passthrough=true

Could you please help me to resolve the "failed authentication due to: Failed to process post-handshake messages" error?

[scholzj]

Assuming the errors from your Ingress log are related to the Kafka client connection, I guess it suggests that the TLS passthrough might not work? Because with TLS passthrough, I don't think the Ingress should be in any way involved in the TLS handshake.

[subinmt]

@scholzj Kafka user is that required for my commands testing? If I disabled the tls, the kafka pod is not creating here.

    listeners:
      - name: external
        port: 9094
        type: ingress
        tls: false
        #authentication:
          #type: tls

[scholzj]

I'm not sure what you mean. But yes, the lines you commented out are what enabled the TLS client authentication. So without them your original commend should be fine and the truststore should be all you need.

[subinmt]

@scholzj I make the changes like below and the produced is accepting the messages.

       tls: true
        #authentication:
          #type: tls

But not sure the consumer is cathing that. The official docs are not providing that command. Do you have that?

[subinmt]

@scholzj The issue got resolved now. I got the consumer commands:

bin/kafka-console-consumer.sh --consumer-property security.protocol=SSL --consumer-property ssl.truststore.password=password --consumer-property ssl.truststore.location=/tmp/truststore.jks --bootstrap-server dev-kafka-bootstrap.mydomain.com:443 --topic kafka-topic --from-beginning

The messages are coming to consumers.

[scholzj]

Great, glad it works now for you.