SSL Error : Tag Mismatch with ingress type #6397

demanghon · 2022-02-18T09:36:52Z

demanghon
Feb 18, 2022

Hello,

I deployed a Kafka Cluster on Kind with NGINX Ingress Controller configured.

I follow the instructions of Accessing Kafka: Part 5 - Ingress to use Kafka from an external client. But I have a SSL Error when I tried to communicate with it.

I enabled SSL Passthrough on NGINX.

spec:
  template:
    spec:
      containers:
      - name: controller
        args:
        - /nginx-ingress-controller
        - --election-id=ingress-controller-leader
        - --controller-class=k8s.io/ingress-nginx
        - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
        - --validating-webhook=:8443
        - --validating-webhook-certificate=/usr/local/certificates/cert
        - --validating-webhook-key=/usr/local/certificates/key
        - --watch-ingress-without-class=true
        - --publish-status-address=localhost
        - --enable-ssl-passthrough

kubectl patch deployments.apps -n ingress-nginx ingress-nginx-controller --patch-file enable-ssl-passthrough.yml

Here the Kakfa definition file:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: kafka-cluster
spec:
  kafka:
    version: 3.1.0
    replicas: 3
    listeners:
      - name: tls
        port: 9093
        type: internal
        tls: true
      - name: external1
        port: 9094
        type: ingress
        tls: true
        authentication:
          type: tls
        configuration:
          bootstrap:
            host: bootstrap.kafka-dev.com
          brokers:
          - broker: 0
            host: broker-0.kafka-dev.com
          - broker: 1
            host: broker-1.kafka-dev.com
          - broker: 2
            host: broker-2.kafka-dev.com
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      inter.broker.protocol.version: "3.1"
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 100Gi
        deleteClaim: false
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 100Gi
      deleteClaim: false
  entityOperator:
    topicOperator: {}
    userOperator: {}

I generated the client certificate:

kubectl get secret kafka-cluster-cluster-ca-cert -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt
keytool -import -trustcacerts -alias root -file ca.crt -keystore truststore.jks -storepass password -noprompt

My configuration file for the client contains:

security.protocol=SSL
ssl.truststore.location=./truststore.jks
ssl.truststore.password=password`

I use kafka client to connect to my kafka boostrap server but I had this error:

kafka_2.13-3.1.0/bin/kafka-topics.sh --bootstrap-server bootstrap.kafka-dev.com:443 --command-config client/client-config.properties --list
javax.net.ssl|DEBUG|01|main|2022-02-18 10:21:08.555 CET|SSLCipher.java:464|jdk.tls.keyLimits:  entry = AES/GCM/NoPadding KeyUpdate 2^37. AES/GCM/NOPADDING:KEYUPDATE = 137438953472
javax.net.ssl|DEBUG|0E|kafka-admin-client-thread | adminclient-1|2022-02-18 10:21:08.877 CET|SSLCipher.java:1866|KeyLimit read side: algorithm = AES/GCM/NOPADDING:KEYUPDATE
countdown value = 137438953472
javax.net.ssl|DEBUG|0E|kafka-admin-client-thread | adminclient-1|2022-02-18 10:21:08.879 CET|SSLCipher.java:2020|KeyLimit write side: algorithm = AES/GCM/NOPADDING:KEYUPDATE
countdown value = 137438953472
javax.net.ssl|DEBUG|0E|kafka-admin-client-thread | adminclient-1|2022-02-18 10:21:08.959 CET|SSLCipher.java:1866|KeyLimit read side: algorithm = AES/GCM/NOPADDING:KEYUPDATE
countdown value = 137438953472
javax.net.ssl|ALL|0E|kafka-admin-client-thread | adminclient-1|2022-02-18 10:21:08.961 CET|X509Authentication.java:246|No X.509 cert selected for EC
javax.net.ssl|ALL|0E|kafka-admin-client-thread | adminclient-1|2022-02-18 10:21:08.961 CET|X509Authentication.java:246|No X.509 cert selected for EC
javax.net.ssl|ALL|0E|kafka-admin-client-thread | adminclient-1|2022-02-18 10:21:08.961 CET|X509Authentication.java:246|No X.509 cert selected for EC
javax.net.ssl|ALL|0E|kafka-admin-client-thread | adminclient-1|2022-02-18 10:21:08.961 CET|X509Authentication.java:246|No X.509 cert selected for RSA
javax.net.ssl|ALL|0E|kafka-admin-client-thread | adminclient-1|2022-02-18 10:21:08.962 CET|X509Authentication.java:246|No X.509 cert selected for RSA
javax.net.ssl|ALL|0E|kafka-admin-client-thread | adminclient-1|2022-02-18 10:21:08.962 CET|X509Authentication.java:246|No X.509 cert selected for RSA
javax.net.ssl|ALL|0E|kafka-admin-client-thread | adminclient-1|2022-02-18 10:21:08.962 CET|X509Authentication.java:246|No X.509 cert selected for RSASSA-PSS
javax.net.ssl|ALL|0E|kafka-admin-client-thread | adminclient-1|2022-02-18 10:21:08.962 CET|X509Authentication.java:246|No X.509 cert selected for RSASSA-PSS
javax.net.ssl|ALL|0E|kafka-admin-client-thread | adminclient-1|2022-02-18 10:21:08.963 CET|X509Authentication.java:246|No X.509 cert selected for RSASSA-PSS
javax.net.ssl|ALL|0E|kafka-admin-client-thread | adminclient-1|2022-02-18 10:21:08.963 CET|X509Authentication.java:246|No X.509 cert selected for RSA
javax.net.ssl|ALL|0E|kafka-admin-client-thread | adminclient-1|2022-02-18 10:21:08.963 CET|X509Authentication.java:246|No X.509 cert selected for RSA
javax.net.ssl|ALL|0E|kafka-admin-client-thread | adminclient-1|2022-02-18 10:21:08.963 CET|X509Authentication.java:246|No X.509 cert selected for RSA
javax.net.ssl|ALL|0E|kafka-admin-client-thread | adminclient-1|2022-02-18 10:21:08.964 CET|X509Authentication.java:246|No X.509 cert selected for EC
javax.net.ssl|ALL|0E|kafka-admin-client-thread | adminclient-1|2022-02-18 10:21:08.964 CET|X509Authentication.java:246|No X.509 cert selected for RSA
javax.net.ssl|DEBUG|0E|kafka-admin-client-thread | adminclient-1|2022-02-18 10:21:08.965 CET|SSLCipher.java:2020|KeyLimit write side: algorithm = AES/GCM/NOPADDING:KEYUPDATE
countdown value = 137438953472
javax.net.ssl|ERROR|0E|kafka-admin-client-thread | adminclient-1|2022-02-18 10:21:08.995 CET|TransportContext.java:341|Fatal (BAD_RECORD_MAC): Tag mismatch! (
"throwable" : {
  javax.crypto.AEADBadTagException: Tag mismatch!
        at java.base/com.sun.crypto.provider.GaloisCounterMode.decryptFinal(GaloisCounterMode.java:623)
        at java.base/com.sun.crypto.provider.CipherCore.finalNoPadding(CipherCore.java:1116)
        at java.base/com.sun.crypto.provider.CipherCore.fillOutputBuffer(CipherCore.java:1053)
        at java.base/com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:941)
        at java.base/com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:491)
        at java.base/javax.crypto.CipherSpi.bufferCrypt(CipherSpi.java:779)
        at java.base/javax.crypto.CipherSpi.engineDoFinal(CipherSpi.java:730)
        at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2497)
        at java.base/sun.security.ssl.SSLCipher$T13GcmReadCipherGenerator$GcmReadCipher.decrypt(SSLCipher.java:1929)
        at java.base/sun.security.ssl.SSLEngineInputRecord.decodeInputRecord(SSLEngineInputRecord.java:240)
        at java.base/sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:197)
        at java.base/sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:160)
        at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111)
        at java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:681)
        at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:636)
        at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:454)
        at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:433)
        at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:637)
        at org.apache.kafka.common.network.SslTransportLayer.read(SslTransportLayer.java:567)
        at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:95)
        at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:452)
        at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:402)
        at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:674)
        at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:576)
        at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
        at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:560)
        at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1400)
        at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1331)
        at java.base/java.lang.Thread.run(Thread.java:829)}

)
javax.net.ssl|WARNING|0E|kafka-admin-client-thread | adminclient-1|2022-02-18 10:21:08.997 CET|SSLEngineOutputRecord.java:168|outbound has closed, ignore outbound application data
[2022-02-18 10:21:09,002] ERROR [AdminClient clientId=adminclient-1] Connection to node -1 (bootstrap.kafka-dev.com/127.0.0.1:443) failed authentication due to: Failed to process post-handshake messages (org.apache.kafka.clients.NetworkClient)
[2022-02-18 10:21:09,004] WARN [AdminClient clientId=adminclient-1] Metadata update failed due to authentication error (org.apache.kafka.clients.admin.internals.AdminMetadataManager)
org.apache.kafka.common.errors.SslAuthenticationException: Failed to process post-handshake messages
Caused by: javax.net.ssl.SSLException: Tag mismatch!
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:133)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:349)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:292)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:287)
        at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:123)
        at java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:681)
        at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:636)
        at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:454)
        at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:433)
        at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:637)
        at org.apache.kafka.common.network.SslTransportLayer.read(SslTransportLayer.java:567)
        at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:95)
        at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:452)
        at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:402)
        at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:674)
        at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:576)
        at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
        at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:560)
        at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1400)
        at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1331)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: javax.crypto.AEADBadTagException: Tag mismatch!
        at java.base/com.sun.crypto.provider.GaloisCounterMode.decryptFinal(GaloisCounterMode.java:623)
        at java.base/com.sun.crypto.provider.CipherCore.finalNoPadding(CipherCore.java:1116)
        at java.base/com.sun.crypto.provider.CipherCore.fillOutputBuffer(CipherCore.java:1053)
        at java.base/com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:941)
        at java.base/com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:491)
        at java.base/javax.crypto.CipherSpi.bufferCrypt(CipherSpi.java:779)
        at java.base/javax.crypto.CipherSpi.engineDoFinal(CipherSpi.java:730)
        at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2497)
        at java.base/sun.security.ssl.SSLCipher$T13GcmReadCipherGenerator$GcmReadCipher.decrypt(SSLCipher.java:1929)
        at java.base/sun.security.ssl.SSLEngineInputRecord.decodeInputRecord(SSLEngineInputRecord.java:240)
        at java.base/sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:197)
        at java.base/sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:160)
        at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111)
        ... 16 more
Error while executing topic command : Failed to process post-handshake messages
[2022-02-18 10:21:09,009] ERROR org.apache.kafka.common.errors.SslAuthenticationException: Failed to process post-handshake messages
Caused by: javax.net.ssl.SSLException: Tag mismatch!
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:133)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:349)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:292)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:287)
        at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:123)
        at java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:681)
        at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:636)
        at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:454)
        at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:433)
        at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:637)
        at org.apache.kafka.common.network.SslTransportLayer.read(SslTransportLayer.java:567)
        at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:95)
        at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:452)
        at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:402)
        at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:674)
        at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:576)
        at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
        at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:560)
        at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1400)
        at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1331)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: javax.crypto.AEADBadTagException: Tag mismatch!
        at java.base/com.sun.crypto.provider.GaloisCounterMode.decryptFinal(GaloisCounterMode.java:623)
        at java.base/com.sun.crypto.provider.CipherCore.finalNoPadding(CipherCore.java:1116)
        at java.base/com.sun.crypto.provider.CipherCore.fillOutputBuffer(CipherCore.java:1053)
        at java.base/com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:941)
        at java.base/com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:491)
        at java.base/javax.crypto.CipherSpi.bufferCrypt(CipherSpi.java:779)
        at java.base/javax.crypto.CipherSpi.engineDoFinal(CipherSpi.java:730)
        at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2497)
        at java.base/sun.security.ssl.SSLCipher$T13GcmReadCipherGenerator$GcmReadCipher.decrypt(SSLCipher.java:1929)
        at java.base/sun.security.ssl.SSLEngineInputRecord.decodeInputRecord(SSLEngineInputRecord.java:240)
        at java.base/sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:197)
        at java.base/sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:160)
        at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111)
        ... 16 more
 (kafka.admin.TopicCommand$)

I don't understand where is my mistake.

Could you help me ?

Jérôme

Answered by scholzj

Feb 18, 2022

Your listener has TLS client authentication configured:

        authentication:
          type: tls

But in your command, you pass only truststore and no keystore for client authentication. You have two options:

Keep the authentication -> in that case you need to create a user using the KafkaUser resource and use the generated user certificate in your client for authentication (as keystore).
Disable the TLS clint authentication by removing the code above and keep using the command as you have it.

View full answer

scholzj · 2022-02-18T10:57:41Z

scholzj
Feb 18, 2022
Maintainer

Your listener has TLS client authentication configured:

        authentication:
          type: tls

But in your command, you pass only truststore and no keystore for client authentication. You have two options:

Keep the authentication -> in that case you need to create a user using the KafkaUser resource and use the generated user certificate in your client for authentication (as keystore).
Disable the TLS clint authentication by removing the code above and keep using the command as you have it.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strimzi

SSL Error : Tag Mismatch with ingress type #6397

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Strimzi

SSL Error : Tag Mismatch with ingress type #6397

Uh oh!

demanghon Feb 18, 2022

Replies: 1 comment

Uh oh!

scholzj Feb 18, 2022 Maintainer

demanghon
Feb 18, 2022

scholzj
Feb 18, 2022
Maintainer