Error connecting to the cluster org.apache.kafka.common.error.SslAuthenticationException: SSL handshake failed

[kmandal-volvo]

vcc-external-esp-sandbox-kafka-0.log
This broker configuration, previously working fine, able connect cluster using Keystore and trust store and certificate generated from cluster secret, now I am getting all three broker pod Failed authentication with /172.20.81.8 (SSL handshake failed) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-0-ListenerName(TLS-9093)-SSL-7] . I am using Strimzi version 0.20.2 kafka 26.0 and kubernet version 1.19.11 . Log file attached

Broker configuration is given below

apiVersion: kafka.strimzi.io/v1alpha1
kind: Kafka
metadata:
  name: vcc-external-esp-sandbox
  namespace: vcc-external-esp
spec:
  kafka:
    replicas: 3
    logging:
      type: inline
      loggers:
        kafka.root.logger.level: "DEBUG"
    listeners:
      plain:
        type: internal
        port: 9092
        tls: false
        configuration:
          useServiceDnsDomain: true
      tls:
        type: internal
        port: 9093
        tls: true
        authentication:
          type: tls
        configuration:
          useServiceDnsDomain: true
      external:
        type: loadbalancer
        port: 9094
        tls: true
        authentication:
          type: tls
        overrides:
          bootstrap:
            dnsAnnotations:
              service.beta.kubernetes.io/azure-load-balancer-internal: "true"
            host: kafka.test.volvocars.biz
          brokers:
            - broker: 0
              dnsAnnotations:
                service.beta.kubernetes.io/azure-load-balancer-internal: "true"
              host: broker0-kafka.test.volvocars.biz
            - broker: 1
              dnsAnnotations:
                service.beta.kubernetes.io/azure-load-balancer-internal: "true"
              host: broker1-kafka.test.volvocars.biz
            - broker: 2
              dnsAnnotations:
                service.beta.kubernetes.io/azure-load-balancer-internal: "true"
              host: broker2-kafka.test.volvocars.biz
    authorization:
      type: simple
      superUsers:
        - CN=icoe-internal-mm-sandbox-user
        - CN=icoe-ext-super-user
        - CN=icoe-bridge
        - CN=vcc-external-esp-connect
    config:
      auto.create.topics.enable: true
      default.replication.factor: 3
      delete.topic.enable: true
      group.initial.rebalance.delay.ms: 0
      log.message.format.version: "2.3"
      log.retention.bytes: 107374182400
      log.retention.check.interval.ms: 300000
      log.retention.hours: 168
      log.segment.bytes: 1073741824
      max.message.bytes: 1000012
      num.io.threads: 8
      num.network.threads: 3
      num.partitions: 1
      num.recovery.threads.per.data.dir: 1
      offsets.topic.replication.factor: 3
      socket.receive.buffer.bytes: 102400
      socket.request.max.bytes: 104857600
      socket.send.buffer.bytes: 102400
      transaction.state.log.min.isr: 3
      transaction.state.log.replication.factor: 3
    storage:
      type: persistent-claim
      persistentVolumeReclaimPolicy: Retain
      size: 30Gi
      class: managed-premium-retain
      deleteClaim: false
    metrics:
      lowercaseOutputName: true
      rules:
      # Special cases and very specific rules
      - pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), topic=(.+), partition=(.*)><>Value
        name: kafka_server_$1_$2
        type: GAUGE
        labels:
          clientId: "$3"
          topic: "$4"
          partition: "$5"
      - pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), brokerHost=(.+), brokerPort=(.+)><>Value
        name: kafka_server_$1_$2
        type: GAUGE
        labels:
          clientId: "$3"
          broker: "$4:$5"
      - pattern: kafka.server<type=(.+), cipher=(.+), protocol=(.+), listener=(.+), networkProcessor=(.+)><>connections
        name: kafka_server_$1_connections_tls_info
        type: GAUGE
        labels:
          listener: "$2"
          networkProcessor: "$3"
          protocol: "$4"
          cipher: "$5"
      - pattern: kafka.server<type=(.+), clientSoftwareName=(.+), clientSoftwareVersion=(.+), listener=(.+), networkProcessor=(.+)><>connections
        name: kafka_server_$1_connections_software
        type: GAUGE
        labels:
          clientSoftwareName: "$2"
          clientSoftwareVersion: "$3"
          listener: "$4"
          networkProcessor: "$5"
      - pattern: "kafka.server<type=(.+), listener=(.+), networkProcessor=(.+)><>(.+):"
        name: kafka_server_$1_$4
        type: GAUGE
        labels:
          listener: "$2"
          networkProcessor: "$3"
      - pattern: kafka.server<type=(.+), listener=(.+), networkProcessor=(.+)><>(.+)
        name: kafka_server_$1_$4
        type: GAUGE
        labels:
          listener: "$2"
          networkProcessor: "$3"
      # Some percent metrics use MeanRate attribute
      # Ex) kafka.server<type=(KafkaRequestHandlerPool), name=(RequestHandlerAvgIdlePercent)><>MeanRate
      - pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*><>MeanRate
        name: kafka_$1_$2_$3_percent
        type: GAUGE
      # Generic gauges for percents
      - pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*><>Value
        name: kafka_$1_$2_$3_percent
        type: GAUGE
      - pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*, (.+)=(.+)><>Value
        name: kafka_$1_$2_$3_percent
        type: GAUGE
        labels:
          "$4": "$5"
      # Generic per-second counters with 0-2 key/value pairs
      - pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*, (.+)=(.+), (.+)=(.+)><>Count
        name: kafka_$1_$2_$3_total
        type: COUNTER
        labels:
          "$4": "$5"
          "$6": "$7"
      - pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*, (.+)=(.+)><>Count
        name: kafka_$1_$2_$3_total
        type: COUNTER
        labels:
          "$4": "$5"
      - pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*><>Count
        name: kafka_$1_$2_$3_total
        type: COUNTER
      # Generic gauges with 0-2 key/value pairs
      - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+), (.+)=(.+)><>Value
        name: kafka_$1_$2_$3
        type: GAUGE
        labels:
          "$4": "$5"
          "$6": "$7"
      - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>Value
        name: kafka_$1_$2_$3
        type: GAUGE
        labels:
          "$4": "$5"
      - pattern: kafka.(\w+)<type=(.+), name=(.+)><>Value
        name: kafka_$1_$2_$3
        type: GAUGE
      # Emulate Prometheus 'Summary' metrics for the exported 'Histogram's.
      # Note that these are missing the '_sum' metric!
      - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+), (.+)=(.+)><>Count
        name: kafka_$1_$2_$3_count
        type: COUNTER
        labels:
          "$4": "$5"
          "$6": "$7"
      - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.*), (.+)=(.+)><>(\d+)thPercentile
        name: kafka_$1_$2_$3
        type: GAUGE
        labels:
          "$4": "$5"
          "$6": "$7"
          quantile: "0.$8"
      - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>Count
        name: kafka_$1_$2_$3_count
        type: COUNTER
        labels:
          "$4": "$5"
      - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.*)><>(\d+)thPercentile
        name: kafka_$1_$2_$3
        type: GAUGE
        labels:
          "$4": "$5"
          quantile: "0.$6"
      - pattern: kafka.(\w+)<type=(.+), name=(.+)><>Count
        name: kafka_$1_$2_$3_count
        type: COUNTER
      - pattern: kafka.(\w+)<type=(.+), name=(.+)><>(\d+)thPercentile
        name: kafka_$1_$2_$3
        type: GAUGE
        labels:
          quantile: "0.$4"
  zookeeper:
    config:
      autopurge.purgeInterval: 24
      autopurge.snapRetainCount: 3
      initLimit: 10
      syncLimit: 5
      timeTick: 2000
    metrics:
      lowercaseOutputName: true
      rules:
      # replicated Zookeeper
      - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+)><>(\\w+)"
        name: "zookeeper_$2"
        type: GAUGE
      - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+)><>(\\w+)"
        name: "zookeeper_$3"
        type: GAUGE
        labels:
          replicaId: "$2"
      - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+), name2=(\\w+)><>(Packets\\w+)"
        name: "zookeeper_$4"
        type: COUNTER
        labels:
          replicaId: "$2"
          memberType: "$3"
      - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+), name2=(\\w+)><>(\\w+)"
        name: "zookeeper_$4"
        type: GAUGE
        labels:
          replicaId: "$2"
          memberType: "$3"
      - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+), name2=(\\w+), name3=(\\w+)><>(\\w+)"
        name: "zookeeper_$4_$5"
        type: GAUGE
        labels:
          replicaId: "$2"
          memberType: "$3"
    replicas: 3
    storage:
      type: persistent-claim
      persistentVolumeReclaimPolicy: Retain
      pod:
        affinity:
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
              - podAffinityTerm:
                  labelSelector:
                    matchLabels:
                      strimzi.io/name: vcc-external-esp
                  topologyKey: topology.kubernetes.io/zone
                weight: 100
        securityContext:
          fsGroup: 1001
      size: 30Gi
      class: managed-premium-retain
      deleteClaim: false
  entityOperator:
    topicOperator:
      reconciliationIntervalSeconds: 60
      resources:
        limits:
          cpu: 500m
          memory: 1Gi
        requests:
          cpu: 150m
          memory: 500M
    userOperator:
      reconciliationIntervalSeconds: 60
      resources:
        limits:
          cpu: 500m
          memory: 1Gi
        requests:
          cpu: 150m
          memory: 500M
  kafkaExporter:
    topicRegex: ".*"
    groupRegex: ".*"

[scholzj]

Sorry, I'm not sure I follow. There are no clients logs, client configuration, nothing. So it is hard to say why something doesn't work. You would need to provide more details.

[kmandal-volvo]

For more details, i am sending the Server and Client log attached.

Broker configuration is given below

apiVersion: kafka.strimzi.io/v1alpha1
kind: Kafka
metadata:
  name: vcc-internal-esp-qa
  namespace: vcc-internal-esp
spec:
  kafka:
    replicas: 3
    logging:
      type: inline
      loggers:
        kafka.root.logger.level: "DEBUG"
    listeners:
      plain:
        type: internal
        port: 9092
        tls: false
        configuration:
          useServiceDnsDomain: true
      tls:
        type: internal
        port: 9093
        tls: true
        authentication:
          type: tls

      external:
        type: loadbalancer
        port: 9094
        tls: true
        authentication:
          type: tls
        overrides:
          bootstrap:
            dnsAnnotations:
              service.beta.kubernetes.io/azure-load-balancer-internal: "true"
            host: kafka.qa.volvocars.net
          brokers:
            - broker: 0
              dnsAnnotations:
                service.beta.kubernetes.io/azure-load-balancer-internal: "true"
              host: broker-0-kafka.qa.volvocars.net
            - broker: 1
              dnsAnnotations:
                service.beta.kubernetes.io/azure-load-balancer-internal: "true"
              host: broker-1-kafka.qa.volvocars.net
            - broker: 2
              dnsAnnotations:
                service.beta.kubernetes.io/azure-load-balancer-internal: "true"
              host: broker-2-kafka.qa.volvocars.net
    authorization:
      type: simple
      superUsers:
        - CN=icoe-int-super-user
        - CN=my-bridge
        - CN=vcc-internal-esp-connect
    config:
      auto.create.topics.enable: true
      default.replication.factor: 3
      delete.topic.enable: true
      group.initial.rebalance.delay.ms: 0
      log.message.format.version: "2.3"
      log.retention.bytes: 107374182400
      log.retention.check.interval.ms: 300000
      log.retention.hours: 168
      log.segment.bytes: 1073741824
      max.message.bytes: 1000012
      num.io.threads: 8
      num.network.threads: 3
      num.partitions: 1
      num.recovery.threads.per.data.dir: 1
      offsets.topic.replication.factor: 3
      socket.receive.buffer.bytes: 102400
      socket.request.max.bytes: 104857600
      socket.send.buffer.bytes: 102400
      transaction.state.log.min.isr: 3
      transaction.state.log.replication.factor: 3
    storage:
      type: persistent-claim
      size: 15Gi
      class: managed-premium-retain
      deleteClaim: false
    metrics:
      lowercaseOutputName: true
      rules:
      # Special cases and very specific rules
      - pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), topic=(.+), partition=(.*)><>Value
        name: kafka_server_$1_$2
        type: GAUGE
        labels:
          clientId: "$3"
          topic: "$4"
          partition: "$5"
      - pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), brokerHost=(.+), brokerPort=(.+)><>Value
        name: kafka_server_$1_$2
        type: GAUGE
        labels:
          clientId: "$3"
          broker: "$4:$5"
      - pattern: kafka.server<type=(.+), cipher=(.+), protocol=(.+), listener=(.+), networkProcessor=(.+)><>connections
        name: kafka_server_$1_connections_tls_info
        type: GAUGE
        labels:
          listener: "$2"
          networkProcessor: "$3"
          protocol: "$4"
          cipher: "$5"
      - pattern: kafka.server<type=(.+), clientSoftwareName=(.+), clientSoftwareVersion=(.+), listener=(.+), networkProcessor=(.+)><>connections
        name: kafka_server_$1_connections_software
        type: GAUGE
        labels:
          clientSoftwareName: "$2"
          clientSoftwareVersion: "$3"
          listener: "$4"
          networkProcessor: "$5"
      - pattern: "kafka.server<type=(.+), listener=(.+), networkProcessor=(.+)><>(.+):"
        name: kafka_server_$1_$4
        type: GAUGE
        labels:
          listener: "$2"
          networkProcessor: "$3"
      - pattern: kafka.server<type=(.+), listener=(.+), networkProcessor=(.+)><>(.+)
        name: kafka_server_$1_$4
        type: GAUGE
        labels:
          listener: "$2"
          networkProcessor: "$3"
      # Some percent metrics use MeanRate attribute
      # Ex) kafka.server<type=(KafkaRequestHandlerPool), name=(RequestHandlerAvgIdlePercent)><>MeanRate
      - pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*><>MeanRate
        name: kafka_$1_$2_$3_percent
        type: GAUGE
      # Generic gauges for percents
      - pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*><>Value
        name: kafka_$1_$2_$3_percent
        type: GAUGE
      - pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*, (.+)=(.+)><>Value
        name: kafka_$1_$2_$3_percent
        type: GAUGE
        labels:
          "$4": "$5"
      # Generic per-second counters with 0-2 key/value pairs
      - pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*, (.+)=(.+), (.+)=(.+)><>Count
        name: kafka_$1_$2_$3_total
        type: COUNTER
        labels:
          "$4": "$5"
          "$6": "$7"
      - pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*, (.+)=(.+)><>Count
        name: kafka_$1_$2_$3_total
        type: COUNTER
        labels:
          "$4": "$5"
      - pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*><>Count
        name: kafka_$1_$2_$3_total
        type: COUNTER
      # Generic gauges with 0-2 key/value pairs
      - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+), (.+)=(.+)><>Value
        name: kafka_$1_$2_$3
        type: GAUGE
        labels:
          "$4": "$5"
          "$6": "$7"
      - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>Value
        name: kafka_$1_$2_$3
        type: GAUGE
        labels:
          "$4": "$5"
      - pattern: kafka.(\w+)<type=(.+), name=(.+)><>Value
        name: kafka_$1_$2_$3
        type: GAUGE
      # Emulate Prometheus 'Summary' metrics for the exported 'Histogram's.
      # Note that these are missing the '_sum' metric!
      - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+), (.+)=(.+)><>Count
        name: kafka_$1_$2_$3_count
        type: COUNTER
        labels:
          "$4": "$5"
          "$6": "$7"
      - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.*), (.+)=(.+)><>(\d+)thPercentile
        name: kafka_$1_$2_$3
        type: GAUGE
        labels:
          "$4": "$5"
          "$6": "$7"
          quantile: "0.$8"
      - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>Count
        name: kafka_$1_$2_$3_count
        type: COUNTER
        labels:
          "$4": "$5"
      - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.*)><>(\d+)thPercentile
        name: kafka_$1_$2_$3
        type: GAUGE
        labels:
          "$4": "$5"
          quantile: "0.$6"
      - pattern: kafka.(\w+)<type=(.+), name=(.+)><>Count
        name: kafka_$1_$2_$3_count
        type: COUNTER
      - pattern: kafka.(\w+)<type=(.+), name=(.+)><>(\d+)thPercentile
        name: kafka_$1_$2_$3
        type: GAUGE
        labels:
          quantile: "0.$4"
  zookeeper:
    config:
      autopurge.purgeInterval: 24
      autopurge.snapRetainCount: 3
      initLimit: 10
      syncLimit: 5
      timeTick: 2000
    metrics:
      lowercaseOutputName: true
      rules:
      # replicated Zookeeper
      - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+)><>(\\w+)"
        name: "zookeeper_$2"
        type: GAUGE
      - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+)><>(\\w+)"
        name: "zookeeper_$3"
        type: GAUGE
        labels:
          replicaId: "$2"
      - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+), name2=(\\w+)><>(Packets\\w+)"
        name: "zookeeper_$4"
        type: COUNTER
        labels:
          replicaId: "$2"
          memberType: "$3"
      - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+), name2=(\\w+)><>(\\w+)"
        name: "zookeeper_$4"
        type: GAUGE
        labels:
          replicaId: "$2"
          memberType: "$3"
      - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+), name2=(\\w+), name3=(\\w+)><>(\\w+)"
        name: "zookeeper_$4_$5"
        type: GAUGE
        labels:
          replicaId: "$2"
          memberType: "$3"
    replicas: 3
    storage:
      type: persistent-claim
      pod:
        affinity:
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
              - podAffinityTerm:
                  labelSelector:
                    matchLabels:
                      strimzi.io/name: vcc-internal-esp
                  topologyKey: topology.kubernetes.io/zone
                weight: 100
        securityContext:
          fsGroup: 1001
      size: 15Gi
      class: managed-premium-retain
      deleteClaim: false
  entityOperator:
    topicOperator:
      reconciliationIntervalSeconds: 60
      resources:
        limits:
          cpu: 500m
          memory: 1Gi
        requests:
          cpu: 150m
          memory: 500M
    userOperator:
      reconciliationIntervalSeconds: 60
      resources:
        limits:
          cpu: 500m
          memory: 1Gi
        requests:
          cpu: 150m
          memory: 500M
  kafkaExporter:
    topicRegex: ".*"
    groupRegex: ".*"

kafka_client.log
vcc-internal-esp-qa-kafka-0.log

[scholzj]

I do not know librdkafka. If you reproduced it with the regular Java clients, I might better understand the errors and the configuration. The error suggests that you seem to be missing the user key for passing the TLS Client Authentication. Did you configured it?

[kmandal-volvo]

You are correct this is a user key issue, as the broker cluster name is updated in some way I missed to update the cluster name due the certificate is not being generated for that user. Thanks for your suggestion.