SSL Handshake Failed: [AdminClient clientId=adminclient-1] Metadata update failed

[alexpollock8]

I am facing a SSL handshake failed issue when trying to use internal tls listener on port 9093.

Here is my Kafka cluster configuration:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: kafka
  namespace: kafka
spec:
  kafka:
    version: 3.0.0
    replicas: 3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
    config:c
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      log.message.format.version: "3.0"
      message.max.bytes: 5242880
      log.retention.hours: 360
      default.replication.factor: 3
      min.insync.replicas: 2
      auto.create.topics.enable: false
    resources:
      requests:
        cpu: "1.0"
        memory: 4Gi
    rack:
      topologyKey: topology.kubernetes.io/zone
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 120Gi
        deleteClaim: false       
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: kafka-metrics-config.yaml
  zookeeper:
    replicas: 3
    resources:
      requests:
        cpu: 0.2
        memory: 512M
    storage:
      type: persistent-claim
      size: 100Gi
      deleteClaim: false
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: zookeeper-metrics-config.yaml
  entityOperator:
    topicOperator: {}
    userOperator: {}

Here is the log output of one of Kafka brokers:

2022-06-13 13:52:17,569 INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Failed authentication with /192.168.52.225 (SSL handshake failed) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-0-ListenerName(TLS-9093)-SSL-7]
2022-06-13 13:52:21,069 INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Failed authentication with /192.168.52.225 (SSL handshake failed) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-0-ListenerName(TLS-9093)-SSL-8]
2022-06-13 13:52:26,339 INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Failed authentication with /192.168.52.225 (SSL handshake failed) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-0-ListenerName(TLS-9093)-SSL-9]
2022-06-13 13:52:29,749 INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Failed authentication with /192.168.52.225 (SSL handshake failed) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-0-ListenerName(TLS-9093)-SSL-8]

Here is log output of AdminClient in pod trying to access broker:

kafka-admin-client-thread | adminclient-1  INFO  - [ r= t= u= ] - [AdminClient clientId=adminclient-1] Metadata update failed 
org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1655128708473) timed out at 1655128708474 after 1 attempt(s)
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
kafka-admin-client-thread | adminclient-1  INFO  - [ r= t= u= ] - [AdminClient clientId=adminclient-1] Metadata update failed 
org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1655128738474) timed out at 1655128738551 after 1 attempt(s)

Here is configuration of AdminClient in pod:

	main  INFO  - [ r= t= u= ] - AdminClientConfig values: 
	bootstrap.servers = [kafka-kafka-bootstrap.kafka.svc.cluster.local:9093]
	client.dns.lookup = default
	client.id = 
	connections.max.idle.ms = 300000
	default.api.timeout.ms = 60000
	metadata.max.age.ms = 300000
	metric.reporters = []
	metrics.num.samples = 2
	metrics.recording.level = INFO
	metrics.sample.window.ms = 30000
	receive.buffer.bytes = 65536
	reconnect.backoff.max.ms = 1000
	reconnect.backoff.ms = 50
	request.timeout.ms = 30000
	retries = 2147483647
	retry.backoff.ms = 100
	sasl.client.callback.handler.class = null
	sasl.jaas.config = null
	sasl.kerberos.kinit.cmd = /usr/bin/kinit
	sasl.kerberos.min.time.before.relogin = 60000
	sasl.kerberos.service.name = null
	sasl.kerberos.ticket.renew.jitter = 0.05
	sasl.kerberos.ticket.renew.window.factor = 0.8
	sasl.login.callback.handler.class = null
	sasl.login.class = null
	sasl.login.refresh.buffer.seconds = 300
	sasl.login.refresh.min.period.seconds = 60
	sasl.login.refresh.window.factor = 0.8
	sasl.login.refresh.window.jitter = 0.05
	sasl.mechanism = GSSAPI
	security.protocol = PLAINTEXT
	security.providers = null
	send.buffer.bytes = 131072
	ssl.cipher.suites = null
	ssl.enabled.protocols = [TLSv1.2]
	ssl.endpoint.identification.algorithm = https
	ssl.key.password = null
	ssl.keymanager.algorithm = SunX509
	ssl.keystore.location = null
	ssl.keystore.password = null
	ssl.keystore.type = JKS
	ssl.protocol = TLSv1.2
	ssl.provider = null
	ssl.secure.random.implementation = null
	ssl.trustmanager.algorithm = PKIX
	ssl.truststore.location = null
	ssl.truststore.password = null
	ssl.truststore.type = JKS

I am using Kafka-Clients version 2.5 and SpringBoot version 2.6.8.

[scholzj]

I think you want to use the port 9092 with your configuration. On port 9093, you have configured TLS encryption (tls: true), but you did not enabled tLS in your client (security.protocol = PLAINTEXT) not provided any truststore to trusts the broker certificates.