[Bug]: SSL handshake failed when MTLS (Mutual TLS) is enabled.

[prashanthisadde]

Bug Description

When attempting to consume topics with the Confluent S3 Sink Connector, I'm facing an error related to SSL handshake when MTLS (Mutual TLS) is enabled.

ERROR [AdminClient clientId=adminclient-1] Connection to node <> failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient) [kafka-admin-client-thread | adminclient-1]
WARN [AdminClient clientId=adminclient-1] Metadata update failed due to authentication error (org.apache.kafka.clients.admin.internals.AdminMetadataManager) [kafka-admin-client-thread | adminclient-1]
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Please check attached log file more information.

Steps to reproduce

Base: I created a Kafka Connect - Confluent S3 Sink Connector Helm chart (with basic config)

1. MSK cluster with AWS PCA generated tls certificate
2. Enable MTLS by configuring the following TLS-related settings in the Kafka Connect config in the Helm chart:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect

  tls:
    trustedCertificates:
      - secretName: <secretname>
        certificate: ca.crt
  authentication:
    type: tls
    tlsTrustedCertificates:
      - certificate: ca.crt
        secretName: <secretname>
    passwordSecret: 
      password: JKS_PASSWORD
      secretName: <secretname>
    certificateAndKey:
      certificate: tls.crt
      key: tls.key
      secretName: <secretname>

4. Deploy the Helm chart to consume topics using the Confluent S3 Sink Connector with MTLS enabled.

Expected behavior

The Kafka Connect worker should be able to connect to the Kafka brokers and consume topics successfully to S3 with MTLS enabled.

Strimzi version

0.28.0

Kubernetes version

Kubernetes 1.24.13

Installation method

Helm chart

Infrastructure

Amazon EKS

Configuration files and logs

I'm attaching the error log file

ps_mtls_error_log.log

Additional context

• TLS configuration and necessary certificates have been verified to be correct.
• There are no apparent network connectivity issues between the Kafka Connect worker and the Kafka brokers.
• The Kafka brokers are running and accessible.
• Topics can be connected and consumed successfully without MTLS.

[scholzj]

I don't think your configuration is valid. The configuration for mTLS should looks something like this:

  tls:
    trustedCertificates:
      - secretName: my-cluster-cluster-ca-cert
        certificate: ca.crt
  authentication:
    type: tls
    certificateAndKey:
      secretName: my-connect
      certificate: user.crt
      key: user.key

You seem to have invalid fields in the authentication section. So the first thing would be to fix those. The error itself suggests that the server certificate is not trusted => so that would mean the ca.crt you configured there is not the right one.

[scholzj]

Well, as I said, than it is likely you have the certificates wrong.

[prashanthisadde]

I have already checked and confirmed that the certificate is valid.

tls:
    trustedCertificates:
      - secretName: <secretname>
        certificate: ca.crt

The certificate being used is from AWS Certificate Authority (CA), and it should already be included in the default truststore. Also, the tls.crt and tls.key contain the correct issuer details.

[scholzj]

If your KAfka cluster uses a server certificates signed by a publi cCA, you could do somehting like this:

tls:
    trustedCertificates: []

That will tell Strimzi to use the default Java truststore.

[prashanthisadde]

Wow, I missed this. Maybe because it was an empty [] for default certificates. It worked. Thank you.

[scholzj]

Great, glad it helped 👍