Connecting MongoDB debezium source connector on Strimzi Kafka Connect cluster with tlsCAFile

[murthy95]

Hi, I am trying to create debezium mongo source connector in Strimzi Kafka Connect cluster. Mongo cluster has tls enabled. To pass the .pem file I created the secrets and loaded them as ext config in Kafka Connect yaml as explained here I verified that these files .pem are actually written in Kafka Connect pods using

kubectl exec kc-pod -- ls /opt/kafka/external-configuration/connector-config 

the file is present but when I deploy the connector I get this error

com.mongodb.MongoSocketWriteException: Exception sending message
        at com.mongodb.internal.connection.InternalStreamConnection.translateWriteException(InternalStreamConnection.java:665)
        at com.mongodb.internal.connection.InternalStreamConnection.sendMessage(InternalStreamConnection.java:542)
        at com.mongodb.internal.connection.InternalStreamConnection.sendCommandMessage(InternalStreamConnection.java:368)
        at com.mongodb.internal.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:317)
        at com.mongodb.internal.connection.CommandHelper.sendAndReceive(CommandHelper.java:88)
        at com.mongodb.internal.connection.CommandHelper.executeCommand(CommandHelper.java:36)
        at com.mongodb.internal.connection.InternalStreamConnectionInitializer.initializeConnectionDescription(InternalStreamConnectionInitializer.java:129)
        at com.mongodb.internal.connection.InternalStreamConnectionInitializer.startHandshake(InternalStreamConnectionInitializer.java:71)
        at com.mongodb.internal.connection.InternalStreamConnection.open(InternalStreamConnection.java:167)
        at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.lookupServerDescription(DefaultServerMonitor.java:195)
        at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:151)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:353)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:296)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:291)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:654)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369)
        at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
        at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
        at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421)
        at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:183)
        at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172)
        at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1506)
        at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1416)
        at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:456)
        at java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:921)
        at java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:1291)
        at com.mongodb.internal.connection.SocketStream.write(SocketStream.java:99)
        at com.mongodb.internal.connection.InternalStreamConnection.sendMessage(InternalStreamConnection.java:539)
        ... 10 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
        at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306)
        at java.base/sun.security.validator.Validator.validate(Validator.java:264)
        at java.base/sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:222)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:129)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:638)
        ... 24 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
        at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
        at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
        at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434)
        ... 30 more

In my connection string I only use tlsCAFile I have only one pem file and I am using mongodb.connection.string as per debezium mongo documentation. in connector yaml file

[scholzj]

This error normally means that the CA you provided does not match the certificate used by the server. So I think you need to doublecheck that you use the right certificates.

[murthy95]

HI @scholzj thanks for the quick reply. I am able to connect with the same file from host machine.

I added this config to my Kafka Connect yaml file

  externalConfiguration:
    volumes:
      - name: ca-pemstore
        secret:
          secretName: ca-pemstore

and using the following connection string.

mongodb.connection.string: "mongodb://xxx:yyy@zzz/?tls=true&retryWrites=false&tlsCAFile=/opt/kafka/external-configuration/ca-pemstore/ca.pem"

[murthy95]

@scholzj
I am able to solve my issues by importing the cert pem file to jks using javakeytool and adding the trust store configs to KafkaConnect pod
with the following config

authentication:
    type: tls
    certificateAndKey:
      secretName: super-user
      certificate: user.crt
      key: user.key
  jvmOptions:
    javaSystemProperties:
      - name: javax.net.ssl.trustStore
        value: "/tmp/certs/rds-truststore.jks"
      - name: javax.net.ssl.trustStorePassword
        value: "changeit"
  config:

right now im going inside the pod and adding the ca files to trust store. Can we run this when creating the connect cluster itself using build option while building the container to run a shell script ?

[scholzj]

You can add it to your own Dockerfile when building the container image if you want. But this is not the right way to do this as it overrides the truststroe for everything. The connector should have instead its own configuration for the TLS truststore for Debezium and not just use the JVM default one.

[murthy95]

"The connector should have instead its own configuration for the TLS truststore for Debezium and not just use the JVM default one."
@scholzj Can you point me to the documentation or example config files explaining the right way to do this ?

[scholzj]

I cannot. Because that will differ for every connector. I'm just saying that it should support it. I do not know if the Mongo client used by Debezium can or cannot do it. Sorry.