KAFKA CONNECT: aused by: java.security.cert.CertificateException: No name matching gpds-dev1-kafka-tls-bootstrap found at

[sreejesh-radhakrishnan-db]

HI Thanks a lot for proving a such a wonderful platform and support.

And we are now LIVE with Strimzi in our PROD and a month and works like a charm.

Now the issue 👍🏽

KAFKA CR: where I am providing a certificate created by by PKI (and GOOGLE Cert ISSUER)

listeners:
  - name: tls
    port: 9093
    type: loadbalancer
    tls: true
    authentication:
      type: tls
    configuration:
      brokerCertChainAndKey:
        secretName: kafka-listener #my-secret
        certificate: tls.crt #my-listener-certificate.crt
        key: tls.key #client-ca.key        

this works well for consumer to connect to us using CERT. But KAFKA CONNECT I am getting SSL exception,

My connect setup:

spec:
bootstrapServers: gpds-dev1-kafka-tls-bootstrap:9093
tls:
trustedCertificates:
- secretName: gpds-dev1-clients-ca-cert
certificate: ca.crt

this secret exist and it has ca.crt. This secret is created by the certificate sameway as I have created for brokerCertChainAndKey:

I am getting the SSL handshake error

org.apache.kafka.connect.errors.ConnectException: Failed to connect to and describe Kafka cluster. Check worker's broker connection and security properties. at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:70) at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:51) at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:97) at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:80) Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999) at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165) at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:64) ... 3 more Caused by: org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed Caused by: javax.net.ssl.SSLHandshakeException: No name matching gpds-dev1-kafka-tls-bootstrap found at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:352) at

[sreejesh-radhakrishnan-db]

org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLHandshakeException: No name matching gpds-dev1-kafka-tls-bootstrap found
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:352)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:295)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:290)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1357)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1232)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1175)
at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1074)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1008)
at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:430)
at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:514)
at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:368)
at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:291)
at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543)
at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:560)
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1400)
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1331)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.security.cert.CertificateException: No name matching gpds-dev1-kafka-tls-bootstrap found
at java.base/sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:234)
at java.base/sun.security.util.HostnameChecker.match(HostnameChecker.java:103)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:455)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:429)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:283)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:141)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1335)
... 19 more

[sreejesh-radhakrishnan-db]

true , to debug this further,. is there any way I can get more on SSL failure what exactly is cause of the issue I am seeing?

[scholzj]

As I said, this is likely an issue with how you configured the TLS client authentication. You can debug TLS using the system property -Djavax.net.debug=ssl -> but not sure if it helps you to understand the issue any better as the output is quite complicated.

[sreejesh-radhakrishnan-db]

how do I set "-Djavax.net.deug=ssl" this on kafkaconnect yaml or some other way? If I could get some detail log, I can trouble some developers in team as well. But currently they are saying - log is not detail to help.

[scholzj]

To be honest, if they cannot figure out how to set a system property for a Java application, I'm a bit sceptical the debug output helps them.

[sreejesh-radhakrishnan-db]

good one, this is happening while starting up KAFKA connect , so was checking . I have set in past for consumer "-Djavax.net.deug=ssl" , not sure how will I ask kafkaconnect to have this set when it starts. I will read more on how to. if stuck will get back.

[sreejesh-radhakrishnan-db]

I did bit more reading and I am now passing the authentication: as well, I do not get any more error on certificate (probably its erroring before that ;-) ) or ssl handshake. but getting this error

"org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: fetchMetadata",

org.apache.kafka.connect.errors.ConnectException: Failed to connect to and describe Kafka cluster. Check worker's broker connection and security properties.
at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:70)
at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:51)
at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:97)
at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:80)
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listNodes
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165)
at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:64)
... 3 more
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listNodes

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
name: my-connect-cluster

spec:
version: 3.1.0
image: artifactory.sdlc.ctl.gcp.db.com/dkr-public-local/com/db/grouppartymaster/oss-kafkaconnect:1.0.0
replicas: 1
authentication:
type: tls
certificateAndKey:
certificate: tls.crt
key: tls.key
secretName: connect-ca-cert1
bootstrapServers: gpds-dev1-kafka-tls-bootstrap:9092 #this is type tls in my setting and not plaintext
tls:
trustedCertificates:
- secretName: connect-ca-cert1
certificate: ca.crt
config:
group.id: connect-cluster
offset.storage.topic: connect-cluster-offsets
config.storage.topic: connect-cluster-configs
status.storage.topic: connect-cluster-status
# -1 means it will use the default replication factor configured in the broker
config.storage.replication.factor: -1
offset.storage.replication.factor: -1
status.storage.replication.factor: -1

[scholzj]

So, I'm not sure I follow ... for what exactly do you want to set the secret?

[sreejesh-radhakrishnan-db]

Ours is a restrictive GKE and hence have no connection to any URL unless we whitelist the egress, and its a pain process. I want to use our Artifactory to pull the PLUGIN by passing my own URL on plugins: section above . hence the original Q.

[scholzj]

If you are pulling a JAR, ZIP or TAR.GZ file, you can use any URL. You cannot really use this for maven type artifacts currently. B ut they need to be open. There is no authentication supported for the URL. If that is not possible, you might need to build the file from Dockerfile in your CI where you are free to pull the artifacts and add them tot he container image any way you want.

[sreejesh-radhakrishnan-db]

Thought same might be case, we need these ZIP to be unzipped onto plugin folder of "/opt/kafka/plugins" and then build the image?

[scholzj]

The Connect Build would unzip it. But when using Dockerfile you have to unzip it your self.

[sreejesh-radhakrishnan-db]

I will try to upload full error stack in sometime

[sreejesh123]

One related q. does the The bootstrap and broker uses same listener certificate during ssl handshake?

[scholzj]

Yes, it is the same certificate. It is always the same brokers on the same port -> it is just a different service. One is to access a particular broker directly to publish or consume data from some partition which the broker is hosting. Other is a service which round-robins between all the broker to get metadata (since you can get them from any broker and by using a separate service which points to all of them you make sure you find some runnign broker even if some of them are down).

[sreejesh123]

yaml.txt
log.txt