Getting Caused by: java.security.cert.CertificateException: No subject alternative DNS name matching

[telia-ankita]

We are getting Caused by: java.security.cert.CertificateException: No subject alternative DNS name matching exception with below KafkaConnect configuration:

kafka_connect.yaml:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
name: debezium-connect-cluster
namespace: {{ .Values.namespace }}
annotations:
strimzi.io/use-connector-resources: "true"
spec:
version: 3.5.0
image: "{{ .Values.image.repository }}/debezium-connect-postgres:{{ .Values.image.tag }}"
replicas: {{ .Values.connect.replicaCount }}
bootstrapServers: {{ .Values.connect.bootstrapServers }}
{{- if .Values.imagePullSecrets }}
template:
pod:
imagePullSecrets:
- name: {{ .Values.imagePullSecrets }}
{{- end }}
authentication:
type: scram-sha-512
username: {{ .Values.connect.authentication.username }}
passwordSecret:
secretName: debezium-secret
password: password
tls:
trustedCertificates:
- certificate: Kafka_Root_CA_v1.crt
secretName: kafka-secret
logging:
type: inline
loggers:
kafka.root.logger.level: DEBUG
config:
config.providers: secrets
config.providers.secrets.class: io.strimzi.kafka.KubernetesSecretConfigProvider
#group.id: 2210
offset.storage.topic: {{ .Values.connect.config.offset_storage_topic }}
config.storage.topic: {{ .Values.connect.config.config_storage_topic }}
status.storage.topic: {{ .Values.connect.config.status_storage_topic }}
# -1 means it will use the default replication factor configured in the broker
config.storage.replication.factor: 1
offset.storage.replication.factor: 1
status.storage.replication.factor: 1
key.converter: org.apache.kafka.connect.json.JsonConverter
value.converter: org.apache.kafka.connect.json.JsonConverter
externalConfiguration:
volumes:
- name: connector
secret:
secretName: connector-secret

Exception:
2024-01-17 13:12:09,586 ERROR Stopping due to error (org.apache.kafka.connect.cli.AbstractConnectCli) [main]
org.apache.kafka.connect.errors.ConnectException: Failed to connect to and describe Kafka cluster. Check worker's broker connection and security properties.
at org.apache.kafka.connect.runtime.WorkerConfig.lookupKafkaClusterId(WorkerConfig.java:305)
at org.apache.kafka.connect.runtime.WorkerConfig.lookupKafkaClusterId(WorkerConfig.java:285)
at org.apache.kafka.connect.runtime.WorkerConfig.kafkaClusterId(WorkerConfig.java:415)
at org.apache.kafka.connect.cli.AbstractConnectCli.startConnect(AbstractConnectCli.java:124)
at org.apache.kafka.connect.cli.AbstractConnectCli.run(AbstractConnectCli.java:94)
at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:116)
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165)
at org.apache.kafka.connect.runtime.WorkerConfig.lookupKafkaClusterId(WorkerConfig.java:299)
... 5 more
Caused by: org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLHandshakeException: No subject alternative DNS name matching Broker_URL found.
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:378)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:316)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1357)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1232)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1175)
at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209)
at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:435)
at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:523)
at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:373)
at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:293)
at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543)
at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:571)
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1381)
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1312)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.security.cert.CertificateException: No subject alternative DNS name matching Broker_URL found.
at java.base/sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:212)
at java.base/sun.security.util.HostnameChecker.match(HostnameChecker.java:103)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:458)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:418)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:292)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1335)
... 19 more

We are using Strimzi Kafka Operator 0.38.0.
Is it possible to disable the DNS verification from Kafka Connect side? If so, what are the configurations for same.

[scholzj]

You should be able to disable the TLs hostname verification using this options: https://kafka.apache.org/documentation/#brokerconfigs_ssl.endpoint.identification.algorithm

[telia-ankita]

@scholzj how we can do it in Strimzi Kafka Connect yaml?

[scholzj]

You would add it to the spec.config section and set it to empty string "". You migth maybe need to prefix it with consumer. and producer. prefixes as well - I never used this option in Connect so not 100% sure, you have to try that.

[telia-ankita]

Thanks @scholzj . I will try

[telia-ankita]

@scholzj I added ssl.endpoint.identification.algorithm: "" inside config of Kafka Connect and it worked.

I am facing one more issue, our Kafka Clusters are deployed in on-prem instance and Debezium Kafka Connect on AWS instance.
We have established connection between on-prem and AWS using Direct Connect. Therefore we have used AWS private link endpoint pointing to on-prem Kafka cluster.
We are using DNS of AWS private link endpoint as bootstrapServers in Kafka Connect instead of actual broker URL of our Kafka Cluster.
Here is flow of connection:
EKS cluster (where debezium kafka connect deployed) ->> AWS private link endpoint >> on-prem kafka cluster

I have checked the connectivity with the help of KafkaCat command and its working.

KafkaCat command:
kcat -b vpce-ewferfsdfsd.vpce.amazonaws.com:9094 -L -X security.protocol=SASL_SSL -X sasl.mechanism=SCRAM-SHA-512 -X sasl.username=admin -X sasl.password=<password> -X ssl.endpoint.identification.algorithm=none

But our debezium kafka connect pod is not coming up, its failing with timeout error.

2024-01-18 10:17:40,047 DEBUG [AdminClient clientId=adminclient-1] Kafka admin client closed. (org.apache.kafka.clients.admin.KafkaAdminClient) [main] 2024-01-18 10:17:40,047 ERROR Stopping due to error (org.apache.kafka.connect.cli.AbstractConnectCli) [main] org.apache.kafka.connect.errors.ConnectException: Failed to connect to and describe Kafka cluster. Check worker's broker connection and security properties. at org.apache.kafka.connect.runtime.WorkerConfig.lookupKafkaClusterId(WorkerConfig.java:305) at org.apache.kafka.connect.runtime.WorkerConfig.lookupKafkaClusterId(WorkerConfig.java:285) at org.apache.kafka.connect.runtime.WorkerConfig.kafkaClusterId(WorkerConfig.java:415) at org.apache.kafka.connect.cli.AbstractConnectCli.startConnect(AbstractConnectCli.java:124) at org.apache.kafka.connect.cli.AbstractConnectCli.run(AbstractConnectCli.java:94) at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:116) Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: listNodes at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165) at org.apache.kafka.connect.runtime.WorkerConfig.lookupKafkaClusterId(WorkerConfig.java:299) ... 5 more Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: listNodes an

Upon looking into logs of kafka connect I found that, it 1st sends request to kafka cluster using vpce-ewferfsdfsd.vpce.amazonaws.com:9094 privatelink URL and collects the metadata of kafka cluster.
In response it gets the list of brokers of kafka cluster with original broker URL, in our case it is

Received METADATA response from node -1 for request with header RequestHeader(apiKey=METADATA, apiVersion=12, clientId=adminclient-1, correlationId=1, headerVersion=2): MetadataResponseData(throttleTimeMs=0, brokers=[MetadataResponseBroker(nodeId=5, host='kafka.company.net', port=9095, rack=null), MetadataResponseBroker(nodeId=4, host='kafka.company.net', port=9094, rack=null), MetadataResponseBroker(nodeId=6, host='kafka.company.net', port=9096, rack=null)], clusterId='m-werwecsfs', controllerId=5, topics=[], clusterAuthorizedOperations=-35657567)

and again it tries to connect to kafka cluster with the broker url received in METADATA response. Which in our case are not reachable.
Therefore it throwing timeout error.

Is there any way by which we can avoid using broker url from METADATA response? Because in our case we have to use the AWS private link endpoint to connect to kafka cluster.

[telia-ankita]

Its not based on Strimzi.

[scholzj]

In Kafka, this is configured in the advertised.listeners options normally. Just keep in mind that if the same listener is used also by some local clients, this might break them for a change. So you might need a separate listener for this.

[telia-ankita]

Thanks. I will work with our kafka team and add required configurations.

[telia-ankita]

@scholzj As an alternative we have added Private DNS zone same as original broker url, for our private link endpoint.
So now connection flow will be like:
Debezium -> Private DNS Zone(kafka.company.net) -> AWS Private Link Endpoint -> on-prem kafka cluster.

With this configurations, we have moved ahead, but still debezium pod is not running and restarting continuously.
We are getting below error in debezium:

2024-01-24 14:14:02,756 WARN Attempt 3 to list offsets for topic partitions resulted in RetriableException; retrying automatically. Reason: Timed out while waiting to get end offsets for topic 'topic.connect.Offset-Storage' on brokers at kafka.company.net:9094,kafka.company.net:9095,kafka.company.net:9096 (org.apache.kafka.connect.util.RetryUtil) [DistributedHerder-connect-1-1]
org.apache.kafka.common.errors.TimeoutException: Timed out while waiting to get end offsets for topic 'topic.connect.Offset-Storage' on brokers at kafka.company.net:9094,kafka.company.net:9095,kafka.company.net:9096
Caused by: org.apache.kafka.common.errors.TimeoutException: Call(callName=listOffsets(api=LIST_OFFSETS), deadlineMs=1706105642754, tries=404, nextAllowedTryMs=1706105642855) timed out at 1706105642755 after 404 attempt(s)
Caused by: org.apache.kafka.common.errors.DisconnectException: Cancelled listOffsets(api=LIST_OFFSETS) request with correlation id 3227 due to node 4 being disconnected

2024-01-24 14:15:13,216 ERROR [Worker clientId=connect-1, groupId=connect-cluster] Uncaught exception in herder work thread, exiting:  (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
org.apache.kafka.common.errors.TimeoutException: Timeout of 60000ms expired before the position for partition topic.connect.Offset-Storage-0 could be determined
2024-01-24 14:15:13,216 INFO Kafka Connect stopping (org.apache.kafka.connect.runtime.Connect) [connect-shutdown-hook]
2

Any idea why this is happening ? Do you have any solution for it?

[scholzj]

I don't think I can identify any issues based on this. TimeoutException_ can mean anything from networking / connectivity issues up to some Kafka cluster or Connect configuration issues. Sorry.