How to specify TLS auth for kafka connect which has debezium mysql connector ?

[alok87]

Hi

Kafka Connector

https://gist.github.com/alok87/d5b2cac48aa5d616a76fdcf2b18cd766

Kafka Connect

https://gist.github.com/alok87/5a8e6533906ce2c164f98f101c223f46

Kafka

 kafka:
    authorization:
      superUsers:
      - CN=kafka-connect
      type: simple
    listeners:
      external:
        authentication:
          type: tls
        tls: true
        type: loadbalancer

Kafka User (super user)

spec:
  authentication:
    type: tls

With the above configurations, kafka starts in TLS mode, Kafka connect comes up!
But as soon as I create the Kafka connector, the connector fails trying to talk to Kafka and the logs of the kafka pods has the following SSL Handshake failed error:

2021-01-12 15:03:24,891 INFO [SocketServer brokerId=0] Failed authentication with /10.2.21.9 (SSL handshake failed) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-0-ListenerName(EXTERNAL-9094)-SSL-4]

TLS is working perfectly from a Golang process using the same certs and keys (same user cert).

Please suggest if I am configuring the TLS correctly? Or Do I also need to pass TLS config like this in the debezium connector configuration https://debezium.io/documentation/reference/1.3/connectors/mysql.html#mysql-connector-pass-through-properties?

Thanks for the help.

[scholzj]

Your configuration is correct as far as Strimzi and Kafka Connect goes. But I do not know Debezium well enough to understand whether they need something more or nor. So that is maybe more a question on Debezium? If they need it, you can probably provide the certificates by mounting them from the secret using the externalConfiguration section?

[alok87]

Can I mount files database.history.producer.ssl.keystore.location=/var/private/ssl/kafka.server.keystore.jks in this /var/private/ssl/... path using externalConfiguration. I see the externalConfiguration mounts the file in a not user controlled location. @scholzj

[scholzj]

No you cannot ... but that is why Debezium gives you the option to configure the path to set it to something else.

[alok87]

Ok that will work. let me try out. I am converting the kafka generated secrets to jks using keytool.

[alok87]

Thanks for the reply.
There configuration does not work with PEM formats, and require a keystore, and truststore https://debezium.io/documentation/reference/1.3/connectors/mysql.html#mysql-connector-pass-through-properties

[scholzj]

You can probably also piggiback on the certificates used by Strimzi directly. If you have running Connect Pod, there should be config file in tmp where you can copy the locations (they are always the same). And you would need to get the passwords using the file provider from this config file.

[alok87]

This article shows the conversions that would be required. https://dzone.com/articles/kafka-on-kubernetes-the-strimzi-way-part-3

[scholzj]

That should not be needed. Kafka / Java can work with PKCS12 stores. And I assume Debezium should as well since they just inherit this from Kafka/Java.

[alok87]

Confused then what I need to do exactly to make it work 😕

[alok87]

Kafka documentation has the keystore type option, i would try P12/PKCS12 format and there might be no need to convert as you mentioned. I would get back by tomorrow and answer this question / or update my findings.

[alok87]

This article does not have the TLS mentioned for the connector https://strimzi.io/blog/2020/01/27/deploying-debezium-with-kafkaconnector-resource/

[scholzj]

I guess it does not use TLS in Kafka.

[alok87]

Yes I would try the keystore and trustore converted certs and get back.

[alok87]

@tombentley hey! saw your debezium article, can you help out here !

[alok87]

So i am little confused. Why does Kafka connect and kafka connector both have certificates and authentication. I thought connectors are plugins, but everything runs as a single process that is Kafka connect!

[scholzj]

That is the case, yes. That is also why Strimzi has no real support for it - because normally it is not needed.

Debezium is the only connector I know which seems to be a bit special in this. But they are smart guys, so I'm sure they had good reasons for it ... but what exactly they were would be question for some Debezium forum I guess.

[alok87]

Debezium team: "Debezium requires to have a separate SSL configuration since Debezium connectors (like MySQL) need to store database history."

[alexmoise-3ib]

@alok87 - any luck with this or feedback from Debezium?

[cyrmue]

@alok87 - have you found a good solution for this?