How to create ingress with cert-manager?

[1nd2rd3st]

Hello,

we are currently running strimzi 0.23.0 and have the demand, that kafka has to be available for services outside of our kubernets cluster.
I thought about using the ingress option provided by strimzi. Our cluster is running the nginx-ingress-controller

For the certificate management we use cert-manager. To create certificates with an ingress, the usual way described in the cert-manager-docu is to provide the secretName for the certificates in the ingress controller under tls.secretName:

tls: # < placing a host in the TLS config will determine what ends up in the cert's subjectAltNames
  - hosts:
    - example.com
    secretName: myingress-cert # < cert-manager will store the created certificate in this secret.

But unfortunatly strimzi is not creating such secretName in the created ingress. This results in an error for the ingress Skipped a TLS block: spec.tls[0].secretName: Required value

Am I overlooking a solution to make cert-manager + strimzi running.

Any help is appreciated.
Regards.

[scholzj]

Kafka is a TCP protocol and in order to use it with Ingress, you need to configure the Ingress to use TLS Passthrough. And when you do TLS passthrough, you cannot configure the certificate on the Ingress level. You have to do it in Kafka. This blog post might help you: https://strimzi.io/blog/2021/05/07/deploying-kafka-with-lets-encrypt-certificates/ ... it focuses on Let's Encrypt, but uses cert-manager as well, so it should not differ much from using some other cert-manager sources.

[1nd2rd3st]

Thank you for your reply.

I already used the provided blog post to create the ingress. So far without any success.
ssl-passthrough is activated in our nginx. I was trying out strimzi 0.23.0 with 2.6.0, 2.8.0 and also updated to strimzi 0.27.1 with 2.8.0 and 3.0.0.

when checking checking certificates wiht openssl i got following error with tls1.3

140231361630656:error:1408F119:SSL routines:ssl3_get_record:decryption failed or bad record mac:../ssl/record/ssl3_record.c:677:

tls1.2 was fine, so i added the configs to force tls1.2

ssl.enabled.protocols: "TLSv1.2"
ssl.protocol: "TLSv1.2"

when im running the strimzi client examples with ingress endpoint. i get no real info for the producer:

2022-02-03 17:03:16 INFO  KafkaProducerExample:69 - Sending messages "Hello world - 34"
2386845 [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Bootstrap broker xxx:443 (id: -1 rack: null) disconnected
2417425 [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Bootstrap broker xxx:443 (id: -1 rack: null) disconnected

or the consumer:

2022-02-03 17:04:00 DEBUG NetworkClient:1042 - [Consumer clientId=consumer-my-group-1, groupId=my-group] Give up sending metadata request since no node is available
2022-02-03 17:04:00 DEBUG NetworkClient:1157 - [Consumer clientId=consumer-my-group-1, groupId=my-group] Initialize connection to node xxxx:443 (id: -1 rack: null) for sending metadata request
2022-02-03 17:04:00 DEBUG NetworkClient:986 - [Consumer clientId=consumer-my-group-1, groupId=my-group] Initiating connection to node xxx:443 (id: -1 rack: null) using address xxx/xxx.xxx.xxx.xxx
2022-02-03 17:04:27 DEBUG NetworkClient:820 - [Consumer clientId=consumer-my-group-1, groupId=my-group] Disconnecting from node -1 due to socket connection setup timeout. The timeout value is 26226 ms.
2022-02-03 17:04:27 WARN  NetworkClient:1060 - [Consumer clientId=consumer-my-group-1, groupId=my-group] Bootstrap broker xxx:443 (id: -1 rack: null) disconnected

To get some more debug output I tried out the local kafka-console-conusmer

$ kafka-console-consumer.sh --bootstrap-server xxx:443 --topic my-topic --consumer-property security.protocol=SSL
[2022-02-03 16:34:42,954] ERROR [Consumer clientId=console-consumer, groupId=console-consumer-86311] Connection to node -1 (xxx/xxx.xxx.xxx.xxx:443) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2022-02-03 16:34:42,955] WARN [Consumer clientId=console-consumer, groupId=console-consumer-86311] Bootstrap broker xxx:443 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2022-02-03 16:34:42,961] ERROR Error processing message, terminating consumer process:  (kafka.tools.ConsoleConsumer$)
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLProtocolException: Unexpected handshake message: server_hello

when I was running kafka with tls1.3 i got a different exception

$ kafka-console-consumer.sh --bootstrap-server xxx:443 --topic my-topic --consumer-property security.protocol=SSL
[2022-02-03 13:43:26,142] ERROR [Consumer clientId=console-consumer, groupId=console-consumer-93267] Connection to node -1 (xxx/xxx.xxx.xxx.xxx:443) failed authentication due to: Failed to process post-handshake messages (org.apache.kafka.clients.NetworkClient)
[2022-02-03 13:43:26,144] WARN [Consumer clientId=console-consumer, groupId=console-consumer-93267] Bootstrap broker xxx:443 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2022-02-03 13:43:26,147] ERROR Error processing message, terminating consumer process:  (kafka.tools.ConsoleConsumer$)
org.apache.kafka.common.errors.SslAuthenticationException: Failed to process post-handshake messages
Caused by: javax.net.ssl.SSLException: Tag mismatch!

also trying out kafkacat

$ kafkacat -C -b xxxx:443 -X security.protocol=ssl -o end -t my-topic
% ERROR: Failed to query metadata for topic my-topic: Local: Broker transport failure

I'm kind of running out of ideas where to look and how to debug the actual problem here.
With internal listeners, the producer & consumer are working fine. So in general the kafka is working.

Kind Regards.

[scholzj]

Its quite hard without knowing the configuration. Tag Mismatch sometimes means that you have TLS client authentication enabled in the broker => so you would need to disable it or pass a user certificate. But without understanding the setup it is just guessing.