Can the livenessProbe and readinessProbe for Kafka Brokers be changed?

[ccastro-ai]

Hi all, I have a Kafka Cluster deployed using strimzi operator on a K8s cluster within AWS.

Strimzi Operator: V 0.28.0
Kafka Version: 3.0.0

The cluster needs to be accesible from outside, so we have set up ExternalDNS to provide server hostname access for the clients using the external-dns annotations for the bootstrap server

    bootstrap:
        ...
        annotations:
            external-dns.alpha.kubernetes.io/hostname: XXXXXXXXXX
            external-dns.alpha.kubernetes.io/ttl: "60"
        alternativeNames:
            - XXXXXXXXXXXX

Now comes the problem, when a doing a rolling restart, a race condition between the brokers restart and the ExternalDNS happens. The DNS entry for the bootstrap server gets deleted as the pods are in a restarting state, when the pods come back alive and reach READY state, the clients cannot send data to them as the DNS entry has not been created again.

A mitigation for this could be to increase the TTL of the DNS entry to a value larger than the expected restarting time but this is just hiding the problem by diminishing the chance of the race condition to be met.

Ideally, we would like to modify the readinessProbe of the Kafka Broker pods, to wait until the DNS entry for the bootstrap server is found (dns lookup), however Kafka resource does not seem to allow modifying what the readinessProbe does, only waiting and frequency durations as well as threshold values can be changed.

Is there a way to modify the readinessProbe behavior and/or extend it to wait until the DNS can be resolved, keeping in mind that this is an issue caused by how ExternalDNS works under the hood?

P.S. Using internal listeners for clients works fine as the brokers can be reached within the cluster, but external clients cannot reach the brokers until the ExternalDNS has created the entry.

[scholzj]

I'm not sure I follow the issue TBH. But if you want, you can modify the timings of the probes. So if you need more time before the pod gets ready, you can for example increase the initial delay. Check the docs for the details (look for something like readinessProbe). You cannot change the probe itself to do something else than what it does.

[ccastro-ai]

Thanks @scholzj

Actually that was my question, whether it was possible to change what the probe does or not (specifically, readinessProbe).

I'm aware of what this probe does, checking for the existence of the kafka-ready file and this file is created by the kafka-agent run as a Java Agent when the kafka process is started.

Ideally, I would also like to check for the bootstrap DNS entry existence as part of the readiness Health Check of the Kafka brokers, so was wondering why can't be the readinessProbe command be overridden?

[scholzj]

The operator relies on the readiness probe and needs to know what it means when it is passing or failing. If we make this configurable, then the readiness would not mean anything to the operator.

[ccastro-ai]

Understood @scholzj, thanks for the support!

[ynaguib]

The main issue here, is that we have a race condition where the kafka broker is ready before k8s external-dns creates the DNS entry.
Our DNS entries are created by annotation. So, until the DNS entry is available, consumer get unknown host, as the host is being sent by the bootstrap.
So, we're looking for a solution to stop broker from starting until the DNS is available.

If you can think of another approach....
Thanks @scholzj

[scholzj]

External DNS is targeting a Service (Load Balancer or NodePort) or an Ingress resource. These are not touched in any way during the rolling updates. So TBH there is absolutely no reason why the DNS entry should be missing during things such as rolling updates. The DNS should be completely unchanged. This is also what I so far saw in all of my environments when using ExternalDNS.

When creating a new cluster, it can for sure happen that the initial creation of the DNS records takes some time. But I still don't see how this would cause any issues. Simply the clients will connect for the first time only some time later.

[ynaguib]

We have been able to reproduce a case where when doing a rolling restart ExternalDNS deleted the DNS entry.
Here is the sequence:

• Broker restart starts
• Endpoint deleted as a result
• ExternalDNS deletes the DNS entry (At the ExternelDNS refresh cycle)
• Broker starts and made available
• DNS created after the ExternelDNS refresh cycle (In our case 10mn)