Strimzi 0.34.0 creates podDisruptionBudget with minAvailable instead of maxUnavailable

[bryan-lw]

We recently upgraded from Strimzi Cluster Operator 0.25.0 to 0.34.0.

Using the same values and deploying and upgrading with Helm, we now have podDisruptionBudget's with minAvailable=3 (the cluster size) instead of maxUnavailable=0, as it was when using 0.25.0.

The Strimzi Kafka documentation says to set podDisruptionBudget=0 to use the Strimzi Drain Cleaner but never says mentions minAvailable. Maybe this was in some release notes, but I couldn't find mention of it and it can be confusing as to why we're setting maxUnavailable=0 but then the PDB show maxUnavailable as "N/A".

Can someone point me to the reason for this change?

[scholzj]

That is correct. This is because with custom pod controllers (such as the StrimziPodSets) you cannot use maxUnavailable. You can only use minAvailable (as a Kubernetes limitation). So we calculate from the number of replicas the inverted value -> how many pods need to be available.

[Migueljfs]

Got it, thanks for clarifying!

If this is the right place to ask, do you have a documentation I can read on what exactly the broker is doing when the drain-cleaner adds the "manual-restart" annotation? I understand kafka-operator will "reconcile" the pod but I don't know where to look

I see it now takes several minutes for a broker do go down and another spawning in, but the broker does not generate logs in the time between the node drain and it going down

[scholzj]

I guess some of it might be described here: https://strimzi.io/blog/2021/09/24/drain-cleaner/ ... but in general, the broker does not do anything. It is all handled by the operator. When the operator sees the manual rolling update annotation, it will connect to the brokers and check the status of them and their topics (and partitions and replicas). It will make sure they are in-sync to not break the availability ad if they are, it will basically delete the broker Pod and wait for it to get back online. Depending on your Strimzi version, new Pod will be started (either by Strimzi or Kubernetes) and will get up and running.

I see it now takes several minutes for a broker do go down and another spawning in

TBH, I would probably need to see more logs and more details on what exactly does this mean. I would expect the pod to be created right away after the old one is deleted. But it it could be that it needs more time to get running. That might depend on your infrastructure (do you have capacity in the right zone to start the pod, what kind of storage you use etc.). So that is not something I can answer without more into I'm afraid.

[Migueljfs]

I did read that blog but it does not fully explain what's going on behind the hood.

It will make sure they are in-sync to not break the availability

So let's assume I have 10 brokers and a replication factor on my topics of 3 and min insync 2. Then I drain a worker node for a given broker. The operator will see that the minimum of 2 replicas is still true and it will kill the pod? In this scenario having drain cleaner or not would not have an impact?

TBH, I would probably need to see more logs and more details on what exactly does this mean

Apologies if I was unclear, when the old pod dies, the new one comes up right away. What I meant is that the time between I run "kubectl node drain.." and the pod actually going down takes 5/6min that I assumed is something to do with the broker cleanly shutting down? But from your description above I guess this time is the operator going "topic1 check, topic2 check, topic3 check..."

[scholzj]

So let's assume I have 10 brokers and a replication factor on my topics of 3 and min insync 2. Then I drain a worker node for a given broker. The operator will see that the minimum of 2 replicas is still true and it will kill the pod? In this scenario having drain cleaner or not would not have an impact?

Yes. The operator will check all topics, their in-sync replicas and make sure they will remain available when it stops the broker. So in your example, if you have a partition with replication factor 3 on brokers 0, 1 and 4 but only replicas 0 and 1 are in-sync, it will not roll the broker 0 until the replica 3 becomes in sync as well. Because if it did that, you would have only one in-sync replica and with min.insync=2 you would not be able to produce messages.

This is important because if you would drain all your worker nodes in a sequence, you would for example first drain broker 0, then broker 1, then broker 3 etc. The operator and its checks give you the guarantees that the replicas are in sync before it rolls the next broker. Without the Drain Cleaner (and the Pod Disruption Budget set to 0), Kubernetes would roll the pods when they are ready. But the pod being ready does not (in case of Strimzi) mean all topics are in-sync. It just means that the broker is able to accept connections from clients.

Apologies if I was unclear, when the old pod dies, the new one comes up right away. What I meant is that the time between I run "kubectl node drain.." and the pod actually going down takes 5/6min that I assumed is something to do with the broker cleanly shutting down? But from your description above I guess this time is the operator going "topic1 check, topic2 check, topic3 check..."

So, this can again have multiple causes. First, there will be delay between the Drain Cleaner annotating the pod and the operator rolling it. That would normally with default settings be for example up to 2 minutes. Second, if you had right before another node drained, the previous broker might not be fully in sync yet and the operator might need to wait for it to sync up which again might take some time.

[Migueljfs]

Understood all your points and what drain-cleaner + operator are doing now, thanks for clarifying!