Zookeeper struggles to come online - operator 0.33.2, .34, .35

[i842536]

I've been attempting to deploy Kafka 3.4 into a K8s cluster via the Strimzi Operator (versions listed below) to satisfy a requirement to upgrade Kafka. All three operator versions yield the same results and the same inconsistencies I'm about to describe.

The Strimzi Operator deployment and pod comes online - this succeeds 100% of the time. Next, the StrimziPodSet attempts to spin up the Zookeeper pods and the problems surface:

• The pods will sit pending for hours if you allow them to.
• The PVs for the zookeeper are created ad nauseam. Every 5 minutes 3 new PVs are created and never attached to any PVCs.
• If you "get pvc" you'll see PVCs creating and terminating in a loop.
• Sometimes you can get away with deleting the ZK pods and upon recreation they come online and enter a running state. If by chance ZK comes online, a second StrimiziPodset will be created and attempt to online the Kafka cluster.
• The Kafka cluster experiences the same issue with infinitely creating PVs too.

The above behaviors happen regardless of the Operator version used. I'm attaching the most recent StrimziOperator log for folks to examine. In this capture the ZK pods never leave a pending state.

Strimzi Operator: 0.33.2, 0.34.0, 0.35.0
Kafka version: 3.4
K8s version: 1.25.6
strimzi-cluster-operator (4).txt

Any advice on where to look / troubleshoot would be welcome and appreciated.

Thanks,

Jarrod

[scholzj]

You should check if you are using something like ArgoCD -> it tends to delete PVCs / PVs.

[i842536]

You're absolutely spot on about running ArgoCD. Upon reading your response and a bit of pondering we disabled the auto "self-heal" and "prune resources" for the Kafa application. Upon disabling those two settings, the repeated PV creation stopped, PVCs were bound, and pods for ZK and Kafka began running.

The logical question is why was Argo acting in such a fashion? What is it about the strimzipodset in 0.33.2 (and greater) that doesn't seem to mesh with Argo? Like I stated in my original post this work is all part of an upgrade. We're currently running live with 0.27.0 and Kafk 3.0 with zero Argo issues detected.

Jarrod

[scholzj]

That is more question for Argo. But as far as I understood it -> you are probably using tracking using owner references. But the PVCs do not have owner references (that is intentional to not garbage collect them). If I remember correctly their recommendation is to switch to annotation-based tracking. There are some previous discussions on this topic here with some more details.

[i842536]

@scholzj

Thanks for the pointers on Argo. We've resolved our issue and learned something new.