kafka is not rollable until we restart strimzi kafka operator

[elvis-cai]

we are using strimzi kafka operator 0.34.0 helm chart, and we found around 07/27, we only have two Kafka brokers running, and strimzi kafka operator keeps Reconciliation and times out until we restart the strimzi operator, here are the logs when the issue happen around 2023-07-27 15:36:51, we guess this issue probably will happen will one of the kubernetes node hosting kafka or zookeeper is scaled up or down by the cluster autoscaler, since we found the same issue for zookeeper this morning, and no new pod is created for zk as well.
The issue may be started when one of the kafka node is changed, here are some logs

1. [AdminClient clientId=adminclient-2110] Connection to node
2. Will temporarily skip verifying pod kafka-operator-kafka-2 is up-to-date due to ForceableProblem:
3. Error waiting for pod strimzi/kafka-operator-kafka-2 to become ready
4. Kafka(strimzi/kafka-operator): createOrUpdate failed
5. continue reconciliation
   The phenomenon is strimzi operator is reporting reconciliation kafka, however there's no new kafka pod created, here are the whole logs https://gist.github.com/elvis-cai/db8d25ae8b60ccd080935fe0ea15147b, can not find any clue on it, would you mind helping have a look, thanks.

[scholzj]

Hard to say for sure without more information. But this looks like it might be the issue with Kubernetes informers. So you should try to upgrade to 0.35.1 or newer.

[elvis-cai]

thanks, @scholzj , will have a go with 0.35.1 🙇

[scholzj]

Just FIY: Tis is the original issue: #8528 ... it also links to some more discussions if you want to dig more into whether it really is this issue or not.

[elvis-cai]

Nice, cheer mate, will take a look, we are using GKE and just upgraded GKE from 1.24 to 1.25 BTW, we haven't had this issue in 1.24 afaik.