Reliably determine if a kafka-cluster restart or rebalance is imminent prior to applying changes through Helm upgrade

[frankgrimes97]

We're trying to improve our Strimzi CI/CD pipeline in such a way that when our engineers create a PR making changes to our Strimzi Helm chart values files we can give an indication that a kafka-cluster restart or rebalance will be triggered by the pending config changes once approved and merged to the main/master branch.
In the short term, this is to give them a heads up that the deployment will likely cause some service disruption.
Longer term we could even consider loosening some monitoring/alerting constraints for the duration of the deploy/rebalance.

Is this something anyone else has solved? If so, how?

Right now we were considering looking into grepping and looking for certain patterns in the helm diff upgrade output.
However, it's not clear to us exactly what output indicates that those more disruptive changes will be applied.

Any help and guidance on this would be much appreciated!

Thanks!

[scholzj]

Are we talking about changes to the Strimzi Cluster Operator (i.e. about the official Strimzi Helm chart?) Or about changes to some custom Helm chart that generates a Kafka CR?

[frankgrimes97]

It's the latter... we deployed the Strimzi Cluster Operator using the Strimzi Helm chart.
We also have deployed many Kafka Custom Resources through our own Helm charts.
As we make changes to our Kafka CR helm charts and apply them through the operator, we'd like to know ahead of time whether the changes to be applied will trigger any Kafka broker restarts and/or (CruiseControl?) topic rebalances.

[scholzj]

Topic Rebalances are triggered only through the KafkaRebalance resource right now. Not through any changes to the Kafka CR.

As for the Kafka rolling updates, that is quite hard I'm afraid. I do not think there is any categorization. If you ask me about specific changes, I might be able to give you a yes or no answer. But I cannot produce a list. In general, for changes to .spec.kafka.config it depends on Kafka if they can be dynamically configured or not. For the other changes, if they change the Pod specification or metadata, they will be rolled. But if they change other resources then typically not. Changes to listeners or authorization will normally always cause rolling updates.