Change Active Controller to help resolve URPs

[Manicben]

Hey all! I haven't been able to find any information as to whether Strimzi supports changing the current active controller. This can be done manually by going into the ZK CLI and deleting the controller znode, which forces all active brokers to attempt to write to that znode, the successful broker then becoming the active controller.
If this doesn't exist, it might be worth adding to make operator's lives easier. That said, there's also a chance we might be doing something wrong and I'd like to hear any suggestions.

For context, we sometimes get a situation after a rolling update where we end up with a small number of underreplicated partitions, usually less than 10. It looks like some followers essentially stop fetching from the leader, leaving the leader as the only in-sync replica. We've found that the easiest way to resolve this is to simply restart that leader, after which the followers replicate and everything gets back in sync. We also prefer to change the active controller in case the leader we are restarting is also the active controller, which helps to reduce disruption and hence this question/request.

Please do let us know if we might be doing something wrong or whether this URP behaviour is unexpected. It occurs rarely, but enough that we've needed to restart leaders and change active controllers a few times. Note that we run many 3 broker Kafka clusters rather than a few large clusters.

[scholzj]

I do not think we support it and do not have any plans to add support for it given ZooKeeper is on its way out. I'm also not sure if it is really something to do as it can have many side effects. That said, I assume you can do it in ZooKeeper yourself if you want to 🤷 .