[mimir-distributed] Rollout operator replicas not configurable

[peschmae]

The mimir-distributed helm-chart allows to use the rollout-operator for zone aware disruption budgets on the ingesters. When performing a Kubernetes you end up draining each node, and if the rollout-operator and a zone aware ingester are running on the same node, this can lead to issues as the AdmissionWebhook for the statefulset will fail until the rollout-operator is started on a new node.

There is currently no option to configure multiple replicas for the rollout-operator to make it HA, which is considered best practice for anything used as a AdmissionWebhook in Kubernetes.

Please add an option to make the rollout-operator HA in mimir-distributed, so it's easier to perform rolling kubernets upgrades.