Pod Anti Affinity is too strict : multiple clusters with same name cannot be scheduled on same nodes

[jggc]

We are running multiple redpanda clusters on the same k8s cluster, for dev, staging, prod, etc.

They are deployed in different namespaces but with the same setup, so they end up with the same names.

With the current strict anti affinity it is impossible to have them scheduled properly in a cluster with less nodes than the total number of redpanda nodes across all redpanda clusters even though they are not part of the same redpanda cluster.

So let's say we deploy two 3 nodes redpanda cluster on a 4 nodes k8s cluster at the same time :

Each redpanda cluster will successfully create the first node on two separate nodes.

Then they will be able to schedule a second node each. But both of their 3rd nodes will never get scheduled. Both clusters are up with 2/3 nodes.

But then you cordon one node and it is impossible to perform the rolling upgrade without downtime since all the 4 workers already have a redpanda node and cannot schedule another one, even if it is not of the same cluster.

So, there are two solutions here :

1. Use preferredDuringSchedulingIgnoredDuringExecution for the anti affinity, which is probably too loose a requirement to be acceptable with the current approach, which is explicitely documented in redpanda docs
2. Use a unique cluster id, or simply the namespace as part of the anti affinity rule to allow separate redpanda clusters to run on the same node
3. Combine the two : use a required anti affinity for redpanda nodes of the same redpanda cluster (by id or namespace) and add to that a preferred anti affinity for any redpanda node.

With all that said, I completely understand that the default is correct for a production cluster. But I think it is reasonable to allow, by default, users to deploy multiple clusters with the same name in different namespaces.

This also aligns with the current permission to deploy multiple redpanda clusters of different names on the same node, even if they are in the same namespace.

I hope that was clear, sorry for the long-ish issue 😅