Kafka pods spread over different zone in k8s

[DavidPeyrton]

Hello everyone
I'm trying to deploy a Kafka instance 3.9.0 with a Strimzi operator 0.45 on an OpenShift cluster.
I have 6 workers with a label topology.kubernetes.io/zone sets with zone1, zone2 and zone3.
I'm trying to equally spread 6 kafka pods over the zones with:
apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka spec: kafka: rack: topologyKey: topology.kubernetes.io/zone pod: **affinity: {}** metadata: labels: app: strimzi-kafka-topology priorityClassName: core-normal-priority-class topologySpreadConstraints: - labelSelector: matchLabels: app: strimzi-kafka-topology maxSkew: 1 nodeAffinityPolicy: Honor nodeTaintsPolicy: Honor topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: DoNotSchedule - labelSelector: matchLabels: app: strimzi-kafka-topology maxSkew: 1 nodeAffinityPolicy: Honor nodeTaintsPolicy: Honor topologyKey: kubernetes.io/hostname whenUnsatisfiable: ScheduleAnyway
When deployed, the topologySpreadConstraints are put into the StrimziPodSet as well as in the Pod objects. But there is also an affinity section:
spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: topology.kubernetes.io/zone operator: Exists podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - podAffinityTerm: labelSelector: matchLabels: strimzi.io/cluster: strimzi strimzi.io/name: strimzi-kafka topologyKey: topology.kubernetes.io/zone weight: 100
I set no affinity in the Kafka object: affinity: {}. So I don't understand why it is added in the created objects.
Is there any reason ?

Initially, the pods are well spread over the zones.
But if I cordon all the nodes of a given zone and restart a pod from this node, then it will respawn on another zone. I was expecting it to wait for a node in its initial zone to become available again.
Did I misunderstood the purpose of the zones or did not configure it properly ?

[scholzj]

The YAML you are sharing is badly formatted. So it is not clear what it really is.

If you enabled the rack awareness, there will be a default preferred affinity that you can combine with additional affinity and spread constraint rules.

[DavidPeyrton]

Thank you for your answer.
I'm sorry, I didn't find a better way to provide the the yaml...
So, even if I set affinity: {} it is overridden.
What is strange to me is that even we all those constraints (affinity + topologyspreadconstraints), a pod can be scheduled in another zone. I don't see what's wrong.

[scholzj]

That is hard to answer without understanding how the rules you have set actually look like, how does your infrastructure look like, and what exactly is wrongly scheduled. It might be that because you cordoned the whole zone it does not see it as a zone that is being used. You might have the labels wrong. And so on. Hard to say without all the details.

Depending on what exactly are you trying to do, you can also try to setup your cluster with 3 KafkaNodePool each one targeting specifcally a particular zone through node affinity. That can help to bypass some of the Kubernetes scheduling complexities around spreading things. See more details for example here: https://strimzi.io/blog/2023/08/28/kafka-node-pools-storage-and-scheduling/

[DavidPeyrton]

"It might be that because you cordoned the whole zone it does not see it as a zone that is being used." I don't know how I can validate this assumption but it make sense.
"you can also try to setup your cluster with 3 KafkaNodePool each one targeting specifcally a particular zone through node affinity." That's a good suggestion. Especially as I started to practice KafkaNodePool and will migrate to Strimzi 0.47 soon.
Thank you.

[scholzj]

"It might be that because you cordoned the whole zone it does not see it as a zone that is being used." I don't know how I can validate this assumption but it make sense.

Yeah, I do not know how to validate that either, I'm afraid. And TBH, in the past I saw more the other way ... that user wanted it to forget the lost AZ and Kubernetes was insisting on it. But it is a long time ago, so things might have changed.

[DavidPeyrton]

I just did a test to validate "It might be that because you cordoned the whole zone it does not see it as a zone that is being used."
3 zones, 6 nodes, 6 Kafka pods. I just cordoned I node and deleted the pod running on it. After 10 minutes the pod respawned on another zone... :-(
Finally, the pod is on the same node as another Kafka pod, but on another zone... I don't understand the scheduling logic...
Thank you for your answer and advice.