anti-affinity with hard scheduling (required doesn't seem to be working)

[elmehdisaniss]

Hi Strimzi team,

I'm using the 0.25.0 of the strimzi operator with its 0.25.0 helm chart. I deployed the kafka/zookeeper clusters with 3 instances for each using the anti-affinity configuration presented in 0.25.0 documentation in an eks cluster that has enough nodes (8 nodes) so the test wasn't relevant. So I used my 1 node minikube cluster so that minikube deploys only one instance and report the other ones as not able to be deployed due to the anti-affinity rule that is not satisfied (1node) but it was not the case. Minikube deployed all instances. I tried this test with the postgres operator and minikube couldn't schedule due to the non existence of other nodes to schedule pods on.

It is the same configuration for the values.yaml file :

kafka_cluster:
  kafka:
    template:
      pod:
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                    - key: strimzi.io/name
                      operator: In
                      values:
                      - nms-kafka
                topologyKey: topology.kubernetes.io/hostname
    replicas: 3
    resources:
      requests:
        cpu: 0m
        memory: 2000Mi
      limits:
        cpu: 0m
        memory: 2000Mi
    loglevel: INFO
  zookeeper:
    template:
      pod:
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                    - key: strimzi.io/name
                      operator: In
                      values:
                      - nms-zookeeper
                topologyKey: topology.kubernetes.io/hostname
    replicas: 3
    resources:
      requests:
        cpu: 0m
        memory: 800Mi
      limits:
        cpu: 0m
        memory: 800Mi
    loglevel: INFO

And as we are using helm charts this is what is in the template file of the kafka resource :

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: nms
spec:
  kafka:
    replicas: {{ .Values.kafka_cluster.kafka.replicas }}
    resources: {{ toYaml .Values.kafka_cluster.kafka.resources | nindent 6 }}
---
    {{- with .Values.kafka_cluster.kafka.template }}
    template:
      {{- toYaml . | nindent 6 }}
  zookeeper:
    replicas: {{ .Values.kafka_cluster.zookeeper.replicas }}
    resources: {{ toYaml .Values.kafka_cluster.zookeeper.resources | nindent 6 }}
---
        {{- with .Values.kafka_cluster.zookeeper.template }}
    template:
      {{- toYaml . | nindent 6 }}
  {{- end }}

Do you see this as an issue or do you have any explanation please ?

Best regards,

[scholzj]

You probably configured it wrong? I'm afraid without you sharing the configuration you used etc., there is not much more one can say.

[scholzj]

Look, all I can tell you from this is that it looks ok. Not much more can be done with a picture which is not even complete and is drawn over.

[scholzj]

BTW: Is topologyKey: topology.kubernetes.io/hostname correct? I do not have it on my cluster. I have kubernetes.io/hostname.

[elmehdisaniss]

You found the issue. No it is not correct. Removed the "topology." and the anti-affinity worked. Thank you so much for the help.

[elmehdisaniss]

Another question please, because I am still having an issue : Is the preferred mode supported ? I used the preferred scheduling configuration in anti-affinity but it is still treated like required(hard scheduling and it is blocking deployment of 2 instances on my minikube environment).

[scholzj]

The whole affinity just mirrors the Kube API and Strimzi just passes it into the pods. So we support anything Kube supports. One thing to keep in mind is that the structure Kube uses for prefered and required differs. So you should make sure you got it right.

[daha]

I think the problem might have been the topologyKey topology.kubernetes.io/hostname. My nodes did not have the label topology.kubernetes.io/hostname, but they had the label kubernetes.io/hostname so when I did set topologyKey to kubernetes.io/hostname it worked for me.