Is it possible to set resource limits to the Kafka init container?

[TalZiv]

Hi,

Is it possible to set resource limits to the Kafka init container?

For some reason, the default resource limit of 512 MB causes OOM on Apple M2 computers.
The same setup works well on Apple I7 Macs but not on M2 not sure why.

I have manually changed the stateful set init container limit to 1GB and Kafka booted past the init container phase
but a couple of minutes later the configuration was reset by the operator.

Thanks and any advise or pointers are appreciated.

[ppatierno]

You should be able to do that by setting the content for kafka.resources (as stated here https://strimzi.io/docs/operators/latest/configuring#type-KafkaClusterSpec-schema-reference). The CPU and memory requests and limits set there will be used for all containers related to the Kafka pod, so Kafka itself and the Kafka init. Of course, the Kafka init container will use this resources and free them because it runs and exits. Those resources will be then used by the Kafka related container.

[TalZiv]

Hi,

The resources set in the Kafka object are set to the Kafka pods, The init container has its own.

[ppatierno]

The resources set in the Kafka object are set to the Kafka pods, The init container has its own.

No it's not this way. The Kafka application and the init container live in the same pod.
Did you try it? This is what happens in the Strimzi code and I can confirm by running an example.

The Kafka pod is built by two containers. The first one is the kafka-init and the other one is about the Kafka application itself. So they live in the same pod as two containers but at different times (because one is an init so it ends before the other one starts).

With a Kafka CR having rack-awareness enabled and the kafka.resources like this:

kafka:
    rack:
      topologyKey: topology.kubernetes.io/zone
    resources:
      requests:
        memory: 1Gi
        cpu: "1"
      limits:
        memory: 1Gi
        cpu: "1"

If you get the definition of a Kafka pod created by the operator you can see two containers. One is the container named kafka which has the following resources definition:

resources:
      limits:
        cpu: "1"
        memory: 1Gi
      requests:
        cpu: "1"
        memory: 1Gi

And one init container in the initContainers section having the same resource definition.

Following the full Kafka pod definition created by the operator to confirm what I said:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    strimzi.io/broker-configuration-hash: fbaf004c
    strimzi.io/clients-ca-cert-generation: "0"
    strimzi.io/cluster-ca-cert-generation: "0"
    strimzi.io/cluster-ca-key-generation: "0"
    strimzi.io/inter-broker-protocol-version: "3.5"
    strimzi.io/kafka-version: 3.5.1
    strimzi.io/log-message-format-version: "3.5"
    strimzi.io/logging-appenders-hash: e893ac9f
    strimzi.io/revision: e9173095
    strimzi.io/server-cert-hash: 86d19927449017f5c24c2bd64a582aff6e0b4ad6
  creationTimestamp: "2023-10-29T17:00:16Z"
  labels:
    app.kubernetes.io/instance: my-cluster
    app.kubernetes.io/managed-by: strimzi-cluster-operator
    app.kubernetes.io/name: kafka
    app.kubernetes.io/part-of: strimzi-my-cluster
    statefulset.kubernetes.io/pod-name: my-cluster-kafka-0
    strimzi.io/broker-role: "true"
    strimzi.io/cluster: my-cluster
    strimzi.io/component-type: kafka
    strimzi.io/controller: strimzipodset
    strimzi.io/controller-name: my-cluster-kafka
    strimzi.io/controller-role: "false"
    strimzi.io/kind: Kafka
    strimzi.io/name: my-cluster-kafka
    strimzi.io/pod-name: my-cluster-kafka-0
    strimzi.io/pool-name: kafka
  name: my-cluster-kafka-0
  namespace: kafka
  ownerReferences:
  - apiVersion: core.strimzi.io/v1beta2
    blockOwnerDeletion: false
    controller: true
    kind: StrimziPodSet
    name: my-cluster-kafka
    uid: bbc3f8da-f20f-4067-a9ac-367022550b3c
  resourceVersion: "1419"
  uid: b381f72a-83bc-412a-93e8-b537be87a534
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: topology.kubernetes.io/zone
            operator: Exists
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - podAffinityTerm:
          labelSelector:
            matchLabels:
              strimzi.io/cluster: my-cluster
              strimzi.io/name: my-cluster-kafka
          topologyKey: topology.kubernetes.io/zone
        weight: 100
  containers:
  - args:
    - /opt/kafka/kafka_run.sh
    env:
    - name: KAFKA_METRICS_ENABLED
      value: "false"
    - name: STRIMZI_KAFKA_GC_LOG_ENABLED
      value: "false"
    - name: STRIMZI_DYNAMIC_HEAP_PERCENTAGE
      value: "50"
    - name: STRIMZI_DYNAMIC_HEAP_MAX
      value: "5368709120"
    image: quay.io/strimzi/kafka:0.37.0-rc1-kafka-3.5.1
    imagePullPolicy: IfNotPresent
    livenessProbe:
      exec:
        command:
        - /opt/kafka/kafka_liveness.sh
      failureThreshold: 3
      initialDelaySeconds: 15
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 5
    name: kafka
    ports:
    - containerPort: 9090
      name: tcp-ctrlplane
      protocol: TCP
    - containerPort: 9091
      name: tcp-replication
      protocol: TCP
    - containerPort: 9092
      name: tcp-clients
      protocol: TCP
    - containerPort: 9093
      name: tcp-clientstls
      protocol: TCP
    readinessProbe:
      exec:
        command:
        - /opt/kafka/kafka_readiness.sh
      failureThreshold: 3
      initialDelaySeconds: 15
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 5
    resources:
      limits:
        cpu: "1"
        memory: 1Gi
      requests:
        cpu: "1"
        memory: 1Gi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/lib/kafka/data
      name: data
    - mountPath: /tmp
      name: strimzi-tmp
    - mountPath: /opt/kafka/cluster-ca-certs
      name: cluster-ca
    - mountPath: /opt/kafka/broker-certs
      name: broker-certs
    - mountPath: /opt/kafka/client-ca-certs
      name: client-ca-cert
    - mountPath: /opt/kafka/custom-config/
      name: kafka-metrics-and-logging
    - mountPath: /var/opt/kafka
      name: ready-files
    - mountPath: /opt/kafka/init
      name: rack-volume
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-5jlkm
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  hostname: my-cluster-kafka-0
  initContainers:
  - args:
    - /opt/strimzi/bin/kafka_init_run.sh
    env:
    - name: NODE_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: spec.nodeName
    - name: RACK_TOPOLOGY_KEY
      value: topology.kubernetes.io/zone
    image: quay.io/strimzi/operator:0.37.0-rc1
    imagePullPolicy: IfNotPresent
    name: kafka-init
    resources:
      limits:
        cpu: "1"
        memory: 1Gi
      requests:
        cpu: "1"
        memory: 1Gi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /opt/kafka/init
      name: rack-volume
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-5jlkm
      readOnly: true
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: my-cluster-kafka
  serviceAccountName: my-cluster-kafka
  subdomain: my-cluster-kafka-brokers
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - emptyDir: {}
    name: data
  - emptyDir:
      medium: Memory
      sizeLimit: 1Mi
    name: rack-volume
  - emptyDir:
      medium: Memory
      sizeLimit: 5Mi
    name: strimzi-tmp
  - name: cluster-ca
    secret:
      defaultMode: 292
      secretName: my-cluster-cluster-ca-cert
  - name: broker-certs
    secret:
      defaultMode: 292
      secretName: my-cluster-kafka-brokers
  - name: client-ca-cert
    secret:
      defaultMode: 292
      secretName: my-cluster-clients-ca-cert
  - configMap:
      defaultMode: 420
      name: my-cluster-kafka-0
    name: kafka-metrics-and-logging
  - emptyDir:
      medium: Memory
      sizeLimit: 1Ki
    name: ready-files
  - name: kube-api-access-5jlkm
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace

[scholzj]

Just to add to what @ppatierno said -> it does not seem to fail on my M2-based Macbook. So I wonder why it happens in the first place in your environment. I usually use Minikube.

[TalZiv]

Hi,

This is what I set in the Kafka object under spec.kafka.resources

    resources:
      limits:
        cpu: "1"
        memory: 512Mi
      requests:
        cpu: "1"
        memory: 128Mi

Here is my pod YAML with two different resources:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    strimzi.io/broker-configuration-hash: c1a19963333933a48b11ecc4a494d2a091fdc4ca8431e618cf5411a6ce797ecb090efefca9251138
    strimzi.io/clients-ca-cert-generation: "0"
    strimzi.io/cluster-ca-cert-generation: "0"
    strimzi.io/custom-listener-cert-thumbprints: '{tls=L5S1D2eCDSQpnCY0/5oU8COrDxdhC626IFn6Yfu4egw=}'
    strimzi.io/generation: "1"
    strimzi.io/inter-broker-protocol-version: "2.6"
    strimzi.io/kafka-version: 2.6.0
    strimzi.io/log-message-format-version: "2.6"
    strimzi.io/logging-appenders-hash: e893ac9f32f7958042d1c82477e3484b0f8ebad5
    strimzi.io/storage: '{"volumes":[{"type":"persistent-claim","size":"0.5Gi","id":0}],"type":"jbod"}'
  creationTimestamp: "2023-10-30T06:22:12Z"
  generateName: kafka-kafka-
  labels:
    app.kubernetes.io/instance: kafka
    app.kubernetes.io/managed-by: strimzi-cluster-operator
    app.kubernetes.io/name: kafka
    app.kubernetes.io/part-of: strimzi-kafka
    controller-revision-hash: kafka-kafka-79d9bff79b
    statefulset.kubernetes.io/pod-name: kafka-kafka-0
    strimzi.io/cluster: kafka
    strimzi.io/kind: Kafka
    strimzi.io/name: kafka-kafka
  name: kafka-kafka-0
  namespace: infra
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: StatefulSet
    name: kafka-kafka
    uid: f6969853-d567-4cdc-b108-b51ec30ad524
  resourceVersion: "2884"
  uid: 720a3d93-faf6-48c2-a348-071757f2963b
spec:
  affinity: {}
  containers:
  - command:
    - /opt/kafka/kafka_run.sh
    env:
    - name: KAFKA_METRICS_ENABLED
      value: "false"
    - name: STRIMZI_KAFKA_GC_LOG_ENABLED
      value: "false"
    - name: DYNAMIC_HEAP_FRACTION
      value: "0.5"
    - name: DYNAMIC_HEAP_MAX
      value: "5368709120"
    image: quay.io/strimzi/kafka:0.23.0-kafka-2.6.0
    imagePullPolicy: IfNotPresent
    livenessProbe:
      exec:
        command:
        - /opt/kafka/kafka_liveness.sh
      failureThreshold: 3
      initialDelaySeconds: 15
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 5
    name: kafka
    ports:
    - containerPort: 9090
      name: tcp-ctrlplane
      protocol: TCP
    - containerPort: 9091
      name: tcp-replication
      protocol: TCP
    - containerPort: 9093
      name: tcp-tls
      protocol: TCP
    readinessProbe:
      exec:
        command:
        - test
        - -f
        - /var/opt/kafka/kafka-ready
      failureThreshold: 3
      initialDelaySeconds: 15
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 5
    resources:
      limits:
        cpu: "1"
        memory: 512Mi
      requests:
        cpu: "1"
        memory: 128Mi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/lib/kafka/data-0
      name: data-0
    - mountPath: /tmp
      name: strimzi-tmp
    - mountPath: /opt/kafka/cluster-ca-certs
      name: cluster-ca
    - mountPath: /opt/kafka/broker-certs
      name: broker-certs
    - mountPath: /opt/kafka/client-ca-certs
      name: client-ca-cert
    - mountPath: /opt/kafka/custom-config/
      name: kafka-metrics-and-logging
    - mountPath: /var/opt/kafka
      name: ready-files
    - mountPath: /opt/kafka/init
      name: rack-volume
    - mountPath: /opt/kafka/certificates/custom-tls-9093-certs
      name: custom-tls-9093-certs
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-wv7np
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  hostname: kafka-kafka-0
  initContainers:
  - args:
    - /opt/strimzi/bin/kafka_init_run.sh
    env:
    - name: NODE_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: spec.nodeName
    - name: EXTERNAL_ADDRESS
      value: "TRUE"
    image: quay.io/strimzi/operator:0.23.0
    imagePullPolicy: IfNotPresent
    name: kafka-init
    resources:
      limits:
        cpu: "1"
        memory: 256Mi
      requests:
        cpu: 100m
        memory: 128Mi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /opt/kafka/init
      name: rack-volume
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-wv7np
      readOnly: true
  nodeName: cwp-dev-env-control-plane
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext:
    fsGroup: 0
  serviceAccount: kafka-kafka
  serviceAccountName: kafka-kafka
  subdomain: kafka-kafka-brokers
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: data-0
    persistentVolumeClaim:
      claimName: data-0-kafka-kafka-0
  - emptyDir: {}
    name: rack-volume
  - emptyDir:
      medium: Memory
    name: strimzi-tmp
  - name: cluster-ca
    secret:
      defaultMode: 292
      secretName: kafka-cluster-ca-cert
  - name: broker-certs
    secret:
      defaultMode: 292
      secretName: kafka-kafka-brokers
  - name: client-ca-cert
    secret:
      defaultMode: 292
      secretName: kafka-clients-ca-cert
  - configMap:
      defaultMode: 420
      name: kafka-kafka-config
    name: kafka-metrics-and-logging
  - emptyDir:
      medium: Memory
    name: ready-files
  - name: custom-tls-9093-certs
    secret:
      defaultMode: 292
      items:
      - key: tls.key
        path: tls.key
      - key: tls.crt
        path: tls.crt
      secretName: strimzi-kafka-tls
  - name: kube-api-access-wv7np
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2023-10-30T06:22:18Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2023-10-30T06:22:33Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2023-10-30T06:22:33Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2023-10-30T06:22:12Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://dee5c83c41cc4d8d363c95e598a68d90dbbb86b3d4c8817e380fbd55d6a31e7f
    image: quay.io/strimzi/kafka:0.23.0-kafka-2.6.0
    imageID: quay.io/strimzi/kafka@sha256:dbc53bb374d2cb9513a5bb8f80fea751c91b249cabd36d8b3c04af5e5c961738
    lastState: {}
    name: kafka
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2023-10-30T06:22:18Z"
  hostIP: 172.18.0.2
  initContainerStatuses:
  - containerID: containerd://2151e3b9bf10b3bf26c2b47303fbd05ae8b5fee637891c16c60c11164192a985
    image: quay.io/strimzi/operator:0.23.0
    imageID: quay.io/strimzi/operator@sha256:59d939f342f96d0e8a54c96fd187832c4b19cd4137bf55ace53d486728ae6c5f
    lastState: {}
    name: kafka-init
    ready: true
    restartCount: 0
    state:
      terminated:
        containerID: containerd://2151e3b9bf10b3bf26c2b47303fbd05ae8b5fee637891c16c60c11164192a985
        exitCode: 0
        finishedAt: "2023-10-30T06:22:18Z"
        reason: Completed
        startedAt: "2023-10-30T06:22:13Z"
  phase: Running
  podIP: 10.244.0.30
  podIPs:
  - ip: 10.244.0.30
  qosClass: Burstable
  startTime: "2023-10-30T06:22:12Z"

[ppatierno]

You are using a pretty old operator AFAICS, it's 0.23.0 you didn't mention.
That time the resources for init container were hard coded, see here https://github.com/strimzi/strimzi-kafka-operator/blob/release-0.23.x/cluster-operator/src/main/java/io/strimzi/operator/cluster/model/KafkaCluster.java

The only way you have to customize is upgrading the operator.

[TalZiv]

Yes, Sorry, once I saw your Yaml I figured out it's probably an old operator issue. Thank you. The one thing I do not get is why the limits that as you confirmed now are hardcoded are not enough just on a M2 CPU but not my I7.

[ppatierno]

why the limits that as you confirmed now are hardcoded are not enough just on a M2 CPU but not my I7.

Tbh I don't know. I don't have an Apple. And as per @scholzj reply, he doesn't have any problems with his same laptop as well.

[TalZiv]

I cannot upgrade the operator as I need the older version of Kafka, How complicated would it be to rebuild a version with that change?
https://github.com/strimzi/strimzi-kafka-operator/tree/release-0.23.x/docker-images/operator ?

[ppatierno]

I need the older version of Kafka

Why? Kafka has a backward compatible protocol, so if you have older clients they will work with newer Kafka. Also 2.7.0 and 2.8.0 are really pretty old Kafkas and you should upgrade it as well for different reasons (first of them CVEs fix).
Said that I don't see an easy way to support new Kafka in so old operator even because it's not just about building an image it's also the API between the operator and the Kafka itself which change.
The Strimzi community doesn't support this anyway, so I think you would be on your own if you try even for any issues.

[TalZiv]

I upgraded Strimzi operator from 23 to 26 and now it works. Thanks