Why the memory consumption of my brokers are constantly going up? Feels like everything is getting stored in memory

[npandeya]

This is my cluster specification. I am writing test data constantly into a kafkatopic on this cluster with 120 partitions. When I check my messages, the memory consumption of each of the broker actually goes above the limit or slowly goes up and constantly. This is the grafana dashboard screenshot as well. Any guidance will be highly appreciated

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: test-cluster
  namespace: kafka
spec:
  kafka:
    template:
      pod:
        tolerations:
          - key: "dedicated"
            operator: "Equal"
            value: "kafka"
            effect: "NoSchedule"
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: node-purpose
                  operator: In
                  values:
                  - kafka
    resources:
      requests:
        memory: 24Gi # changed from 8 on 20 sep
        cpu: "1"
      limits:
        memory: 48Gi
        cpu: "2"
    version: 3.4.0
    replicas: 30
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: true
        authentication:
          type: tls

      - name: tls
        port: 9093
        type: internal
        tls: true
      - name: external
        type: cluster-ip
        tls: true
        port: 9096
        authentication:
          type: tls
    authorization: 
      type: simple
        
    config:
      offsets.topic.replication.factor: 3 
      num.partitions: 30000
      transaction.state.log.replication.factor: 3 
      transaction.state.log.min.isr: 2
      default.replication.factor: 2 
      min.insync.replicas: 1
      log.retention.hours: 24 
      # log.segment.bytes: 2147483648 # 2gb
      log.retention.check.interval.ms: 300000
      num.network.threads: 24 # initially it was 3 changed on 29 jun #12
      num.io.threads: 64 # Initially it was 8 changed on 29 Jun   #16
      num.recovery.threads.per.data.dir: 8 # Initially it was 1 # 4
      inter.broker.protocol.version: "3.4"
      # More config
      auto.create.topics.enable: false 
      auto.leader.rebalance.enable: true 
      compression.type: snappy 
      controlled.shutdown.enable: true 
      controlled.shutdown.max.retries: 3
      controlled.shutdown.retry.backoff.ms: 5000
      delete.topic.enable: true 
      producer.purgatory.purge.interval.requests: 10000
      queued.max.requests: 500
      ssl.enabled.protocols: TLSv1.2

    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        class: kafka
        size: 500Gi
        deleteClaim: false
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: kafka-metrics-config.yml
  zookeeper:
    replicas: 3
    # jvmOptions:
    #   javaSystemProperties:
    #     - name: zookeeper.ssl.hostnameVerification
    #       value: "false"
    #     - name: zookeeper.ssl.quorum.hostnameVerification
    #       value: "false"
    template:
      pod:
        tolerations:
          - key: "dedicated"
            operator: "Equal"
            value: "kafka"
            effect: "NoSchedule"
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: node-purpose
                  operator: In
                  values:
                  - kafka
    storage:
      type: persistent-claim
      size: 125Gi # st1 volume must be at least 125 
      deleteClaim: false
      class: kafka
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: zookeeper-metrics-config.yml
  entityOperator:
    template:
      pod:
        tolerations:
          - key: "dedicated"
            operator: "Equal"
            value: "kafka"
            effect: "NoSchedule"
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: node-purpose
                  operator: In
                  values:
                  - kafka
    topicOperator:
      resources:
        requests:
          memory: 512Mi
          cpu: "1"
        limits:
          memory: 512Mi
          cpu: "1"
    userOperator:
      resources:
        requests:
          memory: 512Mi
          cpu: "1"
        limits:
          memory: 512Mi
          cpu: "1"
  kafkaExporter:
    template:
      pod:
        tolerations:
          - key: "dedicated"
            operator: "Equal"
            value: "kafka"
            effect: "NoSchedule"
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: node-purpose
                  operator: In
                  values:
                  - kafka
    topicRegex: ".*"
    groupRegex: ".*"

[scholzj]

The configuration is badly formatted and not readable. It might depend on many things, but in general yes. Part of the memory will be taken by Java and the rest will be likely filled with the disk page cache keeping the data written to the disk to be able to quickly access them. So with enough time, it will likely eventually consume all the memory you give it. Whether it really needs the memory, that is a different question and will at the end depend a lot on the use-cases, performance you expect etc.

[scholzj]

Right, I remember talking about it at KubeCon. But what is it you see as the issue?

You give the Kafka brokers 24Gi as a request and 48Gi as a limit. The request should be always available as Kubernetes should reserve it. The limit might not be available, but if the node has enough free memory, the broker can get it. So if you use the broker and give it time, it will in the end always consume the memory it gets which in your case will be at least 24Gi and if the node has enough free memory it will get up to the 48Gi.

You do not have any custom JVM memory configuration, so it will use the default which will if I remember correctly configure the Java -Xmx option to 5Gi. In addition to that, Java will take some non-heap memory, so the overall use will be a bit bigger. If you have enabled metrics, the Java memory should be part of them including the split into the different parts.

The resto of the memory will be used for the disk page cache. The operating system will cache everything that Kafka writes to the disk. It is not that it keeps it in memory only, it is sent to the disk as well. But it is kept in this cache. And when you have consumers consuming these messages, when they are still in the cache, the Kafka broker will take them basically from there without reading from disk. This is important as it speeds up the performance. But it is up to you to decide how much speed you need compared to the memory cost of course.

Now, you say that you sometimes see the memory consumption be higher than the limit. I'm not an expert on the containers and Linux memory etc. But as far as I understand, there are various metrics, that might provide slightly different values. A process inside the container might see something slightly different from the container engine itself. There could be some memory shared with other processes in some way and that might affect the calculation. Sometimes there are some conversion issues along the way as well (gibibyte version gigabyte etc.).

I'm not sure what of these you see as an issue.

• Is it that it consumes a lot of memory? Just because it consumes the memory when you give it to it, it does not mean it really needs it. I'm sure your cluster will work fine if you give it only half as much memory as now. You will have to play with it and see what the performance impact is etc. But just the fact that it consumes 48Gi of memory doesn't mean that it needs more or that it won't work with less.
• Is it that you sometimes see the memory consumption above the limit? I'm afraid I do not understand the memory stuff well enough. But a start would be to understand what exact metric you use and where it comes from. That might tell you something. In general, if the difference is small (e.g. 50 instead of 48) I would expect this to be only some different way how the metric is calculated compared to what the container engine uses. If it would say that it consumes 80Gi instead of 48Gi, that would seem weird.

[npandeya]

Thank you so much. I will try to decrease the memory limit and request and may be increase broker. But what makes me more nervous is pods are showing as running on k8s side and brokers are running but when I look at the grafana dashboard the number of online brokers goes down as the memory consumption increases. Is that a normal behavior?

[scholzj]

But what makes me more nervous is pods are showing as running on k8s side and brokers are running but when I look at the grafana dashboard the number of online brokers goes down as the memory consumption increases. Is that a normal behavior?

That sounds weird. But just like that I could not say any reason. I never saw or heard about it. If you would have a lot of topics / partitions, it could be an issue with the performance of the metrics exporter.

[npandeya]

Thank you so much. What would you recommend should I lower the memory by few folds and May be use gp3 volume compared to st1 I have right now?

[scholzj]

I would definitely recommend gp2 or gp3 over st1 (I do not expect this to change anything with regards to memeory, but I think it is a better choice). I would probably try to lower the memory as long as the performance etc. works for you and your usecases.