Kafka Broker Receiver pod keep crashing with Out of of memory exception

[sebastienblanc]

Describe the bug
Once the pod start receiving Kafka Records it crashes with an Out of Memory exception.

You can find the logs of the pod here : https://gist.github.com/sebastienblanc/fcc3c9260d3139ef785efdbac312c060

And the pod's container last status :

  lastState:
      terminated:
        containerID: docker://090d3830ec5283dd2f2e9dede43072c522d05906bdc41b522e8d5ac474e502c5
        exitCode: 137
        finishedAt: "2026-01-17T09:36:19Z"
        message: |
          7Z","@version":"1","message":"[Producer clientId=producer-1] Cluster ID: 703833ba-8097-aa2e-0087-8bb1a7aaaa05","logger_name":"org.apache.kafka.clients.Metadata","thread_name":"kafka-producer-network-thread | producer-1","level":"INFO","level_value":20000}
          {"@timestamp":"2026-01-17T09:36:03.306879758Z","@version":"1","message":"[Producer clientId=producer-8] Cluster ID: 703833ba-8097-aa2e-0087-8bb1a7aaaa05","logger_name":"org.apache.kafka.clients.Metadata","thread_name":"kafka-producer-network-thread | producer-8","level":"INFO","level_value":20000}
          {"@timestamp":"2026-01-17T09:36:03.306880217Z","@version":"1","message":"[Producer clientId=producer-4] Cluster ID: 703833ba-8097-aa2e-0087-8bb1a7aaaa05","logger_name":"org.apache.kafka.clients.Metadata","thread_name":"kafka-producer-network-thread | producer-4","level":"INFO","level_value":20000}
          {"@timestamp":"2026-01-17T09:36:03.306880425Z","@version":"1","message":"[Producer clientId=producer-2] Cluster ID: 703833ba-8097-aa2e-0087-8bb1a7aaaa05","logger_name":"org.apache.kafka.clients.Metadata","thread_name":"kafka-producer-network-thread | producer-2","level":"INFO","level_value":20000}
          {"@timestamp":"2026-01-17T09:36:03.306880425Z","@version":"1","message":"[Producer clientId=producer-3] Cluster ID: 703833ba-8097-aa2e-0087-8bb1a7aaaa05","logger_name":"org.apache.kafka.clients.Metadata","thread_name":"kafka-producer-network-thread | producer-3","level":"INFO","level_value":20000}
          {"@timestamp":"2026-01-17T09:36:03.307032967Z","@version":"1","message":"[Producer clientId=producer-7] Cluster ID: 703833ba-8097-aa2e-0087-8bb1a7aaaa05","logger_name":"org.apache.kafka.clients.Metadata","thread_name":"kafka-producer-network-thread | producer-7","level":"INFO","level_value":20000}
          {"@timestamp":"2026-01-17T09:36:03.3071428Z","@version":"1","message":"[Producer clientId=producer-9] Cluster ID: 703833ba-8097-aa2e-0087-8bb1a7aaaa05","logger_name":"org.apache.kafka.clients.Metadata","thread_name":"kafka-producer-network-thread | producer-9","level":"INFO","level_value":20000}
        reason: OOMKilled
        startedAt: "2026-01-17T09:35:59Z"

Knative release version
version=1.20.1

Additional context
Running on ARM64 on Minikube using the Knative quickstart kn command.

[intojhanurag]

Hi @sebastienblanc! I've identified the root cause of your OOM issue.

The receiver was deploying one verticle instance per CPU core (Runtime.getRuntime().availableProcessors()), which created 10 verticles on your ARM64 system. Each verticle instantiates its own Kafka producer with a 32MB buffer, plus HTTP/HTTPS servers and Kubernetes clients, totaling ~60MB per verticle. This meant the pod needed 700MB (10 × 60MB + 100MB overhead) but only had 450MB allocated, causing the OOMKilled with exit code 137. The crash happened ~2 minutes after startup when Kafka records started flowing and filled the producer buffers, exhausting available memory.

May be I can be wrong.
Can you review my appraoch @matzew , @Cali0707 ? It is the only issue that i can think rn.

[intojhanurag]

/assign

[sebastienblanc]

Thx @intojhanurag for looking into that ! Nice investigation !