Native memory allocation (mmap) failed to map 65536 bytes

[teimyBr]

Kafka Image: strimzi/kafka:0.43.0-kafka-3.8.0
Kubernetes Version: v1.29.4
NodeImage: ubuntu-22.04-03-2023

kafka:
  logging: json
  authorization: true
  cluster:
    version: 3.8.0
    replicas: 5
    offsetsTopicReplicationFactor: 3
    transactionStateLogReplicationFactor: 3
    transactionStateLogMinInServiceReplicas: 2
    autoCreateTopicsEnable: false
   resources:
    limits:
      cpu: 5200m
      memory: 14336Mi
    requests:
      cpu: 2800m
      memory: 14336Mi
  controller:
    replicas: 5
    resources:
      limits:
        cpu: 1000m
        memory: 1440Mi
      requests:
        cpu: 80m
        memory: 1440Mi

Error Message we get in Kafka Broker after 5-25 Minutes Running. before there no really any important logs.

OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f68f9cbf000, 65536, 1) failed; error='Not enough space' (errno=12)                                                                           │
│ #                                                                                                                                                                                                            
│ # There is insufficient memory for the Java Runtime Environment to continue.                                                                                                                                  │
│ # Native memory allocation (mmap) failed to map 65536 bytes. Error detail: committing reserved memory.                                                                                                        │
│ # An error report file with more information is saved as:                                                                                                                                                     │
│ # /tmp/hs_err_pid736.log                                                                                                                                                                                     
│ #                                                                                                                                                                                                             
│ # Compiler replay data is saved as:                                                                                                                                                                           │
│ # /tmp/replay_pid736.log

Strimzi Logs:

{"instant":{"epochSecond":1739436565,"nanoOfSecond":985690338},"thread":"vert.x-eventloop-thread-0","level":"INFO","loggerName":"io.strimzi.operator.cluster.ClusterOperator","message":"Triggering periodic  │
│ reconciliation for namespace *","endOfBatch":false,"loggerFqcn":"org.apache.logging.log4j.spi.AbstractLogger","contextMap":{},"threadId":19,"threadPriority":5}                                               │
│ {"instant":{"epochSecond":1739436574,"nanoOfSecond":642506657},"thread":"vert.x-eventloop-thread-1","level":"INFO","loggerName":"io.strimzi.operator.cluster.operator.assembly.AbstractOperator","marker":{"n │
│ ame":"Kafka(prod-backbone/kafka)"},"message":"Reconciliation #23(watch) Kafka(prod-backbone/kafka): Reconciliation is in progress","endOfBatch":false,"loggerFqcn":"io.strimzi.operator.common.Reconciliation │
│ Logger","contextMap":{},"threadId":1723,"threadPriority":5}                                                                                                                                                   │
│ {"instant":{"epochSecond":1739436634,"nanoOfSecond":642065645},"thread":"vert.x-eventloop-thread-1","level":"INFO","loggerName":"io.strimzi.operator.cluster.operator.assembly.AbstractOperator","marker":{"n │
│ ame":"Kafka(prod-backbone/kafka)"},"message":"Reconciliation #23(watch) Kafka(prod-backbone/kafka): Reconciliation is in progress","endOfBatch":false,"loggerFqcn":"io.strimzi.operator.common.Reconciliation │
│ Logger","contextMap":{},"threadId":1723,"threadPriority":5}

Any Idea if this is a Ubuntu or Kafka Problem ?

[scholzj]

That does not look like Strimzi configuration. You need to share the full Strimzi configuration and full logs. Otherwise people will have no chance to tell you what is going on.

[teimyBr]

Was no strimzi issue.

Issue was in one of the kafka brocker setup via strimzi.

Solution was to set:
vm.max_map_count=524288
vm.overcommit_ratio=100
vm.overcommit_memory=0

on the kubenetes node where kafka node is running