Open Search Fail to run even with host has unlimited ulimit memlock. Podman doesn't even follow the config in containers.conf or docker-compose.yml

[benyaminl]

Hello, I face a problem with podman

$ podman version
Client:       Podman Engine
Version:      5.3.2
API Version:  5.3.2
Go Version:   go1.23.4
Built:        Wed Jan 22 07:00:00 2025
OS/Arch:      linux/amd64

I'm on Fedora 41. I want to run open search with

version: '3'
services:
  opensearch-node1: # This is also the hostname of the container within the Docker network (i.e. https://opensearch-node1/)
    image: opensearchproject/opensearch:latest
    container_name: opensearch-node1
    environment:
      - cluster.name=opensearch-cluster # Name the cluster
      - node.name=opensearch-node1 # Name the node that will run in this container
      - discovery.seed_hosts=opensearch-node1,opensearch-node2 # Nodes to look for when discovering the cluster
      - cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2 # Nodes eligibile to serve as cluster manager
      - bootstrap.memory_lock=true # Disable JVM heap memory swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m -XX:+HeapDumpOnOutOfMemoryError" # Set min and max JVM heap sizes to at least 50% of system RAM
      - OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_INITIAL_ADMIN_PASSWORD} # Sets the demo admin user password when using demo configuration (for OpenSearch 2.12 and later)
    ulimits:
      memlock:
        soft: -1 # Set memlock to unlimited (no soft or hard limit)
        hard: -1
      nofile:
        soft: 65536 # Maximum number of open files for the opensearch user - set to at least 65536
        hard: 65536
    volumes:
      - opensearch-data1:/usr/share/opensearch/data # Creates volume called opensearch-data1 and mounts it to the container
      - ./logs/logs1.log:/usr/share/opensearch/logs
      - ./limits.conf:/etc/security/limits.conf
    ports:
      - 9200:9200 # REST API
      - 9600:9600 # Performance Analyzer
    networks:
      - opensearch-net # All of the containers will join the same Docker bridge network
  opensearch-node2:
    image: opensearchproject/opensearch:latest # This should be the same image used for opensearch-node1 to avoid issues
    container_name: opensearch-node2
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node2
      - discovery.seed_hosts=opensearch-node1,opensearch-node2
      - cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2
      - bootstrap.memory_lock=true
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m -XX:+HeapDumpOnOutOfMemoryError"
      - OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_INITIAL_ADMIN_PASSWORD}
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - opensearch-data2:/usr/share/opensearch/data
      - ./logs/logs2.log:/usr/share/opensearch/logs
      - ./limits.conf:/etc/security/limits.conf
    command: ['sleep', '10000000']
    networks:
      - opensearch-net
  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:latest # Make sure the version of opensearch-dashboards matches the version of opensearch installed on other nodes
    container_name: opensearch-dashboards
    ports:
      - 5601:5601 # Map host port 5601 to container port 5601
    expose:
      - "5601" # Expose port 5601 for web access to OpenSearch Dashboards
    environment:
      OPENSEARCH_HOSTS: '["https://opensearch-node1:9200","https://opensearch-node2:9200"]' # Define the OpenSearch nodes that OpenSearch Dashboards will query
    networks:
      - opensearch-net

volumes:
  opensearch-data1:
  opensearch-data2:

networks:
  opensearch-net:
  

The problem I face is

opensearch-node1       | [2025-02-17T08:11:58,090][WARN ][o.o.b.JNANatives         ] [opensearch-node1] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
opensearch-node1       | [2025-02-17T08:11:58,092][WARN ][o.o.b.JNANatives         ] [opensearch-node1] This can result in part of the JVM being swapped out.
opensearch-node1       | [2025-02-17T08:11:58,092][WARN ][o.o.b.JNANatives         ] [opensearch-node1] Increase RLIMIT_MEMLOCK, soft limit: 8388608, hard limit: 8388608

When investigating with sleep command on one of the node, it output

[opensearch@67937fca6eb9 ~]$ ulimit -a
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) unlimited
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 126728
max locked memory           (kbytes, -l) 8192
max memory size             (kbytes, -m) unlimited
open files                          (-n) 524288
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) 8192
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 126728
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited

The problem is the host already has unlimited ulimit memory lock, and I even set in container config /home/ben/.config/containers/containers.conf

[network]
network_backend="netavark"
[containers]
default_ulimits = ["memlock=-1:-1"]

But it still lock in 8192, where the Host OS is unlimited. (fedora 41).

Any help is appreciated, as I don't want to install docker on my machine.

[baude]

does this work if you exclude the docker yaml and opensearch image ? basically, im thinking lets narrow the problem down without the noise of opensearch ... so for example, if you run the fedora image and check ulimits, do you see what you expect? if so, then create a sample docker.yaml using fedora checks ... and then we can narrow down where the problem might be