BlockingIOError: [Errno 11] Resource temporarily unavailable

[aditya-sanas]

Issue Description

I am running privileged podman container inside my gpu machine. Now when I try to run model training, it first says

OpenBLAS blas_thread_init: pthread_create failed for thread 22 of 48: Resource temporarily unavailable
OpenBLAS blas_thread_init: ensure that your address space and process count limits are big enough (ulimit -a)
OpenBLAS blas_thread_init: or set a smaller OPENBLAS_NUM_THREADS to fit into what you have available
OpenBLAS blas_thread_init: RLIMIT_NPROC 1048576 current, 1048576 max```

after setting `OPENBLAS_NUM_THREADS=1` It gives the folllowing error.

```Traceback (most recent call last):
  File "/home/ubuntu/translators/mel2mel_at/pl_student_mel2vec.py", line 500, in <module>
    train(config)
  File "/home/ubuntu/translators/mel2mel_at/pl_student_mel2vec.py", line 261, in train
    trainer.fit(model=model, train_dataloaders=training_dataloader)
  File "/home/ubuntu/translators/.venv/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 545, in fit
    call._call_and_handle_interrupt(
  File "/home/ubuntu/translators/.venv/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 43, in _call_and_handle_interrupt
    return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
  File "/home/ubuntu/translators/.venv/lib/python3.10/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 98, in launch
    self._call_children_scripts()
  File "/home/ubuntu/translators/.venv/lib/python3.10/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 144, in _call_children_scripts
    new_process = subprocess.Popen(command, env=env_copy, cwd=cwd)
  File "/usr/lib/python3.10/subprocess.py", line 971, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.10/subprocess.py", line 1796, in _execute_child
    self.pid = _posixsubprocess.fork_exec(
BlockingIOError: [Errno 11] Resource temporarily unavailable

Steps to reproduce the issue

Steps to reproduce the issue
1.
2.
3.

Describe the results you received

Describe the results you received

Describe the results you expected

Describe the results you expected

podman info output

host:
  arch: amd64
  buildahVersion: 1.39.2
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon_100:2.1.13-1_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.13, commit: e21e7c85b7637e622f21c57675bf1154fc8b1866'
  cpuUtilization:
    idlePercent: 98.26
    systemPercent: 0.33
    userPercent: 1.41
  cpus: 48
  databaseBackend: sqlite
  distribution:
    codename: jammy
    distribution: ubuntu
    version: "22.04"
  eventLogger: journald
  freeLocks: 2047
  hostname: ts10
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.5.0-35-lowlatency
  linkmode: dynamic
  logDriver: journald
  memFree: 41510346752
  memTotal: 269712904192
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: podman-aardvark-dns_100:1.14.0-1_amd64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.14.0
    package: podman-netavark_100:1.14.0-1_amd64
    path: /usr/libexec/podman/netavark
    version: netavark 1.14.0
  ociRuntime:
    name: runc
    package: runc_1.1.12-0ubuntu2~22.04.1_amd64
    path: /usr/bin/runc
    version: |-
      runc version 1.1.12-0ubuntu2~22.04.1
      spec: 1.0.2-dev
      go: go1.21.1
      libseccomp: 2.5.3
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt_100:0.0+20250217.a1e48a02-1_amd64
    version: ""
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: true
    capabilities: CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_MKNOD,CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_100:1.3.2-1_amd64
    version: |-
      slirp4netns version 1.3.2
      commit: unknown
      libslirp: 4.9.0
      SLIRP_CONFIG_VERSION_MAX: 6
      libseccomp: 2.5.3
  swapFree: 8253861888
  swapTotal: 8589930496
  uptime: 406h 14m 13.00s (Approximately 16.92 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 980666998784
  graphRootUsed: 937707286528
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 5.4.1
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.24.1
  Os: linux
  OsArch: linux/amd64
  Version: 5.4.1

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[Honny1]

Hello, could you please provide more information and steps to reproduce this problem? I think this issue should be reported in the OpenBLAS repository.

[aditya-sanas]

I am running a podman container using

     sudo podman run \
    --name "$container_name" \
    --hostname "$instance_id" \
    --privileged \
    --systemd always \
    --device nvidia.com/gpu=all \
    --cap-add net_admin \
    --cap-add AUDIT_CONTROL \
    --device /dev/net/tun:/dev/net/tun \
    --replace \
    -d \
    image```

Inside the container I am trying to run a distributed training script but seems like it is gettting IOError. Is there some limits which I need to set before starting the container? I already checked OPENBLAS library, no help there.

[aditya-sanas]

Is there some extra flag that I need to pass for distributed training inside the container?

[Honny1]

I am not aware of any specific flags for distributed training. You can try to use --ulimit -1.

[aditya-sanas]

Let me try this and update you. Thanks!

[Honny1]

Also, there could be a problem with memory limit.

[Honny1]

The host machine may also have limited options for running processes, so this can also be a problem.

[aditya-sanas]

The host machine has no such limitations I checked, I will check on the memory constrain part you mentioned.

[Luap99]

Podman by default sets a pid limit in the cgroup of 2048, do you exceed that?
You can try adding --pids-limit -1 for unlimited, in any case this does not sounds like a podman bug so I move it to a discussion.

[aditya-sanas]

If I run with --pids-limit -1, won't it add risk to the machine?

[McG800]

For anyone else that runs into this issue, I added "--pids-limit -1" to our Podman container and it resolved the issues I was seeing.