Unable to see gpu processes pid inside the container

[aditya-sanas]

Issue Description

When I run any gpu process inside my rootful podman container, I see that GPU is getting utilised but the pids are not visible.

Steps to reproduce the issue

Steps to reproduce the issue

1. sudo podman run --name test --privileged --cap-add net_admin --systemd always --device nvidia.com/gpu=all --device /dev/net/tun:/dev/net/tun --replace -d nvidia/cuda:12.1.0-cudnn8-devel-ubuntu22.04
2. sudo podman exec -it test bash
3. nvidia-smi

Describe the results you received

 NVIDIA-SMI 535.183.01             Driver Version: 535.183.01   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 4090        Off | 00000000:41:00.0 Off |                  Off |
| 32%   42C    P2              63W / 450W |   7325MiB / 24564MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce RTX 4090        Off | 00000000:82:00.0 Off |                  Off |
| 32%   42C    P2              67W / 450W |   7555MiB / 24564MiB |    100%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
+---------------------------------------------------------------------------------------+

Describe the results you expected

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.01             Driver Version: 535.183.01   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 4090        Off | 00000000:41:00.0 Off |                  Off |
| 32%   42C    P2              63W / 450W |   7325MiB / 24564MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce RTX 4090        Off | 00000000:82:00.0 Off |                  Off |
| 32%   42C    P2              67W / 450W |   7555MiB / 24564MiB |    100%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A    149767      C   python                                     7318MiB |
|    1   N/A  N/A    150809      C   ...ubuntu/translators/.venv/bin/python     7548MiB |
+---------------------------------------------------------------------------------------+

podman info output

host:
  arch: amd64
  buildahVersion: 1.39.2
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon_100:2.1.13-1_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.13, commit: e21e7c85b7637e622f21c57675bf1154fc8b1866'
  cpuUtilization:
    idlePercent: 95.2
    systemPercent: 1.47
    userPercent: 3.33
  cpus: 48
  databaseBackend: sqlite
  distribution:
    codename: jammy
    distribution: ubuntu
    version: "22.04"
  eventLogger: journald
  freeLocks: 2047
  hostname: ts18
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.15.0-119-generic
  linkmode: dynamic
  logDriver: journald
  memFree: 33020882944
  memTotal: 269667504128
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: podman-aardvark-dns_100:1.14.0-1_amd64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.14.0
    package: podman-netavark_100:1.14.1-1_amd64
    path: /usr/libexec/podman/netavark
    version: netavark 1.14.1
  ociRuntime:
    name: runc
    package: containerd.io_1.7.25-1_amd64
    path: /usr/bin/runc
    version: |-
      runc version 1.2.4
      commit: v1.2.4-0-g6c52b3f
      spec: 1.2.0
      go: go1.22.10
      libseccomp: 2.5.3
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt_100:0.0+20250320.32f62125-1_amd64
    version: ""
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: true
    capabilities: CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_MKNOD,CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_100:1.3.2-1_amd64
    version: ""
  swapFree: 73728
  swapTotal: 8589930496
  uptime: 338h 38m 8.00s (Approximately 14.08 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 105089261568
  graphRootUsed: 87802736640
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 5.4.1
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.24.1
  Os: linux
  OsArch: linux/amd64
  Version: 5.4.1

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[baude]

can you provide a reproducer that doesnt involve running an nvidia tool ? it could help us narrow things down without knowing anything about their tooling.

[Luap99]

This does not sound like a bug with podman, I suggest you ask the nvidia container toolkit about it. I move it to a discussion

(In theory it could be caused by something podman is doing but I don't see what that would be, you are are part of the same pid namespace so the program should see all its processes that run there)

[aditya-sanas]

Okay, on nvidia issues I found they suggest using --pid=host.
But as I am running sudo podman run --name test --privileged --cap-add net_admin --systemd always --device nvidia.com/gpu=all --device /dev/net/tun:/dev/net/tun --pid=host --replace -d nvidia/cuda:12.1.0-cudnn8-devel-ubuntu22.04,
the command fails saying Running in chroot, ignoring request.

[aditya-sanas]

Is there any workaround for this? I am basically running /sbin/init as my entry command for my podman container and now since I want --pid=host to be able to view the gpu threads, I am unable to start the container with error Running in chroot, ignoring request.