Unable to see gpu processes pid inside the container

[aditya-sanas]

When I run any gpu process inside my docker container, I see that GPU is getting utilised but the pids are not visible in the output of nvidia-smi

Steps to reproduce the issue

1. docker run -it --rm --gpus all nvidia/cuda:12.8.0-base-ubuntu22.04 bash
2. Run any process utlising CUDA
3. watch nvidia-smi

Describe the results you received

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.01             Driver Version: 535.183.01   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 4090        Off | 00000000:41:00.0 Off |                  Off |
| 32%   41C    P2              63W / 450W |   7325MiB / 24564MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce RTX 4090        Off | 00000000:82:00.0 Off |                  Off |
| 32%   41C    P2              67W / 450W |   7555MiB / 24564MiB |    100%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
+---------------------------------------------------------------------------------------+

Describe the results you expected

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.01             Driver Version: 535.183.01   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 4090        Off | 00000000:41:00.0 Off |                  Off |
| 32%   41C    P2              63W / 450W |   7325MiB / 24564MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce RTX 4090        Off | 00000000:82:00.0 Off |                  Off |
| 32%   41C    P2              67W / 450W |   7555MiB / 24564MiB |    100%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A    149767      C   python                                     7318MiB |
|    1   N/A  N/A    150809      C   ...ubuntu/translators/.venv/bin/python     7548MiB |
+---------------------------------------------------------------------------------------+

Environment:

• OS: Ubuntu 22.04.3 LTS
• NVIDIA Container Toolkit: NVIDIA Container Runtime Hook version 1.17.5
• Host can detect and use the GPU correctly

[jinxsfe]

I also meet same or similar problem, docker can not utilize the GPU, even I tried with (base) js22bp@js22bp-MS-7D98:$ sudo systemctl restart docker
[sudo] password for js22bp:
(base) js22bp@js22bp-MS-7D98:$ docker info | grep -i runtime
ERROR: Cannot connect to the Docker daemon at unix://var/run/docker.sock. Is the docker daemon running?
errors pretty printing info
(base) js22bp@js22bp-MS-7D98:$ unset DOCKER_HOST
unset DOCKER_CONTEXT
(base) js22bp@js22bp-MS-7D98:$ env | grep DOCKER
(base) js22bp@js22bp-MS-7D98:$ docker info | grep -i runtime
WARNING: daemon is not using the default seccomp profile
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
(base) js22bp@js22bp-MS-7D98:$ sudo apt-get install -y nvidia-container-runtime
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following package was automatically installed and is no longer required:
libnvidia-compute-530
Use 'sudo apt autoremove' to remove it.
The following NEW packages will be installed:
nvidia-container-runtime
0 upgraded, 1 newly installed, 0 to remove and 168 not upgraded.
Need to get 4,988 B of archives.
After this operation, 21.5 kB of additional disk space will be used.
Get:1 https://nvidia.github.io/libnvidia-container/stable/ubuntu18.04/amd64 nvidia-container-runtime 3.13.0-1 [4,988 B]
Fetched 4,988 B in 0s (24.5 kB/s)
Selecting previously unselected package nvidia-container-runtime.
(Reading database ... 371905 files and directories currently installed.)
Preparing to unpack .../nvidia-container-runtime_3.13.0-1_all.deb ...
Unpacking nvidia-container-runtime (3.13.0-1) ...
Setting up nvidia-container-runtime (3.13.0-1) ...
(base) js22bp@js22bp-MS-7D98:$ sudo nano /etc/docker/daemon.json
(base) js22bp@js22bp-MS-7D98:$ docker info | grep -i runtime
WARNING: daemon is not using the default seccomp profile
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
(base) js22bp@js22bp-MS-7D98:$ sudo systemctl daemon-reexec
sudo systemctl daemon-reload
sudo systemctl restart docker
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xeu docker.service" for details.
(base) js22bp@js22bp-MS-7D98:$ docker info | grep -i runtime
WARNING: daemon is not using the default seccomp profile
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
(base) js22bp@js22bp-MS-7D98:$ sudo journalctl -xeu docker.service | tail -n 50
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ An ExecStart= process belonging to unit docker.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 1.
Apr 02 00:42:29 js22bp-MS-7D98 systemd[1]: docker.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit docker.service has entered the 'failed' state with result 'exit-code'.
Apr 02 00:42:29 js22bp-MS-7D98 systemd[1]: Failed to start Docker Application Container Engine.
░░ Subject: A start job for unit docker.service has failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit docker.service has finished with a failure.
░░
░░ The job identifier is 2945 and the job result is failed.
Apr 02 00:42:31 js22bp-MS-7D98 systemd[1]: docker.service: Scheduled restart job, restart counter is at 3.
░░ Subject: Automatic restarting of a unit has been scheduled
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ Automatic restarting of the unit docker.service has been scheduled, as the result for
░░ the configured Restart= setting for the unit.
Apr 02 00:42:31 js22bp-MS-7D98 systemd[1]: Stopped Docker Application Container Engine.
░░ Subject: A stop job for unit docker.service has finished
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A stop job for unit docker.service has finished.
░░
░░ The job identifier is 3024 and the job result is done.
Apr 02 00:42:31 js22bp-MS-7D98 systemd[1]: docker.service: Start request repeated too quickly.
Apr 02 00:42:31 js22bp-MS-7D98 systemd[1]: docker.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit docker.service has entered the 'failed' state with result 'exit-code'.
Apr 02 00:42:31 js22bp-MS-7D98 systemd[1]: Failed to start Docker Application Container Engine.
░░ Subject: A start job for unit docker.service has failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit docker.service has finished with a failure.
░░
░░ The job identifier is 3024 and the job result is failed.
(base) js22bp@js22bp-MS-7D98:$ sudo dockerd
unable to configure the Docker daemon with file /etc/docker/daemon.json: invalid character '#' looking for beginning of value
(base) js22bp@js22bp-MS-7D98:$ sudo nano /etc/docker/daemon.json
(base) js22bp@js22bp-MS-7D98:$ sudo systemctl restart docker
(base) js22bp@js22bp-MS-7D98:$ docker info | grep -i runtime
WARNING: daemon is not using the default seccomp profile
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
(base) js22bp@js22bp-MS-7D98:$ sudo dockerd
INFO[2025-04-02T00:49:23.909005971-04:00] Starting up
failed to start daemon, ensure docker is not running or delete /var/run/docker.pid: process with PID 30870 is still running
(base) js22bp@js22bp-MS-7D98:$ sudo systemctl stop docker
Warning: Stopping docker.service, but it can still be activated by:
docker.socket
(base) js22bp@js22bp-MS-7D98:$ sudo rm -f /var/run/docker.pid
(base) js22bp@js22bp-MS-7D98:$ sudo systemctl start docker
(base) js22bp@js22bp-MS-7D98:$ docker info | grep -i runtime
WARNING: daemon is not using the default seccomp profile
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
(base) js22bp@js22bp-MS-7D98:~$ which nvidia-container-runtime
nvidia-container-runtime --version
/usr/bin/nvidia-container-runtime
NVIDIA Container Runtime version 1.13.5
commit: 6b8589dcb4dead72ab64f14a5912886e6165c079
spec: 1.1.0-rc.2

runc version 1.2.5
commit: v1.2.5-0-g59923ef
spec: 1.2.0
go: go1.23.6
libseccomp: 2.5.3
(base) js22bp@js22bp-MS-7D98:$ cat /etc/docker/daemon.json
{
"default-runtime": "nvidia",
"runtimes": {
"nvidia": {
"path": "nvidia-container-runtime",
"runtimeArgs": []
}
}
}
(base) js22bp@js22bp-MS-7D98:$ sudo systemctl stop docker.socket
sudo systemctl stop docker
sudo rm -f /var/run/docker.pid
sudo systemctl daemon-reexec
sudo systemctl daemon-reload
sudo systemctl start docker
(base) js22bp@js22bp-MS-7D98:$ docker info | grep -i runtime
WARNING: daemon is not using the default seccomp profile
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
(base) js22bp@js22bp-MS-7D98:$ sudo mkdir -p /etc/systemd/system/docker.service.d
sudo nano /etc/systemd/system/docker.service.d/override.conf
(base) js22bp@js22bp-MS-7D98:$ sudo systemctl daemon-reexec
sudo systemctl daemon-reload
sudo systemctl restart docker
(base) js22bp@js22bp-MS-7D98:$ docker info | grep -i runtime
WARNING: daemon is not using the default seccomp profile
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
(base) js22bp@js22bp-MS-7D98:$ sudo nano /etc/systemd/system/docker.service.d/override.conf
(base) js22bp@js22bp-MS-7D98:$ sudo nano /etc/systemd/system/docker.service.d/override.conf
(base) js22bp@js22bp-MS-7D98:$ [200sudo systemctl daemon-reexec
[200sudo: command not found
(base) js22bp@js22bp-MS-7D98:$ sudo systemctl daemon-reload
sudo systemctl stop docker
sudo rm -f /var/run/docker.pid
sudo systemctl start docker
^[[201~(base) js22bp@js22bp-MS-7D98:$
(base) js22bp@js22bp-MS-7D98:$ docker info | grep -i runtime
WARNING: daemon is not using the default seccomp profile
Runtimes: runc io.containerd.runc.v2
Default Runtime: runc
(base) js22bp@js22bp-MS-7D98:$ sudo nano /etc/systemd/system/docker.service.d/override.conf
(base) js22bp@js22bp-MS-7D98:$ sudo nano /etc/systemd/system/docker.service.d/override.conf
(base) js22bp@js22bp-MS-7D98:$ sudo systemctl daemon-reexec
sudo systemctl daemon-reload
sudo systemctl stop docker.socket
sudo systemctl stop docker
sudo rm -f /var/run/docker.pid
sudo systemctl start docker
(base) js22bp@js22bp-MS-7D98:$ docker info | grep -i runtime
WARNING: daemon is not using the default seccomp profile
Runtimes: runc io.containerd.runc.v2
Default Runtime: runc
(base) js22bp@js22bp-MS-7D98:~$ and set

and sudo nano /etc/systemd/system/docker.service.d/override.conf

[Service]
ExecStart=
ExecStart=/usr/bin/dockerd --host=unix:///var/run/docker.sock –config-file=/<etc/docker/daemon.json