docker: nvidia container runtime (--gpus all) broken on NixOS 25.05 #419597
Description
Nixpkgs version
- Stable (25.05)
Describe the bug
Continuation from my comment #371527 (comment). Currently, I am stuck on 24.11 because I am unable to use docker --gpus all with the following error on 25.05:
docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]]
This is with hardware.nvidia-container-toolkit.enable = false and virtualisation.docker.enableNvidia = true. CDI device method (--device nvidia.com/gpu=all) doesn't work when nvidia-container-toolkit is disabled (of course), but still works fine if both are set to true.
Despite CDI working, this is still a huge issue as many Docker Compose files (i.e., Kasm Workspaces, Immich), still rely on the Nvidia Container Runtime method of declaring GPU mounting.
While modifying those is possible (in Immich's case), and really troublesome (in Kasm's case), the main showstopper is VS Code Devcontainers, which also still uses the Nvidia Container Runtime instead of CDI method and is really hard to modify to use CDI. This will probably remain a problem unless Docker transparently changes --devices all to refer to CDI as a form of deprecated backwards support, or somehow a miracle happens and developers start updating to use CDI everywhere.
The following might be relevant (I was trying to see if I could solve it myself):
- virtualisation/docker: fix nvidia container wrapper #407290
- libnvidia-container: include binaries from driver package #372320
- Nvidia Container Runtime version was 1.15.0-rc.3 and part of nvidia-docker in 24.11. It is now part of nvidia-container-toolkit/tools and version 1.17.6.
Steps to reproduce
See above.
Expected behaviour
Should be able to launch a container using --gpus all.
Screenshots
No response
Relevant log output
Additional context
No response
System metadata
24.11:
- system:
"x86_64-linux" - host os:
Linux 6.12.34, NixOS, 24.11 (Vicuna), 24.11.718912.1f426f65ac4e - multi-user?:
yes - sandbox:
yes - version:
nix-env (Nix) 2.24.14 - channels(root):
"nixos-24.11" - nixpkgs:
/nix/var/nix/profiles/per-user/root/channels/nixos
25.05:
- system:
"x86_64-linux" - host os:
Linux 6.12.34, NixOS, 25.05 (Warbler), 25.05.804570.c7ab75210cb8 - multi-user?:
yes - sandbox:
yes - version:
nix-env (Nix) 2.28.3 - channels(root):
"nixos-25.05" - nixpkgs:
/nix/store/aj312mhdmvjxjhky3cxxmxa13jx4mzh6-nixos-25.05/nixos
Notify maintainers
Note for maintainers: Please tag this issue in your pull request description. (i.e. Resolves #ISSUE.)
I assert that this issue is relevant for Nixpkgs
- I assert that this is a bug and not a support request.
- I assert that this is not a duplicate of an existing issue.
- I assert that I have read the NixOS Code of Conduct and agree to abide by it.
Is this issue important to you?
Add a 👍 reaction to issues you find important.