Error running NVIDIA container rootless

[muety]

Issue Description

I installed Podman on my Ubuntu 20.04 machine, as well as nvidia-container-toolkit according to the official instructions.

I got two "types" of users on my system, namely "normal" local users (e.g. user1_local (UID 1001)) and users mapped from LDAP (using SSSD) (e.g. user1 (UID 246740)).

I'm trying to run the following command:

podman run --rm --security-opt=label=disable --hooks-dir=/usr/share/containers/oci/hooks.d/ -nvidia/cuda:11.6.2-base-ubuntu20.04 nvidia-smi

As one of the local users, everything works fine. As a "non-local" user, I'm running into this error, though:

Error: OCI runtime error: error executing hook `/usr/bin/nvidia-container-runtime-hook` (exit code: 1)

This is the according nvidia-container-toolkit.log:

I0818 08:10:19.862831 7982 nvc.c:376] initializing library context (version=1.13.5, build=66607bd046341f7aad7de80a9f022f122d1f2fce)
I0818 08:10:19.862896 7982 nvc.c:350] using root /
I0818 08:10:19.862912 7982 nvc.c:351] using ldcache /etc/ld.so.cache
I0818 08:10:19.862927 7982 nvc.c:352] using unprivileged user 65534:65534
I0818 08:10:19.862955 7982 nvc.c:393] attempting to load dxcore to see if we are running under Windows Subsystem for Linux (WSL)
I0818 08:10:19.863089 7982 nvc.c:395] dxcore initialization failed, continuing assuming a non-WSL environment
W0818 08:10:19.863123 7982 nvc.c:253] skipping kernel modules load due to user namespace
I0818 08:10:19.863293 7988 rpc.c:71] starting driver rpc service
I0818 08:10:19.863636 7982 rpc.c:135] driver rpc service terminated with signal 15
I0818 08:10:19.863707 7982 nvc.c:434] shutting down library context

Note the driver rpc service terminated with signal 15.

What I did in advance:

• Set no-cgroups = true in /etc/nvidia-container-runtime/config.toml
• Configure subuid / subgid mapping for the external users, e.g.:
  • user1:1000000:65536
• Pointed rootless_storage_path to a folder on the local file system (user home is on NFS by default)

As user1, I can run nvidia-smi on the host or any other, non-GPU container (e.g. hello-world) in Podman without any problems.

Any ideas how to resolve this?

Might as well be an issue related to nvidia-container-toolkit, rather than Podman, but I feel like the community is a lot more vibrant here.

Steps to reproduce the issue

Hard to reproduce, because you'd need LDAP users for this. See my configuration details above, though.

Describe the results you received

Can't run NVIDIA containers as externally mapped users.

Describe the results you expected

Be able to run them.

podman info output

host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.1.2, commit: '
  cpus: 48
  distribution:
    codename: focal
    distribution: ubuntu
    version: "20.04"
  eventLogger: journald
  hostname: ufo
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 44350
      size: 1
    - container_id: 1
      host_id: 231073
      size: 11024
    - container_id: 11025
      host_id: 1000000
      size: 11024
    uidmap:
    - container_id: 0
      host_id: 246740
      size: 1
    - container_id: 1
      host_id: 1000000
      size: 65536
  kernel: 5.15.0-79-generic
  linkmode: dynamic
  logDriver: journald
  memFree: 129950957568
  memTotal: 134901301248
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version UNKNOWN
      commit: ea1fe3938eefa14eb707f1d22adff4db670645d6
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/246740/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.1.8
      commit: unknown
      libslirp: 4.3.1-git
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.4.3
  swapFree: 2147479552
  swapTotal: 2147479552
  uptime: 22m 27.19s
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /home/ws/user1/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/local/docker-data/user1
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 1
  runRoot: /run/user/246740/containers
  volumePath: /var/local/docker-data/user1/volumes
version:
  APIVersion: 3.4.2
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.15.2
  OsArch: linux/amd64
  Version: 3.4.2

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[Luap99]

Have you tried using CDI as described here: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#container-device-interface-cdi-support

In general I recommend you ask nvidia about this, we have no idea about what the hook does.