Error: OCI runtime error: error executing hook /usr/bin/nvidia-container-runtime-hook (exit code: 1)

[GKSONG]

o. My run environment
-- RHEL 8.4, gcc 8.4.*, AMD CPU
-- A40 GPU
-- nvidia driver : 510.47.03
-- cgroupv2
-- kvm, GPU passthrough enabled
-- rootless podman

o. When (rootless) podman start, the following message displayed and not started.
" Error: OCI runtime error: error executing hook /usr/bin/nvidia-container-runtime-hook (exit code: 1) "

how can I start debugging this problem ?

Unfortunately, I setup the logging option but not generated.
==> /etc/nvidia-container-runtime/config.toml
debug = "/tmp/nvidia-container-toolkit.log"
..
debug = "/tmp/nvidia-container-runtime.log"

o. My finding and workaround
When the message happend, I tried to "nvidia-container-cli list", but terminal hang.

The workaround method 
    ==> nvidia-smi -i 0 -pm ENABLED
    ==> deviceQuery 

o. Could you share a checkpoint or an experience ?

[giuseppe]

so does it work after your workaround?

Doesn't seem to be a podman issue, so moving to a discussion

[GKSONG]

Yes, It works. ( But It frequently happen in RHEL 8.4, occasionally happen in CentOS8.4 after server rebooting)

Right, It's related to "nvidia-container-*" which is called by "podman".

I don't guess where is the start point for debugging. I fully followed the setup-guideline.

[sazzy4o]

Had the same issue(on Ubuntu 22.04), went away after a reboot

sudo reboot

Edit:
Can also appear if no-cgroups = true is not set in /etc/nvidia-container-runtime/config.toml

[Dixeran]

It's weird that the document from nvidia said

If the user running the containers is a privileged user (e.g. root) this change should not be made and will cause containers using the NVIDIA Container Toolkit to fail.

But for me using root, set no-cgroups = true solved the problem.

[sotiris-bos]

I have the same issue trying to run ghcr.io/blakeblackshear/frigate:stable

The problem is, I am not using any NVIDIA software, in fact I have no NVIDIA hardware installed in the system.

[rhatdan]

Are you saying you are seeing: OCI runtime error: error executing hook /usr/bin/nvidia-container-runtime-hook? Without nvidia code installed?

[VergeDX]

Remove old configuration /usr/share/containers/oci/hooks.d/oci-nvidia-hook.json (from https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/1.9.0/install-guide.html#step-2-1-check-the-installation)
and reconfigure CDI (https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/cdi-support.html) works for me.