[question] Perhaps there is some flaw when using NRI to inject into the GPU

[LavenderQAQ]

What happened:

When using NRI to inject into the GPU, there might be a problem of losing device cgroup permissions.
In our production, we use the NRI injection similar to koordinator for implementation. We set the NVIDIA_VISIBLE_DEVICES environment variable and then let the nvidia-container-runtime handle it.

However, a very strange thing occurred. In the device cgroup of the injected container, the device numbers of these GPU cards would suddenly disappear from the devices.list. After investigation, we found that it was triggered by systemd reload command. Once the node executed systemctl daemon-reload, the device cgroup would be reset, and then the GPU device permissions would be lost directly.

Further research revealed that the corresponding container's permission configuration was not saved in /run/systemd/transient/. The specific location is /run/systemd/transient/*/50-DeviceAllow.conf. This should be written by runc after reading the OCI spec. In the implementation of the device-plugin, kubelet writes the OCI spec so that runc can write to /run/systemd/transient. This logic is not implemented in the NRI mechanism.

This issue only occurs when the nvidia-container-runtime is using the legacy mode; the cdi mode can be used to avoid it.

What you expected to happen:

I'm not sure if koordlet has a similar issue. It's not very convenient for me to simulate an environment. Judging from the code, it seems that there is no logic to supplement the OCI spec. It's very likely that it can be reproduced.

Here, it might be necessary to consider calling adjustment.AddDevice in the NRI injection logic to replace the kubelet's device-plugin mechanism to complete the device information in the OCI spec. Otherwise, systemctl daemon-reload will cause all GPU permissions on the entire node to be lost.

Environment:

Could someone reproduce this scenario conveniently? You could check whether the GPU device permissions of the pod have been persisted in systemd by viewing /run/systemd/transient/. The prerequisite is to set the nvidia-container-runtime to the legacy mode, that is, to use the method of NVIDIA Container Runtime Hook.

[ZiMengSheng]

@LavenderQAQ Excellent point regarding stability considerations. Before discussing the solution, I have another question: How does the CDI (Container Device Interface) mechanism within the kubelet device-plugin framework resolve the issue of lost device permissions? Why doesn’t it require leveraging /run/systemd/transient/*/50-DeviceAllow.conf to control device access? Previously, I understood this systemd transient configuration path to be the sole interface provided by the operating system for governing device visibility and permissions.

[AutuSnow]

@ZiMengSheng Let me answer your question about CDI. Nvidia container runtime no longer uses preset hooks to dynamically modify cgroup configurations, but declares devices through CDI spec files. Containerd directly parses the CDI spec when creating the container and writes the device information into the devices and resources.devices fields of OCI spec. When processing OCI spec, Runc will write these device rules to the transient unit through the systemd API, so the above problem will not occur

[AutuSnow]

This place only appears to be filled with environment variables, without any AddContainerDevices，Of course, I'm not entirely sure if the issue I pointed out is correct