Remove bash/shell dependencies from the final image to reduce security risks

[leiyiz]

FR description

Currently, the k8s-dra-driver-gpu container image relies on several bash scripts during runtime:

1. hack/kubelet-plugin-prestart.sh (Used as the entrypoint for the init-kubelet-plugin initContainer).
2. scripts/bind_to_driver.sh (Executed via os/exec in the Go kubelet plugin).
3. scripts/unbind_from_driver.sh (Executed via os/exec in the Go kubelet plugin).

Because of these runtime dependencies, the Dockerfile has to explicitly download and copy a static bash binary (ghcr.io/nvidia/k8s-dra-driver-gpu:v25.12.0-dev-839e966a) into the final image.

This reliance on a shell environment prevents the project from adopting pure, zero-shell distroless base images (like gcr.io/distroless/static or gcr.io/distroless/base). In strict, highly-regulated enterprise Kubernetes environments (where supply chain security and minimizing the attack surface are critical), deploying containers that contain a shell is increasingly flagged by security policies.

Describe the solution you'd like

I propose we refactor the logic currently contained in these bash scripts directly into the Go codebase. This would allow us to completely eliminate the bash dependency from the final container image.

Proposed Implementation Path:

1. bind_to_driver.sh & unbind_from_driver.sh:

   • These scripts primarily handle reading and writing to sysfs (/sys/bus/pci/...) and procfs (/proc/driver/nvidia/...).
   • We can replace the exec.Command calls in cmd/gpu-kubelet-plugin/vfio-device.go with native Go file I/O (os.WriteFile,
     os.ReadFile, etc.). This will also improve error handling, as we won't have to parse stdout/stderr from a subprocess.

2. kubelet-plugin-prestart.sh: (less important)

   • This script acts as an init container, looping to check for the presence and health of nvidia-smi and libnvidia-ml.so.1 on the host mount.
   • We can implement this logic as a new subcommand in the existing gpu-kubelet-plugin binary (e.g., gpu-kubelet-plugin prestart-init) or as a tiny, separate Go binary built alongside the others.

Additional thoughts

I'd also love to hear if there were some special considerations behind the reason for using shell scripts over golang code.

[jgehrcke]

I find this very aggreeable. A personal wish is that we have to pay attention to retaining or even improving error handling and debuggability characteristics. The whole point of kubelet-plugin-prestart.sh for example was/is to help guide users with actionable input when prerequisites are not met, and I'd love to retain all the goodness and not walk back from any attention to detail given so far :).

[leiyiz]

An alternative for the kubelet-plugin-prestart.sh would be to pull it out to a separate image just for initContainer purpose thus reducing the attack surface. While we can convert the other scripts required during runtime to golang. What do you think?

[jgehrcke]

Addendum: I agreed generally above, but having worked in and around enterprise security -- let's maybe try to identify the precise, hard blockers that we would run into by not taking care of this.

Let's distinguish two security categories:

1. actual security: at this point, I believe there is no actual security concern around how things are structured -- let's think about that a little more.
2. checkbox security: which compliance check / policy precisely is violated, in which environment?

When we have answers to these questions, it becomes easier to justify a re-write in terms of effort per impact.

Maybe we already have an answer for (2) because above you said:

deploying containers that contain a shell is increasingly flagged by security policies

and then it's worth sharing specifics here!

[shivamerla]

Thanks @leiyiz for filing this issue. We’ve been seeing an increasing number of CVE alerts triggered by shell dependencies in the container image, so reducing or eliminating shell scripts is definitely a step in the right direction. I agree with the proposal to convert these scripts into a gpu-kubelet-plugin subcommand like we have added check subcommand for the compute-domain-daemon and remove the bash scripts entirely, especially since we’re targeting a clean distroless base image.

@varunrsekar might be able to provide more context on why the bind/unbind operations were originally implemented as shell scripts. I recall we had a discussion around this during the VFIO feature addition, and there may have been specific constraints or assumptions at the time. It would be helpful to understand that background before refactoring them into Go.

[jgehrcke]

Oh, I forgot to mention what's probably the most relevant aspect in practice (more than security): debuggability.

At the core, a change like this has benefits along some dimensions, and costs along other dimensions. Some dimensions are:

(I took this from #421 which tracked introduction of the distroless base image).

Right now we have busybox in the image (we use the -dev suffix) for debuggability, to get basic commands in the image for free. And we have bash in the image so that we don't need to run shell scripts through the funky busybox shell, but via a proper shell.

If we decide to remove busybox and bash from the container image, there is a significant cost in terms of debuggability. I understand that there are debug containers that can be attached -- but my advice is that one should really see how things feel and work in practice. Just to be explicitly prepared for debugging. In the heat of the moment when debugging in a customer's environment: not an ideal moment to lose time on debug road blocks.

[leiyiz]

My 2 cents on debuggability with bash:

1. If the container is running on a prod cluster, to me it's not a great idea to ssh into the container and just run bash commands for debug purposes
2. for general purpose debuggability, could a 2 container solution be useful? 2 containers otherwise identical but one of them contains bash can be produced at the same time, this might give the flexibility of debuggability while providing a true bash-less container for deployment

[shivamerla]

@leiyiz please go ahead and submit a PR with your proposal. I agree with @jgehrcke there is a trade-off with debugging, but there are standard practices to achieve that both at the pod and node level. We can probably better document those in the troubleshooting guide.