GPU Support - SPIKE

[ravi-shankar-sap]

Compare the current approach used in this repository with the approaches followed by nvidia gpu operator and other available options to investigate the next steps for GpuDriver support.

[ravi-shankar-sap]

Currently, this repository has two helm charts and their details are as follows:

1. Gpu Driver Operator Helm Chart

2. Gpu Driver Helm Chart

[ravi-shankar-sap]

Whereas, the nvidia gpu operator uses the following approach for installing the GPU driver in the k8s environment:

[ravi-shankar-sap]

The gardenlinux-nvidia-installer is recently updated to use the nvidia-gpu-operator for GardenLinux Gpu support. gardenlinux/gardenlinux-nvidia-installer#47. This is similar to the nvidia approach described above.

The roadmap document explains the details: gardenlinux/gardenlinux-nvidia-installer#42

[ravi-shankar-sap]

Based on the gardenlinux-nvidia-installer repository future roadmap, following are the current limitations:

• Compiling and installing nvidia kernel modules is "somewhat fragile as well as adding a significant time to the node startup phase."
• In case of pre-compiled drivers, "images contain proprietary NVIDIA code (the driver is not open source), it is legally difficult to put such images into a publicly-accessible registry for use by all."
• The current approach is "fine for doing a proof-of-concept to build a driver for a cluster with a given version, but day 2 operations require the user to create a build pipeline and deployment system."

In addition,

• Currently, the gardenlinux-nvidia-installer provides values.yaml samples for installing the Gpu Driver utilizing the NVIDIA GPU Operator.
• The future roadmap includes automation of the driver installations on shoot nodes
  • using Node Feature Discovery Operator,
  • NVIDIA GPU Operator, and
  • GPU configuration through shoot.yaml specifications.

[ravi-shankar-sap]

As of now, Garden Linux team started publishing pre-compiled container images to the Garden Linux Container repository: https://github.com/gardenlinux/gardenlinux-nvidia-installer/pkgs/container/gardenlinux-nvidia-installer%2Fdriver

[ravi-shankar-sap]

Based on the current changes on the gardenlinux-nvidia-installer repository to incorporate pre-compiled driver containers and using the NVIDIA GPU Operator for deploying to the SKR runtimes, the SKR users can be able to deploy it using helm install of the NVIDIA GPU driver's helm chart. Alternatively, to make it more simpler, we could offer the following functionalities as part of the Gpu Driver module:

1. Deploy the NVIDIA GPU Operator
   The behavior of installing/upgrading the helm chart can be incorporated into the module. It would make it simple for the SKR users to enable/disable GPU support by adding or removing this module on their SKR. Below is a possible high-level approach for this module controller:

2. GPU Driver Build Pipelines (Optional)
   The gardenlinux-nvidia-installer creates and publishes the pre-compiled driver images for some specific GPU Driver versions. In case the development teams need a specific version of the GPU driver for their application that is not supported, they have to create their own build pipeline and store it in their container registry. This could be supported in the Gpu Driver module with the below approach:

[ravi-shankar-sap]

Based on the discussion with the team, the following approach would be supported:

1. A new repository gpu-module would be created, and it would be used for the new module.
2. As a first-drop, the gpu-module would only support the pre-compiled drivers uploaded by the gardenlinux-nvidia-installer.
3. Requirements for supporting the Gpu Driver Build Pipelines OR directly compiling the other driver versions would be evaluated later.

[ravi-shankar-sap]

Implementing the module may not provide much value-add other than installing the NVIDIA Gpu Operator. So, it is planned to postpone the development of module.