The underlying GCP instance doesn't install gpu drivers

[xuzhao9]

Is your feature request related to a problem? Please describe.
We are working on GHA self-hosted runners to provide A100 instances from GCP.
After deploy actions-runner-controller, we find the A100 GCP instances created doesn't have gpu driver installed as described in https://cloud.google.com/container-optimized-os/docs/how-to/run-gpus.

We also need the nvidia driver to be > 500 to perform our tasks, so the stable release won't work.

Also, the runner container created needs to have access to the Nvidia utilities, therefore they need to be run with options

docker run \
  --volume /var/lib/nvidia/lib64:/usr/local/nvidia/lib64 \
  --volume /var/lib/nvidia/bin:/usr/local/nvidia/bin \
  --device /dev/nvidia0:/dev/nvidia0 \
  --device /dev/nvidia-uvm:/dev/nvidia-uvm \
  --device /dev/nvidiactl:/dev/nvidiactl \
  --cap-add=SYS_ADMIN
  <DOCKER-IMAGE>

Describe the solution you'd like
I would like to the GCP compute instances created with Nvidia gpu driver installed, and the GHA runner container can use Nvidia GPU tools such as nvidia-smi.

[mumoshu]

Please forgive me to move this to Discussions as this isn't a feature request or bug report to ARC itself

[xuzhao9]

Update: I think I can install Nvidia drivers by applying the gpu yaml described on this page: https://cloud.google.com/container-optimized-os/docs/how-to/run-gpus

However, the remaining problem is how to setup docker container startup arguments so that we can access to the gpu resource.

[xuzhao9]

Related discussion: #961

[xuzhao9]

Update: I can now map the host vm's device files and volumes in the runner docker:

  spec:
     # other fields...
      volumes:
        - name: nvidia-lib
          hostPath:
            path: /home/kubernetes/bin/nvidia/lib64
            type: Directory
        - name: nvidia-bin
          hostPath:
            path: /home/kubernetes/bin/nvidia/bin
            type: Directory
        - name: nvidia-card
          hostPath:
            path: /dev/nvidia0
            type: CharDevice
        - name: nvidia-uvm
          hostPath:
            path: /dev/nvidia-uvm
            type: CharDevice
        - name: nvidia-ctl
          hostPath:
            path: /dev/nvidiactl
            type: CharDevice
      volumeMounts:
        - mountPath: /usr/local/nvidia/lib64
          name: nvidia-lib
        - mountPath: /usr/local/nvidia/bin
          name: nvidia-bin
        - mountPath: /dev/nvidia0
          name: nvidia-card
        - mountPath: /dev/nvidia-uvm
          name: nvidia-uvm
        - mountPath: /dev/nvidiactl
          name: nvidia-ctl

However, stilling trying to figure out how to add SYS_ADMIN capacity to the runner container.

[xuzhao9]

The above problem has been resolved with using dind. Closed.