diff --git a/gpu-operator/index.rst b/gpu-operator/index.rst index 678d1b7f9..fd2cedacc 100644 --- a/gpu-operator/index.rst +++ b/gpu-operator/index.rst @@ -31,7 +31,7 @@ gpu-driver-upgrades.rst install-gpu-operator-vgpu.rst install-gpu-operator-nvaie.rst - + Security Considerations .. toctree:: diff --git a/gpu-operator/manifests/input/k-run-cuda.txt b/gpu-operator/manifests/input/k-run-cuda.txt new file mode 100644 index 000000000..1da093772 --- /dev/null +++ b/gpu-operator/manifests/input/k-run-cuda.txt @@ -0,0 +1,84 @@ +# envall-limit-1 +$ kubectl run --rm -it cuda \ + --image=nvcr.io/nvidia/cuda:12.5.1-base-ubuntu20.04 \ + --restart=Never --override-type=strategic \ + --overrides='{"spec":{"containers":[{"name":"cuda","resources":{"limits":{"nvidia.com/gpu":1}}}]}}' \ + --env="NVIDIA_VISIBLE_DEVICES=all" -- nvidia-smi -L +# end + +# default-envall-limit-1 +GPU 0: Tesla T4 (UUID: GPU-6cfb8bf0-b556-d216-90a4-0827b23b1348) +GPU 1: Tesla T4 (UUID: GPU-90ac9649-3c3b-4d9f-e3ab-95c8043e7d48) +# end + +# limit-envall-limit-1 +GPU 0: Tesla T4 (UUID: GPU-6cfb8bf0-b556-d216-90a4-0827b23b1348) +# end + +# noenv-limit-1 +$ kubectl run --rm -it cuda \ + --image=nvcr.io/nvidia/cuda:12.5.1-base-ubuntu20.04 \ + --restart=Never --override-type=strategic \ + --overrides='{"spec":{"containers":[{"name":"cuda","resources":{"limits":{"nvidia.com/gpu":1}}}]}}' \ + -- nvidia-smi -L +# end + +# default-noenv-limit-1 +GPU 0: Tesla T4 (UUID: GPU-6cfb8bf0-b556-d216-90a4-0827b23b1348) +# end + +# limit-noenv-limit-1 +GPU 0: Tesla T4 (UUID: GPU-6cfb8bf0-b556-d216-90a4-0827b23b1348) +# end + +# noenv-limit-0 +$ kubectl run --rm -it cuda \ + --image=nvcr.io/nvidia/cuda:12.5.1-base-ubuntu20.04 \ + --restart=Never --override-type=strategic \ + --overrides='{"spec":{"containers":[{"name":"cuda","resources":{"limits":{"nvidia.com/gpu":0}}}]}}' \ + -- nvidia-smi -L +# end + +# default-noenv-limit-0 +GPU 0: Tesla T4 (UUID: GPU-dfb3f7ba-f6c2-3dd1-e621-46208932a9a7) +GPU 1: Tesla T4 (UUID: GPU-1d37ef30-0861-de64-a06d-73257e247a0d) +# end + +# limit-noenv-limit-0 +pod default/cuda terminated (StartError) +failed to create containerd task: failed to create shim task: OCI runtime create failed: runc +create failed: unable to start container process: exec: "nvidia-smi": executable file not found in $PATH: unknown +# end + +# privileged +$ kubectl run --rm -it cuda \ + --image=nvcr.io/nvidia/cuda:12.5.1-base-ubuntu20.04 \ + --restart=Never --override-type=strategic \ + --overrides='{ + "spec":{ + "containers": [{ + "name":"cuda", + "env": [{ + "name": "NVIDIA_VISIBLE_DEVICES", + "value": "0,1" + }], + "securityContext": { + "capabilities": { + "add": ["SYS_ADMIN"] + } + } + }] + } + }' \ + -- nvidia-smi -L +# end + +# default-privileged +GPU 0: Tesla T4 (UUID: GPU-6cfb8bf0-b556-d216-90a4-0827b23b1348) +GPU 1: Tesla T4 (UUID: GPU-90ac9649-3c3b-4d9f-e3ab-95c8043e7d48) +# end + +# limit-privileged +GPU 0: Tesla T4 (UUID: GPU-6cfb8bf0-b556-d216-90a4-0827b23b1348) +GPU 1: Tesla T4 (UUID: GPU-90ac9649-3c3b-4d9f-e3ab-95c8043e7d48) +# end diff --git a/gpu-operator/overview.rst b/gpu-operator/overview.rst index 226f2d971..69e84eb93 100644 --- a/gpu-operator/overview.rst +++ b/gpu-operator/overview.rst @@ -62,25 +62,6 @@ Platform Support The :ref:`operator-platform-support` describes the supported platform configurations. -Pod Security Context of the Operator and Operands -================================================= - -Several of the NVIDIA GPU Operator operands, such as the driver containers and container toolkit, -require the following elevated privileges: - -- ``privileged: true`` -- ``hostPID: true`` -- ``hostIPC: true`` - -The elevated privileges are required for the following reasons: - -- Access to the host file system and hardware devices, such as NVIDIA GPUs. -- Restart system services such as containerd. -- Permit users to list all GPU clients using the ``nvidia-smi`` utility. - -Only the Kubernetes cluster administrator needs to access or manage the Operator namespace. -As a best practice, establish proper security policies and prevent any other users from accessing the Operator namespace. - Licenses and Contributing ========================= diff --git a/gpu-operator/security-considerations.rst b/gpu-operator/security-considerations.rst new file mode 100644 index 000000000..cc5a19fc5 --- /dev/null +++ b/gpu-operator/security-considerations.rst @@ -0,0 +1,120 @@ +.. headings (h1/h2/h3/h4/h5) are # * = - + +############################################### +Security Considerations for NVIDIA GPU Operator +############################################### + +.. contents:: + :depth: 2 + :local: + :backlinks: none + + +************************************** +Preventing Unprivileged Access to GPUs +************************************** + +A default installation of NVIDIA Container Toolkit and NVIDIA Kubernetes Device Plugin permits unprivileged containers to bypass the Kubernetes Device Plugin API and gain access to GPUs on a node by setting the ``NVIDIA_VISIBLE_DEVICES`` environment variable. +This behavior means that an unprivileged container can gain access to GPUs without specifying resource limits or gain access to more GPUs than were specified through resource limits. + +However, there are valid circumstances to grant containers access to all GPUs on the system without going through the NVIDIA Kubernetes Device Plugin and interfering with Kubernetes accounting for allocatable resources. +Examples of these pods are the device plugin itself and and container that performs monitoring of GPU devices, such as NVIDIA DCGM Exporter. +However, these pods differ from typical workloads because they run software that is critical to the overall Kubernetes infrastructure and should be considered privileged. +Keep in mind that privileged containers always have access to all GPUs on a node. + +You can configure the toolkit and device plugin to prevent unprivileged containers from gaining access to GPUs through the ``NVIDIA_VISIBLE_DEVICES`` environment variable. +The strategy for the toolkit and device plugin are as follows: + +- The toolkit ignores ``NVIDIA_VISIBLE_DEVICES``. +- The device plugin produces a list of GPU devices as volume mounts, according to resource requests, and the toolkit accepts this list exclusively. + + An assumption is made that cluster administrators do not allow host volume mounts for unprivileged containers. + +The combination of configuring the toolkit and device plugin, and not allowing host volume mounts for unprivileged containers prevents those containers from setting up the GPU device list themselves. + +To configure these operands during Operator installation or upgrade, create a ``values.yaml`` file with contents like the following example: + +.. code-block:: yaml + + toolkit: + env: + - name: ACCEPT_NVIDIA_VISIBLE_DEVICES_ENVVAR_WHEN_UNPRIVILEGED + value: "false" + - name: ACCEPT_NVIDIA_VISIBLE_DEVICES_AS_VOLUME_MOUNTS + value: "true" + devicePlugin: + env: + - name: DEVICE_LIST_STRATEGY + value: volume-mounts + +The following comparison shows how setting ``NVIDIA_VISIBLE_DEVICES`` to ``all`` grants access to more GPUs than the resource request +in the default configuration and that access is limited after reconfiguration. +Any image can set ``NVIDIA_VISIBLE_DEVICES`` to ``all``--the base CUDA image in the example does--with the same result as explicitly setting the variable on the command line. + ++------------------------------------------------------------------+-----------------------------------------------------------------+ +| .. literalinclude:: ./manifests/input/k-run-cuda.txt | +| :language: console | +| :start-after: envall-limit-1 | +| :end-before: end | ++------------------------------------------------------------------+-----------------------------------------------------------------+ +| Default Configuration | Limit Unprivileged | ++------------------------------------------------------------------+-----------------------------------------------------------------+ +| .. literalinclude:: ./manifests/input/k-run-cuda.txt | .. literalinclude:: ./manifests/input/k-run-cuda.txt | +| :language: output | :language: output | +| :start-after: default-envall-limit-1 | :start-after: limit-envall-limit-1 | +| :end-before: end | :end-before: end | ++------------------------------------------------------------------+-----------------------------------------------------------------+ + +The following comparison shows how specifying a resource request of ``nvidia.com/gpu: 0`` grants access to GPUs +in the default configuration and that access is limited after reconfiguration. + ++------------------------------------------------------------------+-----------------------------------------------------------------+ +| .. literalinclude:: ./manifests/input/k-run-cuda.txt | +| :language: console | +| :start-after: noenv-limit-0 | +| :end-before: end | ++------------------------------------------------------------------+-----------------------------------------------------------------+ +| Default Configuration | Limit Unprivileged | ++------------------------------------------------------------------+-----------------------------------------------------------------+ +| .. literalinclude:: ./manifests/input/k-run-cuda.txt | .. literalinclude:: ./manifests/input/k-run-cuda.txt | +| :language: output | :language: output | +| :start-after: default-noenv-limit-0 | :start-after: limit-noenv-limit-0 | +| :end-before: end | :end-before: end | ++------------------------------------------------------------------+-----------------------------------------------------------------+ + +The following comparison shows that privileged containers have access to all GPUs regardless of environment variables, resource requests, +or device plugin and toolkit configuration. + ++------------------------------------------------------------------+-----------------------------------------------------------------+ +| .. literalinclude:: ./manifests/input/k-run-cuda.txt | +| :language: text | +| :start-after: privileged | +| :end-before: end | ++------------------------------------------------------------------+-----------------------------------------------------------------+ +| Default Configuration | Limit Unprivileged | ++------------------------------------------------------------------+-----------------------------------------------------------------+ +| .. literalinclude:: ./manifests/input/k-run-cuda.txt | .. literalinclude:: ./manifests/input/k-run-cuda.txt | +| :language: output | :language: output | +| :start-after: default-privileged | :start-after: limit-privileged | +| :end-before: end | :end-before: end | ++------------------------------------------------------------------+-----------------------------------------------------------------+ + +************************************************* +Pod Security Context of the Operator and Operands +************************************************* + +Several of the NVIDIA GPU Operator operands, such as the driver containers and container toolkit, +require the following elevated privileges: + +- ``privileged: true`` +- ``hostPID: true`` +- ``hostIPC: true`` + +The elevated privileges are required for the following reasons: + +- Access to the host file system and hardware devices, such as NVIDIA GPUs. +- Restart system services such as containerd. +- Permit users to list all GPU clients using the ``nvidia-smi`` utility. + +Only the Kubernetes cluster administrator needs to access or manage the Operator namespace. +As a best practice, establish proper security policies and prevent any other users from accessing the Operator namespace.