Merge pull request #42126 from ousleyp/cnv-16295

CNV-16295: Prevent NVIDIA GPU operands from deploying on nodes

Author: ousleyp

modules/virt-preventing-nvidia-gpu-operands-from-deploying-on-nodes.adoc

@@ -0,0 +1,80 @@
+// Module included in the following assembly:
+//
+// * virt/virtual_machines/advanced_vm_management/virt-configuring-pci-passthrough.adoc
+//
+
+:_content-type: PROCEDURE
+[id="virt-preventing-nvidia-operands-from-deploying-on-nodes_{context}"]
+= Preventing NVIDIA GPU operands from deploying on nodes
+
+If you use the link:https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/openshift/contents.html[NVIDIA GPU Operator] in your cluster, you can apply the `nvidia.com/gpu.deploy.operands=false` label to nodes that you do not want to configure for GPU or vGPU operands. This label prevents the creation of the pods that configure GPU or vGPU operands and terminates the pods if they already exist.
+
+ifdef::openshift-enterprise[]
+:FeatureName: Using the NVIDIA GPU Operator with {VirtProductName}
+include::snippets/technology-preview.adoc[]
+endif::[]
+
+.Prerequisites
+
+* The OpenShift CLI (`oc`) is installed.
+
+.Procedure
+
+* Label the node by running the following command:
++
+[source,terminal]
+----
+$ oc label node <node_name> nvidia.com/gpu.deploy.operands=false <1>
+----
+<1> Replace `<node_name>` with the name of a node where you do not want to install the NVIDIA GPU operands.
+
+.Verification
+
+. Verify that the label was added to the node by running the following command:
++
+[source,terminal]
+----
+$ oc describe node <node_name>
+----
+
+. Optional: If GPU operands were previously deployed on the node, verify their removal. 
+
+.. Check the status of the pods in the `nvidia-gpu-operator` namespace by running the following command:
++
+[source,terminal]
+----
+$ oc get pods -n nvidia-gpu-operator
+----
++
+.Example output
+
+[source,terminal]
+----
+NAME                             READY   STATUS        RESTARTS   AGE
+gpu-operator-59469b8c5c-hw9wj    1/1     Running       0          8d
+nvidia-sandbox-validator-7hx98   1/1     Running       0          8d
+nvidia-sandbox-validator-hdb7p   1/1     Running       0          8d
+nvidia-sandbox-validator-kxwj7   1/1     Terminating   0          9d
+nvidia-vfio-manager-7w9fs        1/1     Running       0          8d
+nvidia-vfio-manager-866pz        1/1     Running       0          8d
+nvidia-vfio-manager-zqtck        1/1     Terminating   0          9d
+----
+
+.. Monitor the pod status until the pods with `Terminating` status are removed:
++
+[source,terminal]
+----
+$ oc get pods -n nvidia-gpu-operator
+----
++
+.Example output
+
+[source,terminal]
+----
+NAME                             READY   STATUS    RESTARTS   AGE
+gpu-operator-59469b8c5c-hw9wj    1/1     Running   0          8d
+nvidia-sandbox-validator-7hx98   1/1     Running   0          8d
+nvidia-sandbox-validator-hdb7p   1/1     Running   0          8d
+nvidia-vfio-manager-7w9fs        1/1     Running   0          8d
+nvidia-vfio-manager-866pz        1/1     Running   0          8d
+----

virt/virtual_machines/advanced_vm_management/virt-configuring-pci-passthrough.adoc

@@ -13,14 +13,28 @@ toc::[]
 //When this feature is available in the web console, please
 //add the new content to this assembly.
 
-The Peripheral Component Interconnect (PCI) passthrough feature enables you to access and manage hardware devices from a virtual machine. When PCI passthrough is configured, the PCI devices function as if they were physically attached to the guest operating system.
+The Peripheral Component Interconnect (PCI) passthrough feature enables you to access and manage hardware devices from a virtual machine (VM). When PCI passthrough is configured, the PCI devices function as if they were physically attached to the guest operating system.
 
 Cluster administrators can expose and manage host devices that are permitted to be used in the cluster by using the `oc` command-line interface (CLI).
 
-include::modules/virt-about-pci-passthrough.adoc[leveloffset=+1]
+[id="virt-preparing-nodes-for-gpu-passthrough"]
+== Preparing nodes for GPU passthrough
+
+You can prevent GPU operands from deploying on worker nodes that you designated for GPU passthrough.
+
+include::modules/virt-preventing-nvidia-gpu-operands-from-deploying-on-nodes.adoc[leveloffset=+2]
+
+[id="virt-preparing-host-devices-for-pci-passthrough"]
+== Preparing host devices for PCI passthrough
+
+include::modules/virt-about-pci-passthrough.adoc[leveloffset=+2]
+
 include::modules/virt-adding-kernel-arguments-enable-iommu.adoc[leveloffset=+2]
+
 include::modules/virt-binding-devices-vfio-driver.adoc[leveloffset=+2]
+
 include::modules/virt-exposing-pci-device-in-cluster-cli.adoc[leveloffset=+2]
+
 include::modules/virt-removing-pci-device-from-cluster-cli.adoc[leveloffset=+2]
 
 [id="virt-configuring-vms-for-pci-passthrough"]