Skip to content

Commit 0cbc4bf

Browse files
ousleypousleyp
authored
Merge pull request #32247 from lmandavi/CNV-11006-pci-pass
CNV-11006 PCI passthrough
2 parents ae84bcd + ce64758 commit 0cbc4bf

7 files changed

+340
-0
lines changed

‎_topic_map.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2676,6 +2676,8 @@ Topics:
26762676
File: virt-dedicated-resources-vm
26772677
- Name: Scheduling virtual machines
26782678
File: virt-schedule-vms
2679+
- Name: Configuring PCI passthrough
2680+
File: virt-configuring-pci-passthrough
26792681
# Importing virtual machines
26802682
- Name: Importing virtual machines
26812683
Dir: importing_vms

‎modules/virt-about-pci-passthrough.adoc‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * virt/virtual_machines/advanced_vm_management/virt-configuring-pci-passthrough.adoc
4+
5+
[id="virt-about_pci-passthrough_{context}"]
6+
= About preparing nodes and the cluster for PCI passthrough
7+
8+
The Peripheral Component Interconnect (PCI) passthrough feature enables you to access and manage hardware devices from a virtual machine. When PCI passthrough is configured, the PCI devices function as if they were physically attached to the virtual machine.
9+
10+
To configure a hardware device as a host device for PCI passthrough, create a `MachineConfig` object file and add kernel arguments to enable Input-Output Memory Management Unit (IOMMU). Bind the PCI device to the Virtual Function I/O (VFIO) driver by creating a `MachineConfig` object file. Then, add the PCI device to the HyperConverged Operator.

‎modules/virt-adding-kernel-arguments-enable-iommu.adoc‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * virt/virtual_machines/advanced_vm_management/configuring-pci-passthrough.adoc
4+
5+
[id="virt-adding-kernel-arguments-enable-IOMMU_{context}"]
6+
= Adding kernel arguments to enable the IOMMU driver
7+
8+
To enable the IOMMU (Input-Output Memory Management Unit) driver in the kernel, create the `MachineConfig` object and add the kernel arguments.
9+
10+
.Prerequisites
11+
* Administrative privilege to a working {product-title} cluster.
12+
* Intel or AMD CPU hardware.
13+
* Intel Virtualization Technology for Directed I/O extensions or AMD IOMMU in the BIOS (Basic Input/Output System) is enabled.
14+
15+
.Procedure
16+
. Create a `MachineConfig` object that identifies the kernel argument. The following example shows a kernel argument for an Intel CPU.
17+
18+
+
19+
[source,yaml]
20+
----
21+
apiVersion: machineconfiguration.openshift.io/v1
22+
kind: MachineConfig
23+
metadata:
24+
labels:
25+
machineconfiguration.openshift.io/role: worker <1>
26+
name: 100-worker-iommu <2>
27+
spec:
28+
config:
29+
ignition:
30+
version: 3.2.0
31+
kernelArguments:
32+
- intel_iommu=on <3>
33+
...
34+
----
35+
<1> Applies the new kernel argument only to worker nodes.
36+
<2> The `name` indicates the ranking of this kernel argument (100) among the machine configs and its purpose. If you have an AMD CPU, specify the kernel argument as `amd_iommu=on`.
37+
<3> Identifies the kernel argument as `intel_iommu` for an Intel CPU.
38+
39+
. Create the new `MachineConfig` object:
40+
+
41+
[source,terminal]
42+
----
43+
$ oc create -f 100-worker-kernel-arg-iommu.yaml
44+
----
45+
46+
.Verification
47+
48+
* Verify that the new `MachineConfig` object was added.
49+
+
50+
[source,terminal]
51+
----
52+
$ oc get MachineConfig
53+
----

‎modules/virt-adding-pci-device-hyperconverged-operator.adoc‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * virt/virtual_machines/advanced_vm_management/virt-configuring-pci-passthrough.adoc
4+
5+
[id="virt-adding-pci-device-hyperconverged-operator_{context}"]
6+
= Adding PCI devices to the HyperConverged Operator
7+
To enable the availability of PCI devices in the cluster, add details about the PCI devices to the HyperConverged object. The PCI devices are exposed and permitted to be used in the cluster.
8+
9+
.Procedure
10+
* Run the following command to add the PCI device information to the YAML file.
11+
+
12+
[source,terminal]
13+
----
14+
$ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv -o yaml
15+
----
16+
+
17+
.Example
18+
[source,yaml]
19+
----
20+
apiVersion: hco.kubevirt.io/v1
21+
kind: HyperConverged
22+
metadata:
23+
name: kubevirt-hyperconverged
24+
namespace: openshift-cnv
25+
spec:
26+
permittedHostDevices: <1>
27+
pciHostDevices: <2>
28+
- pciVendorSelector: 10de:1eb8 <3>
29+
resourceName: "nvidia.com/TU104GL_Tesla_T4" <4>
30+
----
31+
<1> The host devices that are to be used in the cluster.
32+
<2> PCI devices available on the node.
33+
<3> The `vendor-ID` and the `device-ID` for the PCI device.
34+
<4> The name of the hardware device.
35+
36+
.Verification
37+
* Use the following command to verify that the PCI device was added to the node.
38+
+
39+
[source,terminal]
40+
----
41+
$ oc describe node cnv-lab.eng.raleigh.redhat.com
42+
----
43+
+
44+
.Example
45+
----
46+
Capacity:
47+
cpu: 64
48+
devices.kubevirt.io/kvm: 110
49+
devices.kubevirt.io/tun: 110
50+
devices.kubevirt.io/vhost-net: 110
51+
ephemeral-storage: 915128Mi
52+
hugepages-1Gi: 0
53+
hugepages-2Mi: 0
54+
memory: 131395264Ki
55+
nvidia.com/TU104GL_Tesla_T4: 1
56+
pods: 250
57+
Allocatable:
58+
cpu: 63500m
59+
devices.kubevirt.io/kvm: 110
60+
devices.kubevirt.io/tun: 110
61+
devices.kubevirt.io/vhost-net: 110
62+
ephemeral-storage: 863623130526
63+
hugepages-1Gi: 0
64+
hugepages-2Mi: 0
65+
memory: 130244288Ki
66+
nvidia.com/TU104GL_Tesla_T4: 1
67+
pods: 250
68+
----

‎modules/virt-assigning-pci-device-virtual-machine.adoc‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * virt/virtual_machines/advanced_vm_management/virt-configuring-pci-passthrough.adoc
4+
5+
[id="virt-assigning-pci-device-virtual-machine_{context}"]
6+
= Assigning a PCI device to a virtual machine
7+
8+
When a PCI device is available in a cluster, you can assign it to a virtual machine and enable PCI passthrough.
9+
10+
.Procedure
11+
* Assign the PCI device to a virtual machine as a host device.
12+
+
13+
.Example
14+
[source,yaml]
15+
----
16+
apiVersion: kubevirt.io/v1
17+
kind: VirtualMachine
18+
spec:
19+
domain:
20+
devices:
21+
hostDevices:
22+
- deviceName: nvidia.com/TU104GL_Tesla_T4 <1>
23+
name: hostdevices1
24+
----
25+
<1> The name of the PCI device that is permitted on the cluster as a host device. The virtual machine can access this host device.
26+
27+
.Verification
28+
* Use the following command to verify that the host device is available from the virtual machine.
29+
+
30+
[source,terminal]
31+
$ lspci -nnk | grep NVIDIA
32+
+
33+
.Example output
34+
[source,terminal]
35+
----
36+
$ 02:01.0 3D controller [0302]: NVIDIA Corporation GV100GL [Tesla V100 PCIe 32GB] [10de:1eb8] (rev a1)
37+
----

‎modules/virt-binding-devices-vfio-driver.adoc‎

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * virt/virtual_machines/advanced_vm_management/virt-configuring-pci-passthrough.adoc
4+
5+
[id="virt-binding-devices-vfio-driver_{context}"]
6+
= Binding PCI devices to the VFIO driver
7+
To bind PCI devices to the VFIO (Virtual Function I/O) driver, obtain the values for `vendor-ID` and `device-ID` from each device and create a list with the values. Add this list to the `MachineConfig` object. The `MachineConfig` Operator generates the `/etc/modprobe.d/vfio.conf` on the nodes with the PCI devices, and binds the PCI devices to the VFIO driver.
8+
9+
.Prerequisites
10+
* You added kernel arguments to enable IOMMU for the CPU.
11+
12+
.Procedure
13+
. Create the `MachineConfig` object to enable the binding of PCI devices to the VFIO driver.
14+
+
15+
.Example
16+
[source,yaml]
17+
----
18+
apiVersion: machineconfiguration.openshift.io/v1
19+
kind: MachineConfig <1>
20+
metadata:
21+
labels:
22+
machineconfiguration.openshift.io/role: worker <2>
23+
name: 100-worker-vfiopci-configuration <3>
24+
spec:
25+
config:
26+
ignition:
27+
version: 3.2.0
28+
storage:
29+
files:
30+
- contents:
31+
source: data:text/plain;charset=utf-8;base64,b3B0aW9ucyB2ZmlvLXBjaSBpZHM9MTBkZToxZWI4Cg== <4>
32+
mode: 420 <5>
33+
overwrite: true
34+
path: /etc/modprobe.d/vfio.conf <6>
35+
- contents:
36+
source: data:text/plain;charset=utf-8;base64,dmZpby1wY2kK <7>
37+
mode: 420
38+
overwrite: true
39+
path: /etc/modules-load.d/vfio-pci.conf <8>
40+
----
41+
<1> The type of object being configured is `MachineConfig`.
42+
<2> Applies the new kernel argument only to worker nodes.
43+
<3> Named to identify where this `MachineConfig` object fits among the other `MachineConfig` objects.
44+
<4> The `echo "options vfio-pci ids=10de:1eb8" | base64` command creates the encoded text. The `vendor-ID` value (10de) and the `device-ID` value (1eb8) applies to a single device that binds to the VFIO driver. You can add a list of multiple devices for `source` with their vendor and device information.
45+
<5> The values specified for `mode` determine the permissions for the file.
46+
<6> The `/etc/modprobe.d/vfio.conf` was created by the `echo "options vfio-pci ids=10de:1eb8 > /etc/modprobe.d/vfio/conf"` command.
47+
<7> The `echo "dmZpby1wY2kK" | base64 -d` command and the `vfio-pci` commands create the encoded text for `source` as `dmZpby1wY2kK`. The encoded text translates to `vfio-pci`.
48+
<8> The file that loads the vfio-pci kernel module on the worker nodes.
49+
50+
. Create a `MachineConfig` object that identifies the kernel argument for the IOMMU configuration:
51+
+
52+
[source,terminal]
53+
----
54+
$ oc create -f 100-worker-kernel-arg-iommu.yaml
55+
----
56+
57+
. Verify that the `MachineConfig` object was added.
58+
+
59+
[source,terminal]
60+
----
61+
$ oc get MachineConfig
62+
----
63+
+
64+
.Example output
65+
[source, terminal]
66+
----
67+
NAME GENERATEDBYCONTROLLER IGNITIONVERSION AGE
68+
00-master d3da910bfa9f4b599af4ed7f5ac270d55950a3a1 3.2.0 25h
69+
00-worker d3da910bfa9f4b599af4ed7f5ac270d55950a3a1 3.2.0 25h
70+
01-master-container-runtime d3da910bfa9f4b599af4ed7f5ac270d55950a3a1 3.2.0 25h
71+
01-master-kubelet d3da910bfa9f4b599af4ed7f5ac270d55950a3a1 3.2.0 25h
72+
01-worker-container-runtime d3da910bfa9f4b599af4ed7f5ac270d55950a3a1 3.2.0 25h
73+
01-worker-kubelet d3da910bfa9f4b599af4ed7f5ac270d55950a3a1 3.2.0 25h
74+
100-worker-iommu 3.2.0 30s
75+
100-worker-vfiopci-configuration 3.2.0 30s
76+
----
77+
78+
. Run the `lspci` command to obtain the `vendor-ID` and the `device-ID` for the PCI device.
79+
+
80+
[source, terminal]
81+
----
82+
$ lspci -nnv | grep -i nvidia
83+
----
84+
+
85+
.Example output
86+
[source, terminal]
87+
----
88+
02:01.0 3D controller [0302]: NVIDIA Corporation GV100GL [Tesla V100 PCIe 32GB] [10de:1eb8] (rev a1)
89+
----
90+
91+
. Specify the values for `vendor-ID` and `device-ID` and convert the `/etc/modprobe.d/vfio.conf` file to base64 format:
92+
+
93+
[source,terminal]
94+
----
95+
$ echo "options vfio-pci ids=10de:1eb8" | base64
96+
----
97+
These values are converted into the `base64 format` and the output is encoded.
98+
+
99+
.Example output
100+
[source,terminal]
101+
----
102+
b3B0aW9ucyB2ZmlvLXBjaSBpZHM9MTBkZToxZWI4Cg==
103+
----
104+
105+
. Add the encoded text to `source` in the `MachineConfig` object.
106+
107+
. Convert `vfio-pci` into base64 format and encode the text:
108+
+
109+
[source,terminal]
110+
----
111+
$ echo vfio-pci | base64
112+
----
113+
These values are converted into the `base64 format` and the output is encoded.
114+
+
115+
.Example output
116+
[source,terminal]
117+
----
118+
dmZpby1wY2kK
119+
----
120+
121+
. Add the encoded text to `source` in the `MachineConfig` object.
122+
123+
.Verification
124+
* Verify that the VFIO driver is loaded.
125+
+
126+
[source,terminal]
127+
----
128+
$ lspci -nnk -d 10de:
129+
----
130+
The output confirms that the VFIO driver is being used.
131+
+
132+
.Example output
133+
----
134+
04:00.0 3D controller [0302]: NVIDIA Corporation GP102GL [Tesla P40] [10de:1eb8] (rev a1)
135+
Subsystem: NVIDIA Corporation Device [10de:1eb8]
136+
Kernel driver in use: vfio-pci
137+
Kernel modules: nouveau
138+
----

‎virt/virtual_machines/advanced_vm_management/virt-configuring-pci-passthrough.adoc‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
[id="virt-configuring-pci-passthrough"]
2+
= Configuring PCI passthrough
3+
include::modules/virt-document-attributes.adoc[]
4+
include::modules/common-attributes.adoc[]
5+
:context: virt-configuring-pci-passthrough
6+
7+
toc::[]
8+
9+
//This assembly contains the content for
10+
//configuring PCI passthrough by using the CLI. There are
11+
//plans to enable PCI passthrough configuration
12+
//by using the web console (next release).
13+
//When this feature is available in the web console, please
14+
//add the new content to this assembly.
15+
16+
include::modules/virt-about-pci-passthrough.adoc[leveloffset=+1]
17+
include::modules/virt-adding-kernel-arguments-enable-iommu.adoc[leveloffset=+2]
18+
include::modules/virt-binding-devices-vfio-driver.adoc[leveloffset=+2]
19+
include::modules/virt-adding-pci-device-hyperconverged-operator.adoc[leveloffset=+2]
20+
21+
[id="virt-configuring-vms-for-pci-passthrough"]
22+
== Configuring virtual machines for PCI passthrough
23+
24+
After the PCI devices have been added to the cluster, you can assign them to virtual machines. The PCI devices are now available as if they are physically connected to the virtual machines.
25+
26+
include::modules/virt-assigning-pci-device-virtual-machine.adoc[leveloffset=+2]
27+
28+
[id="additional-resources_configuring-pci-passthrough"]
29+
== Additional resources
30+
* For information about enabling virtualization hardware extensions, refer to link: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_deployment_and_administration_guide/sect-troubleshooting-enabling_intel_vt_x_and_amd_v_virtualization_hardware_extensions_in_bios[Enabling Intel VT-X and AMD-V Virtualization Hardware Extensions in BIOS].
31+
* For information about file permissions, refer to link: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_basic_system_settings/assembly_managing-file-permissions_configuring-basic-system-settings[Managing file permissions].
32+
* For information about the Machine Config Operator, refer to xref:../../../post_installation_configuration/machine-configuration-tasks.adoc#post-install-machine-configuration-tasks[Post-installation machine configuration tasks].

0 commit comments

Comments
 (0)