Skip to content

Commit 2c8ca73

Browse files
author
Shikha Jhala
committed
CNV-21826: Configuring VM for DPDK
1 parent 1885bd8 commit 2c8ca73

File tree

4 files changed

+318
-1
lines changed

4 files changed

+318
-1
lines changed

‎modules/virt-configuring-cluster-dpdk.adoc

Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * virt/virtual_machines/vm_networking/virt-attaching-vm-to-sriov-network.adoc
4+
5+
:_content-type: PROCEDURE
6+
[id="virt-configuring-cluster-dpdk_{context}"]
7+
= Configuring a cluster for DPDK workloads
8+
9+
You can use the following procedure to configure an {product-title} cluster to run Data Plane Development Kit (DPDK) workloads.
10+
11+
:FeatureName: Configuring a cluster for DPDK workloads
12+
include::snippets/technology-preview.adoc[]
13+
14+
.Prerequisites
15+
* You have access to the cluster as a user with `cluster-admin` permissions.
16+
* You have installed the OpenShift CLI (`oc`).
17+
* You have installed the SR-IOV Network Operator.
18+
* You have installed the Node Tuning Operator.
19+
20+
.Procedure
21+
. Map your compute nodes topology to determine which Non-Uniform Memory Access (NUMA) CPUs are isolated for DPDK applications and which ones are reserved for the operating system (OS).
22+
. Label a subset of the compute nodes with a custom role; for example, `worker-dpdk`:
23+
+
24+
[source,terminal]
25+
----
26+
$ oc label node <node_name> node-role.kubernetes.io/worker-dpdk=""
27+
----
28+
29+
. Create a new `MachineConfigPool` manifest that contains the `worker-dpdk` label in the `spec.machineConfigSelector` object:
30+
+
31+
.Example `MachineConfigPool` manifest
32+
[source,yaml]
33+
----
34+
apiVersion: machineconfiguration.openshift.io/v1
35+
kind: MachineConfigPool
36+
metadata:
37+
name: dpdk-workers
38+
labels:
39+
machineconfiguration.openshift.io/role: dpdk-workers
40+
pools.operator.machineconfiguration.openshift.io/dpdk-workers: ""
41+
spec:
42+
configuration:
43+
machineConfigSelector:
44+
matchExpressions:
45+
- key: machineconfiguration.openshift.io/role
46+
operator: In
47+
values:
48+
- worker
49+
- worker-dpdk
50+
nodeSelector:
51+
matchLabels:
52+
node-role.kubernetes.io/worker-dpdk: ""
53+
----
54+
55+
. Create a `PerformanceProfile` manifest that applies to the labeled nodes and the machine config pool that you created in the previous steps. The performance profile specifies the CPUs that are isolated for DPDK applications and the CPUs that are reserved for house keeping.
56+
+
57+
.Example `PerformanceProfile` manifest
58+
[source,yaml]
59+
----
60+
apiVersion: performance.openshift.io/v2
61+
kind: PerformanceProfile
62+
metadata:
63+
name: profile-1
64+
spec:
65+
cpu:
66+
isolated: 20,22,24,26,28,30,32,34,36,38,60,62,64,66,68,70,72,74,76,78
67+
reserved: 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,21,23,25,27,29,31,33,35,37,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,61,63,65,67,69,71,73,75,77,79
68+
hugepages:
69+
defaultHugepagesSize: 1G
70+
pages:
71+
- count: 32
72+
size: 1G
73+
numa:
74+
topologyPolicy: single-numa-node
75+
nodeSelector:
76+
node-role.kubernetes.io/worker-dpdk: ""
77+
----
78+
+
79+
[NOTE]
80+
====
81+
The compute nodes automatically restart after you apply the `MachineConfigPool` and `PerformanceProfile` manifests.
82+
====
83+
84+
. Create an `SriovNetworkNodePolicy` object with the `spec.deviceType` field set to `vfio-pci`:
85+
+
86+
.Example `SriovNetworkNodePolicy` manifest
87+
[source,yaml]
88+
----
89+
apiVersion: sriovnetwork.openshift.io/v1
90+
kind: SriovNetworkNodePolicy
91+
metadata:
92+
name: policy-1
93+
namespace: openshift-sriov-network-operator
94+
spec:
95+
resourceName: intel_nics_dpdk
96+
deviceType: vfio-pci
97+
mtu: 9000
98+
numVfs: 4
99+
priority: 99
100+
nicSelector:
101+
vendor: "8086"
102+
deviceID: "1572"
103+
pfNames:
104+
- eno3
105+
rootDevices:
106+
- "0000:19:00.2"
107+
nodeSelector:
108+
feature.node.kubernetes.io/network-sriov.capable: "true"
109+
----
110+
111+
. Set the container runtime configuration for the `virt-launcher` pod by adding the following annotation to the `HyperConverged` custom resource (CR):
112+
+
113+
[source,terminal]
114+
----
115+
$ oc annotate --overwrite -n openshift-cnv hco kubevirt-hyperconverged \
116+
kubevirt.kubevirt.io/jsonpatch='[{"op": "add", "path": "/spec/configuration/defaultRuntimeClass", "value": <runtimeclass-name>}]'
117+
----
118+
+
119+
[NOTE]
120+
====
121+
Adding the annotation to the `HyperConverged` CR changes a global setting that affects all VMs that are created after the annotation is applied.
122+
====
123+

‎modules/virt-configuring-vm-dpdk.adoc

Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * virt/virtual_machines/vm_networking/virt-attaching-vm-to-sriov-network.adoc
4+
5+
:_content-type: PROCEDURE
6+
[id="virt-configuring-vm-dpdk_{context}"]
7+
= Configuring a virtual machine for DPDK workloads
8+
9+
You can run Data Packet Development Kit (DPDK) workloads on virtual machines (VMs) to achieve lower latency and higher throughput for faster packet processing in the user space. DPDK uses the SR-IOV network for hardware-based I/O sharing.
10+
11+
.Prerequisites
12+
* Your cluster is configured to run DPDK workloads.
13+
* You have created and configured the project in which the VM will run.
14+
15+
.Procedure
16+
. Edit the `VirtualMachine` manifest to include information about the SR-IOV network interface, CPU topology, CRI-O annotations, and huge pages:
17+
+
18+
.Example `VirtualMachine` manifest
19+
[source,yaml]
20+
----
21+
apiVersion: kubevirt.io/v1
22+
kind: VirtualMachine
23+
metadata:
24+
name: rhel-dpdk-vm
25+
spec:
26+
running: true
27+
template:
28+
metadata:
29+
annotations:
30+
cpu-load-balancing.crio.io: disable <1>
31+
cpu-quota.crio.io: disable <2>
32+
irq-load-balancing.crio.io: disable <3>
33+
spec:
34+
nodeSelector:
35+
node-role.kubernetes.io/worker-dpdk: "" <4>
36+
domain:
37+
cpu:
38+
sockets: 1 <5>
39+
cores: 5 <6>
40+
threads: 2
41+
dedicatedCpuPlacement: true
42+
isolateEmulatorThread: true
43+
interfaces:
44+
- masquerade: {}
45+
name: default
46+
- model: virtio
47+
name: nic-east
48+
pciAddress: '0000:07:00.0'
49+
sriov: {}
50+
networkInterfaceMultiqueue: true
51+
rng: {}
52+
memory:
53+
hugepages:
54+
pageSize: 1Gi <7>
55+
resources:
56+
requests:
57+
memory: 8Gi
58+
networks:
59+
- name: default
60+
pod: {}
61+
- multus:
62+
networkName: dpdk-net <8>
63+
name: nic-east
64+
# ...
65+
----
66+
<1> This annotation specifies that load balancing is disabled for CPUs that are used by the container.
67+
<2> This annotation specifies that the CPU quota is disabled for CPUs that are used by the container.
68+
<3> This annotation specifies that Interrupt Request (IRQ) load balancing is disabled for CPUs that are used by the container.
69+
<4> The label that is used in the `MachineConfigPool` and `PerformanceProfile` manifests that were created when configuring the cluster for DPDK workloads.
70+
<5> The number of sockets inside the VM. This field must be set to `1` for the CPUs to be scheduled from the same Non-Uniform Memory Access (NUMA) node.
71+
<6> The number of cores inside the VM. This must be a value greater than or equal to `1`. In this example, the VM is scheduled with 5 hyper-threads or 10 CPUs.
72+
<7> The size of the huge pages. The possible values for x86-64 architecture are 1Gi and 2Mi. In this example, the request is for 8 huge pages of size 1Gi.
73+
<8> The name of the SR-IOV `NetworkAttachmentDefinition` object.
74+
75+
. Save and exit the editor.
76+
. Apply the `VirtualMachine` manifest:
77+
+
78+
[source,terminal]
79+
----
80+
$ oc apply -f <file_name>.yaml
81+
----
82+
83+
. Configure the guest operating system. The following example shows the configuration steps for {op-system-base} 8 OS:
84+
.. Configure isolated VM CPUs and specify huge pages by using the GRUB bootloader command-line interface. In the following example, 8 1G huge pages are specified. The first two CPUs (0 and 1) are set aside for house keeping tasks and the rest are isolated for the DPDK application.
85+
+
86+
[source,terminal]
87+
----
88+
$ grubby --update-kernel=ALL --args="default_hugepagesz=1GB hugepagesz=1G hugepages=8 isolcpus=2-9"
89+
----
90+
91+
.. To achieve low-latency tuning by using the `cpu-partitioning` profile in the TuneD application, run the following commands:
92+
+
93+
[source,terminal]
94+
----
95+
$ dnf install -y tuned-profiles-cpu-partitioning
96+
----
97+
+
98+
[source,terminal]
99+
----
100+
$ echo isolated_cores=2-9 > /etc/tuned/cpu-partitioning-variables.conf
101+
----
102+
+
103+
[source,terminal]
104+
----
105+
$ tuned-adm profile cpu-partitioning
106+
----
107+
108+
.. Override the SR-IOV NIC driver by using the `driverctl` device driver control utility:
109+
+
110+
[source,terminal]
111+
----
112+
$ dnf install -y driverctl
113+
----
114+
+
115+
[source,terminal]
116+
----
117+
$ driverctl set-override 0000:07:00.0 vfio-pci
118+
----
119+
120+
. Restart the VM to apply the changes.

‎modules/virt-configuring-vm-project-dpdk.adoc

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * virt/virtual_machines/vm_networking/virt-attaching-vm-to-sriov-network.adoc
4+
5+
:_content-type: PROCEDURE
6+
[id="virt-configuring-vm-project-dpdk_{context}"]
7+
= Configuring a project for DPDK workloads
8+
9+
You can configure the project to run DPDK workloads on SR-IOV hardware.
10+
11+
.Prerequisites
12+
* Your cluster is configured to run DPDK workloads.
13+
14+
.Procedure
15+
. Create a namespace for your DPDK applications:
16+
+
17+
[source,terminal]
18+
----
19+
$ oc create ns dpdk-checkup-ns
20+
----
21+
22+
. Create an `SriovNetwork` object that references the `SriovNetworkNodePolicy` object. When you create an `SriovNetwork` object, the SR-IOV Network Operator automatically creates a `NetworkAttachmentDefinition` object.
23+
+
24+
.Example `SriovNetwork` manifest
25+
[source,yaml]
26+
----
27+
apiVersion: sriovnetwork.openshift.io/v1
28+
kind: SriovNetwork
29+
metadata:
30+
name: dpdk-sriovnetwork
31+
namespace: openshift-sriov-network-operator
32+
spec:
33+
ipam: |
34+
{
35+
"type": "host-local",
36+
"subnet": "10.56.217.0/24",
37+
"rangeStart": "10.56.217.171",
38+
"rangeEnd": "10.56.217.181",
39+
"routes": [{
40+
"dst": "0.0.0.0/0"
41+
}],
42+
"gateway": "10.56.217.1"
43+
}
44+
networkNamespace: dpdk-checkup-ns <1>
45+
resourceName: intel_nics_dpdk <2>
46+
spoofChk: "off"
47+
trust: "on"
48+
vlan: 1019
49+
----
50+
<1> The namespace where the `NetworkAttachmentDefinition` object is deployed.
51+
<2> The value of the `spec.resourceName` attribute of the `SriovNetworkNodePolicy` object that was created when configuring the cluster for DPDK workloads.
52+
53+
. Optional: Run the virtual machine latency checkup to verify that the network is properly configured.
54+
. Optional: Run the DPDK checkup to verify that the namespace is ready for DPDK workloads.

‎virt/virtual_machines/vm_networking/virt-attaching-vm-to-sriov-network.adoc

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,27 @@ include::modules/nw-sriov-network-attachment.adoc[leveloffset=+1]
2424

2525
include::modules/virt-attaching-vm-to-sriov-network.adoc[leveloffset=+1]
2626

27+
include::modules/virt-configuring-cluster-dpdk.adoc[leveloffset=+1]
28+
29+
[role="_additional-resources"]
30+
.Additional resources
31+
* xref:../../../scalability_and_performance/using-cpu-manager.adoc#using-cpu-manager[Using CPU Manager and Topology Manager]
32+
* xref:../../../scalability_and_performance/what-huge-pages-do-and-how-they-are-consumed-by-apps.adoc#configuring-huge-pages_huge-pages[Configuring huge pages]
33+
* link:https://access.redhat.com/solutions/5688941[Creating a custom machine config pool]
34+
35+
include::modules/virt-configuring-vm-project-dpdk.adoc[leveloffset=+1]
36+
37+
[role="_additional-resources"]
38+
.Additional resources
39+
* xref:../../../applications/projects/working-with-projects.adoc#working-with-projects[Working with projects]
40+
* xref:../../../virt/support/monitoring/virt-running-cluster-checkups.adoc#virt-measuring-latency-vm-secondary-network_virt-running-cluster-checkups[Virtual machine latency checkup]
41+
* xref:../../../virt/support/monitoring/virt-running-cluster-checkups.adoc#virt-checking-cluster-dpdk-readiness_virt-running-cluster-checkups[DPDK checkup]
42+
43+
include::modules/virt-configuring-vm-dpdk.adoc[leveloffset=+1]
44+
45+
2746
[id="next-steps_virt-attaching-vm-to-sriov-network"]
2847
== Next steps
2948

30-
* xref:../../../virt/virtual_machines/vm_networking/virt-accessing-vm-secondary-network-fqdn.adoc#virt-accessing-vm-secondary-network-fqdn[Accessing a virtual machine on a secondary network by using the cluster domain name]
49+
* xref:../../../virt/virtual_machines/vm_networking/virt-accessing-vm-secondary-network-fqdn.adoc#virt-accessing-vm-secondary-network-fqdn[Accessing a virtual machine on a secondary network by using the cluster domain name]
50+

0 commit comments

Comments
 (0)