Skip to content

Commit 91eff06

Browse files
refactor: Remove CPU pinning config and docs from nova gpu DT/VA
Isolcpus shoudl not be used on non realtime systems or on non nfv systems. We consider that to be a bad practics and technically unsupproted by the compute team. Also neither edpm_tuned_isolated_cores nor isolcpus is required to use cpu pinning. And edpm_tuned_isolated_cores is recommend but still optional. Omit those alltogether for a nova04delta full GPU device passthrough, and its related nova-vfio-passthrough VA. Co-authored-by: aider (gemini/gemini-2.5-pro) <[email protected]>
1 parent 5530831 commit 91eff06

File tree

11 files changed

+4
-110
lines changed

11 files changed

+4
-110
lines changed

dt/nova/nova04delta/edpm/nodeset/kustomization.yaml

Lines changed: 0 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -38,19 +38,6 @@ replacements:
3838
options:
3939
create: true
4040

41-
# Nova compute CPU pinning customization
42-
- source:
43-
kind: ConfigMap
44-
name: edpm-nodeset-values
45-
fieldPath: data.nova.compute.conf
46-
targets:
47-
- select:
48-
kind: ConfigMap
49-
name: cpu-pinning-nova
50-
fieldPaths:
51-
- data.25-cpu-pinning-nova\.conf
52-
options:
53-
create: true
5441
# Nova compute PCI passthrough customization
5542
- source:
5643
kind: ConfigMap

dt/nova/nova04delta/edpm/nodeset/nova_gpu.yaml

Lines changed: 0 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,3 @@
1-
---
2-
apiVersion: v1
3-
kind: ConfigMap
4-
metadata:
5-
name: cpu-pinning-nova
6-
data:
7-
25-cpu-pinning-nova.conf: _replaced_
8-
---
91
apiVersion: v1
102
kind: ConfigMap
113
metadata:
@@ -21,8 +13,6 @@ spec:
2113
label: dataplane-deployment-nova-custom-gpu
2214
edpmServiceType: nova
2315
dataSources:
24-
- configMapRef:
25-
name: cpu-pinning-nova
2616
- configMapRef:
2717
name: gpu-nova
2818
- secretRef:

dt/nova/nova04delta/edpm/nodeset2/kustomization.yaml

Lines changed: 0 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -38,19 +38,6 @@ replacements:
3838
options:
3939
create: true
4040

41-
# Nova compute CPU pinning customization
42-
- source:
43-
kind: ConfigMap
44-
name: edpm-nodeset-values
45-
fieldPath: data.nova.compute.conf
46-
targets:
47-
- select:
48-
kind: ConfigMap
49-
name: cpu-pinning-nova-2
50-
fieldPaths:
51-
- data.25-cpu-pinning-nova\.conf
52-
options:
53-
create: true
5441
# Nova compute PCI passthrough customization
5542
- source:
5643
kind: ConfigMap

dt/nova/nova04delta/edpm/nodeset2/nova_gpu.yaml

Lines changed: 0 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,3 @@
1-
---
2-
apiVersion: v1
3-
kind: ConfigMap
4-
metadata:
5-
name: cpu-pinning-nova-2
6-
data:
7-
25-cpu-pinning-nova.conf: _replaced_
8-
---
91
apiVersion: v1
102
kind: ConfigMap
113
metadata:
@@ -21,8 +13,6 @@ spec:
2113
label: dataplane-deployment-nova-custom-gpu
2214
edpmServiceType: nova
2315
dataSources:
24-
- configMapRef:
25-
name: cpu-pinning-nova-2
2616
- configMapRef:
2717
name: gpu-nova-2
2818
- secretRef:

examples/dt/nova/nova04delta/README.md

Lines changed: 1 addition & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -20,11 +20,6 @@ The following parameters are crucial for host-level configuration:
2020
* `intel_iommu=on iommu=pt`: Enables the IOMMU for device passthrough.
2121
* `vfio-pci.ids=10de:20f1`: Instructs the `vfio-pci` driver to claim the specified GPU(s) by their vendor and product IDs at boot time. The example IDs `10de:20f1` are for an NVIDIA A100 GPU.
2222
* `rd.driver.pre=vfio-pci`: Avoids race conditions during boot by loading vfio-pci kernel module early.
23-
* `isolcpus=...`: Isolates a set of CPUs from the kernel scheduler to be used for dedicated vCPU workloads.
24-
25-
* `edpm_tuned_profile` and `edpm_tuned_isolated_cores`: These parameters configure the `tuned` service.
26-
* `edpm_tuned_profile` is set to `cpu-partitioning-powersave` to enable CPU isolation features.
27-
* `edpm_tuned_isolated_cores` specifies the cores to be isolated. This should match the list of CPUs provided in the `isolcpus` kernel argument.
2823

2924
* **VFIO-PCI Binding Service**: The `vfio-pci-bind` service in `dt/nova/nova04delta/edpm/nodeset/nova_gpu.yaml` blacklists the `nouveau` and `nvidia` kernel modules to ensure they do not interfere with the `vfio-pci` driver. The service also regenerates the initramfs and grub configuration to apply these changes. A reboot is required for these changes to take effect.
3025

@@ -66,12 +61,10 @@ That is a contrary to the legacy mode where PCI devices used to be requested thr
6661
In addition to PCI device configuration, the `nova.compute.conf` section includes parameters for resource management on the compute node:
6762

6863
* `[DEFAULT]reserved_host_memory_mb`: Specifies the amount of memory (in megabytes) to reserve for the host operating system and other non-OpenStack services. This memory will not be available for allocation to virtual machines.
69-
* `[compute]cpu_shared_set`: A list of physical CPUs that are available for host processes and for virtual machines that do not have dedicated CPUs (i.e., unpinned VMs). These should be the CPUs that are **not** isolated by `isolcpus` or `edpm_tuned_isolated_cores`.
70-
* `[compute]cpu_dedicated_set`: A list of physical CPUs that are exclusively reserved for virtual machines with dedicated CPU pinning policies. To ensure performance isolation, this list should correspond directly to the CPUs isolated using the `isolcpus` kernel argument and `edpm_tuned_isolated_cores` parameter.
7164
* `[DEFAULT]reserved_huge_pages`: Defines the number and size of huge pages to reserve for the host, making them unavailable for guest VMs. This configuration works in conjunction with the `hugepages` and `hugepagesz` kernel arguments, which define the total pool of huge pages on the host.
7265

7366
**Note**: In a full device passthrough scenario, the `[devices]enabled_vgpu_types` option in Nova's configuration is not used. This option is specific to mediated device (mdev) configurations.
7467

7568
## Guest VM
7669

77-
To use the passthrough GPU, the guest operating system inside the VM must have the appropriate native NVIDIA driver installed. You will need a standard NVIDIA driver. Do not use vGPU-enabled guest drivers. The GPU will appear as a physical PCI device within the guest.
70+
To use the passthrough GPU, the guest operating system inside the VM must have the appropriate native NVIDIA driver installed. You will need a standard NVIDIA driver. Do not use vGPU-enabled guest drivers. The GPU will appear as a physical PCI device within the guest.

examples/dt/nova/nova04delta/edpm/nodeset/values.yaml

Lines changed: 1 addition & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -40,10 +40,7 @@ data:
4040
# GPU passthrough settings for a PCI Vendor and Device ID pair.
4141
# Here, 10de represents NVIDIA Corporation, and 20f1 corresponds to the GA100 [A100 PCIe 40GB] GPU, according to Device Hunt
4242
# https://devicehunt.com/view/type/pci/vendor/10DE/device/20F1
43-
# The isolcpus=4-23,28-47 is the CPU cores that are isolated from the GPU.
44-
edpm_kernel_args: "default_hugepagesz=1GB hugepagesz=1G hugepages=16 intel_iommu=on iommu=pt isolcpus=4-23,28-47 vfio-pci.ids=10de:20f1 rd.driver.pre=vfio-pci"
45-
edpm_tuned_profile: "cpu-partitioning-powersave"
46-
edpm_tuned_isolated_cores: "4-23,28-47"
43+
edpm_kernel_args: "default_hugepagesz=1GB hugepagesz=1G hugepages=16 intel_iommu=on iommu=pt vfio-pci.ids=10de:20f1 rd.driver.pre=vfio-pci"
4744
# edpm_network_config
4845
# These vars are edpm_network_config role vars
4946
edpm_network_config_hide_sensitive_logs: false
@@ -141,9 +138,6 @@ data:
141138
reserved_host_memory_mb = 4096
142139
reserved_huge_pages = node:0,size:4,count:524160
143140
reserved_huge_pages = node:1,size:4,count:524160
144-
[compute]
145-
cpu_shared_set = 0-3,24-27
146-
cpu_dedicated_set = 8-23,32-47
147141
migration:
148142
ssh_keys:
149143
private: CHANGEME4

examples/dt/nova/nova04delta/edpm/nodeset2/values.yaml

Lines changed: 1 addition & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -40,10 +40,7 @@ data:
4040
# GPU passthrough settings for a PCI Vendor and Device ID pair.
4141
# Here, 10de represents NVIDIA Corporation, and 20f1 corresponds to the GA100 [A100 PCIe 40GB] GPU, according to Device Hunt
4242
# https://devicehunt.com/view/type/pci/vendor/10DE/device/20F1
43-
# The isolcpus=4-23,28-47 is the CPU cores that are isolated from the GPU.
44-
edpm_kernel_args: "default_hugepagesz=1GB hugepagesz=1G hugepages=16 intel_iommu=on iommu=pt isolcpus=4-23,28-47 vfio-pci.ids=10de:20f1 rd.driver.pre=vfio-pci"
45-
edpm_tuned_profile: "cpu-partitioning-powersave"
46-
edpm_tuned_isolated_cores: "4-23,28-47"
43+
edpm_kernel_args: "default_hugepagesz=1GB hugepagesz=1G hugepages=16 intel_iommu=on iommu=pt vfio-pci.ids=10de:20f1 rd.driver.pre=vfio-pci"
4744
# edpm_network_config
4845
# These vars are edpm_network_config role vars
4946
edpm_network_config_hide_sensitive_logs: false
@@ -141,9 +138,6 @@ data:
141138
reserved_host_memory_mb = 4096
142139
reserved_huge_pages = node:0,size:4,count:524160
143140
reserved_huge_pages = node:1,size:4,count:524160
144-
[compute]
145-
cpu_shared_set = 0-3,24-27
146-
cpu_dedicated_set = 8-23,32-47
147141
migration:
148142
ssh_keys:
149143
private: CHANGEME4

examples/va/nvidia-vfio-passthrough/edpm/nodeset/kustomization.yaml

Lines changed: 0 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -48,18 +48,6 @@ replacements:
4848
- spec.baremetalSetTemplate
4949
options:
5050
create: true
51-
- source:
52-
kind: ConfigMap
53-
name: edpm-nodeset-values
54-
fieldPath: data.nova.compute.conf
55-
targets:
56-
- select:
57-
kind: ConfigMap
58-
name: cpu-pinning-nova
59-
fieldPaths:
60-
- data.25-cpu-pinning-nova\.conf
61-
options:
62-
create: true
6351
- source:
6452
kind: ConfigMap
6553
name: edpm-nodeset-values

examples/va/nvidia-vfio-passthrough/edpm/nodeset/values.yaml

Lines changed: 1 addition & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -40,10 +40,7 @@ data:
4040
# GPU passthrough settings for a PCI Vendor and Device ID pair.
4141
# Here, 10de represents NVIDIA Corporation, and 20f1 corresponds to the GA100 [A100 PCIe 40GB] GPU, according to Device Hunt
4242
# https://devicehunt.com/view/type/pci/vendor/10DE/device/20F1
43-
# The isolcpus=4-23,28-47 is the CPU cores that are isolated from the GPU.
44-
edpm_kernel_args: "default_hugepagesz=1GB hugepagesz=1G hugepages=16 intel_iommu=on iommu=pt isolcpus=4-23,28-47 vfio-pci.ids=10de:20f1 rd.driver.pre=vfio-pci"
45-
edpm_tuned_profile: "cpu-partitioning-powersave"
46-
edpm_tuned_isolated_cores: "4-23,28-47"
43+
edpm_kernel_args: "default_hugepagesz=1GB hugepagesz=1G hugepages=16 intel_iommu=on iommu=pt vfio-pci.ids=10de:20f1 rd.driver.pre=vfio-pci"
4744
# edpm_network_config
4845
# These vars are edpm_network_config role vars
4946
edpm_network_config_hide_sensitive_logs: false
@@ -141,9 +138,6 @@ data:
141138
reserved_host_memory_mb = 4096
142139
reserved_huge_pages = node:0,size:4,count:524160
143140
reserved_huge_pages = node:1,size:4,count:524160
144-
[compute]
145-
cpu_shared_set = 0-3,24-27
146-
cpu_dedicated_set = 8-23,32-47
147141
migration:
148142
ssh_keys:
149143
private: CHANGEME4

va/nvidia-vfio-passthrough/edpm/nodeset/kustomization.yaml

Lines changed: 0 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -38,19 +38,6 @@ replacements:
3838
options:
3939
create: true
4040

41-
# Nova compute CPU pinning customization
42-
- source:
43-
kind: ConfigMap
44-
name: edpm-nodeset-values
45-
fieldPath: data.nova.compute.conf
46-
targets:
47-
- select:
48-
kind: ConfigMap
49-
name: cpu-pinning-nova
50-
fieldPaths:
51-
- data.25-cpu-pinning-nova\.conf
52-
options:
53-
create: true
5441
# Nova compute PCI passthrough customization
5542
- source:
5643
kind: ConfigMap

0 commit comments

Comments
 (0)