VFIO/PCIe Passthrough configuration support (#247)

Adam Delo · web-flow · commit a01efb4f3aee · 2024-07-05T23:02:29.000+09:00
* added basic support for PCIe passthrough for Intel and AMD CPUs

Feature can be enabled via `pve_pcie_passthrough_enabled`.
Mediated devices are supported, but disabled by default since not all boards support GVT-g.
Interrupt remapping can also be disabled for boards that do not support it.

* moved GRUB update task to a handler to deduplicate tasks
* added handler for updating initramfs when updating modprobe configuration
* added support for certain PCIe passthrough configurations

Role variables have been added to allow stubbing PCI devices via Vendor:Product ID when GRUB boots, blocking the loading of modules (e.g. nvidia drivers) via `softdep`, enabling GPU OVMF passthrough, and disabling DMA translation by the hypervisor for passthrough devices.

* added new section for PCIe passthrough in documentation
* added ability to configure KVM module to ignore MSRS and disable logging ignored MSRs

This fixes issues with certain applications in Windows guests.
diff --git a/README.md b/README.md
@@ -386,6 +386,15 @@ pve_check_for_kernel_update: true # Runs a script on the host to check kernel ve
 pve_reboot_on_kernel_update: false # If set to true, will automatically reboot the machine on kernel updates
 pve_reboot_on_kernel_update_delay: 60 # Number of seconds to wait before and after a reboot process to proceed with next task in cluster mode
 pve_remove_old_kernels: true # Currently removes kernel from main Debian repository
+pve_pcie_passthrough_enabled: false # Set this to true to enable PCIe passthrough.
+pve_iommu_passthrough_mode: false # Set this to true to allow VMs to bypass the DMA translation. This might increase performance for IOMMU passthrough.
+pve_iommu_unsafe_interrupts: false # Set this to true if your system doesn't support interrupt remapping.
+pve_mediated_devices_enabled: false # Set this to true if your device supports gtv-g and you wish to enable split functionality.
+pve_pcie_ovmf_enabled: false # Set this to true to enable GPU OVMF PCI passthrough.
+pve_pci_device_ids: [] # List of pci device ID's (see https://pve.proxmox.com/wiki/Pci_passthrough#GPU_Passthrough).
+pve_vfio_blacklist_drivers: [] # List of device drivers to blacklist from the Proxmox host (see https://pve.proxmox.com/wiki/PCI(e)_Passthrough).
+pve_pcie_ignore_msrs: false # Set this to true if passing through to Windows machine to prevent VM crashing.
+pve_pcie_report_msrs: true # Set this to false to prevent dmesg system from logging msrs crash reports.
 pve_watchdog: none # Set this to "ipmi" if you want to configure a hardware watchdog. Proxmox uses a software watchdog (nmi_watchdog) by default.
 pve_watchdog_ipmi_action: power_cycle # Can be one of "reset", "power_cycle", and "power_off".
 pve_watchdog_ipmi_timeout: 10 # Number of seconds the watchdog should wait
@@ -760,6 +769,56 @@ nodes).
 `pve_ceph_osds` by default creates unencrypted ceph volumes. To use encrypted
 volumes the parameter `encrypted` has to be set per drive to `true`.
 
+## PCIe Passthrough
+
+This role can be configured to allow PCI device passthrough from the Proxmox host to VMs. This feature is not enabled by default since not all motherboards and CPUs support this feature. To enable passthrough, the devices CPU must support hardware virtualization (VT-d for Intel based systems and AMD-V for AMD based systems). Refer to the manuals of all components to determine whether this feature is supported or not. Naming conventions of will vary, but is usually referred to as IOMMU, VT-d, or AMD-V.
+
+By enabling this feature, dedicated devices (such as a GPU or USB devices) can be passed through to the VMs. Along with dedicated devices, various integrated devices such as Intel or AMD's integrated GPU's are also able to be passed through to VMs.
+
+Some devices are able to take advantage of Mediated usage. Mediated devices are able to be passed through to multiple VMs to share resources, while still remaining usable by the host system. Splitting of devices is not always supported and should be validated before being enabled to prevent errors. Refer to the manual of the device you want to pass through to determine whether the device is capable of mediated usage (Currently this role only supports GVT-g; SR-IOV is not currently supported and must be enable manually after role completion).
+
+The following is an example configuration which enables PCIe passthrough:
+
+```yaml
+pve_pcie_passthrough_enabled: true
+pve_iommu_passthrough_mode: true
+pve_iommu_unsafe_interrupts: false
+pve_mediated_devices_enabled: false
+pve_pcie_ovmf_enabled: false
+pve_pci_device_ids:
+  - id: "10de:1381"
+  - id: "10de:0fbc"
+pve_vfio_blacklist_drivers:
+  - name: "radeon"
+  - name: "nouveau"
+  - name: "nvidia"
+pve_pcie_ignore_msrs: false
+pve_pcie_report_msrs: true
+```
+
+`pve_pcie_passthrough_enabled` is required to use any PCIe passthrough functionality. Without this enabled, all other PCIe related fields will be unused.
+
+`pve_iommu_passthrough_mode` enabling IOMMU passthrough mode might increase device performance. By enabling this feature, it allows VMs to bypass the default DMA translation which would normally be performed by the hyper-visor. Instead, VMs pass DMA requests directly to the hardware IOMMU.
+
+`pve_iommu_unsafe_interrupts` is required to be enabled to allow PCI passthrough if your system doesn't support interrupt remapping. You can find check whether the device supports interrupt remapping by using `dmesg | grep 'remapping'`. If you see one of the following lines:
+
+- "AMD-Vi: Interrupt remapping enabled"
+- "DMAR-IR: Enabled IRQ remapping in x2apic mode" ('x2apic' can be different on old CPUs, but should still work)
+
+Then system interrupt remapping is supported and you do not need to enable unsafe interrupts. Be aware that by enabling this value your system can become unstable.
+
+`pve_mediated_devices_enabled` enables GVT-g support for integrated devices such as Intel iGPU's. Not all devices support GVT-g so it is recommended to check with your specific device beforehand to ensure it is allowed.
+
+`pve_pcie_ovmf_enabled` enables GPU OVMF PCI passthrough. When using OVMF you should select 'OVMF' as the BIOS option for the VM instead of 'SeaBIOS' within Proxmox. This setting will try to opt-out devices from VGA arbitration if possible.
+
+`pve_pci_device_ids` is a list of device and vendor ids that is wished to be passed through to VMs from the host. See the section 'GPU Passthrough' on the [Proxmox WIKI](https://pve.proxmox.com/wiki/Pci_passthrough) to find your specific device and vendor id's. When setting this value, it is required to specify an 'id' for each new element in the array.
+
+`pve_vfio_blacklist_drivers` is a list of drivers to be excluded/blacklisted from the host. This is required when passing through a PCI device to prevent the host from using the device before it can be assigned to a VM. When setting this value, it is required to specify a 'name' for each new element in the array.
+
+`pve_pcie_ignore_msrs` prevents some Windows applications like GeForce Experience, Passmark Performance Test and SiSoftware Sandra from crashing the VM. This value is only required when passing PCI devices to Windows based systems.
+
+`pve_pcie_report_msrs` can be used to enable or disable logging messages of msrs warnings. If you see a lot of warning messages in your 'dmesg' system log, this value can be used to silence msrs warnings.
+
 ## Developer Notes
 
 When developing new features or fixing something in this role, you can test out
@@ -802,6 +861,7 @@ PendaGTP ([@PendaGTP](https://github.com/PendaGTP)) - Ceph support
 John Marion ([@jmariondev](https://github.com/jmariondev))  
 foerkede ([@foerkede](https://github.com/foerkede)) - ZFS storage support  
 Guiffo Joel ([@futuriste](https://github.com/futuriste)) - Pool configuration support  
+Adam Delo ([@ol3d](https://github.com/ol3d)) - PCIe Passthrough Support
 
 [Full list of contributors](https://github.com/lae/ansible-role-proxmox/graphs/contributors)
 
diff --git a/defaults/main.yml b/defaults/main.yml
@@ -10,6 +10,15 @@ pve_reboot_on_kernel_update_delay: 60
 pve_remove_old_kernels: true
 pve_run_system_upgrades: false
 pve_run_proxmox_upgrades: true
+pve_pcie_passthrough_enabled: false
+pve_iommu_passthrough_mode: false
+pve_iommu_unsafe_interrupts: false
+pve_mediated_devices_enabled: false
+pve_pcie_ovmf_enabled: false
+pve_pci_device_ids: []
+pve_vfio_blacklist_drivers: []
+pve_pcie_ignore_msrs: false
+pve_pcie_report_msrs: true
 pve_watchdog: none
 pve_watchdog_ipmi_action: power_cycle
 pve_watchdog_ipmi_timeout: 10
diff --git a/handlers/main.yml b/handlers/main.yml
@@ -32,3 +32,12 @@
     name: ceph.service
     state: restarted
     daemon_reload: true
+
+- name: update-initramfs
+  command: update-initramfs -u -k all
+
+- name: update-grub
+  command: update-grub
+  register: _pve_grub_update
+  failed_when: ('error' in _pve_grub_update.stderr)
+  tags: skiponlxc
diff --git a/tasks/disable_nmi_watchdog.yml b/tasks/disable_nmi_watchdog.yml
@@ -30,11 +30,4 @@
     dest: /etc/default/grub
     line: 'GRUB_CMDLINE_LINUX="$GRUB_CMDLINE_LINUX nmi_watchdog=0"'
     insertafter: '^GRUB_CMDLINE_LINUX="'
-  register: _pve_grub
-
-- name: Update GRUB configuration
-  command: update-grub
-  register: _pve_grub_update
-  failed_when: ('error' in _pve_grub_update.stderr)
-  when: "_pve_grub is changed"
-  tags: skiponlxc
+  notify: update-grub
diff --git a/tasks/kernel_module_cleanup.yml b/tasks/kernel_module_cleanup.yml
@@ -20,14 +20,7 @@
         dest: /etc/default/grub
         line: 'GRUB_CMDLINE_LINUX="$GRUB_CMDLINE_LINUX nmi_watchdog=0"'
         state: absent
-      register: _pve_grub
-
-    - name: Update GRUB configuration
-      command: update-grub
-      register: _pve_grub_update
-      failed_when: ('error' in _pve_grub_update.stderr)
-      when: "_pve_grub is changed"
-      tags: skiponlxc
+      notify: update-grub
 
     - name: Remove ipmi_watchdog modprobe configuration
       file:
@@ -46,3 +39,74 @@
       notify:
         - restart watchdog-mux
   when: "pve_watchdog != 'ipmi'"
+
+- name: Modify vfio IOMMU references and configuration in default grub
+  ansible.builtin.blockinfile:
+    dest: /etc/default/grub
+    state: absent
+    marker: "# {mark}: IOMMU default grub configuration (managed by ansible)."
+  notify: update-grub
+  when: >
+    (not pve_pcie_passthrough_enabled | bool) or
+    ((not 'GenuineIntel' in ansible_processor | unique) and
+    (not pve_iommu_passthrough_mode | bool) and
+    (not pve_mediated_devices_enabled | bool) and
+    (not pve_pci_device_ids | length > 0))
+
+- name: Remove modprobe.d configuration files
+  notify: update-initramfs
+  block:
+    - name: Remove vfio config file
+      ansible.builtin.file:
+        dest: /etc/modprobe.d/vfio.conf
+        state: absent
+      when: >
+        (not pve_pcie_passthrough_enabled | bool) or
+        ((not pve_pci_device_ids | length > 0) and
+        (not pve_vfio_blacklist_drivers | length > 0) and
+        (not pve_pcie_ovmf_enabled | bool))
+
+    - name: Remove driver blacklist config file
+      ansible.builtin.file:
+        dest: /etc/modprobe.d/blacklist.conf
+        state: absent
+      when: >
+        (not pve_pcie_passthrough_enabled | bool) or
+        (not pve_vfio_blacklist_drivers | length > 0)
+
+    - name: Remove kvm config file
+      ansible.builtin.file:
+        dest: /etc/modprobe.d/kvm.conf
+        state: absent
+      when: >
+        (not pve_pcie_passthrough_enabled | bool) or
+        ((not pve_pcie_ignore_msrs | bool) and
+        (pve_pcie_report_msrs | bool))
+
+    - name: Disable declaring IOMMU unsafe interrupts on init
+      ansible.builtin.file:
+        dest: /etc/modprobe.d/iommu_unsafe_interrupts.conf
+        state: absent
+      when: >
+        (not pve_pcie_passthrough_enabled | bool) or
+        (not pve_iommu_unsafe_interrupts | bool)
+
+- name: Remove all GVT-g configuration
+  notify: update-initramfs
+  block:
+    - name: Remove modules list for GVT-g
+      ansible.builtin.blockinfile:
+        dest: /etc/modules
+        state: absent
+        marker: "# {mark}: Modules required for GVT-g (managed by ansible)."
+      when: >
+        (not pve_pcie_passthrough_enabled | bool) or
+        (not pve_mediated_devices_enabled | bool)
+
+    - name: Remove modules list required for PCI passthrough
+      ansible.builtin.blockinfile:
+        dest: /etc/modules
+        state: absent
+        marker: "# {mark}: Modules required for PCI passthrough (managed by ansible)."
+      when: >
+        (not pve_pcie_passthrough_enabled | bool)
diff --git a/tasks/main.yml b/tasks/main.yml
@@ -201,6 +201,9 @@
   when:
     - "'pve-no-subscription' in pve_repository_line"
 
+- import_tasks: pcie_passthrough.yml
+  when: "pve_pcie_passthrough_enabled | bool"
+
 - import_tasks: kernel_updates.yml
 
 - import_tasks: ipmi_watchdog.yml
diff --git a/tasks/pcie_passthrough.yml b/tasks/pcie_passthrough.yml
@@ -0,0 +1,93 @@
+---
+- name: Modify vfio IOMMU references and configuration in default grub
+  ansible.builtin.blockinfile:
+    dest: /etc/default/grub
+    marker: "# {mark}: IOMMU default grub configuration (managed by ansible)."
+    content: "\
+      {% if '\"GenuineIntel\" in ansible_processor | unique' %}GRUB_CMDLINE_LINUX=\"$GRUB_CMDLINE_LINUX intel_iommu=on\"\n{% endif %}\
+      {% if (pve_iommu_passthrough_mode | bool) %}GRUB_CMDLINE_LINUX=\"$GRUB_CMDLINE_LINUX iommu=pt\"\n{% endif %}\
+      {% if (pve_mediated_devices_enabled | bool) %}GRUB_CMDLINE_LINUX=\"$GRUB_CMDLINE_LINUX i915.enable_gvt=1 i915.enable_guc=0\"\n{% endif %}\
+      {% if (pve_pci_device_ids | length > 0) %}GRUB_CMDLINE_LINUX=\"$GRUB_CMDLINE_LINUX vfio-pci.ids={% for k in pve_pci_device_ids %}{{ k.id }}{% if k != (pve_pci_device_ids | last) %},{% endif %}{% endfor %}\"{% endif %}"
+    insertafter: '^GRUB_CMDLINE_LINUX=""'
+    mode: "0640"
+  notify: update-grub
+  when: >
+    ('GenuineIntel' in ansible_processor | unique) or
+    (pve_iommu_passthrough_mode | bool) or
+    (pve_mediated_devices_enabled | bool) or
+    (pve_pci_device_ids | length > 0)
+
+- name: Create/Modify modprobe.d configuration files
+  notify: update-initramfs
+  block:
+    - name: Specify vfio configuration options
+      ansible.builtin.blockinfile:
+        dest: /etc/modprobe.d/vfio.conf
+        marker: "# {mark}: VFIO driver configuration options (managed by ansible)."
+        content: "\
+          {% if (pve_vfio_blacklist_drivers | length > 0) %}{% for k in pve_vfio_blacklist_drivers %}softdep {{ k.name }} pre: vfio-pci\n{% endfor %}{% endif %}\
+          {% if (pve_pcie_ovmf_enabled | bool) %}options vfio-pci disable_vga=1\n{% endif %}\
+          {% if (pve_pci_device_ids | length > 0) %}options vfio-pci
+            ids={% for k in pve_pci_device_ids %}{{ k.id }}{% if k != (pve_pci_device_ids | last) %},{% endif %}{% endfor %}{% endif %}"
+        mode: "0640"
+        create: true
+      when: >
+        (pve_vfio_blacklist_drivers | length > 0) or
+        (pve_pci_device_ids | length > 0) or
+        (pve_pcie_ovmf_enabled | bool)
+
+    - name: Blacklist drivers from host
+      ansible.builtin.blockinfile:
+        dest: /etc/modprobe.d/blacklist.conf
+        marker: "# {mark}: Blacklist drivers from host (managed by ansible)."
+        content: "{% for k in pve_vfio_blacklist_drivers %}blacklist {{ k.name }}\n{% endfor %}"
+        mode: "0640"
+        create: true
+      when: >
+        (pve_vfio_blacklist_drivers | length > 0)
+
+    - name: Specify kvm configuration options
+      ansible.builtin.blockinfile:
+        dest: /etc/modprobe.d/kvm.conf
+        marker: "# {mark}: VFIO driver configuration options (managed by ansible)."
+        content: "\
+          {% if (pve_pcie_ignore_msrs | bool) %}options kvm ignore_msrs=1\n{% endif %}\
+          {% if (not pve_pcie_report_msrs | bool) %}options kvm report_ignored_msrs=0{% endif %}"
+        mode: "0640"
+        create: true
+      when: >
+        (pve_pcie_ignore_msrs | bool) or
+        (not pve_pcie_report_msrs | bool)
+
+    - name: Enable IOMMU Interrupt Remapping
+      ansible.builtin.blockinfile:
+        dest: /etc/modprobe.d/iommu_unsafe_interrupts.conf
+        marker: "# {mark}: IOMMU Interrupt Remapping configuration (managed by ansible)."
+        content: "options vfio_iommu_type1 allow_unsafe_interrupts=1"
+        mode: "0640"
+        create: true
+      when: >
+        (pve_iommu_unsafe_interrupts | bool)
+
+- name: Modify required modules list
+  notify: update-initramfs
+  block:
+    - name: Modify modules list for PCIe Passthrough
+      ansible.builtin.blockinfile:
+        dest: /etc/modules
+        marker: "# {mark}: Modules required for PCI passthrough (managed by ansible)."
+        content: |
+          vfio
+          vfio_iommu_type1
+          vfio_pci
+          vfio_virqfd
+
+    - name: Modify modules list for GVT-g
+      ansible.builtin.blockinfile:
+        dest: /etc/modules
+        marker: "# {mark}: Modules required for GVT-g (managed by ansible)."
+        content: |
+          kvmgt
+          mdev
+      when: >
+        (pve_mediated_devices_enabled | bool)
diff --git a/tests/vagrant/group_vars/all b/tests/vagrant/group_vars/all
@@ -7,6 +7,20 @@ pve_extra_packages:
 pve_check_for_kernel_update: true
 pve_reboot_on_kernel_update: true
 pve_run_system_upgrades: true
+pve_pcie_passthrough_enabled: true
+pve_iommu_passthrough_mode: true
+pve_iommu_unsafe_interrupts: true
+pve_mediated_devices_enabled: true
+pve_pcie_ovmf_enabled: true
+pve_pci_device_ids:
+  - id: "10de:1381"
+  - id: "10de:0fbc"
+pve_vfio_blacklist_drivers:
+  - name: "radeon"
+  - name: "nouveau"
+  - name: "nvidia"
+pve_pcie_ignore_msrs: true
+pve_pcie_report_msrs: false
 pve_zfs_enabled: yes
 pve_zfs_zed_email: root@localhost
 pve_cluster_enabled: yes
