diff --git a/CHANGELOG.md b/CHANGELOG.md index e7eb43cf986..5892614b4a3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -53,6 +53,12 @@ and this project adheres to guest's serial console. Not configuring it means Firecracker will continue to print serial output to stdout. Similarly to the logger, this configuration is not persisted across snapshots. +- [#5364](https://github.com/firecracker-microvm/firecracker/pull/5364): Added + PCI support in Firecracker. PCI support is optional. Users can enable it + passing the `--enable-pci` flag when launching the Firecracker process. When + Firecracker process is launched with PCI support, it will create all VirtIO + devices using a PCI VirtIO transport. If not enabled, Firecracker will use the + MMIO transport instead. ### Changed diff --git a/FAQ.md b/FAQ.md index 9456471df8c..8c47529661d 100644 --- a/FAQ.md +++ b/FAQ.md @@ -129,7 +129,7 @@ Example of a kernel valid command line that enables the serial console (which goes in the `boot_args` field of the `/boot-source` Firecracker API resource): ```console -console=ttyS0 reboot=k panic=1 pci=off nomodule +console=ttyS0 reboot=k panic=1 nomodule ``` ### How can I configure multiple Ethernet devices through the kernel command line? diff --git a/docs/getting-started.md b/docs/getting-started.md index f0b02c6e1b5..17c681da34c 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -194,9 +194,16 @@ API_SOCKET="/tmp/firecracker.socket" sudo rm -f $API_SOCKET # Run firecracker -sudo ./firecracker --api-sock "${API_SOCKET}" +sudo ./firecracker --api-sock "${API_SOCKET} --enable-pci" ``` +The `--enable-pci` flag instructs Firecracker to create all VirtIO devices using +a PCI VirtIO transport. This flag is optional. If not passed, Firecracker will +create devices using the legacy MMIO transport. We suggest that users enable the +PCI transport, as it yields higher throughput and lower latency for VirtIO +devices. For more information regarding guest kernel requirements for using PCI +look at our [kernel policy documentation](./kernel-policy.md). + In a new terminal (do not close the 1st one): ```bash @@ -240,7 +247,7 @@ sudo curl -X PUT --unix-socket "${API_SOCKET}" \ "http://localhost/logger" KERNEL="./$(ls vmlinux* | tail -1)" -KERNEL_BOOT_ARGS="console=ttyS0 reboot=k panic=1 pci=off" +KERNEL_BOOT_ARGS="console=ttyS0 reboot=k panic=1" ARCH=$(uname -m) diff --git a/docs/kernel-policy.md b/docs/kernel-policy.md index 86823459d44..9387244118b 100644 --- a/docs/kernel-policy.md +++ b/docs/kernel-policy.md @@ -69,6 +69,15 @@ The configuration items that may be relevant for Firecracker are: - use CPU RNG instructions (if present) to initialize RNG. Available for >= 5.10 - ACPI support - `CONFIG_ACPI` and `CONFIG_PCI` +- PCI support: + - `CONFIG_BLK_MQ_PCI` + - `CONFIG_PCI` + - `CONFIG_PCI_MMCONFIG` + - `CONFIG_PCI_MSI` + - `CONFIG_PCIEPORTBUS` + - `CONFIG_VIRTIO_PCI` + - `CONFIG_PCI_HOST_COMMON` + - `CONFIG_PCI_HOST_GENERIC` There are also guest config options which are dependant on the platform on which Firecracker is run: @@ -142,6 +151,63 @@ following configurations: - Only legacy mechanisms - Both ACPI and legacy mechanisms +##### Booting with PCI: + +Firecracker supports booting guest microVMs with PCI support. This option is +enabled using the `--enable-pci` flag when launching the Firecracker process. +With PCI enabled, Firecracker will create all VirtIO devices using a PCI VirtIO +transport. The PCI transport typically achieves higher throughput and lower +latency for VirtIO devices. No further, per device, configuration is needed to +enable the PCI transport. + +PCI support is optional; if it is not enabled Firecracker will create VirtIO +devices using the MMIO transport. + +For Firecracker microVMs to boot properly with PCI support, use a guest kernel +built with PCI support. See the relevant Kconfig flags in our list of +[relevant Kconfig options](#guest-kernel-configuration-items): + +> [!IMPORTANT] +> +> Make sure that the kernel command line **does NOT** include the `pci=off` +> slug, which disables PCI support during boot time within the guest. When PCI +> is disabled, Firecracker will add this slug in the command line to instruct +> the guest kernel to skip useless PCI checks. For more info, look into the +> section for [Kernel command line parameters](#kernel-command-line-parameters). + +> [!NOTE] +> +> On x86_64 systems, `CONFIG_PCI` Kconfig option is needed even when booting +> microVMs without PCI support in case users want to use ACPI to boot. See +> [here](#booting-with-acpi-x86_64-only) for more info. + +## Kernel command line parameters + +By default, Firecracker will boot a guest microVM passing the following command +line parameters to the kernel: + +`reboot=k panic=1 nomodule 8250.nr_uarts=0 i8042.noaux i8042.nomux i8042.dumbkbd swiotlb=noforce`. + +- `reboot=k` shut down the guest on reboot, instead of rebooting +- `panic=1` on panic, reboot after 1 second +- `nomodule` disable loadable kernel module support +- `8250.nr_uarts=0` disable 8250 serial interface +- `i8042.noaux` do not probe the i8042 controller for an attached mouse (save + boot time) +- `i8042.nomux` do not probe i8042 for a multiplexing controller (save boot + time) +- `i8042.dumbkbd` do not attempt to control kbd state via the i8042 (save boot + time) +- `swiotlb=noforce` disable software bounce buffers (SWIOTLB) + +When running without [PCI support](#booting-with-pci), Firecracker will also +append `pci=off` to the above list. This option instructs the guest kernel to +avoid PCI probing. + +Users can provide their own command line parameters through the `boot_args` +field of the `/boot-source` +[Firecracker API](../src/firecracker/swagger/firecracker.yaml). + ## Caveats - [Snapshot compatibility across kernel versions](snapshotting/snapshot-support.md#snapshot-compatibility-across-kernel-versions) diff --git a/src/pci/src/bus.rs b/src/pci/src/bus.rs index 01c9b1f1933..12188ee7ba5 100644 --- a/src/pci/src/bus.rs +++ b/src/pci/src/bus.rs @@ -161,6 +161,9 @@ impl PciConfigIo { return 0xffff_ffff; } + // NOTE: Potential contention among vCPU threads on this lock. This should not + // be a problem currently, since we mainly access this when we are setting up devices. + // We might want to do some profiling to ensure this does not become a bottleneck. self.pci_bus .as_ref() .lock() @@ -195,6 +198,9 @@ impl PciConfigIo { return None; } + // NOTE: Potential contention among vCPU threads on this lock. This should not + // be a problem currently, since we mainly access this when we are setting up devices. + // We might want to do some profiling to ensure this does not become a bottleneck. let pci_bus = self.pci_bus.as_ref().lock().unwrap(); if let Some(d) = pci_bus.devices.get(&(device as u32)) { let mut device = d.lock().unwrap();