virtio-mem support

[Manciukic]

What

The main goal of virtio-mem is to allow for dynamic resizing of virtual machine memory. virtio-mem provides a flexible, cross-architecture memory hot(un)plug solution that avoids many limitations imposed by existing technologies, architectures, and interfaces.

How

API

Firecracker will present a new API endpoint, /hotplug/memory, where the following operations are defined:

• PUT { region_size, block_size?, slot_size? } : (before boot) adds a virtio-mem device to the VM with the given region_size
• PATCH { requested_size }: (after boot) changes the requested_size of the virtio-mem device to plug or unplug memory
• GET -> { region_size, requested_size, plugged_size, block_size } : (after boot) returns the current state of the device. In particular, this allows FC users to check the progress of the hot(un)plugging from the guest

Requirements

Firecracker will guarantee the following:

• memory that was never plugged is not accessible by the guest
• memory that is unplugged is freed from the host at block granularity
• weak unplugged memory protection: an unplugged memory area can still be accesses by a misbehaving guest, causing new anonymous pages to be allocated. This risk needs to be mitigated by FC users using cgroups memory limits.
• unplugging the entire memory should make it inaccessible by the guest (this is the only case in which unplugged memory is protected)

Design Decisions

• Firecracker will dedicate a static memory region as the hotpluggable region, starting from 512GB (after the MMIO64 gap)
• The maximum size of the hotpluggable memory is configured by the user at boot and cannot be changed afterwards
• Physical memory blocks are grouped into slots which are dynamically plugged or unplugged when required (like QEMU's dynamicslots option. The block and slot sizes are configurable by users and will default to 2MB and 128MB respectively.

Compatibility with other features

• full snapshot: we will need to add logic to skip over unplugged slots.
• dirty page tracking: slots plugged into kvm would be tracked by KVM. Unplugged slots don’t need to be tracked. Only minor changes are required to handle the unplugged slots.
• huge pages: the only requirement is that block size is >=2MB. No changes.
• vhost-user: as memory is mapped ahead of time, the backend already has all the information. No changes.
• uffd: same as vhost-user. No changes.
• balloon: no issue as vhost-user. No changes.
• secret hiding: we don’t foresee any issue

Development

Proof of Concept

A proof of concept is available at https://github.com/Manciukic/firecracker/tree/poc/virtio-pci-mem.

From an inital analysis we found that:

• hotplugging 1GB takes 30ms on m6i and 55ms on m6g.
• hotplugging 16GB takes 250ms on m6i and 300ms on m6g.
• dynamic memory slots don’t influence the hot-plugging performance, but heavily impact the hot-(un)-plug performance
  • mitigation: we will let customers decide whether they want to favor unplug performance over unplugged memory protection by letting them change slot_size.

Feature branch

We will develop the new feature in the feature/virtio-mem branch, which will be based on top of feature/pcie initially.