[virtio-pmem] Implementation #5463

ShadowCurse · 2025-10-03T18:49:15Z

Changes

Add virtio-pmem device support.

License Acceptance

By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache 2.0 license. For more information on following Developer
Certificate of Origin and signing off your commits, please check
CONTRIBUTING.md.

PR Checklist

This functionality cannot be added in rust-vmm.

codecov · 2025-10-03T18:58:50Z

Codecov Report

❌ Patch coverage is 69.88636% with 159 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.56%. Comparing base (eb05db9) to head (d8c695a).

Files with missing lines	Patch %	Lines
src/vmm/src/devices/virtio/pmem/device.rs	60.66%	83 Missing ⚠️
src/vmm/src/devices/virtio/pmem/event_handler.rs	23.07%	40 Missing ⚠️
src/vmm/src/builder.rs	77.14%	8 Missing ⚠️
src/vmm/src/rpc_interface.rs	0.00%	8 Missing ⚠️
src/firecracker/src/api_server/request/pmem.rs	77.27%	5 Missing ⚠️
src/vmm/src/devices/virtio/pmem/metrics.rs	85.29%	5 Missing ⚠️
src/vmm/src/resources.rs	58.33%	5 Missing ⚠️
src/vmm/src/vmm_config/pmem.rs	93.02%	3 Missing ⚠️
src/vmm/src/device_manager/persist.rs	96.55%	1 Missing ⚠️
src/vmm/src/devices/virtio/pmem/persist.rs	95.23%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5463      +/-   ##
==========================================
- Coverage   82.79%   82.56%   -0.23%     
==========================================
  Files         263      269       +6     
  Lines       27223    27736     +513     
==========================================
+ Hits        22538    22899     +361     
- Misses       4685     4837     +152

Flag	Coverage Δ
5.10-m5n.metal	`82.69% <69.88%> (-0.28%)`	⬇️
5.10-m6a.metal	`81.95% <69.88%> (-0.28%)`	⬇️
5.10-m6g.metal	`79.36% <69.88%> (-0.21%)`	⬇️
5.10-m6i.metal	`82.69% <69.88%> (-0.28%)`	⬇️
5.10-m7a.metal-48xl	`81.94% <69.88%> (-0.28%)`	⬇️
5.10-m7g.metal	`79.36% <69.88%> (-0.21%)`	⬇️
5.10-m7i.metal-24xl	`82.66% <69.88%> (-0.28%)`	⬇️
5.10-m7i.metal-48xl	`82.66% <69.88%> (-0.27%)`	⬇️
5.10-m8g.metal-24xl	`79.36% <69.88%> (-0.21%)`	⬇️
5.10-m8g.metal-48xl	`79.36% <69.88%> (-0.21%)`	⬇️
6.1-m5n.metal	`82.72% <69.88%> (-0.28%)`	⬇️
6.1-m6a.metal	`81.99% <69.88%> (-0.27%)`	⬇️
6.1-m6g.metal	`79.36% <69.88%> (-0.22%)`	⬇️
6.1-m6i.metal	`82.72% <69.88%> (-0.28%)`	⬇️
6.1-m7a.metal-48xl	`81.97% <69.88%> (-0.28%)`	⬇️
6.1-m7g.metal	`79.36% <69.88%> (-0.22%)`	⬇️
6.1-m7i.metal-24xl	`82.73% <69.88%> (-0.28%)`	⬇️
6.1-m7i.metal-48xl	`82.73% <69.88%> (-0.29%)`	⬇️
6.1-m8g.metal-24xl	`79.36% <69.88%> (-0.22%)`	⬇️
6.1-m8g.metal-48xl	`79.36% <69.88%> (-0.22%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Manciukic

we should update docs/device-api.md.
changelog entry
any performance tests? we could check how fast we can read or write the entire pmem or maybe we can integrate it with the block tests using fio

src/vmm/src/devices/virtio/pmem/device.rs

Manciukic · 2025-10-07T10:34:37Z

src/vmm/src/devices/virtio/pmem/device.rs

+        }
+    }
+
+    fn write_config(&mut self, _offset: u64, _data: &[u8]) {}


we could log unexpected attempts to write

we don't log such writes in any device, so don't think we should do it here.

src/vmm/src/vstate/vm.rs

src/vmm/src/devices/virtio/pmem/persist.rs

docs/pmem.md

Manciukic · 2025-10-07T10:56:57Z

docs/pmem.md

+         \"root_device\": true,
+         \"read_only\": false
+    }"
+```


we should probably mention snapshot/restore behaviour as well

and also security considerations about sharing memory (which we do not recommend).

We can also mention performance considerations: ie that even though pages are in memory, the guest still needs to exit to the kernel to set up the pagetable mappings. Using hugetlbfs to back the file would be faster (but will consume memory).

added snapshot, security and performance sections

Regarding hugetlbfs: its main usage is to make sharable memory region and it does not support writes (e.g. you cannot copy file to the hugetblfs, only create and resize them). So I don't think we need to explicitly mention it as a backing for pmem since the main use we expect is to use actual files as backing storage.

yeah, we can leave that aside I believe as long as we mention faulting as a cost to pay. What I actually had in mind was tmpfs with huge=always that internally uses THP to back the files in memory,

added mention of tmpfs and hugetblfs

src/vmm/src/devices/virtio/pmem/device.rs

tests/integration_tests/functional/test_pmem.py

Manciukic · 2025-10-07T11:10:48Z

docs/pmem.md

+> `DAX` support is not uniform for all file systems. Check the documentation for
+> the file system you want to use before enabling `DAX`.


it works on ext4, right? does it need any specific options (ie 4096 block size) or just works?

yes, fs need to have block size to be equal to host page size. Added a link to the kernel docs about DAX support.

bchalios

Went half way through. Here's an initial set of comments

src/vmm/src/devices/virtio/pmem/device.rs

src/vmm/src/vmm_config/pmem.rs

bchalios · 2025-10-07T09:51:12Z

src/vmm/src/vmm_config/pmem.rs

+
+    /// Adds an existing pmem device in the builder.
+    pub fn add_device(&mut self, device: Arc<Mutex<Pmem>>) {
+        self.devices.push(device);


shouldn't we add this to the corresponding index?

In any case, could you also add a unit test for this one as well?

unit test for what?

a unit test which ensures that add_device does what you think it's doing. But back to my initial question, shouldn't add_device add device in the correct place in self.devices, according to the device index?

a unit test which ensures that add_device does what you think it's doing.

is a single line: self.devices.push(device); so elusive to need a unit test?

order of deivces onlt matters during VM boot if any of them is a root device. Otherwise order is not important. The add_device only used during snapshot restore and even in that case the order is preserved since configs for devices are stored in the same order as they are during VM boot (with configs function)

msync is used by virtio-pmem device to trigger sync of mmaped file content to the underlying file. Signed-off-by: Egor Lazarchuk <[email protected]>

Add implementations of device, event handling, metrics. Add device config and builder types for API use. Signed-off-by: Egor Lazarchuk <[email protected]>

Update VmResources type with virtio-pmem configuration field to allow virtio-pmem devices be configured through config files and later through API calls. Signed-off-by: Egor Lazarchuk <[email protected]>

Both virtio-block and virtio-pmem can act as root devices for a VM. Add a check to prevent specifing more than 1 root device for a VM. Signed-off-by: Egor Lazarchuk <[email protected]>

Add /pmem/id PUT request for virtio-pmem configuration. Add corresponding metrics. Signed-off-by: Egor Lazarchuk <[email protected]>

Virtio-pmem devices need to allocate a memory region in guest physical memory. The safe place to do this is past 64bit MMIO region. Signed-off-by: Egor Lazarchuk <[email protected]>

Add a counter for KVM slot ids into VmCommon struct. This is done because virtio-pmem device needs to obtain it's KVM slot id independently from number of slots in GuestMemoryMmap. Signed-off-by: Egor Lazarchuk <[email protected]>

Add methods to attach virtio-pmem devices to Vmm. Add methods to create KVM memory slot for virtio-pmem devices. Signed-off-by: Egor Lazarchuk <[email protected]>

Add logic to store and restore virtio-pmem device information in a snapshot. Signed-off-by: Egor Lazarchuk <[email protected]>

Add functional and API tests for virtio-pmem device and its configuration fields Signed-off-by: Egor Lazarchuk <[email protected]>

Expose virtio-pmem metrics in the logger, so they are exported in metrics.json. Update integration tests to expect new metrics. Signed-off-by: Egor Lazarchuk <[email protected]>

Add description of pmem APIs in swagger file and device-api.md Signed-off-by: Egor Lazarchuk <[email protected]>

Add new document about virtio-pmem configuration and usage. Signed-off-by: Egor Lazarchuk <[email protected]>

Add a note about addition of virtio-pmem device. Signed-off-by: Egor Lazarchuk <[email protected]>

ShadowCurse self-assigned this Oct 3, 2025

ShadowCurse force-pushed the virtio_pmem branch 2 times, most recently from d8f9547 to 5970613 Compare October 6, 2025 11:01

ShadowCurse marked this pull request as ready for review October 6, 2025 12:57

ShadowCurse added Status: Awaiting review Indicates that a pull request is ready to be reviewed Type: Enhancement Indicates new feature requests labels Oct 6, 2025

ShadowCurse force-pushed the virtio_pmem branch 2 times, most recently from a8bedbb to 1d2aeb2 Compare October 6, 2025 14:18

ShadowCurse requested review from bchalios and Manciukic October 6, 2025 14:22

ShadowCurse force-pushed the virtio_pmem branch from 1d2aeb2 to 7d83503 Compare October 7, 2025 08:43

Manciukic reviewed Oct 7, 2025

View reviewed changes

ShadowCurse force-pushed the virtio_pmem branch from 7d83503 to efd93ea Compare October 7, 2025 12:48

ShadowCurse requested review from xmarcalx, kalyazin and pb8o as code owners October 7, 2025 12:48

ShadowCurse force-pushed the virtio_pmem branch 2 times, most recently from 9a554b4 to 4b4779b Compare October 7, 2025 12:59

bchalios requested changes Oct 7, 2025

View reviewed changes

ShadowCurse force-pushed the virtio_pmem branch 10 times, most recently from 7eff29f to 4a19190 Compare October 8, 2025 12:19

ShadowCurse force-pushed the virtio_pmem branch 6 times, most recently from 4f309d6 to de6031e Compare October 8, 2025 16:36

ShadowCurse added 14 commits October 8, 2025 17:36

chore(virtio-pmem): add msync syscall to seccomp filters

77b4e72

msync is used by virtio-pmem device to trigger sync of mmaped file content to the underlying file. Signed-off-by: Egor Lazarchuk <[email protected]>

feat(virtio-pmem): add device configs and implementation

3d2d36f

Add implementations of device, event handling, metrics. Add device config and builder types for API use. Signed-off-by: Egor Lazarchuk <[email protected]>

feat(virtio-pmem): add device to the VmResources

cf97469

Update VmResources type with virtio-pmem configuration field to allow virtio-pmem devices be configured through config files and later through API calls. Signed-off-by: Egor Lazarchuk <[email protected]>

feat: add a check for a single root device

fe34a9b

Both virtio-block and virtio-pmem can act as root devices for a VM. Add a check to prevent specifing more than 1 root device for a VM. Signed-off-by: Egor Lazarchuk <[email protected]>

feat(virtio-pmem): add API requests parsing to Firecracker

3e56247

Add /pmem/id PUT request for virtio-pmem configuration. Add corresponding metrics. Signed-off-by: Egor Lazarchuk <[email protected]>

feat: add allocator for past 64bit memory region

675d0a2

Virtio-pmem devices need to allocate a memory region in guest physical memory. The safe place to do this is past 64bit MMIO region. Signed-off-by: Egor Lazarchuk <[email protected]>

feat: add a counter for KVM slots

13f5aba

Add a counter for KVM slot ids into VmCommon struct. This is done because virtio-pmem device needs to obtain it's KVM slot id independently from number of slots in GuestMemoryMmap. Signed-off-by: Egor Lazarchuk <[email protected]>

feat(virtio-pmem): add device to the Vmm

88303d8

Add methods to attach virtio-pmem devices to Vmm. Add methods to create KVM memory slot for virtio-pmem devices. Signed-off-by: Egor Lazarchuk <[email protected]>

feat(virtio-pmem): add snapshot support

09e6224

Add logic to store and restore virtio-pmem device information in a snapshot. Signed-off-by: Egor Lazarchuk <[email protected]>

feat(virtio-pmem): add integration tests

01fc4b2

Add functional and API tests for virtio-pmem device and its configuration fields Signed-off-by: Egor Lazarchuk <[email protected]>

feat(virtio-pmem): export device metrics

9d91e2b

Expose virtio-pmem metrics in the logger, so they are exported in metrics.json. Update integration tests to expect new metrics. Signed-off-by: Egor Lazarchuk <[email protected]>

chore(virtio-pmem): document new APIs

01ab1a4

Add description of pmem APIs in swagger file and device-api.md Signed-off-by: Egor Lazarchuk <[email protected]>

doc(virtio-pmem): add documentation

3479b63

Add new document about virtio-pmem configuration and usage. Signed-off-by: Egor Lazarchuk <[email protected]>

chore: update CHANGELOG

d8c695a

Add a note about addition of virtio-pmem device. Signed-off-by: Egor Lazarchuk <[email protected]>

ShadowCurse force-pushed the virtio_pmem branch from de6031e to d8c695a Compare October 8, 2025 16:36

		> `DAX` support is not uniform for all file systems. Check the documentation for
		> the file system you want to use before enabling `DAX`.

[virtio-pmem] Implementation #5463

Are you sure you want to change the base?

[virtio-pmem] Implementation #5463

Conversation

ShadowCurse commented Oct 3, 2025

Changes

License Acceptance

PR Checklist

Uh oh!

codecov bot commented Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Manciukic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bchalios left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codecov bot commented Oct 3, 2025 •

edited

Loading