Add Support for Migration Statistics API Call #43

phip1611 · 2025-11-25T17:07:56Z

This extends the internal API with a vm_progress function and adds a vm.migration-progress HTTP endpoint including support in ch-remote via ch-remote migration-process to query the latest migration progress.

The two major pre-requisites were:

asynchronize the internal API vmm: live migration asynchronization (dedicated thread) #24
asynchronize the HTTP API Make HTTP Server multi threaded #52

Now, it is possible to get information about ongoing migrations. The most interesting part is the pre-copy phase. The first version is rather coarse-grained with one update per memory iteration. More to follow.

Steps Before Merge

test it locally using ch-remote
add libvirt-tests testcase and verify everything works
deploy it on a node in SAP land and see if it works

phip1611 · 2025-11-28T08:50:31Z

vm-migration/src/progress.rs

+        /// The final memory transmission info.
+        memory_info: MemoryTransmissionInfo,
+    },
+    Failed {


currently I don't like that this phase is part of this enum. Instead, on the top level we should have

Ongoing(InnerStateA), Cancelled(InnerStateB), Failed(InnerStateC)

tpressure · 2025-11-28T08:50:45Z

vm-migration/src/progress.rs

+    pub memory_bytes_transmitted: u64,
+    pub memory_pages_4k_transmitted: u64,
+    pub memory_bytes_remaining_iteration: u64,
+    pub memory_bytes_remaining_total: u64,


qemu also reports the dirty rate. Do we want to report this too?

what's also missing is some sort of status (migration in-progess, migration-finished, ...)

emu also reports the dirty rate. Do we want to report this too?

100%, sure. I just forgot it here

migration in-progess, migration-finished

I have it already, but I want to refactor it to make it more promiment. See my comment #43 (review)

phip1611 · 2025-12-09T13:04:59Z

src/bin/ch-remote.rs

+            let clear = matches
+                .subcommand_matches("migration-progress")
+                .unwrap()
+                .get_one::<bool>("clear")


I think we can completely nuke that clear complexity - there is no problem with keeping the old state around. As soon as a new migration starts, the state is overwritten anyway

phip1611 · 2025-12-09T13:05:17Z

vmm/src/api/mod.rs

 }

+#[derive(Clone, Deserialize, Serialize, Debug)]
+pub struct VmMigrationProgressData {


I think we can completely nuke that clear complexity - there is no problem with keeping the old state around. As soon as a new migration starts, the state is overwritten anyway

phip1611 · 2026-01-12T16:35:00Z

vmm/src/lib.rs

+        vm.complete_migration()?;
+
+        // Give management software a chance to fetch the migration state.
+        info!("Sleeping five seconds before shutting off.");


TODO remove this before merge

phip1611 · 2026-01-12T16:35:04Z

vmm/src/lib.rs

+        info!("Sleeping five seconds before shutting off.");
+        // TODO right now, the http-server is single-threaded and the blocking
+        // start-migration API call will block other requests here.
+        thread::sleep(Duration::from_secs(5));


TODO remove this before merge

phip1611 · 2026-01-13T07:40:53Z

vm-migration/src/progress.rs

+/// [live-migration protocol]: super::protocol
+#[derive(Clone, Debug, serde::Serialize, serde::Deserialize)]
+pub struct MigrationProgressAndStatus {
+    /// UNIX timestamp of the start of the live-migration process.


please note that this structure will be public API for ever and once we reploy it, it will be hard to change

phip1611 · 2026-01-13T07:41:21Z

vm-migration/src/progress.rs

+///
+/// [live-migration protocol]: super::protocol
+#[derive(Clone, Debug, serde::Serialize, serde::Deserialize)]
+pub struct MigrationProgressAndStatus {


should I add a version parameter here? Similar to live migration which also has (or will have?) a version?

I think that would be reasonable.

On-behalf-of: SAP [email protected] Signed-off-by: Philipp Schuster <[email protected]>

This extends the internal API for an `vm_migration` API call to query information about an ongoing live-migration. This is a crucial feature to enable production-ready live-migration at large-scale deployments with corresponding monitoring. On-behalf-of: SAP [email protected] Signed-off-by: Philipp Schuster <[email protected]>

This adds the HTTP endpoint to export ongoing VM live-migration progress. This work was made possible because of the following fundamental prerequisites: - internal API was made async - http thread was made async This way, one can send requests to fetch the latest state without blocking anywhere. On-behalf-of: SAP [email protected] Signed-off-by: Philipp Schuster <[email protected]>

The first version has the limitation that we populate the latest snapshot once per memory iteration, although this is the most interesting part by far. In a follow-up, we can make this more fine-grained. On-behalf-of: SAP [email protected] Signed-off-by: Philipp Schuster <[email protected]>

On-behalf-of: SAP [email protected] Signed-off-by: Philipp Schuster <[email protected]>

olivereanderson

Thanks for your work on this!

I wish the PR description, or some other form of documentation, explained the intended use cases for this feature.

Is the main use case debugging slow/hanging live migrations, or is this intended to be used to collect metrics for every live migration which will then be used to provide statistical insights? If it is the latter, it would be good to document (somewhere) the compatibility story with standard metrics services like Prometheus. I think @snue kowns a thing or two about such topics 🙂

The metrics crate seems to be the metrics analogue of log/tracing and it would be good to know why that isn't a good fit for the task(s) addressed by this PR. I am assuming that there is some context I am missing and that it is quite possible that I have some misconceptions about the problems you are trying to address.

Otherwise I would just like to say that the code looks good and it was easy to follow the implementation for the most part.

olivereanderson · 2026-01-14T07:13:02Z

vm-migration/src/progress.rs

+    pub memory_bytes_remaining_iteration: u64,
+    /// The amount of transmitted 4k pages.
+    pub memory_pages_4k_transmitted: u64,
+    /// The amount of remaining 4k pages for this iteration.
+    pub memory_pages_4k_remaining_iteration: u64,


Question: What about larger pages?

VMM + KVM transparently splits huge-pages for a migration. I'm however not sure if this also applies to read-only pages. We should discuss this again with Thomas

olivereanderson · 2026-01-14T07:18:54Z

vm-migration/src/progress.rs

+pub enum MigrationProgressState {
+    /// The migration has been cancelled.
+    Cancelled {
+        /// The latest memory transmission info, if any.
+        memory_transmission_info: MemoryTransmissionInfo,
+    },
+    /// The migration has failed.
+    Failed {
+        /// The last memory transmission info, if any.
+        memory_transmission_info: MemoryTransmissionInfo,
+        /// Stringified error.
+        error_msg: String,
+        /// Debug-stringified error.
+        error_msg_debug: String,
+        // TODO this is very tricky because I need clone()
+        // error: Box<dyn Error>,
+    },
+    /// The migration has finished successfully.
+    Finished {
+        /// The last memory transmission info, if any.
+        memory_transmission_info: MemoryTransmissionInfo,
+    },
+    /// The migration is ongoing.
+    Ongoing {
+        phase: MigrationPhase,
+        memory_transmission_info: MemoryTransmissionInfo,
+        /// Percent in range `0..=100`.
+        vcpu_throttle_percent: u8,
+    },
+}


I see that every variant has a field of type MemoryTransmissionInfo. Maybe it would make sense to take that out of the enum?
You could do that either with a wrapper struct, or just work with a tuple e.g.:

pub type MigrationProgressInfo = (MigrationProgressState, MigrationProgressInfo)`;

Let me think about this again!

olivereanderson · 2026-01-14T07:19:43Z

vm-migration/src/progress.rs

+    fn memory_transmission_info(&self) -> MemoryTransmissionInfo {
+        match self {
+            MigrationProgressState::Cancelled {
+                memory_transmission_info,
+                ..
+            } => *memory_transmission_info,
+            MigrationProgressState::Failed {
+                memory_transmission_info,
+                ..
+            } => *memory_transmission_info,
+            MigrationProgressState::Finished {
+                memory_transmission_info,
+                ..
+            } => *memory_transmission_info,
+            MigrationProgressState::Ongoing {
+                memory_transmission_info,
+                ..
+            } => *memory_transmission_info,
+        }
+    }


See my comment above with regards to taking MemoryTransmissionInfo out of the enum.

Let me think about this again! I iterates multiple times over the design. It might also makes sense to move that out of the enum at this place, yes

olivereanderson · 2026-01-14T07:27:07Z

vm-migration/src/progress.rs

+        match self {
+            MigrationProgressState::Ongoing {
+                vcpu_throttle_percent,
+                ..
+            } => Some(*vcpu_throttle_percent),
+            _ => None,
+        }


I think if let would be more idiomatic here. Something like this:

Suggested change

match self {

MigrationProgressState::Ongoing {

vcpu_throttle_percent,

..

} => Some(*vcpu_throttle_percent),

_ => None,

}

if let Self::Ongoing{vcpu_throttle_percent, ..} = self {

Some(*vcpu_throttle_percent)

} else {

None

}

olivereanderson · 2026-01-14T07:29:06Z

vm-migration/src/progress.rs

+fn current_unix_timestamp_ms() -> u64 {
+    SystemTime::now()
+        .duration_since(UNIX_EPOCH)
+        .unwrap()


Consider using expect instead of unwrap in this case.

olivereanderson · 2026-01-14T07:44:52Z

vmm/src/api/mod.rs

The commit message contains:

an vm_migration API call

but it should be "a vm_migration API call".

olivereanderson · 2026-01-14T07:47:48Z

vmm/src/api/mod.rs

+    /// The progress of a possibly ongoing live migration.
+    VmMigrationProgress(Box<Option<MigrationProgressAndStatus>>),


I guess the None variant will only ever be present if the API is called before a live migration has started?

olivereanderson · 2026-01-14T07:58:00Z

vmm/src/api/http/http_endpoint.rs

+            {
+                Ok(info) => {
+                    let mut response = Response::new(Version::Http11, StatusCode::OK);
+                    let info_serialized = serde_json::to_string(&info).unwrap();


Nit: It would be good to either replace unwrap with expect or to add a comment explaining why it is OK to unwrap in this case.

vmm/src/lib.rs

vm-migration/src/progress.rs

olivereanderson · 2026-01-14T12:52:26Z

vmm/src/lib.rs

+            if let Some(snapshot) = lock.as_ref() {
+                match snapshot.state {
+                    MigrationProgressState::Ongoing { .. } => {
+                        // if this fails, we made a programming error in our state handling
+                        panic!("migration already ongoing");
+                    }
+                    MigrationProgressState::Cancelled { .. } => {}
+                    MigrationProgressState::Failed { .. } => {}
+                    MigrationProgressState::Finished { .. } => {}
+                }
+            }


Nit: Something like this might be simpler:

Suggested change

if let Some(snapshot) = lock.as_ref() {

match snapshot.state {

MigrationProgressState::Ongoing { .. } => {

// if this fails, we made a programming error in our state handling

panic!("migration already ongoing");

}

MigrationProgressState::Cancelled { .. } => {}

MigrationProgressState::Failed { .. } => {}

MigrationProgressState::Finished { .. } => {}

}

}

if lock.as_ref().is_some_and(|snapshot| matches!(snapshot, MigrationProgressState::Ongoing{..}) {

// if this fails, we made a programming error in our state handling

panic!("migration already ongoing");

}

phip1611 self-assigned this Nov 25, 2025

phip1611 force-pushed the poc-migration-statistics branch from e29356d to 4101b11 Compare November 28, 2025 08:46

phip1611 commented Nov 28, 2025

View reviewed changes

tpressure reviewed Nov 28, 2025

View reviewed changes

phip1611 force-pushed the poc-migration-statistics branch 5 times, most recently from f49e577 to fdf5858 Compare December 9, 2025 11:42

phip1611 commented Dec 9, 2025

View reviewed changes

phip1611 force-pushed the poc-migration-statistics branch from f612cf6 to e9a3321 Compare December 15, 2025 14:24

phip1611 force-pushed the poc-migration-statistics branch from e9a3321 to ecb5f45 Compare January 8, 2026 14:31

phip1611 changed the base branch from gardenlinux-v48 to gardenlinux January 8, 2026 14:32

phip1611 force-pushed the poc-migration-statistics branch 3 times, most recently from e6c80dd to fe8cd0d Compare January 12, 2026 16:30

phip1611 marked this pull request as ready for review January 12, 2026 16:30

phip1611 changed the title ~~WIP XXX Migration Statistics~~ Add Support for Migration Statistics API Call Jan 12, 2026

phip1611 requested review from amphi and olivereanderson January 12, 2026 16:33

phip1611 commented Jan 12, 2026

View reviewed changes

phip1611 marked this pull request as draft January 12, 2026 16:35

phip1611 force-pushed the poc-migration-statistics branch from fe8cd0d to a87fe95 Compare January 12, 2026 16:41

phip1611 requested a review from tpressure January 12, 2026 16:42

phip1611 force-pushed the poc-migration-statistics branch from a87fe95 to 6d68c0c Compare January 12, 2026 16:43

phip1611 commented Jan 13, 2026

View reviewed changes

phip1611 force-pushed the poc-migration-statistics branch from 6d68c0c to c934e0e Compare January 13, 2026 10:36

phip1611 added 7 commits January 13, 2026 11:41

vmm: silence some messages in http_endpoint

c881f0f

On-behalf-of: SAP [email protected] Signed-off-by: Philipp Schuster <[email protected]>

vm-migration: prepare progress types for new API endpoint

9caf160

On-behalf-of: SAP [email protected] Signed-off-by: Philipp Schuster <[email protected]>

ch-remote: add migration-progress command

152713c

On-behalf-of: SAP [email protected] Signed-off-by: Philipp Schuster <[email protected]>

vmm: remove unnecessary Result

78699eb

On-behalf-of: SAP [email protected] Signed-off-by: Philipp Schuster <[email protected]>

phip1611 force-pushed the poc-migration-statistics branch from c934e0e to 78699eb Compare January 13, 2026 10:42

olivereanderson reviewed Jan 14, 2026

View reviewed changes

		/// The progress of a possibly ongoing live migration.
		VmMigrationProgress(Box<Option<MigrationProgressAndStatus>>),

Add Support for Migration Statistics API Call #43

Are you sure you want to change the base?

Add Support for Migration Statistics API Call #43

Uh oh!

Conversation

phip1611 commented Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Steps Before Merge

Uh oh!

phip1611 Nov 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

phip1611 Nov 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

olivereanderson left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

phip1611 Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

phip1611 commented Nov 25, 2025 •

edited

Loading

phip1611 Nov 28, 2025 •

edited

Loading

phip1611 Nov 28, 2025 •

edited

Loading

olivereanderson left a comment •

edited

Loading

phip1611 Jan 14, 2026 •

edited

Loading