Fix KVM incremental volume snapshot migration between secondary storages #12086

hsato03 · 2025-11-17T20:18:48Z

Description

ACS allows data to be migrated from one secondary storage to another; however, this process disregards incremental volume snapshots created using the KVM hypervisor.

Since then, the migration process has been extended to also allow the migration of these resources.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
Build/CI
Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

N°	Test	API	Result	Expected result (Y/N)
1	Migrate a single incremental volume snapshot	`migrateResourceToAnotherSecondaryStorage`	The snapshot, along with its checkpoint file, was migrated to the destination storage, and its child snapshot, which remained in the old storage, was rebased onto the snapshot in the new storage	Y
2	Migrate the entire chain of incremental volume snapshots to another secondary storage	`migrateSecondaryStorageData`	All snapshots were migrated and the chain was correctly updated with the files from the new storage	Y
3	Migrate a chain of incremental volume snapshots split between 2 secondary storages	`migrateSecondaryStorageData`	The chain was updated based on existing snapshots in the destination storage and the new file from the migrated snapshots	Y

How did you try to break this feature and the system with this change?

codecov · 2025-11-17T20:28:39Z

Codecov Report

❌ Patch coverage is 0.72202% with 275 lines in your changes missing coverage. Please review.
✅ Project coverage is 17.56%. Comparing base (6dc259c) to head (a1c64e3).
⚠️ Report is 23 commits behind head on main.

Files with missing lines	Patch %	Lines
...apshotsBetweenSecondaryStoragesCommandWrapper.java	1.08%	91 Missing ⚠️
...tack/engine/orchestration/StorageOrchestrator.java	0.00%	87 Missing ⚠️
...ack/engine/orchestration/DataMigrationUtility.java	0.00%	31 Missing ⚠️
...grateSnapshotsBetweenSecondaryStoragesCommand.java	0.00%	24 Missing ⚠️
...he/cloudstack/storage/snapshot/SnapshotObject.java	0.00%	17 Missing ⚠️
...java/org/apache/cloudstack/utils/qemu/QemuImg.java	0.00%	12 Missing ⚠️
.../MigrateBetweenSecondaryStoragesCommandAnswer.java	0.00%	9 Missing ⚠️
...tack/storage/snapshot/SnapshotDataFactoryImpl.java	0.00%	3 Missing ⚠️
.../hypervisor/kvm/storage/LibvirtStorageAdaptor.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##               main   #12086    +/-   ##
==========================================
  Coverage     17.55%   17.56%            
- Complexity    15535    15549    +14     
==========================================
  Files          5911     5916     +5     
  Lines        529359   529690   +331     
  Branches      64655    64689    +34     
==========================================
+ Hits          92949    93022    +73     
- Misses       425952   426211   +259     
+ Partials      10458    10457     -1

Flag	Coverage Δ
uitests	`3.58% <ø> (+<0.01%)`	⬆️
unittests	`18.62% <0.72%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

hsato03 · 2025-11-17T20:34:06Z

@blueorangutan package

Copilot

Pull request overview

This PR fixes a bug where KVM incremental volume snapshots were not being migrated when data was transferred between secondary storages. The implementation extends the existing migration process to handle KVM-specific incremental snapshots and their checkpoint files, ensuring the entire snapshot chain is correctly migrated and rebased.

Key changes:

Added support for migrating KVM incremental snapshot chains with their checkpoint files
Implemented single-threaded execution per zone for KVM incremental snapshot migrations to maintain chain integrity
Added new command classes and wrapper for handling snapshot migration between secondary storages

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
StorageManagerImpl.java	Added new configuration key for agent migration timeout
QemuImg.java	Enhanced convert method to support backing file specification during conversion
LibvirtStorageAdaptor.java	Updated convert call to include new backing file parameter
LibvirtMigrateSnapshotsBetweenSecondaryStoragesCommandWrapper.java	New handler for executing snapshot migration commands on KVM agents
SnapshotObject.java	Added methods to retrieve parent and child snapshot chains
SnapshotDataFactoryImpl.java	Sets KVM incremental snapshot flag based on checkpoint path presence
StorageOrchestrator.java	Core migration orchestration logic for handling KVM incremental snapshots
DataMigrationUtility.java	Updated to identify and build KVM incremental snapshot chains for migration
StorageManager.java	Defined new configuration key for migration timeout
SnapshotInfo.java	Added interface methods for retrieving snapshot chain relationships
MigrateSnapshotsBetweenSecondaryStoragesCommand.java	New command class for snapshot migration requests
MigrateBetweenSecondaryStoragesCommandAnswer.java	New answer class for migration command responses

Comments suppressed due to low confidence (1)

plugins/hypervisors/kvm/src/main/java/org/apache/cloudstack/utils/qemu/QemuImg.java:1

[nitpick] Extra whitespace before 'new QemuImageOptions' parameter. Should have consistent single-space separation between parameters.

// Licensed to the Apache Software Foundation (ASF) under one

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

...ns/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/LibvirtStorageAdaptor.java

...hestration/src/main/java/org/apache/cloudstack/engine/orchestration/StorageOrchestrator.java

Co-authored-by: Copilot <[email protected]>

hsato03 · 2025-11-27T21:55:31Z

@blueorangutan package

blueorangutan · 2025-11-27T23:13:53Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 15849

DaanHoogland · 2025-11-28T09:27:05Z

@blueorangutan test

blueorangutan · 2025-11-28T09:28:05Z

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2025-11-28T09:56:08Z

[SF] Trillian Build Failed (tid-14880)

JoaoJandre · 2025-11-28T12:41:17Z

...hestration/src/main/java/org/apache/cloudstack/engine/orchestration/StorageOrchestrator.java

+                    throw new CloudRuntimeException("Unable to migrate KVM incremental snapshots to another secondary storage");
+                }
+            } catch (final OperationTimedoutException | AgentUnavailableException e) {
+                throw new CloudRuntimeException("Error while migrating KVM incremental snapshot chain. Check the logs for more information.", e);


We could inform which snapshot failed.

JoaoJandre · 2025-11-28T12:51:33Z

...isor/kvm/resource/wrapper/LibvirtMigrateSnapshotsBetweenSecondaryStoragesCommandWrapper.java

+                String resourceCurrentPath = imagePool.getLocalPathFor(snapshot.getPath());
+
+                if (imagePoolUrl.equals(srcDataStore.getUrl()) && snapshotsIdToMigrate.contains(snapshot.getId())) {
+                    parentSnapshotPath = copyResourceToDestDataStore(snapshot, resourceCurrentPath, destImagePool, parentSnapshotPath);
+                    parentSnapshotWasMigrated = true;
+                } else {
+                    if (parentSnapshotWasMigrated) {
+                        parentSnapshotPath = rebaseResourceToNewParentPath(resourceCurrentPath, parentSnapshotPath);
+                    } else {
+                        parentSnapshotPath = resourceCurrentPath;
+                    }
+                    parentSnapshotWasMigrated = false;


we could add some logging for these decisions

blueorangutan · 2025-11-29T02:45:23Z

[SF] Trillian test result (tid-14882)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 53722 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr12086-t14882-kvm-ol8.zip
Smoke tests completed. 149 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_list_cpvm_vm	`Failure`	0.04	test_ssvm.py
test_04_cpvm_internals	`Failure`	0.05	test_ssvm.py

Fix KVM incremental snapshot migration between secondary storages

f185688

boring-cyborg bot added component:api component:kvm component:orchestration component:storage labels Nov 17, 2025

apache deleted a comment from blueorangutan Nov 25, 2025

DaanHoogland requested review from JoaoJandre, abh1sar and Copilot November 26, 2025 08:24

Copilot started reviewing on behalf of DaanHoogland November 26, 2025 08:25 View session

Copilot finished reviewing on behalf of DaanHoogland November 26, 2025 08:28

Copilot AI reviewed Nov 26, 2025

View reviewed changes

...ns/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/LibvirtStorageAdaptor.java Outdated Show resolved Hide resolved

...hestration/src/main/java/org/apache/cloudstack/engine/orchestration/StorageOrchestrator.java Outdated Show resolved Hide resolved

Apply suggestions from code review

a1c64e3

Co-authored-by: Copilot <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix KVM incremental volume snapshot migration between secondary storages #12086

Fix KVM incremental volume snapshot migration between secondary storages #12086

hsato03 commented Nov 17, 2025

Uh oh!

codecov bot commented Nov 17, 2025 •

edited

Loading

Uh oh!

hsato03 commented Nov 17, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

hsato03 commented Nov 27, 2025

Uh oh!

blueorangutan commented Nov 27, 2025

Uh oh!

DaanHoogland commented Nov 28, 2025

Uh oh!

blueorangutan commented Nov 28, 2025

Uh oh!

blueorangutan commented Nov 28, 2025

Uh oh!

JoaoJandre Nov 28, 2025

Uh oh!

JoaoJandre Nov 28, 2025

Uh oh!

blueorangutan commented Nov 29, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Fix KVM incremental volume snapshot migration between secondary storages #12086

Are you sure you want to change the base?

Fix KVM incremental volume snapshot migration between secondary storages #12086

Conversation

hsato03 commented Nov 17, 2025

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

codecov bot commented Nov 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

hsato03 commented Nov 17, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

hsato03 commented Nov 27, 2025

Uh oh!

blueorangutan commented Nov 27, 2025

Uh oh!

DaanHoogland commented Nov 28, 2025

Uh oh!

blueorangutan commented Nov 28, 2025

Uh oh!

blueorangutan commented Nov 28, 2025

Uh oh!

JoaoJandre Nov 28, 2025

Choose a reason for hiding this comment

Uh oh!

JoaoJandre Nov 28, 2025

Choose a reason for hiding this comment

Uh oh!

blueorangutan commented Nov 29, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov bot commented Nov 17, 2025 •

edited

Loading