systemvm-template: support on-demand download during setup and registration #11656

shwstppr · 2025-09-17T10:54:54Z

Description

Bundling all hypervisor SystemVM templates in release packages simplifies installs but inflates build time and package size. This change enables downloading templates on demand when they’re not found after package installation. The download path is wired into both cloud-setup-management and the existing SystemVM template registration flow.
cloudstack-setup-management allows the argument --systemvm-templates to download the desired hypervisor system template during management server setup.

For air-gapped environments, a repository URL prefix can be provided to support air-gapped setups: pass --systemvm-templates-repository <URL-prefix> to cloudstack-setup-management, or set system.vm.templates.download.repository=<URL-prefix> in server.properties for post-setup registration. Currently, custom repository should mirror the file structure of the official repository, ie, <BASE_VERSION>/systemvmtemplate-<VERSION>-<ARCH>-<HYPERVISOR>.<FILE_EXTENSION>
If templates are already present (bundled or preseeded), behavior is unchanged and no download is attempted.

Documentation PR: apache/cloudstack-documentation#580

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI
test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

codecov · 2025-09-17T11:04:22Z

Codecov Report

❌ Patch coverage is 62.50000% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 17.56%. Comparing base (bfc4f60) to head (4bd5458).
⚠️ Report is 32 commits behind head on main.

Files with missing lines	Patch %	Lines
...om/cloud/upgrade/SystemVmTemplateRegistration.java	40.00%	4 Missing and 2 partials ⚠️
.../cloudstack/utils/server/ServerPropertiesUtil.java	72.72%	4 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##               main   #11656    +/-   ##
==========================================
  Coverage     17.56%   17.56%            
- Complexity    15540    15541     +1     
==========================================
  Files          5909     5911     +2     
  Lines        529045   529381   +336     
  Branches      64615    64661    +46     
==========================================
+ Hits          92933    93000    +67     
- Misses       425657   425920   +263     
- Partials      10455    10461     +6

Flag	Coverage Δ
uitests	`3.58% <ø> (+<0.01%)`	⬆️
unittests	`18.63% <62.50%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…ration Bundling all hypervisor SystemVM templates in release packages simplifies installs but inflates build time and artifact size. This change enables downloading templates on demand when they’re not found after package installation. The download path is wired into both cloud-setup-management and the existing SystemVM template registration flow. For connected or mirrored environments, a repository URL prefix can be provided to support air-gapped setups: pass --systemvm-templates-repository <URL-prefix> to cloud-setup-management, or set system.vm.templates.download.repository=<URL-prefix> in server.properties for post-setup registration. If templates are already present (bundled or preseeded), behavior is unchanged and no download is attempted. Signed-off-by: Abhishek Kumar <[email protected]>

github-actions · 2025-09-27T03:35:04Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

shwstppr · 2025-09-29T06:24:10Z

@blueorangutan package

blueorangutan · 2025-09-29T06:26:03Z

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-09-29T08:11:50Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 15214

Signed-off-by: Abhishek Kumar <[email protected]>

Copilot

Pull Request Overview

This PR enhances CloudStack's SystemVM template handling by enabling on-demand downloading of templates during setup and registration, reducing package size and build time. The implementation supports both regular and air-gapped environments through configurable repository URLs.

Adds on-demand SystemVM template downloading capability with checksum verification
Refactors template metadata handling to support custom download repositories
Implements centralized server properties utility for configuration management

Reviewed Changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
utils/src/main/java/org/apache/cloudstack/utils/server/ServerPropertiesUtil.java	New utility class for centralized server.properties access with thread-safe caching
plugins/hypervisors/external/src/main/java/org/apache/cloudstack/hypervisor/external/provisioner/ExternalPathPayloadProvisioner.java	Refactored to use new ServerPropertiesUtil instead of duplicated property loading logic
engine/schema/templateConfig.sh	Enhanced template metadata generation with repository URL support and refactored template specification structure
engine/schema/src/test/java/com/cloud/upgrade/SystemVmTemplateRegistrationTest.java	Updated test case to use VMware/arm64 combination instead of KVM/amd64
engine/schema/src/main/java/com/cloud/upgrade/SystemVmTemplateRegistration.java	Added support for custom download repository configuration and URL replacement logic
client/conf/server.properties.in	Added configuration property for custom SystemVM template repository URL
client/bindir/cloud-setup-management.in	Implemented comprehensive template download functionality with metadata parsing and checksum verification

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

client/bindir/cloud-setup-management.in

Signed-off-by: Abhishek Kumar <[email protected]>

shwstppr · 2025-10-13T09:02:26Z

@blueorangutan package

blueorangutan · 2025-10-13T09:04:03Z

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Related apache/cloudstack#11656 Signed-off-by: Abhishek Kumar <[email protected]>

blueorangutan · 2025-10-13T11:28:21Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 15431

shwstppr · 2025-10-27T07:40:26Z

@blueorangutan newpackage

blueorangutan · 2025-10-27T07:42:03Z

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Signed-off-by: Abhishek Kumar <[email protected]>

blueorangutan · 2025-10-27T08:39:02Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✖️ debian ✔️ suse15. SL-JID 15568

shwstppr · 2025-10-27T10:20:20Z

@blueorangutan newpackage

blueorangutan · 2025-10-27T10:22:03Z

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with no systemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-11-10T15:02:23Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 15708

shwstppr · 2025-11-12T06:42:02Z

@blueorangutan newpackage

blueorangutan · 2025-11-12T06:44:05Z

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-11-12T08:28:50Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 15725

client/conf/server.properties.in

client/bindir/cloud-setup-management.in

engine/schema/templateConfig.sh

sureshanaparti

clgtm

sureshanaparti · 2025-11-13T06:38:42Z

@blueorangutan newpackage

blueorangutan · 2025-11-13T06:40:04Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-11-13T07:34:13Z

Packaging result [SF]: ✖️ el8 ✖️ el9 ✔️ debian ✖️ suse15. SL-JID 15740

shwstppr · 2025-11-13T19:24:10Z

@blueorangutan newpackage

blueorangutan · 2025-11-13T19:26:04Z

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-11-13T21:01:32Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 15761

shwstppr · 2025-11-16T07:32:49Z

@blueorangutan test matrix

blueorangutan · 2025-11-16T07:34:04Z

@shwstppr a [SL] Trillian-Jenkins matrix job (EL8 mgmt + EL8 KVM, Ubuntu22 mgmt + Ubuntu22 KVM, EL8 mgmt + VMware 7.0u3, EL9 mgmt + XCP-ng 8.2 ) has been kicked to run smoke tests

blueorangutan · 2025-11-16T08:27:53Z

[SF] Trillian Build Failed (tid-14845)

blueorangutan · 2025-11-16T08:34:48Z

[SF] Trillian Build Failed (tid-14844)

blueorangutan · 2025-11-16T22:37:25Z

[SF] Trillian test result (tid-14842)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 51144 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr11656-t14842-kvm-ol8.zip
Smoke tests completed. 150 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

blueorangutan · 2025-11-17T00:24:50Z

[SF] Trillian test result (tid-14843)
Environment: kvm-ubuntu22 (x2), zone: Advanced Networking with Mgmt server u22
Total time taken: 58009 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr11656-t14843-kvm-ubuntu22.zip
Smoke tests completed. 149 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_list_cpvm_vm	`Failure`	0.04	test_ssvm.py
test_04_cpvm_internals	`Failure`	0.04	test_ssvm.py

blueorangutan · 2025-11-19T01:03:32Z

[SF] Trillian test result (tid-14846)
Environment: vmware-70u3 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 52712 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr11656-t14846-vmware-70u3.zip
Smoke tests completed. 130 look OK, 1 have errors, 19 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_vpn_usage	`Error`	1.10	test_usage.py
all_test_vm_deployment_planner	`Skipped`	---	test_vm_deployment_planner.py
all_test_vm_life_cycle	`Skipped`	---	test_vm_life_cycle.py
all_test_vm_lifecycle_unmanage_import	`Skipped`	---	test_vm_lifecycle_unmanage_import.py
all_test_vm_lifecycle_unmanage_kvm_import	`Skipped`	---	test_vm_lifecycle_unmanage_kvm_import.py
all_test_vm_lifecycle_with_snapshot_or_volume	`Skipped`	---	test_vm_lifecycle_with_snapshot_or_volume.py
all_test_vm_schedule	`Skipped`	---	test_vm_schedule.py
all_test_vm_snapshot_kvm	`Skipped`	---	test_vm_snapshot_kvm.py
all_test_vm_snapshots	`Skipped`	---	test_vm_snapshots.py
all_test_vm_strict_host_tags	`Skipped`	---	test_vm_strict_host_tags.py
all_test_vnf_templates	`Skipped`	---	test_vnf_templates.py
all_test_volumes	`Skipped`	---	test_volumes.py
all_test_vpc_ipv6	`Skipped`	---	test_vpc_ipv6.py
all_test_vpc_redundant	`Skipped`	---	test_vpc_redundant.py
all_test_vpc_router_nics	`Skipped`	---	test_vpc_router_nics.py
all_test_vpc_vpn	`Skipped`	---	test_vpc_vpn.py
all_test_webhook_delivery	`Skipped`	---	test_webhook_delivery.py
all_test_webhook_lifecycle	`Skipped`	---	test_webhook_lifecycle.py
all_test_host_maintenance	`Skipped`	---	test_host_maintenance.py
all_test_hostha_kvm	`Skipped`	---	test_hostha_kvm.py

shwstppr force-pushed the disjoint-systemvmtmpl branch from 5c41125 to df47ea1 Compare September 17, 2025 10:55

shwstppr force-pushed the disjoint-systemvmtmpl branch from df47ea1 to efdea06 Compare September 17, 2025 14:49

github-actions bot added the status:has-conflicts label Sep 27, 2025

Merge remote-tracking branch 'apache/main' into disjoint-systemvmtmpl

1a9ce76

github-actions bot removed the status:has-conflicts label Sep 29, 2025

refactor

9649e71

Signed-off-by: Abhishek Kumar <[email protected]>

shwstppr requested a review from Copilot October 9, 2025 10:35

Copilot AI reviewed Oct 9, 2025

View reviewed changes

client/bindir/cloud-setup-management.in Outdated Show resolved Hide resolved

shwstppr added 2 commits October 13, 2025 12:56

Merge branch 'main' into disjoint-systemvmtmpl

41b3658

add tests, fix comment

94417df

Signed-off-by: Abhishek Kumar <[email protected]>

shwstppr added a commit to shapeblue/cloudstack-documentation that referenced this pull request Oct 13, 2025

notes on system template auto-download, seeding

55cff9f

Related apache/cloudstack#11656 Signed-off-by: Abhishek Kumar <[email protected]>

shwstppr mentioned this pull request Oct 13, 2025

notes on system template auto-download, seeding apache/cloudstack-documentation#580

Draft

shwstppr added 2 commits October 27, 2025 13:21

Merge remote-tracking branch 'apache/main' into disjoint-systemvmtmpl

0833a64

dont use fixed version

2fd18bd

Signed-off-by: Abhishek Kumar <[email protected]>

shwstppr mentioned this pull request Oct 27, 2025

systemvm template download change shapeblue/Trillian#212

Draft

shwstppr closed this Oct 27, 2025

shwstppr closed this Nov 11, 2025

shwstppr reopened this Nov 11, 2025

apache deleted a comment from blueorangutan Nov 12, 2025

shwstppr commented Nov 13, 2025

View reviewed changes

client/conf/server.properties.in Outdated Show resolved Hide resolved

Update client/conf/server.properties.in

4bd5458

sureshanaparti reviewed Nov 13, 2025

View reviewed changes

client/bindir/cloud-setup-management.in Show resolved Hide resolved

sureshanaparti reviewed Nov 13, 2025

View reviewed changes

engine/schema/templateConfig.sh Show resolved Hide resolved

sureshanaparti approved these changes Nov 13, 2025

View reviewed changes

sureshanaparti added component:systemvm-template status:needs-testing labels Nov 13, 2025

apache deleted a comment from blueorangutan Nov 13, 2025

systemvm-template: support on-demand download during setup and registration #11656

Are you sure you want to change the base?

systemvm-template: support on-demand download during setup and registration #11656

Uh oh!

Conversation

shwstppr commented Sep 17, 2025 • edited by sureshanaparti Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

codecov bot commented Sep 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions bot commented Sep 27, 2025

Uh oh!

shwstppr commented Sep 29, 2025

Uh oh!

blueorangutan commented Sep 29, 2025

Uh oh!

blueorangutan commented Sep 29, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

shwstppr commented Oct 13, 2025

Uh oh!

blueorangutan commented Oct 13, 2025

Uh oh!

blueorangutan commented Oct 13, 2025

Uh oh!

shwstppr commented Oct 27, 2025

Uh oh!

blueorangutan commented Oct 27, 2025

Uh oh!

blueorangutan commented Oct 27, 2025

Uh oh!

shwstppr commented Oct 27, 2025

Uh oh!

blueorangutan commented Oct 27, 2025

Uh oh!

blueorangutan commented Nov 10, 2025

Uh oh!

shwstppr commented Nov 12, 2025

Uh oh!

blueorangutan commented Nov 12, 2025

Uh oh!

blueorangutan commented Nov 12, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sureshanaparti left a comment

Choose a reason for hiding this comment

Uh oh!

sureshanaparti commented Nov 13, 2025

Uh oh!

blueorangutan commented Nov 13, 2025

Uh oh!

blueorangutan commented Nov 13, 2025

Uh oh!

shwstppr commented Nov 13, 2025

Uh oh!

blueorangutan commented Nov 13, 2025

Uh oh!

blueorangutan commented Nov 13, 2025

Uh oh!

shwstppr commented Nov 16, 2025

Uh oh!

blueorangutan commented Nov 16, 2025

Uh oh!

shwstppr commented Sep 17, 2025 •

edited by sureshanaparti

Loading

codecov bot commented Sep 17, 2025 •

edited

Loading