Skip to content

Conversation

harikrishna-patnala
Copy link
Contributor

@harikrishna-patnala harikrishna-patnala commented Aug 12, 2025

Description

During upgrades while checking for system VM templates, new hypervisor type External needs to be excluded as there are no system VM templates involved for that type.

Note: This will not effect current on going release 4.21 as hypervisor type External is introduced in the same version.

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • build/CI
  • test (unit or integration test code)

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

How Has This Been Tested?

How did you try to break this feature and the system with this change?

@harikrishna-patnala
Copy link
Contributor Author

@blueorangutan package

@blueorangutan
Copy link

@harikrishna-patnala a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Copy link

codecov bot commented Aug 12, 2025

Codecov Report

❌ Patch coverage is 66.66667% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 17.36%. Comparing base (9fd2b90) to head (0233a9a).
⚠️ Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
...om/cloud/upgrade/SystemVmTemplateRegistration.java 50.00% 1 Missing ⚠️
...ain/java/com/cloud/storage/StorageManagerImpl.java 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff            @@
##               main   #11435   +/-   ##
=========================================
  Coverage     17.36%   17.36%           
  Complexity    15236    15236           
=========================================
  Files          5886     5886           
  Lines        525645   525647    +2     
  Branches      64156    64156           
=========================================
+ Hits          91257    91259    +2     
  Misses       424093   424093           
  Partials      10295    10295           
Flag Coverage Δ
uitests 3.63% <ø> (ø)
unittests 18.40% <66.66%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copy link
Contributor

@DaanHoogland DaanHoogland left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clgtm, I am not sure if we ever need to search including external, but that would be an easy enhancement by the time we need it.

@blueorangutan
Copy link

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 14607

Copy link
Contributor

@sureshanaparti sureshanaparti left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clgtm

Copy link
Contributor

@kiranchavala kiranchavala left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@harikrishna-patnala hitting the following exceptions

2025-08-13 07:17:42,492 DEBUG [c.c.u.d.Upgrade42010to42100] (main:[]) (logid:) Updating System Vm template IDs
2025-08-13 07:17:42,698 DEBUG [c.c.u.SystemVmTemplateRegistration] (main:[]) (logid:) Updating System Vm template IDs
2025-08-13 07:17:42,706 DEBUG [c.c.u.SystemVmTemplateRegistration] (main:[]) (logid:) Updating SystemVM Template Guest OS [Debian GNU/Linux 12 (64-bit)] id
2025-08-13 07:17:42,744 DEBUG [c.c.u.SystemVmTemplateRegistration] (main:[]) (logid:) Updating System VM template for hypervisor: LXC
2025-08-13 07:17:42,749 WARN  [c.c.u.SystemVmTemplateRegistration] (main:[]) (logid:) Cannot upgrade 4.20.2 system VM template for hypervisor: LXC as it is not used, not failing upgrade
2025-08-13 07:17:42,787 DEBUG [c.c.u.SystemVmTemplateRegistration] (main:[]) (logid:) Updating System VM template for hypervisor: VMware
2025-08-13 07:17:42,789 WARN  [c.c.u.SystemVmTemplateRegistration] (main:[]) (logid:) Cannot upgrade 4.20.2 system VM template for hypervisor: VMware as it is not used, not failing upgrade
2025-08-13 07:17:42,797 DEBUG [c.c.u.SystemVmTemplateRegistration] (main:[]) (logid:) Updating System VM template for hypervisor: KVM, arch: x86_64
2025-08-13 07:17:42,799 INFO  [c.c.u.SystemVmTemplateRegistration] (main:[]) (logid:) Grabbing lock to register templates.
2025-08-13 07:17:46,262 DEBUG [c.c.u.s.Script] (main:[]) (logid:) Executing command [/bin/bash -c sudo mount -t nfs 10.0.32.4:/acs/secondary/ref-trl-9218-k-Mol8-harikrishna-patnala/ref-trl-9218-k-Mol8-harikrishna-patnala-sec1 /tmp/tmp9555288554741205640 ].
2025-08-13 07:17:46,440 DEBUG [c.c.u.s.Script] (main:[]) (logid:) Successfully executed process [4532] for command [/bin/bash -c sudo mount -t nfs 10.0.32.4:/acs/secondary/ref-trl-9218-k-Mol8-harikrishna-patnala/ref-trl-9218-k-Mol8-harikrishna-patnala-sec1 /tmp/tmp9555288554741205640 ].
2025-08-13 07:17:46,568 DEBUG [c.c.u.s.Script] (main:[]) (logid:) Looking for setup-sysvm-tmplt in the classpath
2025-08-13 07:17:46,569 DEBUG [c.c.u.s.Script] (main:[]) (logid:) System resource: null
2025-08-13 07:17:46,569 DEBUG [c.c.u.s.Script] (main:[]) (logid:) Classpath resource: file:/usr/share/cloudstack-common/scripts/storage/secondary/setup-sysvm-tmplt
2025-08-13 07:17:46,570 DEBUG [c.c.u.s.Script] (main:[]) (logid:) Absolute path =  /usr/share/cloudstack-common/scripts/storage/secondary/setup-sysvm-tmplt
2025-08-13 07:17:46,570 DEBUG [c.c.u.SystemVmTemplateRegistration] (main:[]) (logid:) Executing command [/usr/share/cloudstack-common/scripts/storage/secondary/setup-sysvm-tmplt -u 74c1f164-dbb1-4c1b-82b9-fbb334d8b235 -f /usr/share/cloudstack-management/templates/systemvm/systemvmtemplate-4.20.2-x86_64-kvm.qcow2.bz2 -h kvm -d /tmp/tmp9555288554741205640/template/tmpl/1/203 ].
2025-08-13 07:17:46,711 WARN  [c.c.u.SystemVmTemplateRegistration] (main:[]) (logid:) Execution of process [4543] for command [/usr/share/cloudstack-common/scripts/storage/secondary/setup-sysvm-tmplt -u 74c1f164-dbb1-4c1b-82b9-fbb334d8b235 -f /usr/share/cloudstack-management/templates/systemvm/systemvmtemplate-4.20.2-x86_64-kvm.qcow2.bz2 -h kvm -d /tmp/tmp9555288554741205640/template/tmpl/1/203 ] failed.
2025-08-13 07:17:46,712 DEBUG [c.c.u.SystemVmTemplateRegistration] (main:[]) (logid:) Exit value of process [4543] for command [/usr/share/cloudstack-common/scripts/storage/secondary/setup-sysvm-tmplt -u 74c1f164-dbb1-4c1b-82b9-fbb334d8b235 -f /usr/share/cloudstack-management/templates/systemvm/systemvmtemplate-4.20.2-x86_64-kvm.qcow2.bz2 -h kvm -d /tmp/tmp9555288554741205640/template/tmpl/1/203 ] is [2].
2025-08-13 07:17:46,713 WARN  [c.c.u.SystemVmTemplateRegistration] (main:[]) (logid:) Process [4543] for command [/usr/share/cloudstack-common/scripts/storage/secondary/setup-sysvm-tmplt -u 74c1f164-dbb1-4c1b-82b9-fbb334d8b235 -f /usr/share/cloudstack-management/templates/systemvm/systemvmtemplate-4.20.2-x86_64-kvm.qcow2.bz2 -h kvm -d /tmp/tmp9555288554741205640/template/tmpl/1/203 ] encountered the error: [+ [[ ! -u 74c1f164-dbb1-4c1b-82b9-fbb334d8b235 -f /usr/share/cloudstack-management/templates/systemvm/systemvmtemplate-4.20.2-x86_64-kvm.qcow2.bz2 -h kvm -d /tmp/tmp9555288554741205640/template/tmpl/1/203 =~ ^-.+ ]]+ OPTERR=0+ DISKSPACE=2120000+ getopts h:f:d:u::# OPTION+ case $OPTION in+ uflag=1+ uuid=74c1f164-dbb1-4c1b-82b9-fbb334d8b235+ getopts h:f:d:u::# OPTION+ case $OPTION in+ fflag=1+ tmpltimg=/usr/share/cloudstack-management/templates/systemvm/systemvmtemplate-4.20.2-x86_64-kvm.qcow2.bz2+ getopts h:f:d:u::# OPTION+ case $OPTION in+ hyper=kvm+ getopts h:f:d:u::# OPTION+ case $OPTION in+ destdir=/tmp/tmp9555288554741205640/template/tmpl/1/203+ getopts h:f:d:u::# OPTION+ [[ kvm == \k\v\m ]]+ ext=qcow2++ which qemu-img+ qemuimgcmd=/usr/bin/qemu-img+ localfile=74c1f164-dbb1-4c1b-82b9-fbb334d8b235.qcow2+ sudo mkdir -p /tmp/tmp9555288554741205640/template/tmpl/1/203+ [[ 0 -ne 0 ]]+ [[ -f /tmp/tmp9555288554741205640/template/tmpl/1/203/template.properties ]]+ failed 2 'Data already exists at destination /tmp/tmp9555288554741205640/template/tmpl/1/203'+ local returnval=2+ local 'returnmsg=Data already exists at destination /tmp/tmp9555288554741205640/template/tmpl/1/203'+ [[ -z Data already exists at destination /tmp/tmp9555288554741205640/template/tmpl/1/203 ]]+ echo -e Data already exists at destination /tmp/tmp9555288554741205640/template/tmpl/1/203Data already exists at destination /tmp/tmp9555288554741205640/template/tmpl/1/203+ [[ 2 -eq 0 ]]+ echo 'Installation failed'Installation failed+ exit 2].
2025-08-13 07:17:46,716 ERROR [c.c.u.SystemVmTemplateRegistration] (main:[]) (logid:) failed to create template: + [[ ! -u 74c1f164-dbb1-4c1b-82b9-fbb334d8b235 -f /usr/share/cloudstack-management/templates/systemvm/systemvmtemplate-4.20.2-x86_64-kvm.qcow2.bz2 -h kvm -d /tmp/tmp9555288554741205640/template/tmpl/1/203 =~ ^-.+ ]]+ OPTERR=0+ DISKSPACE=2120000+ getopts h:f:d:u::# OPTION+ case $OPTION in+ uflag=1+ uuid=74c1f164-dbb1-4c1b-82b9-fbb334d8b235+ getopts h:f:d:u::# OPTION+ case $OPTION in+ fflag=1+ tmpltimg=/usr/share/cloudstack-management/templates/systemvm/systemvmtemplate-4.20.2-x86_64-kvm.qcow2.bz2+ getopts h:f:d:u::# OPTION+ case $OPTION in+ hyper=kvm+ getopts h:f:d:u::# OPTION+ case $OPTION in+ destdir=/tmp/tmp9555288554741205640/template/tmpl/1/203+ getopts h:f:d:u::# OPTION+ [[ kvm == \k\v\m ]]+ ext=qcow2++ which qemu-img+ qemuimgcmd=/usr/bin/qemu-img+ localfile=74c1f164-dbb1-4c1b-82b9-fbb334d8b235.qcow2+ sudo mkdir -p /tmp/tmp9555288554741205640/template/tmpl/1/203+ [[ 0 -ne 0 ]]+ [[ -f /tmp/tmp9555288554741205640/template/tmpl/1/203/template.properties ]]+ failed 2 'Data already exists at destination /tmp/tmp9555288554741205640/template/tmpl/1/203'+ local returnval=2+ local 'returnmsg=Data already exists at destination /tmp/tmp9555288554741205640/template/tmpl/1/203'+ [[ -z Data already exists at destination /tmp/tmp9555288554741205640/template/tmpl/1/203 ]]+ echo -e Data already exists at destination /tmp/tmp9555288554741205640/template/tmpl/1/203Data already exists at destination /tmp/tmp9555288554741205640/template/tmpl/1/203+ [[ 2 -eq 0 ]]+ echo 'Installation failed'Installation failed+ exit 2
2025-08-13 07:17:46,717 ERROR [c.c.u.SystemVmTemplateRegistration] (main:[]) (logid:) Failed to register template for hypervisor: KVM com.cloud.utils.exception.CloudRuntimeException: failed to create template: + [[ ! -u 74c1f164-dbb1-4c1b-82b9-fbb334d8b235 -f /usr/share/cloudstack-management/templates/systemvm/systemvmtemplate-4.20.2-x86_64-kvm.qcow2.bz2 -h kvm -d /tmp/tmp9555288554741205640/template/tmpl/1/203 =~ ^-.+ ]]+ OPTERR=0+ DISKSPACE=2120000+ getopts h:f:d:u::# OPTION+ case $OPTION in+ uflag=1+ uuid=74c1f164-dbb1-4c1b-82b9-fbb334d8b235+ getopts h:f:d:u::# OPTION+ case $OPTION in+ fflag=1+ tmpltimg=/usr/share/cloudstack-management/templates/systemvm/systemvmtemplate-4.20.2-x86_64-kvm.qcow2.bz2+ getopts h:f:d:u::# OPTION+ case $OPTION in+ hyper=kvm+ getopts h:f:d:u::# OPTION+ case $OPTION in+ destdir=/tmp/tmp9555288554741205640/template/tmpl/1/203+ getopts h:f:d:u::# OPTION+ [[ kvm == \k\v\m ]]+ ext=qcow2++ which qemu-img+ qemuimgcmd=/usr/bin/qemu-img+ localfile=74c1f164-dbb1-4c1b-82b9-fbb334d8b235.qcow2+ sudo mkdir -p /tmp/tmp9555288554741205640/template/tmpl/1/203+ [[ 0 -ne 0 ]]+ [[ -f /tmp/tmp9555288554741205640/template/tmpl/1/203/template.properties ]]+ failed 2 'Data already exists at destination /tmp/tmp9555288554741205640/template/tmpl/1/203'+ local returnval=2+ local 'returnmsg=Data already exists at destination /tmp/tmp9555288554741205640/template/tmpl/1/203'+ [[ -z Data already exists at destination /tmp/tmp9555288554741205640/template/tmpl/1/203 ]]+ echo -e Data already exists at destination /tmp/tmp9555288554741205640/template/tmpl/1/203Data already exists at destination /tmp/tmp9555288554741205640/template/tmpl/1/203+ [[ 2 -eq 0 ]]+ echo 'Installation failed'Installation failed+ exit 2
	at com.cloud.upgrade.SystemVmTemplateRegistration.setupTemplate(SystemVmTemplateRegistration.java:701)
	at com.cloud.upgrade.SystemVmTemplateRegistration.performTemplateRegistrationOperations(SystemVmTemplateRegistration.java:728)
	at com.cloud.upgrade.SystemVmTemplateRegistration.registerTemplateForNonExistingEntries(SystemVmTemplateRegistration.java:755)
	at com.cloud.upgrade.SystemVmTemplateRegistration.registerTemplatesForZone(SystemVmTemplateRegistration.java:943)
	at com.cloud.upgrade.SystemVmTemplateRegistration$4.doInTransactionWithoutResult(SystemVmTemplateRegistration.java:969)
	at com.cloud.utils.db.TransactionCallbackNoReturn.doInTransaction(TransactionCallbackNoReturn.java:25)
	at com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:48)
	at com.cloud.utils.db.Transaction.execute(Transaction.java:38)
	at com.cloud.utils.db.Transaction.execute(Transaction.java:45)
	at com.cloud.upgrade.SystemVmTemplateRegistration.registerTemplates(SystemVmTemplateRegistration.java:958)
	at com.cloud.upgrade.SystemVmTemplateRegistration.registerOrUpdateSystemVmTemplate(SystemVmTemplateRegistration.java:1037)
	at com.cloud.upgrade.SystemVmTemplateRegistration$5.doInTransactionWithoutResult(SystemVmTemplateRegistration.java:1075)
	at com.cloud.utils.db.TransactionCallbackNoReturn.doInTransaction(TransactionCallbackNoReturn.java:25)
	at com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:48)
	at com.cloud.utils.db.Transaction.execute(Transaction.java:38)
	at com.cloud.utils.db.Transaction.execute(Transaction.java:45)
	at com.cloud.upgrade.SystemVmTemplateRegistration.updateSystemVmTemplates(SystemVmTemplateRegistration.java:1063)
	at com.cloud.upgrade.dao.Upgrade42010to42100.updateSystemVmTemplates(Upgrade42010to42100.java:90)
	at com.cloud.upgrade.DatabaseUpgradeChecker.updateSystemVmTemplates(DatabaseUpgradeChecker.java:291)
	at com.cloud.upgrade.DatabaseUpgradeChecker.upgrade(DatabaseUpgradeChecker.java:310)
	at com.cloud.upgrade.DatabaseUpgradeChecker.check(DatabaseUpgradeChecker.java:483)
	at org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.checkIntegrity(CloudStackExtendedLifeCycle.java:62)
	at org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.start(CloudStackExtendedLifeCycle.java:52)
	at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:178)
	at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:54)
	at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:356)
	at java.base/java.lang.Iterable.forEach(Iterable.java:75)
	at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:155)
	at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:123)
	at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:937)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:586)
	at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContext(DefaultModuleDefinitionSet.java:174)
	at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet$2.with(DefaultModuleDefinitionSet.java:143)
	at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:274)
	at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:279)
	at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:262)
	at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContexts(DefaultModuleDefinitionSet.java:131)
	at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.load(DefaultModuleDefinitionSet.java:79)
	at org.apache.cloudstack.spring.module.factory.ModuleBasedContextFactory.loadModules(ModuleBasedContextFactory.java:37)
	at org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.init(CloudStackSpringContext.java:71)
	at org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:58)
	at org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:62)
	at org.apache.cloudstack.spring.module.web.CloudStackContextLoaderListener.contextInitialized(CloudStackContextLoaderListener.java:53)
	at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1073)
	at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572)
	at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:1002)
	at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:765)
	at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
	at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449)
	at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414)
	at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:916)
	at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
	at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:426)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
	at org.eclipse.jetty.server.Server.start(Server.java:423)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
	at org.eclipse.jetty.server.Server.doStart(Server.java:387)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at org.apache.cloudstack.ServerDaemon.start(ServerDaemon.java:207)
	at org.apache.cloudstack.ServerDaemon.main(ServerDaemon.java:117)

2025-08-13 07:17:46,718 INFO  [c.c.u.SystemVmTemplateRegistration] (main:[]) (logid:) Unmounting store
2025-08-13 07:17:46,718 DEBUG [c.c.u.s.Script] (main:[]) (logid:) Executing command [/bin/bash -c sudo umount /tmp/tmp9555288554741205640 ].
2025-08-13 07:17:46,977 DEBUG [c.c.u.s.Script] (main:[]) (logid:) Successfully executed process [4550] for command [/bin/bash -c sudo umount /tmp/tmp9555288554741205640 ].
2025-08-13 07:17:46,989 DEBUG [c.c.u.d.T.Transaction] (main:[]) (logid:) Rolling back the transaction: Time = 4490 Name =  Upgrade; called by -TransactionLegacy.rollback:905-TransactionLegacy.removeUpTo:848-TransactionLegacy.close:672-Transaction.execute:36-Transaction.execute:45-SystemVmTemplateRegistration.registerTemplates:958-SystemVmTemplateRegistration.registerOrUpdateSystemVmTemplate:1037-SystemVmTemplateRegistration$5.doInTransactionWithoutResult:1075-TransactionCallbackNoReturn.doInTransaction:25-Transaction$2.doInTransaction:48-Transaction.execute:38-Transaction.execute:45
2025-08-13 07:17:47,005 ERROR [c.c.u.SystemVmTemplateRegistration] (main:[]) (logid:) Exception while registering/updating system VM templates for hypervisors in metadata com.cloud.utils.exception.CloudRuntimeException: Failed to register 4.20.2 templates for hypervisors: [kvm-x86_64]. Cannot upgrade system VMs
	at com.cloud.upgrade.SystemVmTemplateRegistration.registerOrUpdateSystemVmTemplate(SystemVmTemplateRegistration.java:1040)
	at com.cloud.upgrade.SystemVmTemplateRegistration$5.doInTransactionWithoutResult(SystemVmTemplateRegistration.java:1075)
	at com.cloud.utils.db.TransactionCallbackNoReturn.doInTransaction(TransactionCallbackNoReturn.java:25)
	at com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:48)
	at com.cloud.utils.db.Transaction.execute(Transaction.java:38)
	at com.cloud.utils.db.Transaction.execute(Transaction.java:45)
	at com.cloud.upgrade.SystemVmTemplateRegistration.updateSystemVmTemplates(SystemVmTemplateRegistration.java:1063)
	at com.cloud.upgrade.dao.Upgrade42010to42100.updateSystemVmTemplates(Upgrade42010to42100.java:90)
	at com.cloud.upgrade.DatabaseUpgradeChecker.updateSystemVmTemplates(DatabaseUpgradeChecker.java:291)
	at com.cloud.upgrade.DatabaseUpgradeChecker.upgrade(DatabaseUpgradeChecker.java:310)
	at com.cloud.upgrade.DatabaseUpgradeChecker.check(DatabaseUpgradeChecker.java:483)
	
	

*
* @param zoneId the zone ID to filter by, or {@code null} to include all zones
* @return list of unique hypervisor type and CPU architecture pairs
*/
@Override
public List<Pair<HypervisorType, CPU.CPUArch>> listDistinctHypervisorsArchAcrossClusters(Long zoneId) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel the method name should reflect the change to avoid confusion and wrong usage

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shwstppr can you check the new changes with name change

@harikrishna-patnala
Copy link
Contributor Author

@harikrishna-patnala hitting the following exceptions

2025-08-13 07:17:42,492 DEBUG [c.c.u.d.Upgrade42010to42100] (main:[]) (logid:) Updating System Vm template IDs
2025-08-13 07:17:42,698 DEBUG [c.c.u.SystemVmTemplateRegistration] (main:[]) (logid:) Updating System Vm template IDs
2025-08-13 07:17:42,706 DEBUG [c.c.u.SystemVmTemplateRegistration] (main:[]) (logid:) Updating SystemVM Template Guest OS [Debian GNU/Linux 12 (64-bit)] id
2025-08-13 07:17:42,744 DEBUG [c.c.u.SystemVmTemplateRegistration] (main:[]) (logid:) Updating System VM template for hypervisor: LXC
2025-08-13 07:17:42,749 WARN  [c.c.u.SystemVmTemplateRegistration] (main:[]) (logid:) Cannot upgrade 4.20.2 system VM template for hypervisor: LXC as it is not used, not failing upgrade
2025-08-13 07:17:42,787 DEBUG [c.c.u.SystemVmTemplateRegistration] (main:[]) (logid:) Updating System VM template for hypervisor: VMware
2025-08-13 07:17:42,789 WARN  [c.c.u.SystemVmTemplateRegistration] (main:[]) (logid:) Cannot upgrade 4.20.2 system VM template for hypervisor: VMware as it is not used, not failing upgrade
2025-08-13 07:17:42,797 DEBUG [c.c.u.SystemVmTemplateRegistration] (main:[]) (logid:) Updating System VM template for hypervisor: KVM, arch: x86_64
2025-08-13 07:17:42,799 INFO  [c.c.u.SystemVmTemplateRegistration] (main:[]) (logid:) Grabbing lock to register templates.
2025-08-13 07:17:46,262 DEBUG [c.c.u.s.Script] (main:[]) (logid:) Executing command [/bin/bash -c sudo mount -t nfs 10.0.32.4:/acs/secondary/ref-trl-9218-k-Mol8-harikrishna-patnala/ref-trl-9218-k-Mol8-harikrishna-patnala-sec1 /tmp/tmp9555288554741205640 ].
2025-08-13 07:17:46,440 DEBUG [c.c.u.s.Script] (main:[]) (logid:) Successfully executed process [4532] for command [/bin/bash -c sudo mount -t nfs 10.0.32.4:/acs/secondary/ref-trl-9218-k-Mol8-harikrishna-patnala/ref-trl-9218-k-Mol8-harikrishna-patnala-sec1 /tmp/tmp9555288554741205640 ].
2025-08-13 07:17:46,568 DEBUG [c.c.u.s.Script] (main:[]) (logid:) Looking for setup-sysvm-tmplt in the classpath
2025-08-13 07:17:46,569 DEBUG [c.c.u.s.Script] (main:[]) (logid:) System resource: null
2025-08-13 07:17:46,569 DEBUG [c.c.u.s.Script] (main:[]) (logid:) Classpath resource: file:/usr/share/cloudstack-common/scripts/storage/secondary/setup-sysvm-tmplt
2025-08-13 07:17:46,570 DEBUG [c.c.u.s.Script] (main:[]) (logid:) Absolute path =  /usr/share/cloudstack-common/scripts/storage/secondary/setup-sysvm-tmplt
2025-08-13 07:17:46,570 DEBUG [c.c.u.SystemVmTemplateRegistration] (main:[]) (logid:) Executing command [/usr/share/cloudstack-common/scripts/storage/secondary/setup-sysvm-tmplt -u 74c1f164-dbb1-4c1b-82b9-fbb334d8b235 -f /usr/share/cloudstack-management/templates/systemvm/systemvmtemplate-4.20.2-x86_64-kvm.qcow2.bz2 -h kvm -d /tmp/tmp9555288554741205640/template/tmpl/1/203 ].
2025-08-13 07:17:46,711 WARN  [c.c.u.SystemVmTemplateRegistration] (main:[]) (logid:) Execution of process [4543] for command [/usr/share/cloudstack-common/scripts/storage/secondary/setup-sysvm-tmplt -u 74c1f164-dbb1-4c1b-82b9-fbb334d8b235 -f /usr/share/cloudstack-management/templates/systemvm/systemvmtemplate-4.20.2-x86_64-kvm.qcow2.bz2 -h kvm -d /tmp/tmp9555288554741205640/template/tmpl/1/203 ] failed.
2025-08-13 07:17:46,712 DEBUG [c.c.u.SystemVmTemplateRegistration] (main:[]) (logid:) Exit value of process [4543] for command [/usr/share/cloudstack-common/scripts/storage/secondary/setup-sysvm-tmplt -u 74c1f164-dbb1-4c1b-82b9-fbb334d8b235 -f /usr/share/cloudstack-management/templates/systemvm/systemvmtemplate-4.20.2-x86_64-kvm.qcow2.bz2 -h kvm -d /tmp/tmp9555288554741205640/template/tmpl/1/203 ] is [2].
2025-08-13 07:17:46,713 WARN  [c.c.u.SystemVmTemplateRegistration] (main:[]) (logid:) Process [4543] for command [/usr/share/cloudstack-common/scripts/storage/secondary/setup-sysvm-tmplt -u 74c1f164-dbb1-4c1b-82b9-fbb334d8b235 -f /usr/share/cloudstack-management/templates/systemvm/systemvmtemplate-4.20.2-x86_64-kvm.qcow2.bz2 -h kvm -d /tmp/tmp9555288554741205640/template/tmpl/1/203 ] encountered the error: [+ [[ ! -u 74c1f164-dbb1-4c1b-82b9-fbb334d8b235 -f /usr/share/cloudstack-management/templates/systemvm/systemvmtemplate-4.20.2-x86_64-kvm.qcow2.bz2 -h kvm -d /tmp/tmp9555288554741205640/template/tmpl/1/203 =~ ^-.+ ]]+ OPTERR=0+ DISKSPACE=2120000+ getopts h:f:d:u::# OPTION+ case $OPTION in+ uflag=1+ uuid=74c1f164-dbb1-4c1b-82b9-fbb334d8b235+ getopts h:f:d:u::# OPTION+ case $OPTION in+ fflag=1+ tmpltimg=/usr/share/cloudstack-management/templates/systemvm/systemvmtemplate-4.20.2-x86_64-kvm.qcow2.bz2+ getopts h:f:d:u::# OPTION+ case $OPTION in+ hyper=kvm+ getopts h:f:d:u::# OPTION+ case $OPTION in+ destdir=/tmp/tmp9555288554741205640/template/tmpl/1/203+ getopts h:f:d:u::# OPTION+ [[ kvm == \k\v\m ]]+ ext=qcow2++ which qemu-img+ qemuimgcmd=/usr/bin/qemu-img+ localfile=74c1f164-dbb1-4c1b-82b9-fbb334d8b235.qcow2+ sudo mkdir -p /tmp/tmp9555288554741205640/template/tmpl/1/203+ [[ 0 -ne 0 ]]+ [[ -f /tmp/tmp9555288554741205640/template/tmpl/1/203/template.properties ]]+ failed 2 'Data already exists at destination /tmp/tmp9555288554741205640/template/tmpl/1/203'+ local returnval=2+ local 'returnmsg=Data already exists at destination /tmp/tmp9555288554741205640/template/tmpl/1/203'+ [[ -z Data already exists at destination /tmp/tmp9555288554741205640/template/tmpl/1/203 ]]+ echo -e Data already exists at destination /tmp/tmp9555288554741205640/template/tmpl/1/203Data already exists at destination /tmp/tmp9555288554741205640/template/tmpl/1/203+ [[ 2 -eq 0 ]]+ echo 'Installation failed'Installation failed+ exit 2].
2025-08-13 07:17:46,716 ERROR [c.c.u.SystemVmTemplateRegistration] (main:[]) (logid:) failed to create template: + [[ ! -u 74c1f164-dbb1-4c1b-82b9-fbb334d8b235 -f /usr/share/cloudstack-management/templates/systemvm/systemvmtemplate-4.20.2-x86_64-kvm.qcow2.bz2 -h kvm -d /tmp/tmp9555288554741205640/template/tmpl/1/203 =~ ^-.+ ]]+ OPTERR=0+ DISKSPACE=2120000+ getopts h:f:d:u::# OPTION+ case $OPTION in+ uflag=1+ uuid=74c1f164-dbb1-4c1b-82b9-fbb334d8b235+ getopts h:f:d:u::# OPTION+ case $OPTION in+ fflag=1+ tmpltimg=/usr/share/cloudstack-management/templates/systemvm/systemvmtemplate-4.20.2-x86_64-kvm.qcow2.bz2+ getopts h:f:d:u::# OPTION+ case $OPTION in+ hyper=kvm+ getopts h:f:d:u::# OPTION+ case $OPTION in+ destdir=/tmp/tmp9555288554741205640/template/tmpl/1/203+ getopts h:f:d:u::# OPTION+ [[ kvm == \k\v\m ]]+ ext=qcow2++ which qemu-img+ qemuimgcmd=/usr/bin/qemu-img+ localfile=74c1f164-dbb1-4c1b-82b9-fbb334d8b235.qcow2+ sudo mkdir -p /tmp/tmp9555288554741205640/template/tmpl/1/203+ [[ 0 -ne 0 ]]+ [[ -f /tmp/tmp9555288554741205640/template/tmpl/1/203/template.properties ]]+ failed 2 'Data already exists at destination /tmp/tmp9555288554741205640/template/tmpl/1/203'+ local returnval=2+ local 'returnmsg=Data already exists at destination /tmp/tmp9555288554741205640/template/tmpl/1/203'+ [[ -z Data already exists at destination /tmp/tmp9555288554741205640/template/tmpl/1/203 ]]+ echo -e Data already exists at destination /tmp/tmp9555288554741205640/template/tmpl/1/203Data already exists at destination /tmp/tmp9555288554741205640/template/tmpl/1/203+ [[ 2 -eq 0 ]]+ echo 'Installation failed'Installation failed+ exit 2
2025-08-13 07:17:46,717 ERROR [c.c.u.SystemVmTemplateRegistration] (main:[]) (logid:) Failed to register template for hypervisor: KVM com.cloud.utils.exception.CloudRuntimeException: failed to create template: + [[ ! -u 74c1f164-dbb1-4c1b-82b9-fbb334d8b235 -f /usr/share/cloudstack-management/templates/systemvm/systemvmtemplate-4.20.2-x86_64-kvm.qcow2.bz2 -h kvm -d /tmp/tmp9555288554741205640/template/tmpl/1/203 =~ ^-.+ ]]+ OPTERR=0+ DISKSPACE=2120000+ getopts h:f:d:u::# OPTION+ case $OPTION in+ uflag=1+ uuid=74c1f164-dbb1-4c1b-82b9-fbb334d8b235+ getopts h:f:d:u::# OPTION+ case $OPTION in+ fflag=1+ tmpltimg=/usr/share/cloudstack-management/templates/systemvm/systemvmtemplate-4.20.2-x86_64-kvm.qcow2.bz2+ getopts h:f:d:u::# OPTION+ case $OPTION in+ hyper=kvm+ getopts h:f:d:u::# OPTION+ case $OPTION in+ destdir=/tmp/tmp9555288554741205640/template/tmpl/1/203+ getopts h:f:d:u::# OPTION+ [[ kvm == \k\v\m ]]+ ext=qcow2++ which qemu-img+ qemuimgcmd=/usr/bin/qemu-img+ localfile=74c1f164-dbb1-4c1b-82b9-fbb334d8b235.qcow2+ sudo mkdir -p /tmp/tmp9555288554741205640/template/tmpl/1/203+ [[ 0 -ne 0 ]]+ [[ -f /tmp/tmp9555288554741205640/template/tmpl/1/203/template.properties ]]+ failed 2 'Data already exists at destination /tmp/tmp9555288554741205640/template/tmpl/1/203'+ local returnval=2+ local 'returnmsg=Data already exists at destination /tmp/tmp9555288554741205640/template/tmpl/1/203'+ [[ -z Data already exists at destination /tmp/tmp9555288554741205640/template/tmpl/1/203 ]]+ echo -e Data already exists at destination /tmp/tmp9555288554741205640/template/tmpl/1/203Data already exists at destination /tmp/tmp9555288554741205640/template/tmpl/1/203+ [[ 2 -eq 0 ]]+ echo 'Installation failed'Installation failed+ exit 2
	at com.cloud.upgrade.SystemVmTemplateRegistration.setupTemplate(SystemVmTemplateRegistration.java:701)
	at com.cloud.upgrade.SystemVmTemplateRegistration.performTemplateRegistrationOperations(SystemVmTemplateRegistration.java:728)
	at com.cloud.upgrade.SystemVmTemplateRegistration.registerTemplateForNonExistingEntries(SystemVmTemplateRegistration.java:755)
	at com.cloud.upgrade.SystemVmTemplateRegistration.registerTemplatesForZone(SystemVmTemplateRegistration.java:943)
	at com.cloud.upgrade.SystemVmTemplateRegistration$4.doInTransactionWithoutResult(SystemVmTemplateRegistration.java:969)
	at com.cloud.utils.db.TransactionCallbackNoReturn.doInTransaction(TransactionCallbackNoReturn.java:25)
	at com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:48)
	at com.cloud.utils.db.Transaction.execute(Transaction.java:38)
	at com.cloud.utils.db.Transaction.execute(Transaction.java:45)
	at com.cloud.upgrade.SystemVmTemplateRegistration.registerTemplates(SystemVmTemplateRegistration.java:958)
	at com.cloud.upgrade.SystemVmTemplateRegistration.registerOrUpdateSystemVmTemplate(SystemVmTemplateRegistration.java:1037)
	at com.cloud.upgrade.SystemVmTemplateRegistration$5.doInTransactionWithoutResult(SystemVmTemplateRegistration.java:1075)
	at com.cloud.utils.db.TransactionCallbackNoReturn.doInTransaction(TransactionCallbackNoReturn.java:25)
	at com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:48)
	at com.cloud.utils.db.Transaction.execute(Transaction.java:38)
	at com.cloud.utils.db.Transaction.execute(Transaction.java:45)
	at com.cloud.upgrade.SystemVmTemplateRegistration.updateSystemVmTemplates(SystemVmTemplateRegistration.java:1063)
	at com.cloud.upgrade.dao.Upgrade42010to42100.updateSystemVmTemplates(Upgrade42010to42100.java:90)
	at com.cloud.upgrade.DatabaseUpgradeChecker.updateSystemVmTemplates(DatabaseUpgradeChecker.java:291)
	at com.cloud.upgrade.DatabaseUpgradeChecker.upgrade(DatabaseUpgradeChecker.java:310)
	at com.cloud.upgrade.DatabaseUpgradeChecker.check(DatabaseUpgradeChecker.java:483)
	at org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.checkIntegrity(CloudStackExtendedLifeCycle.java:62)
	at org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.start(CloudStackExtendedLifeCycle.java:52)
	at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:178)
	at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:54)
	at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:356)
	at java.base/java.lang.Iterable.forEach(Iterable.java:75)
	at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:155)
	at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:123)
	at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:937)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:586)
	at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContext(DefaultModuleDefinitionSet.java:174)
	at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet$2.with(DefaultModuleDefinitionSet.java:143)
	at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:274)
	at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:279)
	at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:262)
	at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContexts(DefaultModuleDefinitionSet.java:131)
	at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.load(DefaultModuleDefinitionSet.java:79)
	at org.apache.cloudstack.spring.module.factory.ModuleBasedContextFactory.loadModules(ModuleBasedContextFactory.java:37)
	at org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.init(CloudStackSpringContext.java:71)
	at org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:58)
	at org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:62)
	at org.apache.cloudstack.spring.module.web.CloudStackContextLoaderListener.contextInitialized(CloudStackContextLoaderListener.java:53)
	at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1073)
	at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572)
	at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:1002)
	at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:765)
	at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
	at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449)
	at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414)
	at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:916)
	at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
	at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:426)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
	at org.eclipse.jetty.server.Server.start(Server.java:423)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
	at org.eclipse.jetty.server.Server.doStart(Server.java:387)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at org.apache.cloudstack.ServerDaemon.start(ServerDaemon.java:207)
	at org.apache.cloudstack.ServerDaemon.main(ServerDaemon.java:117)

2025-08-13 07:17:46,718 INFO  [c.c.u.SystemVmTemplateRegistration] (main:[]) (logid:) Unmounting store
2025-08-13 07:17:46,718 DEBUG [c.c.u.s.Script] (main:[]) (logid:) Executing command [/bin/bash -c sudo umount /tmp/tmp9555288554741205640 ].
2025-08-13 07:17:46,977 DEBUG [c.c.u.s.Script] (main:[]) (logid:) Successfully executed process [4550] for command [/bin/bash -c sudo umount /tmp/tmp9555288554741205640 ].
2025-08-13 07:17:46,989 DEBUG [c.c.u.d.T.Transaction] (main:[]) (logid:) Rolling back the transaction: Time = 4490 Name =  Upgrade; called by -TransactionLegacy.rollback:905-TransactionLegacy.removeUpTo:848-TransactionLegacy.close:672-Transaction.execute:36-Transaction.execute:45-SystemVmTemplateRegistration.registerTemplates:958-SystemVmTemplateRegistration.registerOrUpdateSystemVmTemplate:1037-SystemVmTemplateRegistration$5.doInTransactionWithoutResult:1075-TransactionCallbackNoReturn.doInTransaction:25-Transaction$2.doInTransaction:48-Transaction.execute:38-Transaction.execute:45
2025-08-13 07:17:47,005 ERROR [c.c.u.SystemVmTemplateRegistration] (main:[]) (logid:) Exception while registering/updating system VM templates for hypervisors in metadata com.cloud.utils.exception.CloudRuntimeException: Failed to register 4.20.2 templates for hypervisors: [kvm-x86_64]. Cannot upgrade system VMs
	at com.cloud.upgrade.SystemVmTemplateRegistration.registerOrUpdateSystemVmTemplate(SystemVmTemplateRegistration.java:1040)
	at com.cloud.upgrade.SystemVmTemplateRegistration$5.doInTransactionWithoutResult(SystemVmTemplateRegistration.java:1075)
	at com.cloud.utils.db.TransactionCallbackNoReturn.doInTransaction(TransactionCallbackNoReturn.java:25)
	at com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:48)
	at com.cloud.utils.db.Transaction.execute(Transaction.java:38)
	at com.cloud.utils.db.Transaction.execute(Transaction.java:45)
	at com.cloud.upgrade.SystemVmTemplateRegistration.updateSystemVmTemplates(SystemVmTemplateRegistration.java:1063)
	at com.cloud.upgrade.dao.Upgrade42010to42100.updateSystemVmTemplates(Upgrade42010to42100.java:90)
	at com.cloud.upgrade.DatabaseUpgradeChecker.updateSystemVmTemplates(DatabaseUpgradeChecker.java:291)
	at com.cloud.upgrade.DatabaseUpgradeChecker.upgrade(DatabaseUpgradeChecker.java:310)
	at com.cloud.upgrade.DatabaseUpgradeChecker.check(DatabaseUpgradeChecker.java:483)
	
	

@kiranchavala I see this error "'Data already exists at destination /tmp/tmp9555288554741205640/template/tmpl/1/203'" based on this, I think the environment seems to have reused for upgrade testing where one of the attempts was successful. Please delete this folder in the secondary storage and try upgrade again.

Copy link
Contributor

@kiranchavala kiranchavala left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Thanks @harikrishna-patnala

2025-08-13 13:04:13,124 INFO  [c.c.u.DatabaseUpgradeChecker] (main:[]) (logid:) Finished execution of VIEW scripts that are under resource directory [META-INF/db/views].
2025-08-13 13:04:13,124 DEBUG [c.c.u.d.Upgrade42010to42100] (main:[]) (logid:) Updating System Vm template IDs
2025-08-13 13:04:13,267 DEBUG [c.c.u.SystemVmTemplateRegistration] (main:[]) (logid:) Updating System Vm template IDs
2025-08-13 13:04:13,272 DEBUG [c.c.u.SystemVmTemplateRegistration] (main:[]) (logid:) Updating SystemVM Template Guest OS [Debian GNU/Linux 12 (64-bit)] id
2025-08-13 13:04:13,295 DEBUG [c.c.u.SystemVmTemplateRegistration] (main:[]) (logid:) Updating System VM template for hypervisor: LXC
2025-08-13 13:04:13,299 WARN  [c.c.u.SystemVmTemplateRegistration] (main:[]) (logid:) Cannot upgrade 4.20.2 system VM template for hypervisor: LXC as it is not used, not failing upgrade
2025-08-13 13:04:13,342 DEBUG [c.c.u.SystemVmTemplateRegistration] (main:[]) (logid:) Updating System VM template for hypervisor: VMware
2025-08-13 13:04:13,343 WARN  [c.c.u.SystemVmTemplateRegistration] (main:[]) (logid:) Cannot upgrade 4.20.2 system VM template for hypervisor: VMware as it is not used, not failing upgrade
2025-08-13 13:04:13,358 DEBUG [c.c.u.SystemVmTemplateRegistration] (main:[]) (logid:) Updating System VM template for hypervisor: KVM, arch: x86_64
2025-08-13 13:04:13,361 INFO  [c.c.u.SystemVmTemplateRegistration] (main:[]) (logid:) Grabbing lock to register templates.
2025-08-13 13:04:15,923 DEBUG [c.c.u.s.Script] (main:[]) (logid:) Executing command [/bin/bash -c sudo mount -t nfs 10.0.32.4:/acs/secondary/ref-trl-9230-k-Mol8-kiran-chavala/ref-trl-9230-k-Mol8-kiran-chavala-sec1 /tmp/tmp790604977550814143 ].
2025-08-13 13:04:16,145 DEBUG [c.c.u.s.Script] (main:[]) (logid:) Successfully executed process [40094] for command [/bin/bash -c sudo mount -t nfs 10.0.32.4:/acs/secondary/ref-trl-9230-k-Mol8-kiran-chavala/ref-trl-9230-k-Mol8-kiran-chavala-sec1 /tmp/tmp790604977550814143 ].
2025-08-13 13:04:16,224 DEBUG [c.c.u.s.Script] (main:[]) (logid:) Looking for setup-sysvm-tmplt in the classpath
2025-08-13 13:04:16,225 DEBUG [c.c.u.s.Script] (main:[]) (logid:) System resource: null
2025-08-13 13:04:16,225 DEBUG [c.c.u.s.Script] (main:[]) (logid:) Classpath resource: file:/usr/share/cloudstack-common/scripts/storage/secondary/setup-sysvm-tmplt
2025-08-13 13:04:16,225 DEBUG [c.c.u.s.Script] (main:[]) (logid:) Absolute path =  /usr/share/cloudstack-common/scripts/storage/secondary/setup-sysvm-tmplt
2025-08-13 13:04:16,226 DEBUG [c.c.u.SystemVmTemplateRegistration] (main:[]) (logid:) Executing command [/usr/share/cloudstack-common/scripts/storage/secondary/setup-sysvm-tmplt -u 51ffe658-e2b2-4ad6-80ae-013191bd2288 -f /usr/share/cloudstack-management/templates/systemvm/systemvmtemplate-4.20.2-x86_64-kvm.qcow2.bz2 -h kvm -d /tmp/tmp790604977550814143/template/tmpl/1/203 ].
2025-08-13 13:05:19,362 DEBUG [c.c.u.SystemVmTemplateRegistration] (main:[]) (logid:) Successfully executed process [40106] for command [/usr/share/cloudstack-common/scripts/storage/secondary/setup-sysvm-tmplt -u 51ffe658-e2b2-4ad6-80ae-013191bd2288 -f /usr/share/cloudstack-management/templates/systemvm/systemvmtemplate-4.20.2-x86_64-kvm.qcow2.bz2 -h kvm -d /tmp/tmp790604977550814143/template/tmpl/1/203 ].
2025-08-13 13:05:19,363 DEBUG [c.c.u.SystemVmTemplateRegistration] (main:[]) (logid:) + [[ ! -u 51ffe658-e2b2-4ad6-80ae-013191bd2288 -f /usr/share/cloudstack-management/templates/systemvm/systemvmtemplate-4.20.2-x86_64-kvm.qcow2.bz2 -h kvm -d /tmp/tmp790604977550814143/template/tmpl/1/203 =~ ^-.+ ]]
+ OPTERR=0
+ DISKSPACE=2120000
+ getopts h:f:d:u::# OPTION
+ case $OPTION in
+ uflag=1
+ uuid=51ffe658-e2b2-4ad6-80ae-013191bd2288
+ getopts h:f:d:u::# OPTION
+ case $OPTION in
+ fflag=1
+ tmpltimg=/usr/share/cloudstack-management/templates/systemvm/systemvmtemplate-4.20.2-x86_64-kvm.qcow2.bz2
+ getopts h:f:d:u::# OPTION
+ case $OPTION in
+ hyper=kvm
+ getopts h:f:d:u::# OPTION
+ case $OPTION in
+ destdir=/tmp/tmp790604977550814143/template/tmpl/1/203
+ getopts h:f:d:u::# OPTION
+ [[ kvm == \k\v\m ]]
+ ext=qcow2
++ which qemu-img
+ qemuimgcmd=/usr/bin/qemu-img
+ localfile=51ffe658-e2b2-4ad6-80ae-013191bd2288.qcow2
+ sudo mkdir -p /tmp/tmp790604977550814143/template/tmpl/1/203
+ [[ 0 -ne 0 ]]
+ [[ -f /tmp/tmp790604977550814143/template/tmpl/1/203/template.properties ]]
++ sudo find /tmp/tmp790604977550814143/template/tmpl/1/203 -name '*.qcow2'
+ destfiles=
+ [[ '' != '' ]]
+ tmpfolder=/tmp/cloud/templates/
+ mkdir -p /tmp/cloud/templates/
+ tmplfile=/tmp/cloud/templates//51ffe658-e2b2-4ad6-80ae-013191bd2288.qcow2
+ touch /tmp/cloud/templates//51ffe658-e2b2-4ad6-80ae-013191bd2288.qcow2
+ [[ 0 -ne 0 ]]
++ sudo df -P /tmp/tmp790604977550814143/template/tmpl/1/203
++ awk '{print $4}'
++ tail -1
+ destcap=359815168
+ '[' 359815168 -lt 2120000 ']'
++ df -P /tmp/cloud/templates/
++ tail -1
++ awk '{print $4}'
+ localcap=11385352
+ '[' 11385352 -lt 2120000 ']'
+ [[ 1 == \1 ]]
+ sudo cp /usr/share/cloudstack-management/templates/systemvm/systemvmtemplate-4.20.2-x86_64-kvm.qcow2.bz2 /tmp/cloud/templates//51ffe658-e2b2-4ad6-80ae-013191bd2288.qcow2
+ [[ 0 -ne 0 ]]
+++ dirname /usr/share/cloudstack-common/scripts/storage/secondary/setup-sysvm-tmplt
++ /usr/share/cloudstack-common/scripts/storage/secondary/createtmplt.sh -s 2 -d 'SystemVM Template ( kvm )' -n 51ffe658-e2b2-4ad6-80ae-013191bd2288.qcow2 -t /tmp/tmp790604977550814143/template/tmpl/1/203/ -f /tmp/cloud/templates//51ffe658-e2b2-4ad6-80ae-013191bd2288.qcow2 -u -v -c
Uncompressing to /tmp/cloud/templates//51ffe658-e2b2-4ad6-80ae-013191bd2288.qcow2.tmp (type bz2)...could take a long time
Copying to /tmp/tmp790604977550814143/template/tmpl/1/203//51ffe658-e2b2-4ad6-80ae-013191bd2288.qcow2...could take a while
+ installrslt=
+ [[ 0 -ne 0 ]]
+ tmpdestdir=/tmp/cloud/templates/
+ '[' qcow2 == ova ']'
+ rm -rf /tmp/cloud/templates//51ffe658-e2b2-4ad6-80ae-013191bd2288.qcow2.tmp
+ tmpltfile=/tmp/tmp790604977550814143/template/tmpl/1/203/51ffe658-e2b2-4ad6-80ae-013191bd2288.qcow2
++ sudo ls -l /tmp/tmp790604977550814143/template/tmpl/1/203/51ffe658-e2b2-4ad6-80ae-013191bd2288.qcow2
++ awk '-F ' '{print $5}'
+ tmpltsize=567616512
+ [[ qcow2 == \q\c\o\w\2 ]]
++ sudo /usr/bin/qemu-img info /tmp/tmp790604977550814143/template/tmpl/1/203/51ffe658-e2b2-4ad6-80ae-013191bd2288.qcow2
++ grep -i 'virtual size'
++ xargs
++ sed -ne 's/.*(\([0-9]*\).*/\1/p'
+ vrtmpltsize=5242880000
+ templateId=203
+ sudo touch /tmp/tmp790604977550814143/template/tmpl/1/203/template.properties
+ echo qcow2=true
+ echo id=203
+ echo public=true
+ echo qcow2.filename=51ffe658-e2b2-4ad6-80ae-013191bd2288.qcow2
+ echo uniquename=routing-203
+ echo qcow2.virtualsize=5242880000
+ echo virtualsize=5242880000
+ echo qcow2.size=567616512
+ sudo cp /tmp/cloud/templates//template.properties /tmp/tmp790604977550814143/template/tmpl/1/203/template.properties
+ '[' -f /tmp/cloud/templates//template.properties ']'
+ rm -rf /tmp/cloud/templates//template.properties
+ echo 'Successfully installed system VM template /usr/share/cloudstack-management/templates/systemvm/systemvmtemplate-4.20.2-x86_64-kvm.qcow2.bz2 and template.properties to /tmp/tmp790604977550814143/template/tmpl/1/203'
Successfully installed system VM template /usr/share/cloudstack-management/templates/systemvm/systemvmtemplate-4.20.2-x86_64-kvm.qcow2.bz2 and template.properties to /tmp/tmp790604977550814143/template/tmpl/1/203
+ exit 0

2025-08-13 13:05:19,389 INFO  [c.c.u.SystemVmTemplateRegistration] (main:[]) (logid:) Unmounting store
2025-08-13 13:05:19,389 DEBUG [c.c.u.s.Script] (main:[]) (logid:) Executing command [/bin/bash -c sudo umount /tmp/tmp790604977550814143 ].
2025-08-13 13:05:19,572 DEBUG [c.c.u.s.Script] (main:[]) (logid:) Successfully executed process [40216] for command [/bin/bash -c sudo umount /tmp/tmp790604977550814143 ].
2025-08-13 13:05:19,576 DEBUG [c.c.u.SystemVmTemplateRegistration] (main:[]) (logid:) Updating System Vm Template IDs Complete
2025-08-13 13:05:19,581 INFO  [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components
2025-08-13 13:05:19,582 INFO  [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VersionDaoImpl.
2025-08-13 13:05:19,584 INFO  [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean null.
2025-08-13 13:05:19,585 INFO  [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components
2025-08-13 13:05:19,585 INFO  [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [system] in 90475 ms

@sureshanaparti
Copy link
Contributor

@blueorangutan test

@blueorangutan
Copy link

@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

@sureshanaparti sureshanaparti added this to the 4.21.0 milestone Aug 15, 2025
@sureshanaparti sureshanaparti merged commit 0b381d4 into apache:main Aug 15, 2025
43 of 49 checks passed
@github-project-automation github-project-automation bot moved this from In Progress to Done in Apache CloudStack 4.21.0 Aug 15, 2025
@sureshanaparti sureshanaparti deleted the ExcludeExternalHypervisorDuringUpgrades branch August 15, 2025 10:59
@blueorangutan
Copy link

[SF] Trillian test result (tid-14064)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 51248 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr11435-t14064-kvm-ol8.zip
Smoke tests completed. 145 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File
test_13_retrieve_vr_default_files Error 1.11 test_diagnostics.py
test_14_retrieve_vr_one_file Error 1.12 test_diagnostics.py
test_15_retrieve_ssvm_default_files Error 1.11 test_diagnostics.py
test_16_retrieve_ssvm_single_file Error 0.10 test_diagnostics.py
test_17_retrieve_cpvm_default_files Error 1.11 test_diagnostics.py
test_18_retrieve_cpvm_single_file Error 1.12 test_diagnostics.py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
No open projects
Status: Done
Development

Successfully merging this pull request may close these issues.

6 participants