VMware 80u2 and 80u3 updates/fixes #10586

sureshanaparti · 2025-03-19T08:10:38Z

Description

This PR addresses power on & move datastore issues with VMware 80u2 and 80u3, and fixes tests for restore vm (disk cleanup issue), ssvm/cpvm (wait time to disconnect). Following are the details:

Power On issue
lock issue during PowerOn operation (for ROOT disk, which is cloned from seeded template)

https://knowledge.broadcom.com/external/article/316576/troubleshooting-issues-resulting-from-lo.html

Now, PowerOn operation is being tried after lock issue, and succeeds after few attempts.

Move datastore file issue
access issue with move datastore file operation (when creating volume from snapshot)

Now, Move datastore file operation is being tried after access issue, and succeeds after few attempts.

Restore VM test behavior/issue
test checks for old root disk existence and trying to cleanup.

But for VMware, old root disk is force expunged irrespective of the expunge parameter in restoreVirtualMachine API (which was introduced in PR #8800 - 4.19.1.0), so no need to check for old root disk exists when expunge is false.

cloudstack/engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/VolumeOrchestrator.java

Lines 1187 to 1191 in 2674da2

    
           // In case of VMware VM will continue to use the old root disk until expunged, so force expunge old root disk 
        
           // For system VM we do not need volume entry in Destroy state 
        
           if (vm.getHypervisorType() == HypervisorType.VMware || vm.getType().isUsedBySystem()) { 
        
               logger.info(String.format("Trying to expunge volume [%s] from primary data storage.", volumeToString)); 
        
               AsyncCallFuture<VolumeApiResult> future = volService.expungeVolumeAsync(volFactory.getVolume(existingVolume.getId()));

SSVM / CPVM test issue
Agent will be in Up state for a while after reboot, so need to wait for sometime for the agent to Disconnect and back to Up. Added wait time after reboot.

Other PR references:

[VMware] Add support for VMware 8.0u2 (8.0.2.x) and 8.0u3 (8.0.3.x) #9591 (4.20.0.0) - Updates hypervisor capabilities and guest os mappings for VMware 80u2 and 80u3.
SystemVM template changes - updated debian version & other changes #9625 (4.20.0.0) - Addresses SystemVM template issues for VMware 80u2 and 80u3.

VMware version references:

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI
test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Tested VM/Volume operations manually, and smoke tests.

How did you try to break this feature and the system with this change?

sureshanaparti · 2025-03-19T08:12:06Z

@blueorangutan package

codecov · 2025-03-19T08:18:50Z

Codecov Report

Attention: Patch coverage is 0% with 129 lines in your changes missing coverage. Please review.

Project coverage is 16.99%. Comparing base (dd84c74) to head (69d2c10).
Report is 18 commits behind head on 4.20.

Files with missing lines	Patch %	Lines
...oud/hypervisor/vmware/resource/VmwareResource.java	0.00%	63 Missing ⚠️
...ud/storage/resource/VmwareStorageLayoutHelper.java	0.00%	27 Missing ⚠️
...m/cloud/hypervisor/vmware/mo/VirtualMachineMO.java	0.00%	22 Missing ⚠️
...cloud/storage/resource/VmwareStorageProcessor.java	0.00%	16 Missing ⚠️
.../src/main/java/com/cloud/vm/UserVmManagerImpl.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##              4.20   #10586       +/-   ##
============================================
+ Coverage     4.00%   16.99%   +12.98%     
- Complexity       0    13238    +13238     
============================================
  Files          400     5255     +4855     
  Lines        32555   464813   +432258     
  Branches      5770    54518    +48748     
============================================
+ Hits          1305    78974    +77669     
- Misses       31101   377045   +345944     
- Partials       149     8794     +8645

Flag	Coverage Δ
uitests	`?`
unittests	`16.99% <0.00%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

sureshanaparti · 2025-03-19T08:34:29Z

@blueorangutan package

blueorangutan · 2025-03-19T08:36:04Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

...ns/hypervisors/vmware/src/main/java/com/cloud/hypervisor/vmware/resource/VmwareResource.java

weizhouapache

lgtm

blueorangutan · 2025-03-19T09:42:28Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12828

sureshanaparti · 2025-03-19T10:22:11Z

@blueorangutan package

blueorangutan · 2025-03-19T10:24:04Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-03-19T11:28:04Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12829

sureshanaparti · 2025-03-19T11:30:02Z

@blueorangutan test ol8 vmware-80u3

blueorangutan · 2025-03-19T11:32:04Z

@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + vmware-80u3) has been kicked to run smoke tests

sureshanaparti · 2025-03-19T12:26:37Z

@blueorangutan test ol8 vmware-80u2

blueorangutan · 2025-03-19T12:28:04Z

@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + vmware-80u2) has been kicked to run smoke tests

sureshanaparti · 2025-03-19T12:28:23Z

@blueorangutan test ol8 vmware-80u1

blueorangutan · 2025-03-19T12:30:03Z

@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + vmware-80u1) has been kicked to run smoke tests

blueorangutan · 2025-03-20T10:04:25Z

[SF] Trillian test result (tid-12768)
Environment: vmware-80u3 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 78805 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10586-t12768-vmware-80u3.zip
Smoke tests completed. 134 look OK, 7 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_deploy_more_vms_than_limit_allows	`Error`	223.40	test_deploy_vms_in_parallel.py
test_03_deploy_and_scale_kubernetes_cluster	`Failure`	102.81	test_kubernetes_clusters.py
test_05_basic_lifecycle_kubernetes_cluster	`Error`	88.31	test_kubernetes_clusters.py
test_06_delete_kubernetes_cluster	`Failure`	255.87	test_kubernetes_clusters.py
test_reboot_router	`Error`	459.86	test_network.py
ContextSuite context=TestSharedNetworkWithConfigDrive>:setup	`Error`	1521.98	test_network.py
test_02_restore_vm_with_disk_offering	`Error`	104.27	test_restore_vm.py
test_03_restore_vm_with_disk_offering_custom_size	`Error`	60.25	test_restore_vm.py
test_01_snapshot_to_volume	`Error`	9.37	test_snapshots.py
test_08_reboot_cpvm	`Failure`	39.37	test_ssvm.py
test_01_volume_usage	`Error`	95.45	test_usage.py

blueorangutan · 2025-03-20T10:22:57Z

[SF] Trillian test result (tid-12771)
Environment: vmware-80u1 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 76196 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10586-t12771-vmware-80u1.zip
Smoke tests completed. 136 look OK, 5 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_09_connectivity_between_network_and_vpc_tier	`Failure`	34.56	test_ipv4_routing.py
test_11_isolated_network_with_dynamic_routed_mode	`Failure`	163.88	test_ipv4_routing.py
test_12_vpc_and_tier_with_dynamic_routed_mode	`Failure`	411.53	test_ipv4_routing.py
test_01_ping_in_vr_success	`Failure`	60.52	test_diagnostics.py
test_04_autoscale_kubernetes_cluster	`Failure`	158.03	test_kubernetes_clusters.py
test_05_basic_lifecycle_kubernetes_cluster	`Failure`	800.65	test_kubernetes_clusters.py
ContextSuite context=TestSharedNetworkWithConfigDrive>:setup	`Error`	1524.38	test_network.py
test_02_restore_vm_with_disk_offering	`Error`	58.20	test_restore_vm.py
test_03_restore_vm_with_disk_offering_custom_size	`Error`	114.73	test_restore_vm.py

weizhouapache · 2025-03-20T10:34:44Z

@blueorangutan package

blueorangutan · 2025-03-20T10:36:03Z

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-03-20T11:49:32Z

Packaging result [SF]: ✔️ el8 ✖️ el9 ✔️ debian ✖️ suse15. SL-JID 12844

blueorangutan · 2025-03-20T11:54:54Z

[SF] Trillian test result (tid-12770)
Environment: vmware-80u2 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 80994 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10586-t12770-vmware-80u2.zip
Smoke tests completed. 134 look OK, 7 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestClusterDRS>:setup	`Error`	0.00	test_cluster_drs.py
test_02_internallb_roundrobin_1RVPC_3VM_HTTP_port80	`Failure`	488.47	test_internal_lb.py
test_02_internallb_roundrobin_1RVPC_3VM_HTTP_port80	`Error`	488.49	test_internal_lb.py
test_03_vpc_internallb_haproxy_stats_on_all_interfaces	`Error`	123.15	test_internal_lb.py
test_03_vpc_internallb_haproxy_stats_on_all_interfaces	`Error`	123.16	test_internal_lb.py
test_04_rvpc_internallb_haproxy_stats_on_all_interfaces	`Error`	331.55	test_internal_lb.py
test_04_rvpc_internallb_haproxy_stats_on_all_interfaces	`Error`	331.58	test_internal_lb.py
test_09_connectivity_between_network_and_vpc_tier	`Error`	149.42	test_ipv4_routing.py
test_08_upgrade_kubernetes_ha_cluster	`Failure`	3692.24	test_kubernetes_clusters.py
ContextSuite context=TestSharedNetworkWithConfigDrive>:setup	`Error`	1524.99	test_network.py
test_02_restore_vm_with_disk_offering	`Error`	60.36	test_restore_vm.py
test_03_restore_vm_with_disk_offering_custom_size	`Error`	102.21	test_restore_vm.py
test_01_volume_usage	`Error`	88.37	test_usage.py

Pearl1594 · 2025-03-21T19:31:22Z

@blueorangutan test ol8 vmware-80u1

blueorangutan · 2025-05-10T06:39:35Z

[SF] Trillian test result (tid-13243)
Environment: vmware-80u3 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 87874 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10586-t13243-vmware-80u3.zip
Smoke tests completed. 137 look OK, 4 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestClusterDRS>:setup	`Error`	0.00	test_cluster_drs.py
test_deploy_more_vms_than_limit_allows	`Error`	217.70	test_deploy_vms_in_parallel.py
test_01_deployVMInSharedNetwork	`Failure`	3624.54	test_network.py
ContextSuite context=TestSharedNetworkWithConfigDrive>:teardown	`Error`	3625.85	test_network.py
test_03_restore_vm_with_disk_offering_custom_size	`Error`	73.04	test_restore_vm.py

…n't exists

…ume from snapshot

…h more size than the template size

sureshanaparti · 2025-05-14T11:00:09Z

@blueorangutan package

blueorangutan · 2025-05-14T11:02:03Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-05-14T12:07:50Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13381

DaanHoogland · 2025-05-14T13:19:04Z

@blueorangutan test ol8 vmware-80u3

blueorangutan · 2025-05-14T13:20:04Z

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + vmware-80u3) has been kicked to run smoke tests

blueorangutan · 2025-05-16T00:07:42Z

[SF] Trillian test result (tid-13307)
Environment: vmware-80u3 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 122502 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10586-t13307-vmware-80u3.zip
Smoke tests completed. 137 look OK, 4 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_08_upgrade_kubernetes_ha_cluster	`Failure`	3711.40	test_kubernetes_clusters.py
test_01_deployVMInSharedNetwork	`Error`	414.13	test_network.py
test_03_restore_vm_with_disk_offering_custom_size	`Error`	67.66	test_restore_vm.py
test_01_migrate_vm_strict_tags_success	`Error`	3607.88	test_vm_strict_host_tags.py
test_02_migrate_vm_strict_tags_failure	`Error`	4.12	test_vm_strict_host_tags.py
test_01_restore_vm_strict_tags_success	`Error`	23.90	test_vm_strict_host_tags.py
test_02_restore_vm_strict_tags_failure	`Error`	3604.35	test_vm_strict_host_tags.py
test_01_scale_vm_strict_tags_success	`Error`	24.86	test_vm_strict_host_tags.py
test_02_scale_vm_strict_tags_failure	`Error`	3607.30	test_vm_strict_host_tags.py
test_01_deploy_vm_on_specific_host_without_strict_tags	`Error`	23.42	test_vm_strict_host_tags.py
test_02_deploy_vm_on_any_host_without_strict_tags	`Error`	3605.85	test_vm_strict_host_tags.py
test_03_deploy_vm_on_specific_host_with_strict_tags_success	`Error`	6.14	test_vm_strict_host_tags.py
test_04_deploy_vm_on_any_host_with_strict_tags_success	`Error`	6.03	test_vm_strict_host_tags.py

sureshanaparti · 2025-05-16T07:26:58Z

@blueorangutan test ol8 vmware-80u2

blueorangutan · 2025-05-16T07:28:04Z

@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + vmware-80u2) has been kicked to run smoke tests

sureshanaparti · 2025-05-16T07:28:28Z

@blueorangutan test ol8 vmware-80u1

blueorangutan · 2025-05-16T07:30:04Z

@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + vmware-80u1) has been kicked to run smoke tests

vladimirpetrov

LGTM based on manual testing.

blueorangutan · 2025-05-17T05:17:57Z

[SF] Trillian test result (tid-13328)
Environment: vmware-80u2 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 75117 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10586-t13328-vmware-80u2.zip
Smoke tests completed. 137 look OK, 4 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_deploy_more_vms_than_limit_allows	`Error`	196.18	test_deploy_vms_in_parallel.py
test_01_deployVMInSharedNetwork	`Failure`	3604.36	test_network.py
ContextSuite context=TestSharedNetworkWithConfigDrive>:teardown	`Error`	3605.64	test_network.py
test_03_restore_vm_with_disk_offering_custom_size	`Error`	57.38	test_restore_vm.py
test_01_scale_vm	`Error`	1.53	test_scale_vm.py
test_02_scale_vm_negative_offering_disable_scaling	`Error`	1.49	test_scale_vm.py
test_03_scale_vm_negative_vm_disable_scaling	`Error`	1.46	test_scale_vm.py

* VMware - Ignore disk not found error on cleanup when the VM disk doesn't exists * VMware - Retry powerOn on lock issues * addressed comments * Update CPVM reboot tests - wait for the agent to Disconnect and back Up * Retry moveDatastoreFile when any file access issue while creating volume from snapshot * Update full clone flag when restoring vm using root disk offering with more size than the template size * refactored (mainly,for diskInfo - causing NPE in some cases) * Retry moveDatastoreFile when there is any file access issue

boring-cyborg bot added the component:vmware label Mar 19, 2025

sureshanaparti changed the base branch from main to 4.20 March 19, 2025 08:10

sureshanaparti requested a review from weizhouapache March 19, 2025 08:13

weizhouapache reviewed Mar 19, 2025

View reviewed changes

...ns/hypervisors/vmware/src/main/java/com/cloud/hypervisor/vmware/resource/VmwareResource.java Outdated Show resolved Hide resolved

weizhouapache approved these changes Mar 19, 2025

View reviewed changes

rohityadavcloud added this to the 4.20.1 milestone Mar 19, 2025

rohityadavcloud approved these changes Mar 19, 2025

View reviewed changes

Pearl1594 added this to ACS 4.20.1 Mar 19, 2025

Pearl1594 moved this to In Progress in ACS 4.20.1 Mar 19, 2025

sureshanaparti added 6 commits May 13, 2025 15:57

VMware - Ignore disk not found error on cleanup when the VM disk does…

6993e91

…n't exists

VMware - Retry powerOn on lock issues

7846c18

addressed comments

470d3ad

Update CPVM reboot tests - wait for the agent to Disconnect and back Up

06248c5

Retry moveDatastoreFile when any file access issue while creating vol…

22fbe4e

…ume from snapshot

Update full clone flag when restoring vm using root disk offering wit…

e8ada29

…h more size than the template size

DaanHoogland assigned sureshanaparti May 13, 2025

sureshanaparti added 2 commits May 14, 2025 15:48

refactored (mainly,for diskInfo - causing NPE in some cases)

c9073c4

Retry moveDatastoreFile when there is any file access issue

69d2c10

sureshanaparti force-pushed the vmware-80u2-and-80u3-updates branch from f0d113e to 69d2c10 Compare May 14, 2025 10:59

apache deleted a comment from blueorangutan May 16, 2025

vladimirpetrov approved these changes May 16, 2025

View reviewed changes

rohityadavcloud merged commit 90316b2 into apache:4.20 May 16, 2025
21 of 26 checks passed

github-project-automation bot moved this from In Progress to Done in ACS 4.20.1 May 16, 2025

rohityadavcloud deleted the vmware-80u2-and-80u3-updates branch May 16, 2025 19:09

sureshanaparti mentioned this pull request May 19, 2025

test: fix test_restore_vm failure on vmware #10885

Merged

14 tasks

sureshanaparti mentioned this pull request Nov 12, 2025

Unable to access vmdk sice it is locked #12035

Open

	// In case of VMware VM will continue to use the old root disk until expunged, so force expunge old root disk
	// For system VM we do not need volume entry in Destroy state
	if (vm.getHypervisorType() == HypervisorType.VMware \|\| vm.getType().isUsedBySystem()) {
	logger.info(String.format("Trying to expunge volume [%s] from primary data storage.", volumeToString));
	AsyncCallFuture<VolumeApiResult> future = volService.expungeVolumeAsync(volFactory.getVolume(existingVolume.getId()));

VMware 80u2 and 80u3 updates/fixes #10586

VMware 80u2 and 80u3 updates/fixes #10586

Uh oh!

Conversation

sureshanaparti commented Mar 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

sureshanaparti commented Mar 19, 2025

Uh oh!

codecov bot commented Mar 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sureshanaparti commented Mar 19, 2025

Uh oh!

blueorangutan commented Mar 19, 2025

Uh oh!

Uh oh!

weizhouapache left a comment

Choose a reason for hiding this comment

Uh oh!

blueorangutan commented Mar 19, 2025

Uh oh!

sureshanaparti commented Mar 19, 2025

Uh oh!

blueorangutan commented Mar 19, 2025

Uh oh!

blueorangutan commented Mar 19, 2025

Uh oh!

sureshanaparti commented Mar 19, 2025

Uh oh!

blueorangutan commented Mar 19, 2025

Uh oh!

sureshanaparti commented Mar 19, 2025

Uh oh!

blueorangutan commented Mar 19, 2025

Uh oh!

sureshanaparti commented Mar 19, 2025

Uh oh!

blueorangutan commented Mar 19, 2025

Uh oh!

blueorangutan commented Mar 20, 2025

Uh oh!

blueorangutan commented Mar 20, 2025

Uh oh!

weizhouapache commented Mar 20, 2025

Uh oh!

blueorangutan commented Mar 20, 2025

Uh oh!

blueorangutan commented Mar 20, 2025

Uh oh!

blueorangutan commented Mar 20, 2025

Uh oh!

Pearl1594 commented Mar 21, 2025

Uh oh!

blueorangutan commented May 10, 2025

Uh oh!

sureshanaparti commented May 14, 2025

Uh oh!

blueorangutan commented May 14, 2025

Uh oh!

blueorangutan commented May 14, 2025

Uh oh!

DaanHoogland commented May 14, 2025

Uh oh!

blueorangutan commented May 14, 2025

Uh oh!

blueorangutan commented May 16, 2025

Uh oh!

sureshanaparti commented May 16, 2025

Uh oh!

blueorangutan commented May 16, 2025

sureshanaparti commented Mar 19, 2025 •

edited

Loading

codecov bot commented Mar 19, 2025 •

edited

Loading