Resize volume: add pool capacity disablethreshold for resize and allow volume auto migration #9761

weizhouapache · 2024-10-03T09:07:31Z

Description

Currently, when resize a volume, ACS checks if the current storage pool can accommodate the new size, and simply return true if pool has enough capacity, or false if there are not enough capacity.

This PR

add global and zone setting volume.resize.allowed.beyond.allocation and pool.storage.allocated.resize.capacity.disablethreshold
add API parameter automigrate to resizeVolume API.

when resize a volume

check if the pool capacity (with new volume size) is above disable threshold pool.storage.allocated.capacity.disablethreshold
if pool capacity is below disable threshold, operation succeeds.
if pool capacity is above disable threshold
- if volume.resize.allowed.beyond.allocation is false, go to migration step
- if volume.resize.allowed.beyond.allocation is true, it will check pool capacity for resize threshold pool.storage.allocated.resize.capacity.disablethreshold.
  - if yes, operation succeeds.
  - if not, go to migration step
migrate volume to another pool
- find storage pools by tags, storage types, pool allocation capacity (consider current volume size), and etc
- filter storage pool by resize capacity threshold (consider new volume size)
- if no pool found, operation fails.
- if automigrate is false, operation fails
- if automigrate is true, try to migrate volume to another pool
  - if migration succeeds (for example vm is stopped), operation succeeds
  - if migration fails (for example vm is running on KVM), operation succeeds

prior to this change

With this change

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI
test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

…e/ResizeVolumeCmd.java Co-authored-by: Suresh Kumar Anaparti <[email protected]>

weizhouapache · 2024-10-03T09:10:26Z

@blueorangutan package

blueorangutan · 2024-10-03T09:12:03Z

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

codecov · 2024-10-03T09:15:16Z

Codecov Report

Attention: Patch coverage is 50.76923% with 32 lines in your changes missing coverage. Please review.

Project coverage is 15.91%. Comparing base (046870e) to head (e92f38e).
Report is 72 commits behind head on 4.20.

Files with missing lines	Patch %	Lines
...n/java/com/cloud/storage/VolumeApiServiceImpl.java	40.00%	19 Missing and 8 partials ⚠️
...ain/java/com/cloud/storage/StorageManagerImpl.java	78.57%	3 Missing ⚠️
...stack/api/command/user/volume/ResizeVolumeCmd.java	33.33%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               4.20    #9761      +/-   ##
============================================
+ Coverage     15.78%   15.91%   +0.12%     
- Complexity    12552    12722     +170     
============================================
  Files          5625     5627       +2     
  Lines        491972   499027    +7055     
  Branches      63764    63574     -190     
============================================
+ Hits          77664    79425    +1761     
- Misses       405849   410931    +5082     
- Partials       8459     8671     +212

Flag	Coverage Δ
uitests	`4.04% <ø> (+<0.01%)`	⬆️
unittests	`16.73% <50.76%> (+0.12%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

blueorangutan · 2024-10-03T10:15:22Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11261

weizhouapache · 2024-10-03T10:55:59Z

@blueorangutan test

blueorangutan · 2024-10-03T10:58:03Z

@weizhouapache a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

weizhouapache · 2024-10-03T11:25:48Z

@blueorangutan package

blueorangutan · 2024-10-03T11:28:03Z

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-10-03T11:41:25Z

[SF] Trillian test result (tid-11602)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 380 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9761-t11602-kvm-ol8.zip
Smoke tests completed. 0 look OK, 0 have errors, 81 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
all_test_deploy_vm	`Skipped`	---	test_deploy_vm.py
all_test_escalations_templates	`Skipped`	---	test_escalations_templates.py
all_test_vm_ha	`Skipped`	---	test_vm_ha.py
all_test_vm_sync	`Skipped`	---	test_vm_sync.py
all_test_accounts	`Skipped`	---	test_accounts.py
all_test_affinity_groups_projects	`Skipped`	---	test_affinity_groups_projects.py
all_test_affinity_groups	`Skipped`	---	test_affinity_groups.py
all_test_async_job	`Skipped`	---	test_async_job.py
all_test_certauthority_root	`Skipped`	---	test_certauthority_root.py
all_test_create_list_domain_account_project	`Skipped`	---	test_create_list_domain_account_project.py
all_test_deploy_vgpu_enabled_vm	`Skipped`	---	test_deploy_vgpu_enabled_vm.py
all_test_deploy_virtio_scsi_vm	`Skipped`	---	test_deploy_virtio_scsi_vm.py
all_test_deploy_vm_iso	`Skipped`	---	test_deploy_vm_iso.py
all_test_deploy_vm_root_resize	`Skipped`	---	test_deploy_vm_root_resize.py
all_test_deploy_vms_with_varied_deploymentplanners	`Skipped`	---	test_deploy_vms_with_varied_deploymentplanners.py
all_test_deploy_vm_with_userdata	`Skipped`	---	test_deploy_vm_with_userdata.py
all_test_diagnostics	`Skipped`	---	test_diagnostics.py
all_test_direct_download	`Skipped`	---	test_direct_download.py
all_test_disk_offerings	`Skipped`	---	test_disk_offerings.py
all_test_domain_disk_offerings	`Skipped`	---	test_domain_disk_offerings.py
all_test_domain_network_offerings	`Skipped`	---	test_domain_network_offerings.py
all_test_domain_service_offerings	`Skipped`	---	test_domain_service_offerings.py
all_test_domain_vpc_offerings	`Skipped`	---	test_domain_vpc_offerings.py
all_test_dynamicroles	`Skipped`	---	test_dynamicroles.py
all_test_global_settings	`Skipped`	---	test_global_settings.py
all_test_guest_vlan_range	`Skipped`	---	test_guest_vlan_range.py
all_test_host_annotations	`Skipped`	---	test_host_annotations.py
all_test_hostha_simulator	`Skipped`	---	test_hostha_simulator.py
all_test_internal_lb	`Skipped`	---	test_internal_lb.py
all_test_iso	`Skipped`	---	test_iso.py
all_test_list_ids_parameter	`Skipped`	---	test_list_ids_parameter.py
all_test_loadbalance	`Skipped`	---	test_loadbalance.py
all_test_login	`Skipped`	---	test_login.py
all_test_metrics_api	`Skipped`	---	test_metrics_api.py
all_test_migration	`Skipped`	---	test_migration.py
all_test_multipleips_per_nic	`Skipped`	---	test_multipleips_per_nic.py
all_test_nested_virtualization	`Skipped`	---	test_nested_virtualization.py
all_test_network_acl	`Skipped`	---	test_network_acl.py
all_test_network	`Skipped`	---	test_network.py
all_test_nic_adapter_type	`Skipped`	---	test_nic_adapter_type.py
all_test_nic	`Skipped`	---	test_nic.py
all_test_non_contigiousvlan	`Skipped`	---	test_non_contigiousvlan.py
all_test_outofbandmanagement_nestedplugin	`Skipped`	---	test_outofbandmanagement_nestedplugin.py
all_test_outofbandmanagement	`Skipped`	---	test_outofbandmanagement.py
all_test_over_provisioning	`Skipped`	---	test_over_provisioning.py
all_test_password_server	`Skipped`	---	test_password_server.py
all_test_portable_publicip	`Skipped`	---	test_portable_publicip.py
all_test_portforwardingrules	`Skipped`	---	test_portforwardingrules.py
all_test_primary_storage	`Skipped`	---	test_primary_storage.py
all_test_privategw_acl	`Skipped`	---	test_privategw_acl.py
all_test_projects	`Skipped`	---	test_projects.py
all_test_public_ip_range	`Skipped`	---	test_public_ip_range.py
all_test_pvlan	`Skipped`	---	test_pvlan.py
all_test_regions	`Skipped`	---	test_regions.py
all_test_reset_vm_on_reboot	`Skipped`	---	test_reset_vm_on_reboot.py
all_test_resource_accounting	`Skipped`	---	test_resource_accounting.py
all_test_resource_detail	`Skipped`	---	test_resource_detail.py
all_test_router_dhcphosts	`Skipped`	---	test_router_dhcphosts.py
all_test_router_dns	`Skipped`	---	test_router_dns.py
all_test_router_dnsservice	`Skipped`	---	test_router_dnsservice.py
all_test_routers_iptables_default_policy	`Skipped`	---	test_routers_iptables_default_policy.py
all_test_routers_network_ops	`Skipped`	---	test_routers_network_ops.py
all_test_routers	`Skipped`	---	test_routers.py
all_test_scale_vm	`Skipped`	---	test_scale_vm.py
all_test_secondary_storage	`Skipped`	---	test_secondary_storage.py
all_test_service_offerings	`Skipped`	---	test_service_offerings.py
all_test_snapshots	`Skipped`	---	test_snapshots.py
all_test_ssvm	`Skipped`	---	test_ssvm.py
all_test_staticroles	`Skipped`	---	test_staticroles.py
all_test_templates	`Skipped`	---	test_templates.py
all_test_usage_events	`Skipped`	---	test_usage_events.py
all_test_usage	`Skipped`	---	test_usage.py
all_test_vm_deployment_planner	`Skipped`	---	test_vm_deployment_planner.py
all_test_vm_life_cycle	`Skipped`	---	test_vm_life_cycle.py
all_test_vm_snapshots	`Skipped`	---	test_vm_snapshots.py
all_test_volumes	`Skipped`	---	test_volumes.py
all_test_vpc_redundant	`Skipped`	---	test_vpc_redundant.py
all_test_vpc_router_nics	`Skipped`	---	test_vpc_router_nics.py
all_test_vpc_vpn	`Skipped`	---	test_vpc_vpn.py
all_test_host_maintenance	`Skipped`	---	test_host_maintenance.py
all_test_hostha_kvm	`Skipped`	---	test_hostha_kvm.py

sureshanaparti

clgtm

blueorangutan · 2024-10-03T12:41:36Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11264

sureshanaparti · 2024-10-03T13:11:48Z

@blueorangutan test

blueorangutan · 2024-10-03T13:14:03Z

@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

weizhouapache · 2024-10-03T15:07:36Z

@blueorangutan test

blueorangutan · 2024-10-03T15:08:03Z

@weizhouapache a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2024-10-04T06:14:08Z

[SF] Trillian test result (tid-11607)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 52087 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9761-t11607-kvm-ol8.zip
Smoke tests completed. 140 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_secure_vm_migration	`Error`	377.75	test_vm_life_cycle.py

blueorangutan · 2024-10-04T07:56:05Z

[SF] Trillian test result (tid-11603)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 65044 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9761-t11603-kvm-ol8.zip
Smoke tests completed. 139 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_secure_vm_migration	`Error`	134.20	test_vm_life_cycle.py
test_01_secure_vm_migration	`Error`	134.21	test_vm_life_cycle.py
test_04_deploy_vnf_appliance	`Error`	477.32	test_vnf_templates.py
test_04_deploy_vnf_appliance	`Error`	477.32	test_vnf_templates.py
test_05_delete_vnf_template	`Error`	1.22	test_vnf_templates.py
ContextSuite context=TestVnfTemplates>:teardown	`Error`	1.43	test_vnf_templates.py

blueorangutan · 2024-11-27T19:56:04Z

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-11-27T20:49:05Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✖️ debian ✔️ suse15. SL-JID 11617

weizhouapache · 2024-11-28T08:04:17Z

@blueorangutan package

blueorangutan · 2024-11-28T08:06:08Z

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-11-28T09:07:02Z

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 11625

blueorangutan · 2024-11-28T10:17:47Z

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 11630

blueorangutan · 2024-11-28T13:01:15Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11635

weizhouapache · 2024-11-28T13:02:56Z

@blueorangutan test

blueorangutan · 2024-11-28T13:04:07Z

@weizhouapache a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2024-11-28T14:05:25Z

[SF] Trillian Build Failed (tid-11811)

JoaoJandre · 2024-11-28T17:52:10Z

@weizhouapache does the migration take flexible tags into account (not only normal tags)?

weizhouapache · 2024-11-28T18:11:43Z

@weizhouapache does the migration take flexible tags into account (not only normal tags)?

To be frank, I have not tested flexible tags.
This PR calls an existing method listStoragePoolsForSystemMigrationOfVolume to list suitable storage pools.

If there are some issues with the method, we can fix it in a separated PR.

rohityadavcloud · 2024-11-29T07:31:43Z

@blueorangutan package

blueorangutan · 2024-11-29T07:34:06Z

@rohityadavcloud a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

weizhouapache · 2024-11-29T08:09:14Z

@blueorangutan package

blueorangutan · 2024-11-29T08:10:06Z

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-11-29T08:29:12Z

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 11647

blueorangutan · 2024-11-29T10:05:03Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11650

weizhouapache · 2024-11-29T10:22:05Z

@blueorangutan test

blueorangutan · 2024-11-29T10:24:07Z

@weizhouapache a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

JoaoJandre · 2024-11-29T11:51:06Z

@weizhouapache does the migration take flexible tags into account (not only normal tags)?

To be frank, I have not tested flexible tags. This PR calls an existing method listStoragePoolsForSystemMigrationOfVolume to list suitable storage pools.

If there are some issues with the method, we can fix it in a separated PR.

@weizhouapache from what I have gathered reviewing the code. If the volume is attached to a running VM, flexible tags will be considered. Otherwise not... This should be fixed.

weizhouapache · 2024-11-29T12:43:04Z

@weizhouapache from what I have gathered reviewing the code. If the volume is attached to a running VM, flexible tags will be considered. Otherwise not... This should be fixed.

@JoaoJandre
thanks for looking into it. Agree it should be fixed.

since you are the author of flexible tags (PR #7489), could you please have a look ?
I am not familiar with flexible tags.

blueorangutan · 2024-11-30T03:07:38Z

[SF] Trillian test result (tid-11823)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 57637 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9761-t11823-kvm-ol8.zip
Smoke tests completed. 140 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_secure_vm_migration	`Error`	134.14	test_vm_life_cycle.py
test_01_secure_vm_migration	`Error`	134.15	test_vm_life_cycle.py

…w volume auto migration (apache#9761) * server: add global settings for volume resize * resizeVolume: support automigrate * Address Suresh's comments * Update api/src/main/java/org/apache/cloudstack/api/command/user/volume/ResizeVolumeCmd.java Co-authored-by: Suresh Kumar Anaparti <[email protected]> * address Suresh's comments * UI: add autoMigrate to resizeVolume * resizevolume: add unit tests * resizevolume: add unit test for Allocated volume --------- Co-authored-by: Suresh Kumar Anaparti <[email protected]>

weizhouapache and others added 5 commits October 2, 2024 16:20

server: add global settings for volume resize

a84e65e

resizeVolume: support automigrate

97a64ab

Address Suresh's comments

0afa9cf

Update api/src/main/java/org/apache/cloudstack/api/command/user/volum…

30bdc38

…e/ResizeVolumeCmd.java Co-authored-by: Suresh Kumar Anaparti <[email protected]>

address Suresh's comments

b07f591

boring-cyborg bot added component:api component:storage labels Oct 3, 2024

UI: add autoMigrate to resizeVolume

2733d10

boring-cyborg bot added the component:UI label Oct 3, 2024

sureshanaparti added this to the 4.20.1.0 milestone Oct 3, 2024

sureshanaparti added the status:needs-testing label Oct 3, 2024

sureshanaparti approved these changes Oct 3, 2024

View reviewed changes

weizhouapache mentioned this pull request Oct 3, 2024

Resize volume: add pool capacity disablethreshold for resize and allow volume auto migration apache/cloudstack-documentation#443

Merged

resizevolume: add unit tests

ae6acc1

rohityadavcloud merged commit 8a1da38 into apache:4.20 Dec 2, 2024
25 checks passed

rohityadavcloud removed the status:ready-for-merge label Dec 2, 2024

DaanHoogland deleted the 4.20-resize-volume-threshold-automigrate branch December 2, 2024 07:55

JoaoJandre mentioned this pull request Dec 2, 2024

Fix com.cloud.server.ManagementServerImpl#listStoragePoolsForSystemMigrationOfVolume #10021

Open

Pearl1594 added this to ACS 4.20.1 Mar 17, 2025

Pearl1594 moved this to Done in ACS 4.20.1 Mar 17, 2025

Resize volume: add pool capacity disablethreshold for resize and allow volume auto migration #9761

Resize volume: add pool capacity disablethreshold for resize and allow volume auto migration #9761

Uh oh!

Conversation

weizhouapache commented Oct 3, 2024

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

weizhouapache commented Oct 3, 2024

Uh oh!

blueorangutan commented Oct 3, 2024

Uh oh!

codecov bot commented Oct 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

blueorangutan commented Oct 3, 2024

Uh oh!

weizhouapache commented Oct 3, 2024

Uh oh!

blueorangutan commented Oct 3, 2024

Uh oh!

weizhouapache commented Oct 3, 2024

Uh oh!

blueorangutan commented Oct 3, 2024

Uh oh!

blueorangutan commented Oct 3, 2024

Uh oh!

sureshanaparti left a comment

Choose a reason for hiding this comment

Uh oh!

blueorangutan commented Oct 3, 2024

Uh oh!

sureshanaparti commented Oct 3, 2024

Uh oh!

blueorangutan commented Oct 3, 2024

Uh oh!

weizhouapache commented Oct 3, 2024

Uh oh!

blueorangutan commented Oct 3, 2024

Uh oh!

blueorangutan commented Oct 4, 2024

Uh oh!

blueorangutan commented Oct 4, 2024

Uh oh!

blueorangutan commented Nov 27, 2024

Uh oh!

blueorangutan commented Nov 27, 2024

Uh oh!

weizhouapache commented Nov 28, 2024

Uh oh!

blueorangutan commented Nov 28, 2024

Uh oh!

blueorangutan commented Nov 28, 2024

Uh oh!

blueorangutan commented Nov 28, 2024

Uh oh!

blueorangutan commented Nov 28, 2024

Uh oh!

weizhouapache commented Nov 28, 2024

Uh oh!

blueorangutan commented Nov 28, 2024

Uh oh!

blueorangutan commented Nov 28, 2024

Uh oh!

JoaoJandre commented Nov 28, 2024

Uh oh!

weizhouapache commented Nov 28, 2024

Uh oh!

rohityadavcloud commented Nov 29, 2024

Uh oh!

blueorangutan commented Nov 29, 2024

Uh oh!

weizhouapache commented Nov 29, 2024

codecov bot commented Oct 3, 2024 •

edited

Loading