Fix ordering of secondary storages with the algorithm `firstfitleastconsumed` #8557

GaOrtiga · 2024-01-23T18:21:07Z

The algorithm firstfitleastconsumed orders the secondary storages based on how much free capacity each has and returns the first item of the list that has enough free capacity. However, the ordering of this list is inverted, putting the storages with the least free capacity at the top.

This behaviour has been adjusted, altering the order of the list to guarantee that the secondary storages with the most free capacity get allocated first.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

codecov · 2024-01-24T04:35:10Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 16.92%. Comparing base (fd24509) to head (5f325a7).
⚠️ Report is 653 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #8557      +/-   ##
============================================
+ Coverage     16.07%   16.92%   +0.85%     
- Complexity    12885    12893       +8     
============================================
  Files          5642     5248     -394     
  Lines        494039   461620   -32419     
  Branches      59912    54182    -5730     
============================================
- Hits          79408    78135    -1273     
+ Misses       405828   374830   -30998     
+ Partials       8803     8655     -148

Flag	Coverage Δ
uitests	`?`
unittests	`16.92% <100.00%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

...src/main/java/org/apache/cloudstack/storage/image/manager/ImageStoreProviderManagerImpl.java

sureshanaparti · 2024-01-24T06:04:05Z

@blueorangutan package

blueorangutan · 2024-01-24T06:06:04Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-01-24T07:16:41Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 8428

DaanHoogland

clgtm

github-actions · 2024-02-08T13:19:11Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

GutoVeronezi

CLGTM

GaOrtiga · 2024-03-01T17:06:22Z

@blueorangutan package

blueorangutan · 2024-03-01T17:08:03Z

@GaOrtiga a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-03-01T18:18:54Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 8833

DaanHoogland · 2024-03-04T10:59:54Z

@sureshanaparti is this ok now? do we need more testing?

JoaoJandre · 2024-04-16T12:49:48Z

@blueorangutan package

blueorangutan · 2024-04-16T12:52:03Z

@JoaoJandre a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-04-16T14:04:38Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 9278

JoaoJandre · 2024-04-16T14:55:31Z

@DaanHoogland @sureshanaparti @rohityadavcloud @shwstppr could we run the CI here?

DaanHoogland · 2024-04-16T15:05:21Z

ping @sureshanaparti

BryanMLima · 2024-04-24T13:44:53Z

@sureshanaparti could you trigger the CI here?
cc @DaanHoogland

DaanHoogland · 2024-04-25T06:44:54Z

@sureshanaparti could you trigger the CI here? cc @DaanHoogland

sorry @BryanMLima , our lab is a bit tired at the moment. I first wanted to hear from @sureshanaparti if he is alright with the current state of the PR

DaanHoogland · 2024-04-25T06:45:02Z

@blueorangutan test

blueorangutan · 2024-04-25T06:46:05Z

@DaanHoogland a [SL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2024-04-25T21:15:23Z

[SF] Trillian test result (tid-10013)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 50003 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8557-t10013-kvm-centos7.zip
Smoke tests completed. 126 look OK, 3 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_events_resource	`Error`	312.05	test_events_resource.py
test_01_events_resource	`Error`	312.06	test_events_resource.py
test_04_deploy_vm_for_other_user_and_test_vm_operations	`Failure`	94.86	test_network_permissions.py
ContextSuite context=TestNetworkPermissions>:teardown	`Error`	1.50	test_network_permissions.py
test_hostha_enable_ha_when_host_disabled	`Error`	4.99	test_hostha_kvm.py
test_hostha_enable_ha_when_host_in_maintenance	`Error`	304.10	test_hostha_kvm.py

BryanMLima · 2024-06-12T18:22:23Z

@DaanHoogland @sureshanaparti, could you trigger the CI here?

github-actions · 2024-06-25T14:12:07Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

sureshanaparti · 2024-06-25T15:17:29Z

@BryanMLima @GaOrtiga pls resolve the conflicts. thanks.

blueorangutan · 2024-09-18T20:23:07Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11147

DaanHoogland · 2024-09-19T18:35:44Z

@blueorangutan LLtest

blueorangutan · 2024-09-19T18:38:03Z

@DaanHoogland a [LL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

…rages_with_firstfitleastconsumed

winterhazel · 2025-01-02T15:18:45Z

@blueorangutan package

blueorangutan · 2025-01-02T15:20:04Z

@winterhazel a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-01-02T16:28:56Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11974

winterhazel

Looks good, did some tests manually in an environment with two secondary storages (one with 26 GB available and another with 50 GB available) using firstfitleastconsumed. Before the changes, the storage with 26 GB would be listed first. After the patch, the 50 GB storage gets listed first.

winterhazel · 2025-01-02T16:32:32Z

@DaanHoogland could we run the CI to proceed with the merge?

sureshanaparti

clgtm

DaanHoogland · 2025-01-03T08:20:26Z

@blueorangutan test

blueorangutan · 2025-01-03T08:22:04Z

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2025-01-04T14:54:21Z

[SF] Trillian test result (tid-12035)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 107360 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8557-t12035-kvm-ol8.zip
Smoke tests completed. 139 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_invalid_upgrade_kubernetes_cluster	`Failure`	3721.71	test_kubernetes_clusters.py
test_02_upgrade_kubernetes_cluster	`Failure`	3639.49	test_kubernetes_clusters.py
test_03_deploy_and_scale_kubernetes_cluster	`Failure`	3646.34	test_kubernetes_clusters.py
test_04_autoscale_kubernetes_cluster	`Failure`	3641.96	test_kubernetes_clusters.py
test_05_basic_lifecycle_kubernetes_cluster	`Failure`	3652.30	test_kubernetes_clusters.py
test_06_delete_kubernetes_cluster	`Failure`	3637.52	test_kubernetes_clusters.py
test_08_upgrade_kubernetes_ha_cluster	`Failure`	3671.51	test_kubernetes_clusters.py
test_10_vpc_tier_kubernetes_cluster	`Failure`	3698.47	test_kubernetes_clusters.py
test_create_pvlan_network	`Error`	0.10	test_pvlan.py

winterhazel · 2025-01-15T12:58:41Z

@DaanHoogland test failures seem unrelated and also failed in other PRs, could we rerun or proceed with the merge?

DaanHoogland · 2025-01-15T14:17:57Z

@blueorangutan package

blueorangutan · 2025-01-15T14:20:03Z

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

DaanHoogland · 2025-01-15T14:30:48Z

@DaanHoogland test failures seem unrelated and also failed in other PRs, could we rerun or proceed with the merge?

@winterhazel , the kubernetes failures were an issue on main that should be fixed. I'm doing an extra round just to be sure about the pvlan error. I don't think it is related though.

ok?

blueorangutan · 2025-01-15T15:23:44Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12089

DaanHoogland · 2025-01-15T16:06:39Z

@blueorangutan test

blueorangutan · 2025-01-15T16:08:03Z

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2025-01-15T16:10:44Z

[SF] Trillian Build Failed (tid-12111)

blueorangutan · 2025-01-15T16:55:40Z

[SF] Trillian Build Failed (tid-12112)

blueorangutan · 2025-01-17T02:44:53Z

[LL]Trillian test result (tid-7055)
Environment: kvm-rocky8 (x2), Advanced Networking with Mgmt server r8
Total time taken: 52493 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8557-t7055-kvm-rocky8.zip
Smoke tests completed. 138 look OK, 3 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_11_isolated_network_with_dynamic_routed_mode	`Error`	0.23	test_ipv4_routing.py
test_12_vpc_and_tier_with_dynamic_routed_mode	`Error`	3.27	test_ipv4_routing.py
test_12_vpc_and_tier_with_dynamic_routed_mode	`Error`	3.27	test_ipv4_routing.py
test_02_list_cpvm_vm	`Failure`	0.02	test_ssvm.py
test_04_cpvm_internals	`Failure`	0.02	test_ssvm.py
test_01_redundant_vpc_site2site_vpn	`Failure`	407.42	test_vpc_vpn.py

…onsumed` (apache#8557) * Fix ordering of secondary storages with the algorithm `firstfitleastconsumed` * return store without checking all * Add unit tests --------- Co-authored-by: Gabriel <[email protected]> Co-authored-by: Fabricio Duarte <[email protected]>

boring-cyborg bot added the component:storage label Jan 23, 2024

sureshanaparti reviewed Jan 24, 2024

View reviewed changes

...src/main/java/org/apache/cloudstack/storage/image/manager/ImageStoreProviderManagerImpl.java Outdated Show resolved Hide resolved

DaanHoogland approved these changes Jan 24, 2024

View reviewed changes

github-actions bot added status:has-conflicts and removed status:has-conflicts labels Feb 8, 2024

GutoVeronezi approved these changes Feb 14, 2024

View reviewed changes

rohityadavcloud added this to the 20.0.0 milestone Feb 16, 2024

github-actions bot added the status:has-conflicts label Jun 25, 2024

winterhazel added 2 commits January 2, 2025 12:17

Merge remote-tracking branch 'upstream/main' into fix_ordering_of_sto…

58fe9df

…rages_with_firstfitleastconsumed

Add unit tests

5f325a7

winterhazel approved these changes Jan 2, 2025

View reviewed changes

sureshanaparti approved these changes Jan 3, 2025

View reviewed changes

DaanHoogland added the status:ready-for-merge label Jan 17, 2025

JoaoJandre merged commit 864751d into apache:main Feb 14, 2025
26 checks passed

sureshanaparti added this to Apache CloudStack 4.21.0 Jun 5, 2025

sureshanaparti moved this to Done in Apache CloudStack 4.21.0 Jul 14, 2025

sureshanaparti added the type:improvement label Aug 20, 2025

Fix ordering of secondary storages with the algorithm firstfitleastconsumed #8557

Fix ordering of secondary storages with the algorithm firstfitleastconsumed #8557

Uh oh!

Conversation

GaOrtiga commented Jan 23, 2024

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Uh oh!

codecov bot commented Jan 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

sureshanaparti commented Jan 24, 2024

Uh oh!

blueorangutan commented Jan 24, 2024

Uh oh!

blueorangutan commented Jan 24, 2024

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Feb 8, 2024

Uh oh!

GutoVeronezi left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

GaOrtiga commented Mar 1, 2024

Uh oh!

blueorangutan commented Mar 1, 2024

Uh oh!

blueorangutan commented Mar 1, 2024

Uh oh!

DaanHoogland commented Mar 4, 2024

Uh oh!

JoaoJandre commented Apr 16, 2024

Uh oh!

blueorangutan commented Apr 16, 2024

Uh oh!

blueorangutan commented Apr 16, 2024

Uh oh!

JoaoJandre commented Apr 16, 2024

Uh oh!

DaanHoogland commented Apr 16, 2024

Uh oh!

BryanMLima commented Apr 24, 2024

Uh oh!

DaanHoogland commented Apr 25, 2024

Uh oh!

DaanHoogland commented Apr 25, 2024

Uh oh!

blueorangutan commented Apr 25, 2024

Uh oh!

blueorangutan commented Apr 25, 2024

Uh oh!

BryanMLima commented Jun 12, 2024

Uh oh!

github-actions bot commented Jun 25, 2024

Uh oh!

sureshanaparti commented Jun 25, 2024

Uh oh!

blueorangutan commented Sep 18, 2024

Uh oh!

DaanHoogland commented Sep 19, 2024

Uh oh!

blueorangutan commented Sep 19, 2024

Uh oh!

winterhazel commented Jan 2, 2025

Uh oh!

blueorangutan commented Jan 2, 2025

Uh oh!

blueorangutan commented Jan 2, 2025

Uh oh!

winterhazel left a comment

Choose a reason for hiding this comment

Uh oh!

winterhazel commented Jan 2, 2025

Fix ordering of secondary storages with the algorithm `firstfitleastconsumed` #8557

Fix ordering of secondary storages with the algorithm `firstfitleastconsumed` #8557

codecov bot commented Jan 24, 2024 •

edited

Loading

GutoVeronezi left a comment •

edited

Loading