Add support to RBD erasure code pools #9808

BryanMLima · 2024-10-16T13:38:27Z

Description

This PR adds support for Ceph erasure code pools, allowing users to specify the data pool required for using this type of Ceph pool.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI
test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Screenshots (if appropriate):

Details of the data pool in the UI

How Has This Been Tested?

The following tests were executed to verify this patch; tests were executed on replicated pools as well to check if this patch would interfere in it.

Deploy of VM in a Ceph replicated pool
Deploy of VM in a Ceph erasure code pool
Volume resize in a Ceph replicated pool
Volume resize in a Ceph erasure code pool
Live migration of VM with volume allocated in a Ceph replicated pool
Live migration of VM with volume allocated in a Ceph erasure code pool
Attach, mount and utilization of volume in a Ceph replicated pool
Attach, mount and utilization of volume in a Ceph erasure code pool
Template creation from a volume in a Ceph replicated pool
Deploy of VM from the template based on volume in a Ceph replicated pool
Template creation from a volume in a Ceph erasure code pool
Deploy of VM from the template based on volume in a Ceph erasure code pool
Volume migration from/to Ceph replicated pool to/from NFS
Volume migration from/to Ceph replicated pool to/from iSCSI (SharedMountPoint)
Volume migration from/to Ceph erasure code pool to/from NFS
Volume migration from/to Ceph erasure code pool to/from iSCSI (SharedMountPoint)
Volume migration from/to Ceph erasure code pool to/from Ceph replicated pool
Snapshot of a volume from Ceph replicated pool
Revert snapshot of a volume from Ceph replicated pool
Snapshot of a volume from Ceph erasure code pool
Revert snapshot of a volume from Ceph erasure code pool

How did you try to break this feature and the system with this change?

I tried to migrate volumes from different type of pool types (NFS, iSCSI and RBD) to each other, verifying that the migration was successful.

codecov · 2024-10-16T13:46:46Z

Codecov Report

Attention: Patch coverage is 6.42202% with 102 lines in your changes missing coverage. Please review.

Project coverage is 16.18%. Comparing base (81e052c) to head (fce6e14).
Report is 360 commits behind head on main.

Files with missing lines	Patch %	Lines
...hypervisor/kvm/storage/StoragePoolInformation.java	0.00%	35 Missing ⚠️
.../hypervisor/kvm/storage/LibvirtStorageAdaptor.java	0.00%	18 Missing ⚠️
...ain/java/com/cloud/api/query/QueryManagerImpl.java	0.00%	16 Missing ⚠️
.../hypervisor/kvm/storage/KVMStoragePoolManager.java	0.00%	10 Missing ⚠️
...e/cloudstack/api/response/StoragePoolResponse.java	0.00%	6 Missing ⚠️
...torage/datastore/provider/DefaultHostListener.java	0.00%	4 Missing ⚠️
...oud/hypervisor/kvm/storage/LibvirtStoragePool.java	0.00%	4 Missing ⚠️
.../cloud/hypervisor/kvm/storage/KVMPhysicalDisk.java	70.00%	1 Missing and 2 partials ⚠️
...ud/hypervisor/kvm/storage/KVMStorageProcessor.java	0.00%	2 Missing ⚠️
...CreatePrivateTemplateFromVolumeCommandWrapper.java	0.00%	1 Missing ⚠️
... and 3 more

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #9808      +/-   ##
============================================
- Coverage     16.18%   16.18%   -0.01%     
+ Complexity    13046    13045       -1     
============================================
  Files          5645     5646       +1     
  Lines        494795   494841      +46     
  Branches      59955    59963       +8     
============================================
+ Hits          80065    80069       +4     
- Misses       405897   405937      +40     
- Partials       8833     8835       +2

Flag	Coverage Δ
uitests	`4.01% <ø> (-0.01%)`	⬇️
unittests	`17.03% <6.42%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

BryanMLima · 2024-10-16T16:41:33Z

@blueorangutan package

blueorangutan · 2024-10-16T16:42:04Z

@BryanMLima a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-10-16T17:55:47Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11365

...ns/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/LibvirtStorageAdaptor.java

DaanHoogland

clgtm, but needs testing

JoaoJandre

LGTM, did not test it. Only left a small nitpick

...ns/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/LibvirtStorageAdaptor.java

JoaoJandre

LGTM, did not test it

github-actions · 2025-01-08T13:23:29Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

… data pool

…l is of erasure code type. Also added javadoc for createPhysicalDisk method

BryanMLima · 2025-01-22T17:26:42Z

@blueorangutan package

blueorangutan · 2025-01-22T17:28:03Z

@BryanMLima a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-01-22T18:52:09Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12174

winterhazel

Tested for both replicated and erasure coded pools:

VM deployment
VM deletion
Snapshot creation
Reversion to snapshot
Template creation from snapshot
VM deployment based on template created from snapshot
Volume creation from snapshot
Attachment of root disk created from snapshot
Data disk attachment with VM running and stopped
Volume resize
Volume migration
- To/from local storage
- To/from NFS
- To/from another RBD pool
- Of volumes that have snapshots on primary (the snapshots get deleted)
Live migration

BryanMLima · 2025-03-20T10:00:59Z

@DaanHoogland @sureshanaparti @JoaoJandre, I think we can merge this one, right?

JoaoJandre · 2025-03-20T12:44:39Z

@blueorangutan package

JoaoJandre · 2025-03-20T12:45:55Z

@DaanHoogland @sureshanaparti @JoaoJandre, I think we can merge this one, right?

@BryanMLima For formality's sake, we should run the CI still

blueorangutan · 2025-03-20T12:46:03Z

@JoaoJandre a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

DaanHoogland · 2025-03-20T13:10:07Z

@blueorangutan test

blueorangutan · 2025-03-20T13:12:03Z

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2025-03-20T14:12:53Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12848

DaanHoogland · 2025-03-24T09:47:11Z

@blueorangutan test

blueorangutan · 2025-03-24T09:48:03Z

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2025-03-25T05:46:11Z

[SF] Trillian test result (tid-12807)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 69190 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9808-t12807-kvm-ol8.zip
Smoke tests completed. 135 look OK, 6 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestPrivateGwACL>:setup	`Error`	0.00	test_privategw_acl.py
test_01_create_template	`Error`	1.15	test_templates.py
test_CreateTemplateWithDuplicateName	`Error`	1.14	test_templates.py
test_02_create_template_with_checksum_sha1	`Error`	65.61	test_templates.py
test_03_create_template_with_checksum_sha256	`Error`	65.65	test_templates.py
test_04_create_template_with_checksum_md5	`Error`	65.64	test_templates.py
test_05_create_template_with_no_checksum	`Error`	65.62	test_templates.py
ContextSuite context=TestTemplates>:setup	`Error`	270.58	test_templates.py
ContextSuite context=TestISOUsage>:setup	`Error`	0.00	test_usage.py
test_01_snapshot_usage	`Error`	24.76	test_usage.py
test_06_download_detached_volume	`Error`	314.07	test_volumes.py
test_13_migrate_volume_and_change_offering	`Error`	124.18	test_volumes.py
ContextSuite context=TestIpv6Vpc>:setup	`Error`	0.00	test_vpc_ipv6.py
ContextSuite context=TestVPCRedundancy>:setup	`Error`	0.00	test_vpc_redundant.py

JoaoJandre · 2025-03-31T12:04:51Z

@blueorangutan package

blueorangutan · 2025-03-31T12:06:03Z

@JoaoJandre a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-03-31T13:13:21Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12934

JoaoJandre · 2025-04-01T11:45:35Z

@DaanHoogland could we try the CI again?

DaanHoogland · 2025-04-01T17:15:41Z

@blueorangutan test

blueorangutan · 2025-04-01T17:18:03Z

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2025-04-02T08:41:01Z

[SF] Trillian test result (tid-12877)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 52440 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9808-t12877-kvm-ol8.zip
Smoke tests completed. 141 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

* Readd filename string on qemuimg create * Remove empty object on the data pool details of storage pools with no data pool * Only use the method createPhysicalDiskByLibVirt with RBD when the pool is of erasure code type. Also added javadoc for createPhysicalDisk method * Change literal '/' string to File.separator * Add support for erasure code pools * Fix null on putAll

boring-cyborg bot added component:api component:iscsi component:kvm component:primary-storage component:storage component:UI labels Oct 16, 2024

BryanMLima requested review from DaanHoogland, JoaoJandre and sureshanaparti and removed request for DaanHoogland October 16, 2024 16:41

weizhouapache reviewed Oct 16, 2024

View reviewed changes

...ns/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/LibvirtStorageAdaptor.java Outdated Show resolved Hide resolved

DaanHoogland approved these changes Oct 28, 2024

View reviewed changes

DaanHoogland added the status:needs-testing label Oct 28, 2024

JoaoJandre approved these changes Oct 29, 2024

View reviewed changes

...ns/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/LibvirtStorageAdaptor.java Outdated Show resolved Hide resolved

slavkap assigned slavkap and unassigned slavkap Oct 30, 2024

slavkap self-requested a review October 30, 2024 10:35

JoaoJandre approved these changes Nov 26, 2024

View reviewed changes

github-actions bot added the status:has-conflicts label Jan 8, 2025

BryanMLima added 5 commits January 22, 2025 14:22

Readd filename string on qemuimg create

4c74326

Remove empty object on the data pool details of storage pools with no…

7c196d9

… data pool

Only use the method createPhysicalDiskByLibVirt with RBD when the poo…

83ef09e

…l is of erasure code type. Also added javadoc for createPhysicalDisk method

Change literal '/' string to File.separator

fce6e14

Add support for erasure code pools

62f8d27

Fix null on putAll

399b023

BryanMLima force-pushed the add-support-to-erasure-code-rbd-pools branch from cea613c to fce6e14 Compare January 22, 2025 17:23

github-actions bot removed the status:has-conflicts label Jan 22, 2025

winterhazel approved these changes Jan 24, 2025

View reviewed changes

BryanMLima added this to the 4.21.0 milestone Mar 20, 2025

JoaoJandre merged commit cb4848b into apache:main Apr 2, 2025
25 of 26 checks passed

sureshanaparti added the type:improvement label Jun 4, 2025

sureshanaparti added this to Apache CloudStack 4.21.0 Jun 5, 2025

sureshanaparti moved this to Done in Apache CloudStack 4.21.0 Jul 14, 2025

Add support to RBD erasure code pools #9808

Add support to RBD erasure code pools #9808

Uh oh!

Conversation

BryanMLima commented Oct 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

codecov bot commented Oct 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

BryanMLima commented Oct 16, 2024

Uh oh!

blueorangutan commented Oct 16, 2024

Uh oh!

blueorangutan commented Oct 16, 2024

Uh oh!

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

JoaoJandre left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

JoaoJandre left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jan 8, 2025

Uh oh!

BryanMLima commented Jan 22, 2025

Uh oh!

blueorangutan commented Jan 22, 2025

Uh oh!

blueorangutan commented Jan 22, 2025

Uh oh!

winterhazel left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

BryanMLima commented Mar 20, 2025

Uh oh!

JoaoJandre commented Mar 20, 2025

Uh oh!

JoaoJandre commented Mar 20, 2025

Uh oh!

blueorangutan commented Mar 20, 2025

Uh oh!

DaanHoogland commented Mar 20, 2025

Uh oh!

blueorangutan commented Mar 20, 2025

Uh oh!

blueorangutan commented Mar 20, 2025

Uh oh!

DaanHoogland commented Mar 24, 2025

Uh oh!

blueorangutan commented Mar 24, 2025

Uh oh!

blueorangutan commented Mar 25, 2025

Uh oh!

JoaoJandre commented Mar 31, 2025

Uh oh!

blueorangutan commented Mar 31, 2025

Uh oh!

blueorangutan commented Mar 31, 2025

Uh oh!

JoaoJandre commented Apr 1, 2025

Uh oh!

DaanHoogland commented Apr 1, 2025

Uh oh!

blueorangutan commented Apr 1, 2025

Uh oh!

blueorangutan commented Apr 2, 2025

BryanMLima commented Oct 16, 2024 •

edited

Loading

codecov bot commented Oct 16, 2024 •

edited

Loading

winterhazel left a comment •

edited

Loading