Enforce strict host tag check #9017

vishesh92 · 2024-04-30T16:44:04Z

Description

Documentation PR: apache/cloudstack-documentation#398

Currently, an administrator can break host tag compatibility for a VM administrator by certain operations:

deploy/start VM on a specific host
migrate VM
restore VM
scale VM

This PR allows the user to specify tags which must be checked during these operations.

Global Settings

vm.strict.host.tags - A comma-separated list of tags for strict host check (Default - empty)
vm.strict.resource.limit.host.tag.check - Determines whether the resource limits tags are considered strict or not (Default - true)

During the above operations, we now check and throw an error if host tags compatibility is being broken for tags specified in vm.strict.host.tags. If vm.strict.resource.limit.host.tag.check is set to true, tags set in resource.limit.host.tags are also checked during these operations.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Screenshots (if appropriate):

How Has This Been Tested?

Manually tested

How did you try to break this feature and the system with this change?

codecov-commenter · 2024-04-30T17:04:15Z

Codecov Report

Attention: Patch coverage is 62.66667% with 28 lines in your changes missing coverage. Please review.

Project coverage is 15.28%. Comparing base (19f9635) to head (73d6888).
Report is 103 commits behind head on main.

Files	Patch %	Lines
.../src/main/java/com/cloud/vm/UserVmManagerImpl.java	47.05%	18 Missing ⚠️
...rver/src/main/java/com/cloud/vm/UserVmManager.java	63.63%	2 Missing and 2 partials ⚠️
...ne/schema/src/main/java/com/cloud/host/HostVO.java	88.88%	0 Missing and 3 partials ⚠️
...om/cloud/deploy/DeploymentPlanningManagerImpl.java	0.00%	1 Missing and 1 partial ⚠️
.../apache/cloudstack/vm/UnmanagedVMsManagerImpl.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##               main    #9017    +/-   ##
==========================================
  Coverage     15.28%   15.28%            
- Complexity    11535    11542     +7     
==========================================
  Files          5424     5424            
  Lines        474334   474390    +56     
  Branches      60970    61603   +633     
==========================================
+ Hits          72498    72524    +26     
- Misses       393778   393810    +32     
+ Partials       8058     8056     -2

Flag	Coverage Δ
uitests	`4.25% <ø> (ø)`
unittests	`16.02% <62.66%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

vishesh92 · 2024-04-30T20:46:41Z

@blueorangutan package

blueorangutan · 2024-04-30T20:48:03Z

@vishesh92 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-04-30T22:13:14Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 9491

vishesh92 · 2024-05-01T05:52:38Z

@blueorangutan test

blueorangutan · 2024-05-01T05:54:03Z

@vishesh92 a [SL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

vishesh92 · 2024-05-01T17:20:41Z

@blueorangutan package

blueorangutan · 2024-05-01T17:22:03Z

@vishesh92 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-05-01T18:33:23Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 9507

vishesh92 · 2024-05-01T18:38:18Z

@blueorangutan test

blueorangutan · 2024-05-01T18:40:03Z

@vishesh92 a [SL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2024-05-01T19:31:46Z

[SF] Trillian test result (tid-10102)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 47445 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9017-t10102-kvm-centos7.zip
Smoke tests completed. 128 look OK, 4 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_events_resource	`Error`	426.75	test_events_resource.py
test_01_restore_vm	`Error`	0.25	test_restore_vm.py
test_02_restore_vm_allocated_root	`Error`	0.15	test_restore_vm.py
ContextSuite context=TestRestoreVM>:teardown	`Error`	1.35	test_restore_vm.py
test_02_trigger_shutdown	`Failure`	346.70	test_safe_shutdown.py
test_02_restore_vm_strict_tags_failure	`Failure`	41.64	test_vm_strict_host_tags.py
test_02_scale_vm_strict_tags_failure	`Failure`	44.85	test_vm_strict_host_tags.py
test_06_deploy_vm_on_any_host_with_strict_tags_failure	`Failure`	1.30	test_vm_strict_host_tags.py
ContextSuite context=TestVMDeploymentPlannerStrictTags>:teardown	`Error`	53.96	test_vm_strict_host_tags.py

blueorangutan · 2024-05-02T09:44:50Z

[SF] Trillian test result (tid-10124)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 52549 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9017-t10124-kvm-centos7.zip
Smoke tests completed. 127 look OK, 5 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_events_resource	`Error`	428.10	test_events_resource.py
test_01_restore_vm	`Error`	0.24	test_restore_vm.py
test_02_restore_vm_allocated_root	`Error`	0.14	test_restore_vm.py
ContextSuite context=TestRestoreVM>:teardown	`Error`	1.23	test_restore_vm.py
test_02_list_cpvm_vm	`Failure`	0.05	test_ssvm.py
test_04_cpvm_internals	`Failure`	0.04	test_ssvm.py
test_01_secure_vm_migration	`Error`	135.30	test_vm_life_cycle.py
test_01_secure_vm_migration	`Error`	135.31	test_vm_life_cycle.py
test_02_restore_vm_strict_tags_failure	`Failure`	46.75	test_vm_strict_host_tags.py
test_02_scale_vm_strict_tags_failure	`Failure`	51.04	test_vm_strict_host_tags.py
test_06_deploy_vm_on_any_host_with_strict_tags_failure	`Failure`	1.31	test_vm_strict_host_tags.py
ContextSuite context=TestVMDeploymentPlannerStrictTags>:teardown	`Error`	77.52	test_vm_strict_host_tags.py

vishesh92 · 2024-05-02T13:18:58Z

@blueorangutan package

blueorangutan · 2024-05-02T13:20:03Z

@vishesh92 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-05-02T14:38:36Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 9517

vishesh92 · 2024-05-02T18:13:21Z

@blueorangutan test

blueorangutan · 2024-05-02T18:14:03Z

@vishesh92 a [SL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2024-05-03T08:24:58Z

[SF] Trillian test result (tid-10146)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 49258 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9017-t10146-kvm-centos7.zip
Smoke tests completed. 129 look OK, 3 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_events_resource	`Error`	437.35	test_events_resource.py
test_01_restore_vm	`Error`	0.21	test_restore_vm.py
test_02_restore_vm_allocated_root	`Error`	0.14	test_restore_vm.py
ContextSuite context=TestRestoreVM>:teardown	`Error`	1.23	test_restore_vm.py
ContextSuite context=TestVMDeploymentPlannerStrictTags>:teardown	`Error`	59.13	test_vm_strict_host_tags.py

vishesh92 · 2024-05-04T06:37:22Z

@blueorangutan package

blueorangutan · 2024-05-04T06:38:04Z

@vishesh92 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-05-04T07:54:08Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 9532

vishesh92 · 2024-05-04T07:57:08Z

@blueorangutan test

vishesh92 · 2024-05-07T11:41:14Z

@blueorangutan test

blueorangutan · 2024-05-07T11:42:04Z

@vishesh92 a [SL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2024-05-08T01:33:43Z

[SF] Trillian test result (tid-10182)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 48103 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9017-t10182-kvm-centos7.zip
Smoke tests completed. 131 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_events_resource	`Error`	428.02	test_events_resource.py

github-actions · 2024-05-08T15:29:29Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

vishesh92 · 2024-05-21T08:20:06Z

@blueorangutan package

blueorangutan · 2024-05-21T08:22:03Z

@vishesh92 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-05-21T09:38:20Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 9647

vishesh92 · 2024-06-06T20:43:51Z

@blueorangutan package

blueorangutan · 2024-06-06T20:46:03Z

@vishesh92 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

vladimirpetrov · 2024-06-06T21:49:25Z

@blueorangutan test matrix

blueorangutan · 2024-06-06T21:50:03Z

@vladimirpetrov a [SL] Trillian-Jenkins matrix job (centos7 mgmt + xenserver71, rocky8 mgmt + vmware67u3, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

blueorangutan · 2024-06-06T21:56:42Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 9811

blueorangutan · 2024-06-07T11:06:56Z

[SF] Trillian test result (tid-10388)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 46016 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9017-t10388-kvm-centos7.zip
Smoke tests completed. 133 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

blueorangutan · 2024-06-07T11:26:47Z

[SF] Trillian test result (tid-10386)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 47326 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9017-t10386-xenserver-71.zip
Smoke tests completed. 133 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

blueorangutan · 2024-06-07T12:12:47Z

[SF] Trillian test result (tid-10387)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 50026 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9017-t10387-vmware-67u3.zip
Smoke tests completed. 131 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_balanced_drs_algorithm	`Error`	424.56	test_cluster_drs.py
test_02_restore_vm_with_disk_offering	`Error`	57.29	test_restore_vm.py
test_03_restore_vm_with_disk_offering_custom_size	`Error`	58.21	test_restore_vm.py

vladimirpetrov

LGTM based on manual testing. Tested deployment, start/stop, migration with and without storage, reinstal and scaling to another template with and without strict host tagging.

rohityadavcloud · 2024-06-25T09:10:30Z

Merging based on manual QA, review and smoketests.

Documentation PR: apache/cloudstack-documentation#398 Currently, an administrator can break host tag compatibility for a VM administrator by certain operations: * deploy/start VM on a specific host * migrate VM * restore VM * scale VM This PR allows the user to specify tags which must be checked during these operations. Global Settings 1. `vm.strict.host.tags` - A comma-separated list of tags for strict host check (Default - empty) 2. `vm.strict.resource.limit.host.tag.check` - Determines whether the resource limits tags are considered strict or not (Default - true) During the above operations, we now check and throw an error if host tags compatibility is being broken for tags specified in `vm.strict.host.tags`. If `vm.strict.resource.limit.host.tag.check` is set to `true`, tags set in `resource.limit.host.tags` are also checked during these operations.

boring-cyborg bot added component:integration-test component:integration component:marvin Python Warning... Python code Ahead! labels Apr 30, 2024

vishesh92 force-pushed the enforce-strict-host-tag-check branch from 5e42531 to 3c53cf4 Compare April 30, 2024 20:46

vishesh92 mentioned this pull request May 1, 2024

Update documentation for strict host tags apache/cloudstack-documentation#398

Merged

vishesh92 force-pushed the enforce-strict-host-tag-check branch from a1091b0 to 8769a93 Compare May 2, 2024 12:54

github-actions bot added the status:has-conflicts label May 8, 2024

Merge branch 'main' into enforce-strict-host-tag-check

5ae9fc3

github-actions bot removed the status:has-conflicts label May 21, 2024

Merge branch 'main' into enforce-strict-host-tag-check

73d6888

vladimirpetrov self-assigned this Jun 10, 2024

vladimirpetrov approved these changes Jun 11, 2024

View reviewed changes

rohityadavcloud added this to the 4.20.0.0 milestone Jun 25, 2024

rohityadavcloud added status:ready-for-merge and removed component:integration-test labels Jun 25, 2024

rohityadavcloud marked this pull request as ready for review June 25, 2024 09:10

rohityadavcloud merged commit f6ceeab into apache:main Jun 25, 2024

rohityadavcloud deleted the enforce-strict-host-tag-check branch June 25, 2024 09:12

DaanHoogland removed the status:ready-for-merge label Dec 4, 2024

Enforce strict host tag check #9017

Enforce strict host tag check #9017

Uh oh!

Conversation

vishesh92 commented Apr 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Global Settings

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

codecov-commenter commented Apr 30, 2024 • edited by codecov bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

vishesh92 commented Apr 30, 2024

Uh oh!

blueorangutan commented Apr 30, 2024

Uh oh!

blueorangutan commented Apr 30, 2024

Uh oh!

vishesh92 commented May 1, 2024

Uh oh!

blueorangutan commented May 1, 2024

Uh oh!

vishesh92 commented May 1, 2024

Uh oh!

blueorangutan commented May 1, 2024

Uh oh!

blueorangutan commented May 1, 2024

Uh oh!

vishesh92 commented May 1, 2024

Uh oh!

blueorangutan commented May 1, 2024

Uh oh!

blueorangutan commented May 1, 2024

Uh oh!

blueorangutan commented May 2, 2024

Uh oh!

vishesh92 commented May 2, 2024

Uh oh!

blueorangutan commented May 2, 2024

Uh oh!

blueorangutan commented May 2, 2024

Uh oh!

vishesh92 commented May 2, 2024

Uh oh!

blueorangutan commented May 2, 2024

Uh oh!

blueorangutan commented May 3, 2024

Uh oh!

vishesh92 commented May 4, 2024

Uh oh!

blueorangutan commented May 4, 2024

Uh oh!

blueorangutan commented May 4, 2024

Uh oh!

vishesh92 commented May 4, 2024

Uh oh!

vishesh92 commented May 7, 2024

Uh oh!

blueorangutan commented May 7, 2024

Uh oh!

blueorangutan commented May 8, 2024

Uh oh!

github-actions bot commented May 8, 2024

Uh oh!

vishesh92 commented May 21, 2024

Uh oh!

blueorangutan commented May 21, 2024

Uh oh!

blueorangutan commented May 21, 2024

Uh oh!

vishesh92 commented Jun 6, 2024

Uh oh!

blueorangutan commented Jun 6, 2024

vishesh92 commented Apr 30, 2024 •

edited

Loading

codecov-commenter commented Apr 30, 2024 •

edited by codecov bot

Loading