Support for Management Server Maintenance Mode #9854

sureshanaparti · 2024-10-25T06:14:27Z

Description

This PR adds support for Management Server Maintenance Mode. More details below.

New APIs: prepareForMaintenance and cancelMaintenance, with required parameter - managementserverid.
New management server states for maintenance: PreparingForMaintenance, Maintenance.
listHosts API with optional parameter – managementserverid, to list the hosts connected to the management server.
Support management server maintenance when more than one active management servers available.
Triggers transfer agents to other available management servers for maintenance, new agent command MigrateAgentConnectionCommand to initiate transfer of indirect agents.
New global config 'management.server.maintenance.timeout', to set the timeout (in mins) for the management server maintenance window, default: 60 mins.
UI changes: Prepare and Cancel Maintenance in Management Server section, Connected Agents tab, New fields for hosts and management servers.

This also updates the shutdown plugin introduced here: #6755, to maintenance plugin (as common plugin for MS maintenance and shutdown operations).

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI
test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Manually tested prepare and cancel management server maintenance operations.

How did you try to break this feature and the system with this change?

sureshanaparti · 2024-10-25T06:18:24Z

@blueorangutan package

blueorangutan · 2024-10-25T06:20:04Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

codecov · 2024-10-25T06:21:46Z

Codecov Report

Attention: Patch coverage is 8.83095% with 1084 lines in your changes missing coverage. Please review.

Project coverage is 16.18%. Comparing base (98f5663) to head (d60f8d1).
Report is 345 commits behind head on main.

Files with missing lines	Patch %	Lines
...enance/ManagementServerMaintenanceManagerImpl.java	14.48%	298 Missing and 3 partials ⚠️
...cloud/agent/manager/ClusteredAgentManagerImpl.java	0.00%	139 Missing ⚠️
...loudstack/agent/lb/IndirectAgentLBServiceImpl.java	12.24%	84 Missing and 2 partials ⚠️
agent/src/main/java/com/cloud/agent/Agent.java	0.00%	64 Missing ⚠️
...java/com/cloud/agent/manager/AgentManagerImpl.java	6.66%	56 Missing ⚠️
.../response/ManagementServerMaintenanceResponse.java	0.00%	35 Missing ⚠️
...stack/framework/jobs/impl/AsyncJobManagerImpl.java	6.06%	31 Missing ⚠️
...cloud/agent/api/MigrateAgentConnectionCommand.java	0.00%	24 Missing ⚠️
...cloud/cluster/dao/ManagementServerHostDaoImpl.java	15.38%	22 Missing ⚠️
...d/cluster/dao/ManagementServerHostPeerDaoImpl.java	0.00%	22 Missing ⚠️
... and 47 more

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #9854      +/-   ##
============================================
- Coverage     16.19%   16.18%   -0.01%     
- Complexity    13051    13066      +15     
============================================
  Files          5645     5653       +8     
  Lines        494567   495556     +989     
  Branches      59955    60070     +115     
============================================
+ Hits          80088    80204     +116     
- Misses       405642   406512     +870     
- Partials       8837     8840       +3

Flag	Coverage Δ
uitests	`4.01% <ø> (-0.01%)`	⬇️
unittests	`17.03% <8.83%> (-0.02%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

blueorangutan · 2024-10-25T07:34:37Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11437

sureshanaparti · 2024-10-25T07:38:39Z

@blueorangutan test

blueorangutan · 2024-10-25T07:40:03Z

@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2024-10-25T22:52:47Z

[SF] Trillian test result (tid-11701)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 52486 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9854-t11701-kvm-ol8.zip
Smoke tests completed. 139 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_prepare_and_cancel_maintenance	`Error`	0.05	test_ms_maintenance_and_safe_shutdown.py
test_02_unsecure_vm_migration	`Error`	449.14	test_vm_life_cycle.py
test_03_secured_to_nonsecured_vm_migration	`Error`	382.76	test_vm_life_cycle.py
test_04_nonsecured_to_secured_vm_migration	`Error`	378.56	test_vm_life_cycle.py

sureshanaparti · 2024-11-05T13:01:02Z

@blueorangutan package

blueorangutan · 2024-11-05T13:02:04Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-11-05T14:01:53Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11509

sureshanaparti · 2024-11-05T15:55:58Z

@blueorangutan test

blueorangutan · 2024-11-05T15:58:04Z

@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

sureshanaparti · 2024-11-06T04:35:41Z

@blueorangutan package

blueorangutan · 2024-11-06T04:38:03Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-11-06T06:02:41Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11514

blueorangutan · 2024-11-06T06:49:49Z

[SF] Trillian test result (tid-11748)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 51108 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9854-t11748-kvm-ol8.zip
Smoke tests completed. 140 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_prepare_and_cancel_maintenance	`Error`	0.04	test_ms_maintenance_and_safe_shutdown.py

sureshanaparti · 2024-11-18T05:22:49Z

@blueorangutan package

blueorangutan · 2024-11-18T05:24:03Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-11-18T06:27:35Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11577

sureshanaparti · 2024-11-18T07:34:18Z

@blueorangutan test

blueorangutan · 2024-11-18T07:36:03Z

@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2024-11-19T01:10:52Z

[SF] Trillian test result (tid-11784)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 60592 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9854-t11784-kvm-ol8.zip
Smoke tests completed. 139 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_prepare_and_cancel_maintenance	`Error`	0.04	test_ms_maintenance_and_safe_shutdown.py
test_01_secure_vm_migration	`Error`	408.30	test_vm_life_cycle.py

sureshanaparti · 2024-11-20T13:38:23Z

@blueorangutan package

blueorangutan · 2025-01-27T08:44:04Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-01-27T09:48:31Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12218

rohityadavcloud · 2025-01-28T11:51:42Z

@blueorangutan test

blueorangutan · 2025-01-28T11:54:04Z

@rohityadavcloud a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

sureshanaparti · 2025-01-28T13:11:43Z

@blueorangutan package

blueorangutan · 2025-01-28T13:14:03Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-01-28T14:27:16Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12240

…ble, and not configured in host setting for kvm

sureshanaparti · 2025-01-28T16:30:14Z

@blueorangutan package

blueorangutan · 2025-01-28T16:32:04Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-01-28T17:39:10Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12245

blueorangutan · 2025-01-29T03:47:18Z

[SF] Trillian test result (tid-12210)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 54557 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9854-t12210-kvm-ol8.zip
Smoke tests completed. 140 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_11_isolated_network_with_dynamic_routed_mode	`Error`	1.26	test_ipv4_routing.py
test_12_vpc_and_tier_with_dynamic_routed_mode	`Error`	3.40	test_ipv4_routing.py
test_12_vpc_and_tier_with_dynamic_routed_mode	`Error`	3.40	test_ipv4_routing.py

sureshanaparti · 2025-01-29T04:12:12Z

[SF] Trillian test result (tid-12210) Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8 Total time taken: 54557 seconds Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9854-t12210-kvm-ol8.zip Smoke tests completed. 140 look OK, 1 have errors, 0 did not run Only failed and skipped tests results shown below:

Test Result Time (s) Test File
test_11_isolated_network_with_dynamic_routed_mode Error 1.26 test_ipv4_routing.py
test_12_vpc_and_tier_with_dynamic_routed_mode Error 3.40 test_ipv4_routing.py
test_12_vpc_and_tier_with_dynamic_routed_mode Error 3.40 test_ipv4_routing.py

these failures are not related to the changes in this PR.

harikrishna-patnala

code LGTM

* Support for Management Server Maintenance - New APIs: prepareForMaintenance and cancelMaintenance, with required parameter - managementserverid. - New management server states for maintenance: PreparingForMaintenance, Maintenance. - listHosts API with optional parameter – managementserverid, to list the hosts connected to the management server. - Support management server maintenance when more than one active management servers available. - Triggers transfer agents to other available management servers for maintenance, new agent command MigrateAgentConnectionCommand to initiate transfer of indirect agents. - New global config 'management.server.maintenance.timeout', to set the timeout (in mins) for the management server maintenance window, default: 60 mins. - UI changes: Prepare and Cancel Maintenance in Management Server section, Connected Agents tab, New fields for hosts and management servers. * Updated pending jobs check timer task with ScheduledExecutorService * keep maintenance state on trigger shutdown call when ms is in maintenance * add pending jobs count to ms response * during ms heartbeat, update state to up only when it's down * allow vm work jobs of async job created before prepare for maintenance * Revert "keep maintenance state on trigger shutdown call when ms is in maintenance" This reverts commit 607e133. * skip maintenance test when multiple management servers are not available, and not configured in host setting for kvm

boring-cyborg bot added component:agent component:api component:build component:orchestration labels Oct 25, 2024

sureshanaparti added this to the 4.21.0.0 milestone Oct 25, 2024

sureshanaparti force-pushed the cs-mgmt-server-maintenance branch from 4c5f10c to 2285bf8 Compare November 18, 2024 05:20

sureshanaparti force-pushed the cs-mgmt-server-maintenance branch from 2285bf8 to 023ca4c Compare November 20, 2024 13:37

skip maintenance test when multiple management servers are not availa…

d60f8d1

…ble, and not configured in host setting for kvm

sureshanaparti force-pushed the cs-mgmt-server-maintenance branch from 386532e to d60f8d1 Compare January 28, 2025 16:29

harikrishna-patnala approved these changes Jan 29, 2025

View reviewed changes

sureshanaparti added the status:ready-for-merge label Jan 29, 2025

harikrishna-patnala merged commit 3b108b9 into apache:main Jan 29, 2025
23 of 26 checks passed

harikrishna-patnala deleted the cs-mgmt-server-maintenance branch January 29, 2025 08:01

harikrishna-patnala removed the status:ready-for-merge label Jan 29, 2025

DaanHoogland mentioned this pull request Feb 13, 2025

Setting for session.timeout not working for Admin user after upgrade to 4.19 #9328

Open

This was referenced Apr 11, 2025

managementserverid value inconsistency in shutdown APIs #10669

Closed

prepareForShutdown reflects invalid state for shutdowntriggered #10670

Closed

DaanHoogland linked an issue Apr 15, 2025 that may be closed by this pull request

prepareForShutdown reflects invalid state for shutdowntriggered #10670

Closed

sureshanaparti mentioned this pull request Apr 16, 2025

plugin/shutdown: use mgmt server uuid in the shutdown response #10717

Merged

14 tasks

DaanHoogland removed a link to an issue Apr 18, 2025

prepareForShutdown reflects invalid state for shutdowntriggered #10670

Closed

sureshanaparti added the type:new-feature label Jun 4, 2025

sureshanaparti added this to Apache CloudStack 4.21.0 Jun 5, 2025

sureshanaparti moved this to Done in Apache CloudStack 4.21.0 Jul 14, 2025

shwstppr mentioned this pull request Nov 27, 2025

[UI] Infrastructure > Management Servers > Number Of connected agents #12125

Closed

Support for Management Server Maintenance Mode #9854

Support for Management Server Maintenance Mode #9854

Uh oh!

Conversation

sureshanaparti commented Oct 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

sureshanaparti commented Oct 25, 2024

Uh oh!

blueorangutan commented Oct 25, 2024

Uh oh!

codecov bot commented Oct 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

blueorangutan commented Oct 25, 2024

Uh oh!

sureshanaparti commented Oct 25, 2024

Uh oh!

blueorangutan commented Oct 25, 2024

Uh oh!

blueorangutan commented Oct 25, 2024

Uh oh!

sureshanaparti commented Nov 5, 2024

Uh oh!

blueorangutan commented Nov 5, 2024

Uh oh!

blueorangutan commented Nov 5, 2024

Uh oh!

sureshanaparti commented Nov 5, 2024

Uh oh!

blueorangutan commented Nov 5, 2024

Uh oh!

sureshanaparti commented Nov 6, 2024

Uh oh!

blueorangutan commented Nov 6, 2024

Uh oh!

blueorangutan commented Nov 6, 2024

Uh oh!

blueorangutan commented Nov 6, 2024

Uh oh!

sureshanaparti commented Nov 18, 2024

Uh oh!

blueorangutan commented Nov 18, 2024

Uh oh!

blueorangutan commented Nov 18, 2024

Uh oh!

sureshanaparti commented Nov 18, 2024

Uh oh!

blueorangutan commented Nov 18, 2024

Uh oh!

blueorangutan commented Nov 19, 2024

Uh oh!

sureshanaparti commented Nov 20, 2024

Uh oh!

blueorangutan commented Jan 27, 2025

Uh oh!

blueorangutan commented Jan 27, 2025

Uh oh!

rohityadavcloud commented Jan 28, 2025

Uh oh!

blueorangutan commented Jan 28, 2025

Uh oh!

sureshanaparti commented Jan 28, 2025

Uh oh!

blueorangutan commented Jan 28, 2025

Uh oh!

blueorangutan commented Jan 28, 2025

Uh oh!

sureshanaparti commented Jan 28, 2025

Uh oh!

blueorangutan commented Jan 28, 2025

sureshanaparti commented Oct 25, 2024 •

edited

Loading

codecov bot commented Oct 25, 2024 •

edited

Loading