Support for Management Server Maintenance

[sureshanaparti]

Description

This PR adds support for Management Server Maintenance. More details below.

• New APIs: prepareForMaintenance and cancelMaintenance, with required parameter - managementserverid.

• New management server states for maintenance: PreparingForMaintenance, Maintenance.

• listHosts API with optional parameter – managementserverid, to list the hosts connected to the management server.

• Support management server maintenance when more than one active management servers available.

• Triggers transfer agents to other available management servers for maintenance, new agent command MigrateAgentConnectionCommand to initiate transfer of indirect agents.

• New global config 'management.server.maintenance.timeout', to set the timeout (in mins) for the management server maintenance window, default: 60 mins.

• UI changes: Prepare and Cancel Maintenance in Management Server section, Connected Agents tab, New fields for hosts and management servers.

This also updates the shutdown plugin introduced here: #6755, to maintenance plugin (as common plugin for MS maintenance and shutdown operations).

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• build/CI
• test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

Screenshots (if appropriate):

How Has This Been Tested?

Manually tested prepare and cancel management server maintenance operations.

How did you try to break this feature and the system with this change?

[sureshanaparti]

@blueorangutan package

[blueorangutan]

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[codecov]

Codecov Report

Attention: Patch coverage is 9.16667% with 981 lines in your changes missing coverage. Please review.

Project coverage is 15.79%. Comparing base (20901c7) to head (89c335f).
Report is 5 commits behind head on main.

Files with missing lines	Patch %	Lines
...enance/ManagementServerMaintenanceManagerImpl.java	14.12%	299 Missing and 5 partials ⚠️
...cloud/agent/manager/ClusteredAgentManagerImpl.java	0.00%	114 Missing ⚠️
...loudstack/agent/lb/IndirectAgentLBServiceImpl.java	12.24%	84 Missing and 2 partials ⚠️
agent/src/main/java/com/cloud/agent/Agent.java	0.00%	64 Missing ⚠️
...java/com/cloud/agent/manager/AgentManagerImpl.java	6.55%	57 Missing ⚠️
.../response/ManagementServerMaintenanceResponse.java	0.00%	35 Missing ⚠️
...cloud/agent/api/MigrateAgentConnectionCommand.java	0.00%	24 Missing ⚠️
...d/cluster/dao/ManagementServerHostPeerDaoImpl.java	0.00%	22 Missing ⚠️
.../src/main/java/com/cloud/host/dao/HostDaoImpl.java	36.66%	19 Missing ⚠️
...oudstack/api/command/PrepareForMaintenanceCmd.java	0.00%	19 Missing ⚠️
... and 41 more

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #9854      +/-   ##
============================================
+ Coverage     15.78%   15.79%   +0.01%     
- Complexity    12564    12585      +21     
============================================
  Files          5627     5635       +8     
  Lines        492266   493084     +818     
  Branches      63860    62935     -925     
============================================
+ Hits          77707    77891     +184     
- Misses       406084   406666     +582     
- Partials       8475     8527      +52     

Flag	Coverage Δ
uitests	4.04% <ø> (-0.01%)	⬇️
unittests	16.61% <9.16%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11437

[sureshanaparti]

@blueorangutan test

[blueorangutan]

@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

[blueorangutan]

[SF] Trillian test result (tid-11701)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 52486 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9854-t11701-kvm-ol8.zip
Smoke tests completed. 139 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_prepare_and_cancel_maintenance	Error	0.05	test_ms_maintenance_and_safe_shutdown.py
test_02_unsecure_vm_migration	Error	449.14	test_vm_life_cycle.py
test_03_secured_to_nonsecured_vm_migration	Error	382.76	test_vm_life_cycle.py
test_04_nonsecured_to_secured_vm_migration	Error	378.56	test_vm_life_cycle.py

[sureshanaparti]

@blueorangutan package

[blueorangutan]

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11509

[sureshanaparti]

@blueorangutan test

[blueorangutan]

@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests