engine/orchestration: fix missing vm powerstate update vm state

[shwstppr]

Description

Fixes #10406
VMs were not moving to Stopped state when PowerReportMissing is processed.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• build/CI
• test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

[shwstppr]

@blueorangutan package

[blueorangutan]

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 15.99%. Comparing base (789f94b) to head (00a13ca).
Report is 15 commits behind head on 4.20.

Additional details and impacted files

@@            Coverage Diff            @@
##               4.20   #10407   +/-   ##
=========================================
  Coverage     15.98%   15.99%           
- Complexity    13077    13083    +6     
=========================================
  Files          5649     5649           
  Lines        495624   495625    +1     
  Branches      60003    60003           
=========================================
+ Hits          79250    79268   +18     
+ Misses       407526   407506   -20     
- Partials       8848     8851    +3     

Flag	Coverage Δ
uitests	4.01% <ø> (ø)
unittests	16.83% <100.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[DaanHoogland]

clgtm, but no idea if this has side effects.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12487

[weizhouapache]

code lgtm

2025-02-17T14:14:55,772 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) VM state report is updated. Host {"id":2,"name":"ref-trl-6851-k-Mu24-wei-zhou-kvm2","type":"Routing","uuid":"eb8661f9-6a36-4449-8923-cfe3fb849500"}, VM instance {"id":1159,"instanceName":"i-2-1159-VM","type":"User","uuid":"9e1de80e-ddcf-4c8b-87bc-1c1412292a46"}, power state: PowerReportMissing
2025-02-17T14:14:55,783 INFO  [c.c.v.ClusteredVirtualMachineManagerImpl] (AgentManager-Handler-1:[ctx-d817875a]) (logid:) VM VM instance {"id":1159,"instanceName":"i-2-1159-VM","type":"User","uuid":"9e1de80e-ddcf-4c8b-87bc-1c1412292a46"} is at Running and we received a PowerReportMissing report while there is no pending jobs on it
...
2025-02-17T14:14:55,843 INFO  [c.c.v.ClusteredVirtualMachineManagerImpl] (AgentManager-Handler-1:[ctx-d817875a]) (logid:) VM VM instance {"id":1159,"instanceName":"i-2-1159-VM","type":"User","uuid":"9e1de80e-ddcf-4c8b-87bc-1c1412292a46"} is sync-ed to at Stopped state according to power-off report from hypervisor.

[sureshanaparti]

clgtm

[harikrishna-patnala]

code LGTM

[shwstppr]

@blueorangutan package

[blueorangutan]

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 12498

[shwstppr]

@blueorangutan package

[blueorangutan]

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12499

[DaanHoogland]

@blueorangutan test

[blueorangutan]

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

[blueorangutan]

[SF] Trillian test result (tid-12461)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 56397 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10407-t12461-kvm-ol8.zip
Smoke tests completed. 139 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_11_isolated_network_with_dynamic_routed_mode	Error	2.27	test_ipv4_routing.py
test_12_vpc_and_tier_with_dynamic_routed_mode	Error	2.35	test_ipv4_routing.py
test_12_vpc_and_tier_with_dynamic_routed_mode	Error	2.35	test_ipv4_routing.py
test_06_purge_expunged_vm_background_task	Failure	392.82	test_purge_expunged_vms.py

[kiranchavala]

LGTM

Steps to reproduce the issue

1. Deploy a vm from cloudstack
2. Login to the kvm host and destory the vm using virsh (virsh destory domain id)
3. The vm should get stopped in cloudstack after some time

Before the fix (the vm continued to be in running state)

2025-02-25` 06:59:13,941 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-16:null) (logid:) Run missing VM report. current time: 1740466753941
2025-02-25 06:59:13,942 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-16:null) (logid:) Detected missing VM. Host {"id":1,"name":"ol8.localdomain","type":"Routing","uuid":"23ad85e8-e338-4b1a-958c-9a65b2532e75"}, VM instance {"id":3,"instanceName":"i-2-3-VM","type":"User","uuid":"6ae77659-2f5f-492c-b616-6d78ef134dcb"}, power state: PowerReportMissing, last state update: 2025-02-25T06:58:13+0000
2025-02-25 06:59:13,942 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-16:null) (logid:) VM instance {"id":3,"instanceName":"i-2-3-VM","type":"User","uuid":"6ae77659-2f5f-492c-b616-6d78ef134dcb"} - time since last state update(60941 ms) has not passed graceful period yet
2025-02-25 06:59:13,942 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-16:null) (logid:) Done with process of VM state report. Host {"id":1,"name":"ol8.localdomain","type":"Routing","uuid":"23ad85e8-e338-4b1a-958c-9a65b2532e75"}


2025-02-25 07:00:13,989 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:null) (logid:) Run missing VM report. current time: 1740466813989
2025-02-25 07:00:13,990 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:null) (logid:) Detected missing VM. Host {"id":1,"name":"ol8.localdomain","type":"Routing","uuid":"23ad85e8-e338-4b1a-958c-9a65b2532e75"}, VM instance {"id":3,"instanceName":"i-2-3-VM","type":"User","uuid":"6ae77659-2f5f-492c-b616-6d78ef134dcb"}, power state: PowerReportMissing, last state update: 2025-02-25T06:58:13+0000
2025-02-25 07:00:13,990 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:null) (logid:) VM instance {"id":3,"instanceName":"i-2-3-VM","type":"User","uuid":"6ae77659-2f5f-492c-b616-6d78ef134dcb"} - time since last state update(120989 ms) has passed graceful period
2025-02-25 07:00:13,994 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:null) (logid:) Done with process of VM state report. Host {"id":1,"name":"ol8.localdomain","type":"Routing","uuid":"23ad85e8-e338-4b1a-958c-9a65b2532e75"}

After the fix (the vm goes to stop state)

2025-02-25 06:39:20,361 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:null) (logid:) Run missing VM report. current time: 1740465560361

2025-02-25 06:39:20,362 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:null) (logid:) Detected missing VM. Host {"id":1,"name":"ol8.localdomain","type":"Routing","uuid":"22e66fdc-690b-4ae1-bfd6-a5677315cea0"}, VM instance {"id":3,"instanceName":"i-2-3-VM","type":"User","uuid":"9368adc3-6b06-48db-8fa4-4ad8dbb20a81"}, power state: PowerReportMissing, last state update: 2025-02-25T06:38:20+0000

2025-02-25 06:39:20,362 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:null) (logid:) VM instance {"id":3,"instanceName":"i-2-3-VM","type":"User","uuid":"9368adc3-6b06-48db-8fa4-4ad8dbb20a81"} - time since last state update(60361 ms) has not passed graceful period yet

2025-02-25 06:39:20,362 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:null) (logid:) Done with process of VM state report. Host {"id":1,"name":"ol8.localdomain","type":"Routing","uuid":"22e66fdc-690b-4ae1-bfd6-a5677315cea0"


2025-02-25 06:40:20,331 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:null) (logid:) VM state report is updated. Host {"id":1,"name":"ol8.localdomain","type":"Routing","uuid":"22e66fdc-690b-4ae1-bfd6-a5677315cea0"}, VM instance {"id":3,"instanceName":"i-2-3-VM","type":"User","uuid":"9368adc3-6b06-48db-8fa4-4ad8dbb20a81"}, power state: PowerReportMissing

2025-02-25 06:40:20,341 INFO  [c.c.v.VirtualMachineManagerImpl] (AgentManager-Handler-1:null) (logid:) VM i-2-3-VM is at Running and we received a PowerReportMissing report while there is no pending jobs on it

2025-02-25 06:40:20,430 INFO  [c.c.v.VirtualMachineManagerImpl] (AgentManager-Handler-1:null) (logid:) VM i-2-3-VM is sync-ed to at Stopped state according to power-off report from hypervisor

2025-02-25 06:40:20,431 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:null) (logid:) Done with process of VM state report. Host {"id":1,"name":"ol8.localdomain","type":"Routing","uuid":"22e66fdc-690b-4ae1-bfd6-a5677315cea0"}