-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Description
problem
Zone-wide storage:
If there are virtual machines on specific primary storage and that storage is set to Maintenance mode it is not possible to cancel the maintenance mode if the agents in the zone were restarted.
Cluster-wide storage pool:
If there are virtual machines on such pool, and the pool is set to maintenance and after that if we try to cancel the maintenance mode it's not possible
versions
4.21
Zone-wide primary storage - StorPool and NFS
Cluster-wide primary storage - NFS
The steps to reproduce the bug
All VMs are created with service offering that have storage tags. I haven't tested it without tags
Zone-wide pool:
- deploy a VM
- enable maintenance mode of the storage pool
- force reconnect all host in the zone from the UI (or restart the cloudstack-agent service)
- cancle maintenance mode of the storage pool
Cluster-wide
- deploy a VM
- enable maintenance mode
- cancle maintenance mode
2025-09-30 14:09:42,578 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-15:[ctx-6b4bcd04, job-5444/job-5445, ctx-59178917]) (logid:51ee9166) Found pools [[]] that match with tags [[nfs-c
luster2]].
2025-09-30 14:09:42,578 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-15:[ctx-6b4bcd04, job-5444/job-5445, ctx-59178917]) (logid:51ee9166) No storage pools available for [shared] volum
e allocation.
2025-09-30 14:09:42,578 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-15:[ctx-6b4bcd04, job-5444/job-5445, ctx-59178917]) (logid:51ee9166) Using volume allocation algorithm random to r
eorder pools.
2025-09-30 14:09:42,578 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-15:[ctx-6b4bcd04, job-5444/job-5445, ctx-59178917]) (logid:51ee9166) Trying to find suitable pools to allocate volume [Vo
lume {"id":562,"instanceId":380,"name":"ROOT-380","uuid":"bb7f3c0b-19f8-4be1-a6ce-bf29cff9a0ff","volumeType":"ROOT"}] necessary to deploy VM [VM instance {"id":380,"instanceName":"i-2-380-VM","state":"Startin
g","type":"User","uuid":"f4f66160-a169-4b4a-8729-df528c516c33"}], using StoragePoolAllocator: [ZoneWideStoragePoolAllocator].
2025-09-30 14:09:42,580 DEBUG [o.a.c.s.a.ZoneWideStoragePoolAllocator] (Work-Job-Executor-15:[ctx-6b4bcd04, job-5444/job-5445, ctx-59178917]) (logid:51ee9166) Could not find any zone wide storage pool that ma
tched with any of the following tags [[nfs-cluster2]].
2025-09-30 14:09:42,581 DEBUG [o.a.c.s.a.ZoneWideStoragePoolAllocator] (Work-Job-Executor-15:[ctx-6b4bcd04, job-5444/job-5445, ctx-59178917]) (logid:51ee9166) ZoneWideStoragePoolAllocator is returning [0] sui
table storage pools [[]].
2025-09-30 14:09:42,581 DEBUG [o.a.c.s.a.ZoneWideStoragePoolAllocator] (Work-Job-Executor-15:[ctx-6b4bcd04, job-5444/job-5445, ctx-59178917]) (logid:51ee9166) Using volume allocation algorithm random to reord
er pools.
2025-09-30 14:09:42,581 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-15:[ctx-6b4bcd04, job-5444/job-5445, ctx-59178917]) (logid:51ee9166) No suitable pools found for volume [Volume {"id":562
,"instanceId":380,"name":"ROOT-380","uuid":"bb7f3c0b-19f8-4be1-a6ce-bf29cff9a0ff","volumeType":"ROOT"}] used by VM [VM instance {"id":380,"instanceName":"i-2-380-VM","state":"Starting","type":"User","uuid":"f
4f66160-a169-4b4a-8729-df528c516c33"}] under cluster: [3].
2025-09-30 14:09:42,581 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-15:[ctx-6b4bcd04, job-5444/job-5445, ctx-59178917]) (logid:51ee9166) No suitable pools found
2025-09-30 14:09:42,581 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-15:[ctx-6b4bcd04, job-5444/job-5445, ctx-59178917]) (logid:51ee9166) No suitable storagePools found under this Cluster: C
luster {id: "3", name: "Cluster-A-2", uuid: "bd8f0dce-1699-4879-8804-04713a1a87ee"}
2025-09-30 14:09:42,583 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-15:[ctx-6b4bcd04, job-5444/job-5445, ctx-59178917]) (logid:51ee9166) Could not find suitable Deployment Destination for t
his VM under any clusters, returning.
2025-09-30 14:09:42,584 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-15:[ctx-6b4bcd04, job-5444/job-5445, ctx-59178917]) (logid:51ee9166) Searching resources only under specified Pod: HostPod {"id":1,"nam
e":"Pod-A","uuid":"f1cb86a7-ea4f-4211-a104-dd501578d7e4"}
2025-09-30 14:09:42,584 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-15:[ctx-6b4bcd04, job-5444/job-5445, ctx-59178917]) (logid:51ee9166) Listing clusters in order of aggregate capacity, that have (at lea
st one host with) enough CPU and RAM capacity under this Pod: 1
2025-09-30 14:09:42,585 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-15:[ctx-6b4bcd04, job-5444/job-5445, ctx-59178917]) (logid:51ee9166) CapacityType: CPU is used for Cluster ordering
2025-09-30 14:09:42,586 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-15:[ctx-6b4bcd04, job-5444/job-5445, ctx-59178917]) (logid:51ee9166) Removing from the clusterId list these clusters from avoid set: [1
, 3]
2025-09-30 14:09:42,587 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-15:[ctx-6b4bcd04, job-5444/job-5445, ctx-59178917]) (logid:51ee9166) No clusters found after removing disabled clusters and clusters in
avoid list, returning.
2025-09-30 14:09:42,594 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-15:[ctx-6b4bcd04, job-5444/job-5445, ctx-59178917]) (logid:51ee9166) VM instance {"id":380,"instanceName":"i-2-380-VM","state":"Sto
pped","type":"User","uuid":"f4f66160-a169-4b4a-8729-df528c516c33"} state transited from [Starting] to [Stopped] with event [OperationFailed]. VM's original host: Host {"id":7,"name":"cs-dev-kvm-3-4","type":"R
outing","uuid":"e078cdb7-8819-43b2-a2c9-262788ad6099"}, new host: null, host before state transition: null
2025-09-30 14:09:42,598 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-15:[ctx-6b4bcd04, job-5444/job-5445, ctx-59178917]) (logid:51ee9166) Invocation exception, caused by: com.cloud.exception.Insuffi
cientServerCapacityException: Unable to create a deployment for VM instance {"id":380,"instanceName":"i-2-380-VM","state":"Starting","type":"User","uuid":"f4f66160-a169-4b4a-8729-df528c516c33"}Scope=interface com.cloud.dc.DataCenter; id=1
2025-09-30 14:09:42,599 INFO [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-15:[ctx-6b4bcd04, job-5444/job-5445, ctx-59178917]) (logid:51ee9166) Rethrow exception com.cloud.exception.InsufficientServerCapa
cityException: Unable to create a deployment for VM instance {"id":380,"instanceName":"i-2-380-VM","state":"Starting","type":"User","uuid":"f4f66160-a169-4b4a-8729-df528c516c33"}Scope=interface com.cloud.dc.D
ataCenter; id=1
2025-09-30 14:09:42,599 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-15:[ctx-6b4bcd04, job-5444/job-5445]) (logid:51ee9166) Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 380, job origi
n: 5444
2025-09-30 14:09:42,599 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-15:[ctx-6b4bcd04, job-5444/job-5445]) (logid:51ee9166) Unable to complete AsyncJob {"accountId":2,"cmd":"com.cloud.vm.VmWorkStart",
"cmdInfo":"rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zd
ElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkc
QB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAABfHQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwc
HA","cmdVersion":0,"completeMsid":null,"created":"Tue Sep 30 14:09:41 UTC 2025","id":5445,"initMsid":2199191191555,"instanceId":null,"instanceType":null,"lastPolled":null,"lastUpdated":null,"processStatus":0,
"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"68e14887-3529-4f40-a73d-1960007af720"}, job origin: 5444 com.cloud.exception.InsufficientServerCapacityException: Unable t
o create a deployment for VM instance {"id":380,"instanceName":"i-2-380-VM","state":"Starting","type":"User","uuid":"f4f66160-a169-4b4a-8729-df528c516c33"}Scope=interface com.cloud.dc.DataCenter; id=1
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1451)
What to do about it?
No response