Unable to deploy an instance on PowerFlex storage, with powerflex.connect.on.demand enabled

[sureshanaparti]

problem

Instance deployment failed on PowerFlex storage, with powerflex.connect.on.demand enabled. The MDMs are not configured while preparing the SDC connections.

2025-08-15 14:56:03,074 DEBUG [o.a.c.s.d.m.ScaleIOSDCManagerImpl] (Work-Job-Executor-15:[ctx-a2b46ad8, job-58/job-60, ctx-6c69cd89]) (logid:329970ca) Preparing SDC on the host Host {"id":2,"name":"pr11452-t14067-kvm-ol8-kvm2","type":"Routing","uuid":"bc1f3a08-ad99-4b67-852e-628a02158b53"}
2025-08-15 14:56:03,076 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (Work-Job-Executor-15:[ctx-a2b46ad8, job-58/job-60, ctx-6c69cd89]) (logid:329970ca) Wait time setting on com.cloud.agent.api.PrepareStorageClientCommand is 60 seconds
2025-08-15 14:56:03,077 DEBUG [c.c.a.m.ClusteredAgentAttache] (Work-Job-Executor-15:[ctx-a2b46ad8, job-58/job-60, ctx-6c69cd89]) (logid:329970ca) Seq 4-7097391537759191139: Routed from 32989056599078
2025-08-15 14:56:03,077 DEBUG [c.c.a.t.Request] (Work-Job-Executor-15:[ctx-a2b46ad8, job-58/job-60, ctx-6c69cd89]) (logid:329970ca) Seq 2-7097391537759191139: Sending  { Cmd , MgmtId: 32989056599078, via: 2(pr11452-t14067-kvm-ol8-kvm2), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.PrepareStorageClientCommand":{"poolType":"PowerFlex","poolUuid":"6a31478b-3fcd-4be2-9ee5-f57971b1bbe1","details":{"powerflex.block.sdc.unprepare":"false","powerflex.storagepool.system.id":"3a9d888f33e3880f","powerflex.storagepool.mdms":"10.0.35.115,10.0.34.88","powerflex.mdm.change.apply.wait":"3000","powerflex.mdm.validate.on.connect":"false"},"wait":"60","bypassHostMaintenance":"false"}}] }
...
2025-08-15 14:56:17,335 DEBUG [o.a.c.s.d.m.ScaleIOSDCManagerImpl] (Work-Job-Executor-15:[ctx-a2b46ad8, job-58/job-60, ctx-6c69cd89]) (logid:329970ca) Try to get host SDC Id for pool: StoragePool {"id":3,"name":"pflexpool","poolType":"PowerFlex","uuid":"6a31478b-3fcd-4be2-9ee5-f57971b1bbe1"}, with SDC guid E6E12AD5-B83C-46D0-B413-4E89D74BD0A1
2025-08-15 14:56:17,336 DEBUG [o.a.c.s.d.c.ScaleIOGatewayClientImpl] (Work-Job-Executor-15:[ctx-a2b46ad8, job-58/job-60, ctx-6c69cd89]) (logid:329970ca) Sending GET request: GET https://10.0.34.88/api/types/Sdc/instances HTTP/1.1
2025-08-15 14:56:17,370 DEBUG [o.a.c.s.d.c.ScaleIOGatewayClientImpl] (Work-Job-Executor-15:[ctx-a2b46ad8, job-58/job-60, ctx-6c69cd89]) (logid:329970ca) Received response: 200 , for the sent GET request: GET https://10.0.34.88/api/types/Sdc/instances HTTP/1.1
2025-08-15 14:56:17,374 WARN  [o.a.c.s.d.m.ScaleIOSDCManagerImpl] (Work-Job-Executor-15:[ctx-a2b46ad8, job-58/job-60, ctx-6c69cd89]) (logid:329970ca) Couldn't retrieve PowerFlex storage SDC details from the host: Host {"id":2,"name":"pr11452-t14067-kvm-ol8-kvm2","type":"Routing","uuid":"bc1f3a08-ad99-4b67-852e-628a02158b53"}, add MDMs if On-demand connect disabled or try (re)install SDC & restart agent
2025-08-15 14:56:17,377 WARN  [o.a.c.s.d.d.ScaleIOPrimaryDataStoreDriver] (Work-Job-Executor-15:[ctx-a2b46ad8, job-58/job-60, ctx-6c69cd89]) (logid:329970ca) SDC not connected on the host: Host {"id":2,"name":"pr11452-t14067-kvm-ol8-kvm2","type":"Routing","uuid":"bc1f3a08-ad99-4b67-852e-628a02158b53"}
...
2025-08-15 14:56:17,835 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-40:[ctx-975d4b48, job-58]) (logid:329970ca) Unexpected exception while executing org.apache.cloudstack.api.command.admin.vm.DeployVMCmdByAdmin com.cloud.utils.exception.CloudRuntimeException: Unable to orchestrate the start of VM instance {"instanceName":"i-2-8-VM","uuid":"db3b2dbf-c2ce-410c-8bbe-3a7ec8faf3ab"}.
	at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5830)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:102)
	at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5951)
	at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:99)
	at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:689)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
	at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:637)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

2025-08-15 14:56:17,836 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-40:[ctx-975d4b48, job-58]) (logid:329970ca) Complete async job-58, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Unable to orchestrate the start of VM instance {"instanceName":"i-2-8-VM","uuid":"db3b2dbf-c2ce-410c-8bbe-3a7ec8faf3ab"}."}

versions

ACS 4.21.0 RC3 / KVM / PowerFlex storage

The steps to reproduce the bug

1. Add the PowerFlex storage as primary storage
2. Install SDCs on the hosts.
3. Update configuration 'powerflex.connect.on.demand' to true.
4. Disable primary storages other than PowerFlex storage
5. Create an instance

What to do about it?

Instance deployment should succeed on PowerFlex storage, after MDMs of the PowerFlex storage are added on the host (when 'powerflex.connect.on.demand' is true.)

[sureshanaparti]

Fixed in #11458