[Linstor] Stop vm error after remove linstor primary storage and linstor controller

[hulucc]

problem

Stop vm error after remove linstor storage from cloudstack and stop linstor controller

cloudstack-0 cloudstack com.cloud.utils.exception.CloudRuntimeException: Unable to stop VM instance {"id":234,"instanceName":"i-2-234-VM","type":"User","uuid":"0775e293-5101-4c04-9636-719109de1368"}
cloudstack-0 cloudstack         at com.cloud.vm.VirtualMachineManagerImpl.advanceStop(VirtualMachineManagerImpl.java:2191)
cloudstack-0 cloudstack         at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStop(VirtualMachineManagerImpl.java:1980)
cloudstack-0 cloudstack         at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStop(VirtualMachineManagerImpl.java:5505)
cloudstack-0 cloudstack         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
cloudstack-0 cloudstack         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
cloudstack-0 cloudstack         at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
cloudstack-0 cloudstack         at java.base/java.lang.reflect.Method.invoke(Method.java:566)
cloudstack-0 cloudstack         at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:105)
cloudstack-0 cloudstack         at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5610)
cloudstack-0 cloudstack         at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
cloudstack-0 cloudstack         at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
cloudstack-0 cloudstack         at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
cloudstack-0 cloudstack         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
cloudstack-0 cloudstack         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
cloudstack-0 cloudstack         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
cloudstack-0 cloudstack         at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
cloudstack-0 cloudstack         at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
cloudstack-0 cloudstack         at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
cloudstack-0 cloudstack         at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
cloudstack-0 cloudstack         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
cloudstack-0 cloudstack         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
cloudstack-0 cloudstack         at java.base/java.lang.Thread.run(Thread.java:829)
cloudstack-0 cloudstack INFO  [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-21:ctx-a035adc2 job-3248/job-3249) (logid:e7885641) Remove job-3249 from job monitoring
cloudstack-0 cloudstack ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-47:ctx-354ffcf7 job-3248) (logid:e7885641) Unexpected exception while executing org.apache.cloudstack.api.command.admin.vm.StopVMCmdByAdmin
cloudstack-0 cloudstack com.cloud.utils.exception.CloudRuntimeException: Unable to stop VM instance {"id":234,"instanceName":"i-2-234-VM","type":"User","uuid":"0775e293-5101-4c04-9636-719109de1368"}
cloudstack-0 cloudstack         at com.cloud.vm.VirtualMachineManagerImpl.advanceStop(VirtualMachineManagerImpl.java:2191)
cloudstack-0 cloudstack         at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStop(VirtualMachineManagerImpl.java:1980)
cloudstack-0 cloudstack         at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStop(VirtualMachineManagerImpl.java:5505)
cloudstack-0 cloudstack         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
cloudstack-0 cloudstack         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
cloudstack-0 cloudstack         at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
cloudstack-0 cloudstack         at java.base/java.lang.reflect.Method.invoke(Method.java:566)
cloudstack-0 cloudstack         at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:105)
cloudstack-0 cloudstack         at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5610)
cloudstack-0 cloudstack         at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
cloudstack-0 cloudstack         at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
cloudstack-0 cloudstack         at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
cloudstack-0 cloudstack         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
cloudstack-0 cloudstack         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
cloudstack-0 cloudstack         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
cloudstack-0 cloudstack         at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
cloudstack-0 cloudstack         at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
cloudstack-0 cloudstack         at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
cloudstack-0 cloudstack         at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
cloudstack-0 cloudstack         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
cloudstack-0 cloudstack         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
cloudstack-0 cloudstack         at java.base/java.lang.Thread.run(Thread.java:829)

versions

4.19.1.2 kvm centos8.5

The steps to reproduce the bug

1. Create linstor primary storage, create vm1 with linstor disk on hostA
2. Expunge vm1, delete linstor storage, stop linstor controller
3. Create vm2 with other disk on hostA
4. Stop vm2, will cause the error

What to do about it?

No response

[slavkap]

@hulucc, what storage uses the volume for the second VM (vm2)?

[DaanHoogland]

@hulucc , why do you think the error is related to the deleted storage?

[hulucc]

@slavkap , what storage uses the volume for the second VM (vm2)?

any other storage, it doesn't matter.

@DaanHoogland , why do you think the error is related to the deleted storage?

Because the error only ocurrs after linstor storage get deleted, If i never add linstor storage or didn't delete it on a new cloudstack instance, everything is fine.

[DaanHoogland]

@rp- can you have a look?

[rp-]

I can reproduce it, even with 4.19.2.
The more interesting exception would have been this:

javax.ws.rs.ProcessingException: java.net.ConnectException: Connection refused (Connection refused)
       at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:275)                                                                                                                   
       at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:300)                                                                                                                                        
       at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$0(JerseyInvocation.java:662)                                                                                                                  
       at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697)                                                                                                                            
       at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691
       at org.glassfish.jersey.internal.Errors.process(Errors.java:292
       at org.glassfish.jersey.internal.Errors.process(Errors.java:274
       at org.glassfish.jersey.internal.Errors.process(Errors.java:205
       at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390
       at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691
       at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:661
       at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:413
       at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:313
       at com.linbit.linstor.api.ApiClient.invokeAPI(ApiClient.java:704
       at com.linbit.linstor.api.DevelopersApi.viewResources(DevelopersApi.java:6606
       at com.cloud.hypervisor.kvm.storage.LinstorStorageAdaptor.tryDisconnectLinstor(LinstorStorageAdaptor.java:388
       at com.cloud.hypervisor.kvm.storage.LinstorStorageAdaptor.disconnectPhysicalDiskByPath(LinstorStorageAdaptor.java:479
       at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.disconnectPhysicalDiskByPath(KVMStoragePoolManager.java:207
       at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.cleanupDisk(LibvirtComputingResource.java:3400
       at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStopCommandWrapper.execute(LibvirtStopCommandWrapper.java:100
       at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStopCommandWrapper.execute(LibvirtStopCommandWrapper.java:48
       at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78
       at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1930
       at com.cloud.agent.Agent.processRequest(Agent.java:683
       at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1106
       at com.cloud.utils.nio.Task.call(Task.java:83
       at com.cloud.utils.nio.Task.call(Task.java:29
       at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264
       at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128
       at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628
       at java.base/java.lang.Thread.run(Thread.java:829
Caused by: java.net.ConnectException: Connection refused (Connection refused
       at java.base/java.net.PlainSocketImpl.socketConnect(Native Method
       at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412
       at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255
       at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237
       at java.base/java.net.Socket.connect(Socket.java:609
       at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:177)
       ...

Seems the Linstor provider is still loaded in the agent and tries to disconnect the disk that doesn't belong to it.
As a workaround you can restart the agents, after removing the Linstor storage, as this will not load the provider anymore.

I'll try to work on a fix ignoring, such disconnect requests.

[Pearl1594]

Fixed with :#10561