Infinite loading of system virtual machines

[miloserdoff]

Hello. Today I noticed that one of my system VMs (console proxy) is unavailable. Its Agent State was Down. I decided to reboot the system VMs and now they are in the Starting state for a long time.

The Cloudstack Management Server logs said:
WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-0:[ctx-e8343f22]) (logid:fed07982) Task (job-389) has been pending for 1618 seconds
WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-0:[ctx-e8343f22]) (logid:fed07982) Task (job-390) has been pending for 1618 seconds

Now, they are still in the Starting state.

I decided to check other functions of this host, and saw that I can't view the contents of the NFS server:
ERROR [c.c.a.ApiServer] (qtp1404565079-82973:[ctx-1d9fd248, ctx-eddcf4ec]) (logid:c913a3ec) unhandled exception executing api command: [Ljava.lang.String;@1863a332 com.cloud.utils.exception.CloudRuntimeException: No remote endpoint to send command

Please tell me how to solve the problem and restore the host's functionality?

[DaanHoogland]

@miloserdoff , first stepd would be

• Have tried trouble shooting the network connectivity from both the MS and the host?, and
• Have you tried to manually mount the secondary storage?

[bhouse-nexthop]

Did you recently update the system, such as going from 4.20.0 to 4.20.1 (maybe inadvertently) ? If so you likely need to install the newer systemvm.

Also if you check the agent log of the host the systemvm is being deployed on you'll usually get more information.

As a side note, I recently did the 4.20.0 -> 4.20.1 upgrade on my test cluster as it was auto pulled in and I had that same behavior. Installing the updated systemvm appeared to get rid of the error in the agent log when it tried to nfs mount the path but the systemvm never actually would start for me. Since this is a test cluster I'm just running my ansible playbooks to redeploy everything from bare metal. I often just hit the power on my test cluster on all nodes since it sits behind me and is loud, so I figured I might have corrupted something .... though I probably should have investigated further.