Cloudstack 4.19.2.0 - KVM Cloudstack Agent Stuck

[wverleger]

Hi,

We are running ACS 4.19.2.0 with KVM hypervisors (Ubuntu 24.04 LTS), using linstor as hyperconverged storage and nfs as secondary storage.

We have the phenomen, that one of our hosts seems to have a cloudstack agent problem.
In the UI the "State" is "Up" and the "Resource state" is "Enabled" of the host but for some reason the agent seems to be unresponsive.

We have noticed this because the agent.log stopped working (no new entries and no logrotate) and volume snapshots started to fail and ended in error state.

What we also can see is, that the communication to (redundant) virtual routers on this host is disrupted, health checks fail, and the requested host is not repsonding.

After failed health checks the router is getting replaced and the old one should be expunged but the router is still in expunging status.

I have attached a management-server log entry regarding virtual router "r-749-VM" not able to perform health checks.
The cloudstack-agent log on the host is inconspicuous.

cloudstack_management_log.txt

After restarting the cloudstack-agent the logging starts to work again. But the router state is still "expunging".

Has anyone had any experience with it?
Does anyone know a good way to troubleshoot?

Thanks!

BR
Wilken

[wverleger]

Hi,

As an additional info we receive the following log entries from cloudstack-agent log now:

cloudstack-agent_log.txt

While creating a volume snapshot:

2025-04-24 11:55:00,183 INFO [resource.wrapper.LinstorBackupSnapshotCommandWrapper] (agentRequest-Handler-5:null) (logid:23fbdeeb) Src: /dev/mapper/vg_nvme-cs--53eced61--c5b3--4cb0--85c3--b7013e1d6ca6_00000_cs--a2db7f5c--b44c--48b4--bbdf--c39ad8e49318 | wilken-test
2025-04-24 11:55:00,186 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:23fbdeeb) Attempting to create storage pool 638de611-0552-36a1-91ca-6df1c493dc8f (NetworkFilesystem) in libvirt

I can not send the command: virsh pool-list --all
Seems to load forever. On other hosts it works instantly.

[DaanHoogland]

these “ERROR”s you can ignore. The agent tries several strategies to retrieve the cpu-speed and the last one you see succeeding:

2025-04-24 10:14:18,062 INFO  [utils.linux.KVMHostInfo] (Agent-Handler-1:null) (logid:) Fetching CPU speed from "host capabilities"
2025-04-24 10:14:18,065 INFO  [utils.linux.KVMHostInfo] (Agent-Handler-1:null) (logid:) Retrieved value [2800000000] from "host capabilities". This corresponds to a CPU speed of [2800] MHz.

[DaanHoogland]

@wverleger , I think you should maybe also restart qemu/libvirtd to resolve the issues. If there are no vital VMs on the machine , even reboot it, though this should not be necessary.

• Can you migrate VMs away? and
• What happens if you put the host in maintenance?

[wverleger]

Hi Daan,

Thank you!

I've restarted qemu/libvirtd and cloudstack-agent and it works as intended for now.
I will have a close look and double check next week.