cloudstack-agent in restart loop

[rohityadavcloud]

problem

On my arm64/KVM host, ovftool & virt-v2v isn't available and it kills the agent when the version isn't found:

2025-09-26 13:05:09,277 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-2:[]) (logid:075c27e6) Attempting to create storage pool dc9afbc9-aec9-30ff-81c8-9a9aa1485c68 (NetworkFilesystem) in libvirt
2025-09-26 13:05:09,279 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-2:[]) (logid:075c27e6) Found existing defined storage pool dc9afbc9-aec9-30ff-81c8-9a9aa1485c68, using it.
2025-09-26 13:05:09,290 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-2:[]) (logid:075c27e6) Trying to fetch storage pool dc9afbc9-aec9-30ff-81c8-9a9aa1485c68 from libvirt
2025-09-26 13:05:09,570 INFO  [cloud.agent.Agent] (AgentRequest-Handler-3:[]) (logid:075c27e6) Processing agent ready command, agent id = 179, uuid = 9b3bc304-101e-46f5-a399-ad2d596a3dcd, name = kvmpi.onprem.in
2025-09-26 13:05:09,573 INFO  [cloud.agent.Agent] (AgentRequest-Handler-3:[]) (logid:075c27e6) Ready command is processed for agent [id: 179, uuid: 9b3bc304-101e-46f5-a399-ad2d596a3dcd, name: kvmpi.onprem.in]
2025-09-26 13:05:09,838 WARN  [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:075c27e6) Execution of process [3140165] for command [/bin/bash -c ovftool --version ] failed.
2025-09-26 13:05:09,839 WARN  [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:075c27e6) Process [3140165] for command [/bin/bash -c ovftool --version ] encountered the error: [127].
2025-09-26 13:05:09,888 INFO  [cloud.agent.Agent] (AgentShutdownThread:[]) (logid:) Stopping the agent: Reason = sig.kill

versions

ACS 4.21, KVM on both arm64 & x86

The steps to reproduce the bug

No response

What to do about it?

It should report tools arent' available but not kill the agent; for env that don't want to use virt-v2v or ovftool

[rohityadavcloud]

Suprisingly, the host that is disabled isn't killed, but the host that's enabled gets disconnected:

[rohityadavcloud]

Debug logs show another reason:

2025-09-26 13:13:55,921 DEBUG [kvm.resource.LibvirtComputingResource] (AgentRequest-Handler-3:[]) (logid:70ede39b) Successfully executed process [3152355] for command [/usr/share/cloudstack-common/scripts/vm/hypervisor/versions.sh ].
2025-09-26 13:13:55,921 DEBUG [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:70ede39b) Executing command [/bin/bash -c dpkg -l nbdkit ].
2025-09-26 13:13:55,946 DEBUG [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:70ede39b) Successfully executed process [3152374] for command [/bin/bash -c dpkg -l nbdkit ].
2025-09-26 13:13:55,946 DEBUG [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:70ede39b) Executing command [/bin/bash -c virt-v2v --version | awk '{print $2}' ].
2025-09-26 13:13:55,983 DEBUG [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:70ede39b) Successfully executed process [3152375] for command [/bin/bash -c virt-v2v --version | awk '{print $2}' ].
2025-09-26 13:13:55,983 DEBUG [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:70ede39b) Executing command [/bin/bash -c ovftool --version ].
2025-09-26 13:13:55,989 WARN  [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:70ede39b) Execution of process [3152381] for command [/bin/bash -c ovftool --version ] failed.
2025-09-26 13:13:55,989 DEBUG [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:70ede39b) Exit value of process [3152381] for command [/bin/bash -c ovftool --version ] is [127].
2025-09-26 13:13:55,989 WARN  [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:70ede39b) Process [3152381] for command [/bin/bash -c ovftool --version ] encountered the error: [127].
2025-09-26 13:13:55,991 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-3:[]) (logid:70ede39b) Seq 179-8821425770111959047:  { Ans: , MgmtId: 104339268210986, via: 179, Ver: v1, Flags: 110, [{"com.cloud.agent.api.ReadyAnswer":{"detailsMap":{"host.virtv2v.version":"2.4.0"},"result":"true","wait":"0","bypassHostMaintenance":"false"}}] }
2025-09-26 13:13:56,038 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-4:[]) (logid:70ede39b) Request:Seq -1--1:  { Cmd , MgmtId: -1, via: -1, Ver: v1, Flags: 111, [{"com.cloud.agent.api.ReadyCommand":{"_details":"java.lang.NullPointerException: Cannot invoke "String.equals(Object)" because "uefiEnabled" is null","wait":"0","bypassHostMaintenance":"false"}}] }
2025-09-26 13:13:56,038 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-4:[]) (logid:70ede39b) Processing command: com.cloud.agent.api.ReadyCommand
2025-09-26 13:13:56,039 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-4:[]) (logid:70ede39b) Not ready to connect to mgt server: java.lang.NullPointerException: Cannot invoke "String.equals(Object)" because "uefiEnabled" is null
2025-09-26 13:13:56,039 INFO  [cloud.agent.Agent] (AgentShutdownThread:[]) (logid:) Stopping the agent: Reason = sig.kill

And management server logs have this:

...
2025-09-26 13:15:45,783 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-8:[]) (logid:) Seq 179-6141502516849803271: No more commands found
2025-09-26 13:15:45,783 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-49:[ctx-31b11afa]) (logid:4550268b) Seq 179-6141502516849803271: Received:  { Ans: , MgmtId: 104339268210986, via: 179(kvmpi.onprem.in), Ver: v1, Flags: 110, { ReadyAnswer } }
2025-09-26 13:15:45,783 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-49:[ctx-31b11afa]) (logid:4550268b) Got HOST_UEFI_ENABLE [null] for host [Host {"id":179,"name":"kvmpi.onprem.in","type":"Routing","uuid":"9b3bc304-101e-46f5-a399-ad2d596a3dcd"}]:
2025-09-26 13:15:45,785 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-49:[ctx-31b11afa]) (logid:4550268b) Failed to handle host connection: java.lang.NullPointerException: Cannot invoke "String.equals(Object)" because "uefiEnabled" is null
        at com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:812)
        at com.cloud.agent.manager.AgentManagerImpl.connectHostAgent(AgentManagerImpl.java:1403)
        at com.cloud.agent.manager.AgentManagerImpl.sendReadyAndGetAttache(AgentManagerImpl.java:1365)
        at com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:1419)
        at com.cloud.agent.manager.AgentManagerImpl$HandleAgentConnectTask.runInContext(AgentManagerImpl.java:1515)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)

[rohityadavcloud]

uefi.properties was missing, and originally not required for arm64 hosts:

guest.nvram.template.secure=/usr/share/OVMF/OVMF_VARS_4M.ms.fd
guest.nvram.template.legacy=/usr/share/OVMF/OVMF_VARS_4M.fd
guest.nvram.path=/var/lib/libvirt/qemu/nvram/
guest.loader.secure=/usr/share/OVMF/OVMF_CODE_4M.secboot.fd
guest.loader.legacy=/usr/share/OVMF/OVMF_CODE_4M.fd

Adding this at /etc/cloudstack/agent/uefi.properties - fixed the issue.

[weizhouapache]

this is same as #11604 which has been fixed by #11610