CloudStack 4.20 UI Unable to add KVM-Host on VXLAN Logic structure - Errors

[tatay188]

problem

On CloudStack 4.20 with a KVM host,
modified the agent.properties private.network.device=cloudbr1 as originally have not resources available for cloudbr0

I have my VXLAN 1 on Cloudbr1 that reaches management and SS, We only have cloudbr1 created, No other bridge, and everything is BGP.

Then at the UI creating the ZONE I got errors:

We have had tryed the following Options:

Option 1: No tags
I put only [Physical Network 1] [VXLAN] Management, throws a popup error " All required traffic types should be added and with multiple physical networks each traffic type should have a label"

Option 2: No Tags, Throws and Error " All required traffic types should be added and with multiple physical networks each traffic type should have a label"
I put :
[Physical Network 1] [VXLAN] Management
[Physical Network 2] [VXLAN] Public
[Physical Network 3] [VXLAN] Guest
[Physical Network 4] [VXLAN] Storage

Option 3:
I put :
[Physical Network 1] [VXLAN] Management, Public, Guest, it goes to the next window - which is Great

Going with Option 3:

The the system throws exceptions like:
Something went wrong; please correct the following:
Unable to add the host: Cannot find the server resources at http://

At this point libvirt is always up and running.

LOGS:

Management logs shows:
Succeeded to import certificate in the keystore for agent on the KVM host: . Agent secured and trusted

The Agent Logs:

2025-03-28 16:48:42,550 DEBUG [commons.beanutils.ConvertUtils] (main:[]) (logid:) Convert string ',' to class 'java.lang.String'
2025-03-28 16:48:42,550 DEBUG [beanutils.converters.StringConverter] (main:[]) (logid:) Converting 'String' value ',' to type 'String'
2025-03-28 16:48:42,551 DEBUG [agent.properties.AgentPropertiesFileHandler] (main:[]) (logid:) Property [guid] has empty or null value. Using default value [null].
2025-03-28 16:48:42,552 DEBUG [agent.properties.AgentPropertiesFileHandler] (main:[]) (logid:) Property [developer] has empty or null value. Using default value [false].
2025-03-28 16:48:53,897 INFO [cloud.agent.AgentShell] (main:[]) (logid:) Agent started
2025-03-28 16:48:53,902 INFO [cloud.agent.AgentShell] (main:[]) (logid:) Implementation Version is 4.20.0.0
2025-03-28 16:48:53,904 INFO [cloud.agent.AgentShell] (main:[]) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2025-03-28 16:48:53,913 DEBUG [agent.properties.AgentPropertiesFileHandler] (main:[]) (logid:) Property [host] was altered. Now using the value [10.23.123.61,10.23.123.62].
2025-03-28 16:48:53,923 DEBUG [beanutils.converters.BooleanConverter] (main:[]) (logid:) Setting default value: false
2025-03-28 16:48:53,923 DEBUG [beanutils.converters.BooleanConverter] (main:[]) (logid:) Converting 'Boolean' value 'false' to type 'Boolean'
2025-03-28 16:48:53,923 DEBUG [beanutils.converters.BooleanConverter] (main:[]) (logid:) No conversion required, value is already a Boolean
2025-03-28 16:48:53,925 DEBUG [beanutils.converters.ByteConverter] (main:[]) (logid:) Setting default value: 0
2025-03-28 16:48:53,925 DEBUG [beanutils.converters.ByteConverter] (main:[]) (logid:) Converting 'Integer' value '0' to type 'Byte'
2025-03-28 16:48:53,925 DEBUG [beanutils.converters.ByteConverter] (main:[]) (logid:) Converted to Byte value '0'
2025-03-28 16:48:53,926 DEBUG [beanutils.converters.CharacterConverter] (main:[]) (logid:) Setting default value:
2025-03-28 16:48:53,926 DEBUG [beanutils.converters.CharacterConverter] (main:[]) (logid:) Converting 'Character' value ' ' to type 'Character'
2025-03-28 16:48:53,926 DEBUG [beanutils.converters.CharacterConverter] (main:[]) (logid:) No conversion required, value is already a Character
2025-03-28 16:48:53,926 DEBUG [beanutils.converters.DoubleConverter] (main:[]) (logid:) Setting default value: 0
2025-03-28 16:48:53,926 DEBUG [beanutils.converters.DoubleConverter] (main:[]) (logid:) Converting 'Integer' value '0' to type 'Double'
2025-03-28 16:48:53,927 DEBUG [beanutils.converters.DoubleConverter] (main:[]) (logid:) Converted to Double value '0.0'
2025-03-28 16:48:53,927 DEBUG [beanutils.converters.FloatConverter] (main:[]) (logid:) Setting default value: 0
2025-03-28 16:48:53,927 DEBUG [beanutils.converters.FloatConverter] (main:[]) (logid:) Converting 'Integer' value '0' to type 'Float'
2025-03-28 16:48:53,927 DEBUG [beanutils.converters.FloatConverter] (main:[]) (logid:) Converted to Float value '0.0'
2025-03-28 16:48:53,928 DEBUG [beanutils.converters.IntegerConverter] (main:[]) (logid:) Setting default value: 0

AGENT out :
Unable to find the guid
Unable to find the guid
Unable to find the guid
Unable to find the guid
Unable to find the guid

There is nothing there

And the agent come up, and immediately goes down!!

Tata Y.

versions

ACS 4.20.0.0
Management1 Ubuntu 22.04
Management2 ubuntu 22.04
Primary Storage CEPH
Secondary Storage EMC NFS
Hypervisor: KVM UBUNTU 22.04

The steps to reproduce the bug

1. Adding the first Zone Go over the process
2. on network [Physical Network 1] [VXLAN] Management, Public, Guest
3. Add hosts, Kvm host keeps bouncing,
4. enable debug logs: - there are no additional logs -

What to do about it?

No response

[Gunseerat]

Can you try entering guid from libvirt file, actually libvirt is responsible for sending this guid to cloudstack-agent.
Open /etc/libvirt/libvirtd.conf file
find host_uuid
copy this value
go to /etc/cloudstack/agent/agent.properties
in guid field which should be empty for you .... you can paste this.
Although this is sort of injecting values deliberately

[weizhouapache]

@tatay188
there is log file setup.log in the same folder as agent.log
you can have a look

[tatay188]

### On the AGENT side:

Not sure if this is just related to Ubuntu systems:

there is an error on the management server every time i reload it:

DEBUG:root:execute:sudo /usr/sbin/service cloudstack-management stop
DEBUG:root:execute:sudo update-rc.d -f cloudstack-management remove
DEBUG:root:execute:sudo update-rc.d -f cloudstack-management defaults
DEBUG:root:Failed to execute:update-rc.d: ### error: unable to read /etc/init.d/cloudstack-management
DEBUG:root:execute:sudo /usr/sbin/service cloudstack-management status
DEBUG:root:Failed to execute:× cloudstack-management.service - CloudStack Management Server
Loaded: loaded (/lib/systemd/system/cloudstack-management.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2025-04-02 13:13:48 UTC; 1 day 8h ago
Main PID: 537731 (code=exited, status=143)
CPU: 27min 54.476s

Obviously there is no such file: Should I create One?

ls -al /etc/init.d/
cryptdisks grub-common iwpmd nfs-common plymouth rng-tools-debian ufw
apparmor cryptdisks-early hwclock.sh keyboard-setup.sh ntp plymouth-log rpcbind unattended-upgrades
apport dbus ipmievd kmod open-iscsi procps rsync uuidd
console-setup.sh dpdk irqbalance lvm2 openipmi quota ssh x11-common
cron frr iscsid lvm2-lvmpolld openvswitch-switch

### On the management side, only when i restart the agent:

Apr 03 18:41:23 kvmvcompatl2001 systemd[1]: Starting Uncomplicated firewall...
Apr 03 18:41:23 kvmvcompatl2001 systemd[1]: Finished Uncomplicated firewall.
Apr 03 23:30:14 kvmvcompatl2001 systemd[1]: Stopping Uncomplicated firewall...
Apr 03 23:30:14 kvmvcompatl2001 ufw-init[56782]: Skip stopping firewall: ufw (not enabled)
Apr 03 23:30:14 kvmvcompatl2001 systemd[1]: ufw.service: Deactivated successfully.
Apr 03 23:30:14 kvmvcompatl2001 systemd[1]: Stopped Uncomplicated firewall.
DEBUG:root:execute:sudo /usr/sbin/service ufw start
DEBUG:root:execute:sudo /usr/sbin/service cloudstack-agent status
DEBUG:root:execute:sudo /usr/sbin/service cloudstack-agent stop
DEBUG:root:execute:sleep 30
DEBUG:root:execute:sudo update-rc.d -f cloudstack-agent remove
DEBUG:root:execute:sudo update-rc.d -f cloudstack-agent defaults
DEBUG:root:### Failed to execute:update-rc.d: error: unable to read /etc/init.d/cloudstack-agent
DEBUG:root:execute:sudo /usr/sbin/service cloudstack-agent status
DEBUG:root:Failed to execute:○ cloudstack-agent.service - CloudStack Agent
Loaded: loaded (/lib/systemd/system/cloudstack-agent.service; enabled; vendor preset: enabled)
Active: inactive (dead) (Result: exit-code) since Thu 2025-04-03 23:30:17 UTC; 31s ago
Docs: http://www.cloudstack.org/
Main PID: 56558 (code=exited, status=1/FAILURE)
CPU: 14.716s

Apr 03 23:30:17 kvmvcompatl2001 systemd[1]: Stopped CloudStack Agent.
Apr 03 23:30:17 kvmvcompatl2001 systemd[1]: cloudstack-agent.service: Consumed 14.716s CPU time.
DEBUG:root:execute:sudo /usr/sbin/service cloudstack-agent start

### Regarding the VXLAN i found the Cloudstack does not link the Interface Loopback to the VXLAN interface,
I did manually and now it propagates the routes

CLOUDSTACK ALONE DOES NOT ADD THE PARENT Loopback interface:
vtysh -c 'show interface vxlan1000'
Interface vxlan1000 is up, line protocol is up
Link ups: 4 last: 2025/04/06 03:16:37.88
Link downs: 0 last: (never)
vrf: default
index 143 metric 0 mtu 9050 speed 0 txqlen 1000
flags: <UP,LOWER_UP,BROADCAST,RUNNING,MULTICAST>
Type: Ethernet
HWaddr: 7e:05:79:2f:33:7d
Interface Type Vxlan
Interface Slave Type Bridge
VTEP IP: 10.23.133.14 VxLAN Id 1000 Access VLAN Id 1

Master interface: brvx-1000
protodown: off

Once I manually Link the parent Loopback interface:
vtysh -c 'show interface vxlan1000'
Interface vxlan1000 is up, line protocol is up
Link ups: 0 last: (never)
Link downs: 0 last: (never)
vrf: default
index 17 metric 0 mtu 9050 speed 0 txqlen 1000
flags: <UP,LOWER_UP,BROADCAST,RUNNING,MULTICAST>
Type: Ethernet
HWaddr: b6:9d:d9:66:b1:60
Interface Type Vxlan
Interface Slave Type Bridge
VTEP IP: 10.23.133.14 Link Interface lo VxLAN Id 1000 Access VLAN Id 1

Master interface: brvx-1000
protodown: off
Parent interface: lo

Is propagating the routes, and I can reach from the remote VTEP
Now the System VMs agent-state shows green

But still the System Vms' Interface name are not populated, should it be populated ? what is the normal status for this?

Please let me know.

Tata Y.

[weizhouapache]

it is normal, the public IPs are used by system vms, not guest networks

same on my env