BUGS: CloudStack 4.20 VXLAN-EVPN - Management VXLAN generates error - Public VXLAN routes not advertised - is Bug LINKED TO #9920? - Initial VMs running but Grey State for Agent State

[tatay188]

problem

Using [Network 1] Only for all three [Management], [guest],[public] Tagged as VXLAN and with cloudbr1

• **Should I use a separate Network for each ? ? **
  

What should I check, in addition to the official manual: I reread again the instructions and saw videos
https://www.youtube.com/watch?v=9gXEmWbgX2o <<Actually this video shown both VMs are down'
https://www.youtube.com/watch?v=DlJg3LYvIIs << this shows both on green.
https://www.youtube.com/watch?v=vZfHfoYEMdY << this one shows the [Agent State] green while the Vms are in starting mode with a bue dot.
I have Read the https://rohityadav.cloud/blog/cloudstack-kvm/

• Management servers are two physical separated servers. all Ubuntu 22.04
• DB is mysql running on a separate server. Mysql 8
• Hypervisor is running on a separate server. Ubuntu22.0 KVM
• Cloudstack 4.20.0.0

Issue 1) I can't add a second Management server, No idea why !!
Thank you Wei, as soon i shut the second Management server the initial VMs proxy and secondary storage come up
even when I enable and configure the "MultipleServer" = true and have the servers there.

Issue 2) Management Network is Tagged as VXLAN, the UI shows vlan://untagged - "no typo vlan" every-time i use the UI to change to vxlan://untagged
I did try vxlan://untagged parameter Got the message: Vlan parameter : vxlan://untagged is not in valid format
I did try vxlan:// Got the message: Unable to convert to broadcast URI: vxlan://
I did try vxlan:// Got the message: Vlan parameter : vxlan://1000 is not in valid format'

Issue 3) s-VM and Proxy come up but Public IP addresses are not being routed Advertised.
s-1-VM <control-ip-169.x.x.x> [State-Running] [Agent State-Gray]
v-2-VM <control-ip-169.x.x.y> [State-Running] [Agent State-Gray]
If both [State] and [Agent State] need to be green, What other debug can I enable ??

Is grey radio button a normal state as shown in the picture ?

NOTE The VXLAN parameter for the Public network was taken vxlan://1000, But IP addresses are not routed.

Issue 4) The VNI's I Choose for Guest are showing as VLANs,
I try to use the format of vxlan://1024 for vxlan the start and vxlan://2048 for the end and got the same error messages, not valid format

Issue 5) VXLAN 1000 No route to the public IP addresses
The Public IPs are assigned, but there is no network assigned to it: - Is there maybe a secret button on the global configuration ?

I have to use a Single management server as everything messes UP when I run a second Management server with the Same Database.

I reiterate The initial VMs are running:
s-1-VM <control-ip-169.x.x.x>
v-2-VM <control-ip-169.x.x.y>

I do not see the Public IP anywhere, not even advertised on vni1000 or any other, the public VNI is there was created by the system, but nothing is there - No IP addresses advertised

I used VLAN/VNI vxlan://1000

vxlan1000: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9050 master brvx-1000 state forwarding priority
There are 2 VNETS assigned

virsh domiflist s-1-VM
Interface Type Source Model MAC

vnet3 bridge cloud0 virtio 0e:00:a9:fe:a1:14
vnet4 bridge cloudbr1 virtio 1e:00:3f:00:03:33
vnet5 bridge brvx-1000 virtio 1e:00:b8:00:0c:fe

virsh domifaddr s-1-VM
Name MAC address Protocol Address

----- Nothing here -----

the command arp -n, does not show Public IP addresses neither the management IP addresses, just the private IP addresses and the <control-ip-169.x.x.x>

sh bgp evpn route vni 1000
BGP table version is 1, local router ID is 10.1.1.14
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[EthTag]:[ESI]:[IPlen]:[VTEP-IP]:[Frag-id]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
*> [3]:[0]:[32]:[10.1.1.14]
10.1.1.14(hv1)
32768 i
ET:8 RT:5641:1000

Displayed 1 prefixes (1 paths)

At the Agent side only WARNINGS I am not using uefi.

2025-04-03 00:01:44,744 ERROR [kvm.resource.LibvirtComputingResource] (main:[]) (logid:) uefi properties file not found due to: Unable to find file uefi.properties.
2025-04-03 00:01:46,815 WARN [utils.script.Script] (Agent-Handler-1:[]) (logid:) Process [15539] for command [/bin/bash -c virt-v2v --version ] encountered the error: [127].
2025-04-03 00:01:47,921 ERROR [cloud.agent.Agent] (agentRequest-Handler-5:[]) (logid:09929bd6) Unexpected arch null, expected x86_64
2025-04-03 00:01:46,815 WARN [utils.script.Script] (Agent-Handler-1:[]) (logid:) Execution of process [15539] for command [/bin/bash -c virt-v2v --version ] failed.
2025-04-03 00:01:46,815 WARN [utils.script.Script] (Agent-Handler-1:[]) (logid:) Process [15539] for command [/bin/bash -c virt-v2v --version ] encountered the error: [127].

AT the management the rror is:
2025-04-03 00:00:28,115 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-c96bca9e]) (logid:3fbff637) Executing ClusterServicePdu with service URL: https://10.1.1.1:9090/clusterservice
2025-04-03 00:21:28,122 ERROR [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-c96bca9e]) (logid:3fbff637) Exception from : https://10.1.1.1:9090/clusterservice, method : null, exception : javax.net.ssl.SSLPeerUnverifiedException: Certificate for <10.1.1.1> doesn't match any of the subject alternative names: [fde0:f:2897:1:1:0:0:1, 172.1.1.1, fe80:0:0:0:e643:4bff:fe81:9660, cs1.myinternaldomain.int, cloudstack.internal]

versions

• Management servers are two physical separated servers. all Ubuntu 22.04
• DB is mysql running on a separate server. Mysql 8
• Hypervisor is running on a separate server. Ubuntu22.0 KVM
• Cloudstack 4.20.0.0
• Primary Storage CEPH 19.2.1.1 RBD
• Secondary Storage EMC NFS

The steps to reproduce the bug

1. As described above

...

What to do about it?

No response

[rohityadavcloud]

@tatay188 can you try keeping the input as blank where you want to specify untagged vlan/vxlan? It may be possible that vxlan isn't supported for any network other than guest (cc @wido @weizhouapache ... others can advise)

[tatay188]

Hello Rohit, thank you for your input, The storage and management seems to be working fine. From the UI I am Using 2 Interfaces, I did try with a single interface, and same results, I have a single "Traffic-Label" for all [Guest][Public][Management] cloudbr1 for the Public I specifically stated vxlan://1000 and it was accepted, and cloudstack creates it, but there are no routes advertised at all from the KVM. as I show on the Outputs. for the Guest I put a range, but the prefix vxlan:// is not accepted durint he Zone creation, and after the zone creation the zone shows as reserved VLANs not VXLANs. The system VMs shows green running and grey button on the agent-state - is this a normal state Do I need to enable something special in the configuration. Global-settings ? I have degbug mode in the management and the agent and neither of them shows warnings or errors. Maybe I need one Those are the major issues I see, Guido can you share the version of cloudstack you are using, you can send me privately please. #10659 Tata Y.

…

On Apr 7, 2025, at 6:45 AM, Rohit Yadav ***@***.***> wrote: rohityadavcloud left a comment (apache/cloudstack#10659) @tatay188 can you try keeping the input as blank where you want to specify untagged vlan/vxlan? It may be possible that vxlan isn't supported for any network other than guest (cc @wido @weizhouapache ... others can advise) — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned. <https://github.com/tatay188> <https://github.com/wido> <https://github.com/weizhouapache> <#10659 (comment)> <https://github.com/notifications/unsubscribe-auth/ACNEALMN6LNXLYICN7P2JIL2YJJOLAVCNFSM6AAAAAB2MWQERCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOOBSHA4TEOBYGY> rohityadavcloud left a comment (apache/cloudstack#10659) <#10659 (comment)> @tatay188 <https://github.com/tatay188> can you try keeping the input as blank where you want to specify untagged vlan/vxlan? It may be possible that vxlan isn't supported for any network other than guest (cc @wido <https://github.com/wido> @weizhouapache <https://github.com/weizhouapache> ... others can advise) — Reply to this email directly, view it on GitHub <#10659 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACNEALMN6LNXLYICN7P2JIL2YJJOLAVCNFSM6AAAAAB2MWQERCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOOBSHA4TEOBYGY>. You are receiving this because you were mentioned.

[tatay188]

**### I noticed the cloudstack does not link the physical** lo loopback interface when creating a vxlan using the script. **### Regarding the VXLAN** i found the Cloudstack does not link the Interface Loopback to the VXLAN interface, I did manually and now it propagates the routes ### **CLOUDSTACK ALONE DOES NOT ADD THE PARENT Loopback interface vxlan:** vtysh -c 'show interface vxlan1000' Interface vxlan1000 is up, line protocol is up Link ups: 4 last: 2025/04/06 03:16:37.88 Link downs: 0 last: (never) vrf: default index 143 metric 0 mtu 9050 speed 0 txqlen 1000 flags: <UP,LOWER_UP,BROADCAST,RUNNING,MULTICAST> Type: Ethernet HWaddr: 7e:05:79:2f:33:7d Interface Type Vxlan Interface Slave Type Bridge VTEP IP: 10.23.133.14 VxLAN Id 1000 Access VLAN Id 1 Master interface: brvx-1000 protodown: off ### **After I manually Link the parent Loopback interface:** vtysh -c 'show interface vxlan1000' Interface vxlan1000 is up, line protocol is up Link ups: 0 last: (never) Link downs: 0 last: (never) vrf: default index 17 metric 0 mtu 9050 speed 0 txqlen 1000 flags: <UP,LOWER_UP,BROADCAST,RUNNING,MULTICAST> Type: Ethernet HWaddr: b6:9d:d9:66:b1:60 Interface Type Vxlan Interface Slave Type Bridge VTEP IP: 10.23.133.14 Link Interface lo VxLAN Id 1000 Access VLAN Id 1 Master interface: brvx-1000 protodown: off **Parent interface: lo** Is propagating the routes, and I can reach from the remote VTEP And even the Gray buttons for the system VMs are now green. It's a domino effect. But still for the Systems VMs the network Name is blank- is this a normal behavior, should not have a network name for the System VMs?  regarding setup logs, I got a fail after I reboot, but maybe related to be using UBUNTU: **### On the AGENT side:** Not sure if this is just related to Ubuntu systems: there is an error on the management server every time i reload it: DEBUG:root:execute:sudo /usr/sbin/service cloudstack-management stop DEBUG:root:execute:sudo update-rc.d -f cloudstack-management remove DEBUG:root:execute:sudo update-rc.d -f cloudstack-management defaults DEBUG:root:Failed to execute:update-rc.d: ### error: unable to read /etc/init.d/cloudstack-management DEBUG:root:execute:sudo /usr/sbin/service cloudstack-management status DEBUG:root:Failed to execute:× cloudstack-management.service - CloudStack Management Server Loaded: loaded (/lib/systemd/system/cloudstack-management.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Wed 2025-04-02 13:13:48 UTC; 1 day 8h ago Main PID: 537731 (code=exited, status=143) CPU: 27min 54.476s Obviously there is no such file: Should I create One? ls -al /etc/init.d/ cryptdisks grub-common iwpmd nfs-common plymouth rng-tools-debian ufw apparmor cryptdisks-early hwclock.sh keyboard-setup.sh ntp plymouth-log rpcbind unattended-upgrades apport dbus ipmievd kmod open-iscsi procps rsync uuidd console-setup.sh dpdk irqbalance lvm2 openipmi quota ssh x11-common cron frr iscsid lvm2-lvmpolld openvswitch-switch ### **On the management side, only when i restart the agent:** Apr 03 18:41:23 kvmvcompatl2001 systemd[1]: Starting Uncomplicated firewall... Apr 03 18:41:23 kvmvcompatl2001 systemd[1]: Finished Uncomplicated firewall. Apr 03 23:30:14 kvmvcompatl2001 systemd[1]: Stopping Uncomplicated firewall... Apr 03 23:30:14 kvmvcompatl2001 ufw-init[56782]: Skip stopping firewall: ufw (not enabled) Apr 03 23:30:14 kvmvcompatl2001 systemd[1]: ufw.service: Deactivated successfully. Apr 03 23:30:14 kvmvcompatl2001 systemd[1]: Stopped Uncomplicated firewall. DEBUG:root:execute:sudo /usr/sbin/service ufw start DEBUG:root:execute:sudo /usr/sbin/service cloudstack-agent status DEBUG:root:execute:sudo /usr/sbin/service cloudstack-agent stop DEBUG:root:execute:sleep 30 DEBUG:root:execute:sudo update-rc.d -f cloudstack-agent remove DEBUG:root:execute:sudo update-rc.d -f cloudstack-agent defaults DEBUG:root:### Failed to execute:update-rc.d: error: unable to read /etc/init.d/cloudstack-agent DEBUG:root:execute:sudo /usr/sbin/service cloudstack-agent status DEBUG:root:Failed to execute:○ cloudstack-agent.service - CloudStack Agent Loaded: loaded (/lib/systemd/system/cloudstack-agent.service; enabled; vendor preset: enabled) Active: inactive (dead) (Result: exit-code) since Thu 2025-04-03 23:30:17 UTC; 31s ago Docs: http://www.cloudstack.org/ Main PID: 56558 (code=exited, status=1/FAILURE) CPU: 14.716s Apr 03 23:30:17 kvmvcompatl2001 systemd[1]: Stopped CloudStack Agent. Apr 03 23:30:17 kvmvcompatl2001 systemd[1]: cloudstack-agent.service: Consumed 14.716s CPU time. DEBUG:root:execute:sudo /usr/sbin/service cloudstack-agent start Please let me know your thoughts. Thank you Tata Y.

…

On Apr 7, 2025, at 6:45 AM, Rohit Yadav ***@***.***> wrote: rohityadavcloud left a comment (apache/cloudstack#10659) @tatay188 can you try keeping the input as blank where you want to specify untagged vlan/vxlan? It may be possible that vxlan isn't supported for any network other than guest (cc @wido @weizhouapache ... others can advise) — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned. <https://github.com/tatay188> <https://github.com/wido> <https://github.com/weizhouapache> <#10659 (comment)> <https://github.com/notifications/unsubscribe-auth/ACNEALMN6LNXLYICN7P2JIL2YJJOLAVCNFSM6AAAAAB2MWQERCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOOBSHA4TEOBYGY> rohityadavcloud left a comment (apache/cloudstack#10659) <#10659 (comment)> @tatay188 <https://github.com/tatay188> can you try keeping the input as blank where you want to specify untagged vlan/vxlan? It may be possible that vxlan isn't supported for any network other than guest (cc @wido <https://github.com/wido> @weizhouapache <https://github.com/weizhouapache> ... others can advise) — Reply to this email directly, view it on GitHub <#10659 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACNEALMN6LNXLYICN7P2JIL2YJJOLAVCNFSM6AAAAAB2MWQERCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOOBSHA4TEOBYGY>. You are receiving this because you were mentioned.

[wido]

There is no need to link a VXLAN device to the loopback. The loopback should just contain the VTEP address where other hypervisors can reach it. Take a look at this blogpost: https://blog.widodh.nl/2022/03/proxmox-with-bgpevpnvxlan/

Under CloudStack it works exactly the same. CloudStack itself doesn't do anything with VXLAN. It just creates a bridge and that's it. The rest is up to your BGP+EVPN configuration.

I would suggest that you get that part working properly prior to trying to add it into CloudStack.