Upgrading Broke vpc and kubernetes (4.18 to 4.20)

[carbon-arts]

problem

After upgrading from 4.18.1 version to 4.20.0 we are unable to add vpc vpn which has split connection enabled. Without a split connection we were able to connect vpn. Can anyone confirm that 4.20.0 doesn't support split connection vpc connection

mgm_log.txt
cloud_log.txt

versions

The versions of ACS: 4.20

The steps to reproduce the bug

1.upgraded from 4.18 to 4.20 following the official upgradation documentation
2.upgraded the system vms from 4.18 to 4.20
3.upgraded the virtual routers from 4.18 to 4.20
then check and , vpc was broken , so was kubernetes

What to do about it?

No response

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[weizhouapache]

Is it fixed by #10067 ?

[DaanHoogland]

tnx @wei@carbon-arts,
@carbon-arts, this means the fix is in 4.19.2 and will be in 4.20.1 (unfortunately not in 4.20.0)

[carbon-arts]

how to incorporate this patch into cloudstack 4.20? currently stuck without an idea , deleted all the old templates and added the new one from here , then deleted the system vms , now what should i do ?

[weizhouapache]

how to incorporate this patch into cloudstack 4.20? currently stuck without an idea , deleted all the old templates and added the new one from here , then deleted the system vms , now what should i do ?

@carbon-arts
You can refer to #10184 (reply in thread)

[carbon-arts]

thanks for the help guys , i tried that solution , restarted the cloudstack-management after that.

after that i tried to create an site to site vpn but the error :

Resource [Site2SiteVpnConnection:6] is unreachable: Failed to apply site-to-site VPN

comes up

in management log :

2025-05-14 07:28:40,805 DEBUG [c.c.a.t.Request] (AgentManager-Handler-1:[]) (logid:) Seq 24-7736339734892388827: Processing: { Ans: , MgmtId: 52237573761, via: 24, Ver: v1, Flags: 100, [{"com.cloud.agent.api.routing.GroupAnswer":{"results":["null - success: Creating file in VR, with ip: 169.254.42.87, file: site_2_site_vpn.json.e8907165-bd67-4916-b73c-5e34c51db82e","null - failed: java.io.IOException: Stream closed

[carbon-arts]

update*:

manually went inside the router changed the lines in configure.py as mentioned above , and restarted the router , currently when creating new vpn connection , instead of showing error in status , it shows disconnected .

/var/log/cloud.log of a virtual router used for vpc .

cloud.log

[weizhouapache]

thanks for the help guys , i tried that solution , restarted the cloudstack-management after that.

you need to restart VR or restart network with cleanup (to recreate VR)

[weizhouapache]

update*:

manually went inside the router changed the lines in configure.py as mentioned above , and restarted the router , currently when creating new vpn connection , instead of showing error in status , it shows disconnected .

/var/log/cloud.log of a virtual router used for vpc .

cloud.log

can you reset the VPN connection and share the output of ipsec statusall in the VR ?

[carbon-arts]

you need to restart VR or restart network with cleanup (to recreate VR)

did that , same issue persists , one more thing i have noticed is that when i did the restart with cleanup on the Virtual Router , the line:

if splitconnections and len(peerlistarr) > 1:

somehow reverts to
if splitconnections and peerlistarr.count > 1:

and the output of ipsec statusall is empty.

[carbon-arts]

/var/log/cloud.log , didn't see this issue before, i just rebooted the vm after changing peerlistarr.count > 1: to
len(peerlistarr) > 1:

root@r-563-VM:~# tail -f /var/log/cloud.log 
Thu May 15 11:50:45 AM UTC 2025 Could not find patch file, retrying
Thu May 15 11:50:47 AM UTC 2025 Could not find patch file, retrying

also couldn't delete vpn's now , had this issue before , but don't know how it got rectified
Delete VPN connection (b954a974-1b79-4e4c-8b3d-1bde704ffcef) Resource [Site2SiteVpnConnection:17] is unreachable: Failed to apply site-to-site VPN

[weizhouapache]

you need to restart VR or restart network with cleanup (to recreate VR)

did that , same issue persists , one more thing i have noticed is that when i did the restart with cleanup on the Virtual Router , the line:

if splitconnections and len(peerlistarr) > 1:

somehow reverts to if splitconnections and peerlistarr.count > 1:

and the output of ipsec statusall is empty.

I think it should work, you might have missed a step or so.

@carbon-arts can you share what you have done ?