Failed to apply site-to-site VPN

[tuanhoangth1603]

Hello,
I am experiencing an issue with S2S VPN after upgrading from version 4.19 to 4.20.

Context:

• My VPC has a single-tier network.
• Both RVRs are in a running state.
• All S2S VPN connections were working fine before upgrading CloudStack, but after the upgrade, I encountered errors (as shown in the logs below) whenever I restart the VPN connection or even when I delete the VPN, which results in a similar error.
  I have tried restarting the VPC with cleanup, deleting, and recreating the RVR, but none of these actions have resolved the issue.
  Please help me with this problem. Thanks !!

2025-03-06 21:02:19,264 DEBUG [c.c.a.t.Request] (AgentManager-Handler-5:[]) (logid:) Seq 10-7107524636920774788: Processing:  { Ans: , MgmtId: 345052580983, via: 10, Ver: v1, Flags: 100, [{"com.cloud.agent.api.routing.GroupAnswer":{"results":["null - success: Creating file in VR, with ip: 169.254.100.35, file: site_2_site_vpn.json.a8fa2457-69d5-4250-9d9e-498c0d22b73d","null - failed: java.io.IOException: Stream closed
        at java.base/java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:168)
        at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:334)
        at java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:287)
        at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:330)
        at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:190)
        at java.base/java.io.InputStreamReader.read(InputStreamReader.java:177)
        at java.base/java.io.BufferedReader.fill(BufferedReader.java:162)
        at java.base/java.io.BufferedReader.readLine(BufferedReader.java:329)
        at java.base/java.io.BufferedReader.readLine(BufferedReader.java:396)
        at com.cloud.utils.script.OutputInterpreter.processError(OutputInterpreter.java:41)
        at com.cloud.utils.script.Script.execute(Script.java:314)
        at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeInVR(LibvirtComputingResource.java:553)
        at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.applyConfigToVR(VirtualRoutingResource.java:303)
        at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.applyConfig(VirtualRoutingResource.java:318)
        at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.executeRequest(VirtualRoutingResource.java:165)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtNetworkElementCommandWrapper.execute(LibvirtNetworkElementCommandWrapper.java:35)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtNetworkElementCommandWrapper.execute(LibvirtNetworkElementCommandWrapper.java:29)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
        at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1945)
        at com.cloud.agent.Agent.processRequest(Agent.java:686)
        at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1109)
        at com.cloud.utils.nio.Task.call(Task.java:83)
        at com.cloud.utils.nio.Task.call(Task.java:29)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
"],"result":"false","wait":"0","bypassHostMaintenance":"false"}}] }
2025-03-06 21:02:19,264 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-5:[]) (logid:) Seq 1-7107524636920774788: No more commands found
2025-03-06 21:02:19,265 DEBUG [c.c.a.t.Request] (API-Job-Executor-22:[ctx-b5423d6a, job-29372, ctx-95a3513d]) (logid:9e09c55c) Seq 10-7107524636920774788: Received:  { Ans: , MgmtId: 345052580983, via: 10(host134.cs.cfox.local), Ver: v1, Flags: 100, { GroupAnswer } }
2025-03-06 21:02:19,298 WARN  [o.a.c.a.c.u.v.DeleteVpnConnectionCmd] (API-Job-Executor-22:[ctx-b5423d6a, job-29372, ctx-95a3513d]) (logid:9e09c55c) Exception: com.cloud.exception.ResourceUnavailableException: Resource [Site2SiteVpnConnection:20] is unreachable: Failed to apply site-to-site VPN
        at com.cloud.network.vpn.Site2SiteVpnManagerImpl.stopVpnConnection(Site2SiteVpnManagerImpl.java:613)
        at com.cloud.network.vpn.Site2SiteVpnManagerImpl.deleteVpnConnection(Site2SiteVpnManagerImpl.java:585)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:105)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at jdk.proxy3/jdk.proxy3.$Proxy198.deleteVpnConnection(Unknown Source)
        at org.apache.cloudstack.api.command.user.vpn.DeleteVpnConnectionCmd.execute(DeleteVpnConnectionCmd.java:79)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:173)
        at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:110)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:652)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:600)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)

2025-03-06 21:02:19,300 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-22:[ctx-b5423d6a, job-29372]) (logid:9e09c55c) Complete async job-29372, jobStatus: FAILED, resultCode: 534, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"534","errortext":"Resource [Site2SiteVpnConnection:20] is unreachable: Failed to apply site-to-site VPN"}

[tuanhoangth1603]

And there are some logs in VR:

2025-03-06 14:21:10,953 INFO     Adding route table: 103 Table_eth3 to /etc/iproute2/rt_tables if not present
2025-03-06 14:21:10,953 INFO     Executing: ip rule show
2025-03-06 14:21:10,955 INFO     Executing: ip rule show
2025-03-06 14:21:10,957 INFO     Executing: ip link show eth3 | grep ' state '
2025-03-06 14:21:10,961 INFO     Executing2: arping -c 1 -I eth3 -A -U -s 10.1.1.13 10.1.1.1
2025-03-06 14:21:10,961 INFO     Adding route: dev eth3 table: Table_eth3 network: 10.1.1.0/24 if not present
2025-03-06 14:21:10,961 INFO     Executing: ip route show  10.1.1.0/24 table Table_eth3 proto static
2025-03-06 14:21:10,964 INFO     Executing: sudo ip route flush cache
2025-03-06 14:21:10,971 ERROR    Not able to setup source-nat for a regular router yet
2025-03-06 14:21:10,971 INFO     Making dns publicly available
2025-03-06 14:21:10,972 INFO     Executing: systemctl start cloud-password-server@10.1.1.1,10.1.1.13
2025-03-06 14:21:10,980 INFO     Service cloud-password-server@10.1.1.1,10.1.1.13 start
2025-03-06 14:21:10,980 INFO     Flush all IPv6 ACL rules
2025-03-06 14:21:10,980 INFO     Executing: nft list tables ip6 | grep ip6_acl
2025-03-06 14:21:10,989 ERROR    Command 'nft list tables ip6 | grep ip6_acl' returned non-zero exit status 1.
2025-03-06 14:21:10,989 INFO     Executing: iptables-save | grep '^:FW_EGRESS_RULES' || iptables -t filter -N FW_EGRESS_RULES
2025-03-06 14:21:10,994 INFO     Executing: iptables-save | grep '^-A FW_EGRESS_RULES -j ACCEPT$' | sed 's/^-A/iptables -t filter -D/g' | bash
2025-03-06 14:21:11,001 INFO     Executing: iptables -F FW_EGRESS_RULES
2025-03-06 14:21:11,003 INFO     Executing: ipset -L | grep Name:  | awk {'print $2'} | ipset flush
2025-03-06 14:21:11,008 INFO     Executing: ipset -L | grep Name:  | awk {'print $2'} | ipset destroy
2025-03-06 14:21:11,013 INFO     Flush all IPv6 firewall rules
2025-03-06 14:21:11,013 INFO     Executing: nft list tables ip6 | grep ip6_firewall
2025-03-06 14:21:11,020 ERROR    Command 'nft list tables ip6 | grep ip6_firewall' returned non-zero exit status 1.

[weizhouapache]

@tuanhoangth1603
can you run the following commands inside the VR ?

cd /var/cache/cloud
cp processed/site_2_site_vpn.json.a8fa2457-69d5-4250-9d9e-498c0d22b73d.gz .
gzip -dk site_2_site_vpn.json.a8fa2457-69d5-4250-9d9e-498c0d22b73d.gz
/opt/cloud/bin/configure.py site_2_site_vpn.json.a8fa2457-69d5-4250-9d9e-498c0d22b73d

[tuanhoangth1603]

Hi @weizhouapache,
I ran and see the output:

Invalid unit name "cloud-password-server@10.1.1.1,10.1.1.13" escaped as "cloud-password-server@10.1.1.1\x2c10.1.1.13" (maybe you should use systemd-escape?).
Traceback (most recent call last):
  File "/opt/cloud/bin/configure.py", line 1680, in <module>
    main(sys.argv)
  File "/opt/cloud/bin/configure.py", line 1670, in main
    execIptables(config)
  File "/opt/cloud/bin/configure.py", line 1660, in execIptables
    iptables_executor.process()
  File "/opt/cloud/bin/configure.py", line 1566, in process
    vpns.process()
  File "/opt/cloud/bin/configure.py", line 1030, in process
    self.configure_ipsec(self.dbag[vpn])
  File "/opt/cloud/bin/configure.py", line 1103, in configure_ipsec
    if splitconnections and peerlistarr.count > 1:
                            ^^^^^^^^^^^^^^^^^^^^^
TypeError: '>' not supported between instances of 'builtin_function_or_method' and 'int'

[weizhouapache]

Hi @weizhouapache, I ran and see the output:

Invalid unit name "cloud-password-server@10.1.1.1,10.1.1.13" escaped as "cloud-password-server@10.1.1.1\x2c10.1.1.13" (maybe you should use systemd-escape?).
Traceback (most recent call last):
  File "/opt/cloud/bin/configure.py", line 1680, in <module>
    main(sys.argv)
  File "/opt/cloud/bin/configure.py", line 1670, in main
    execIptables(config)
  File "/opt/cloud/bin/configure.py", line 1660, in execIptables
    iptables_executor.process()
  File "/opt/cloud/bin/configure.py", line 1566, in process
    vpns.process()
  File "/opt/cloud/bin/configure.py", line 1030, in process
    self.configure_ipsec(self.dbag[vpn])
  File "/opt/cloud/bin/configure.py", line 1103, in configure_ipsec
    if splitconnections and peerlistarr.count > 1:
                            ^^^^^^^^^^^^^^^^^^^^^
TypeError: '>' not supported between instances of 'builtin_function_or_method' and 'int'

@tuanhoangth1603
this issue has been fixed by #10067

[tuanhoangth1603]

I upgraded to 4.20 a few days ago, but the issue still occurs.
What should I do?

[weizhouapache]

4.20.0.0 does not have the fix.

you can refer to #10184 (reply in thread)

[tuanhoangth1603]

Thanks @weizhouapache !

4.20.0.0 does not have the fix.

you can refer to #10184 (reply in thread)