Skip to content

Commit 8b242ad

Browse files
rambkrambk
committed
Create ExpressRoute VPN Backup Doc
1 parent 20703d4 commit 8b242ad

File tree

4 files changed

+272
-0
lines changed

4 files changed

+272
-0
lines changed

‎articles/expressroute/TOC.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,8 @@
7070
items:
7171
- name: Private Peering
7272
href: designing-for-disaster-recovery-with-expressroute-privatepeering.md
73+
- name: Maintaining VPN Backup
74+
href: active-validation-of-s2s-vpn-to-backup-expressroute-privatepeering.md
7375
- name: How-to guides
7476
items:
7577
- name: Create and modify a circuit

‎articles/expressroute/active-validation-of-s2s-vpn-to-backup-expressroute-privatepeering.md

Lines changed: 270 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,270 @@
1+
---
2+
title: 'Leveraging S2S VPN to backup Azure ExpressRoute Private Peering | Microsoft Docs'
3+
description: This page provides architectural recommendations for backing up Azure ExpressRoute private peering with S2S VPN.
4+
documentationcenter: na
5+
services: networking
6+
author: rambk
7+
manager: tracsman
8+
9+
ms.service: expressroute
10+
ms.topic: article
11+
ms.workload: infrastructure-services
12+
ms.date: 02/05/2020
13+
ms.author: rambala
14+
15+
---
16+
17+
# Active validation of S2S VPN to backup ExpressRoute private peering
18+
19+
In the article titled [Designing for disaster recovery with ExpressRoute private peering][DR-PP], we discussed the need for backup connectivity solution for an ExpressRoute private peering connectivity and how to use geo-redundant ExpressRoute circuits for the purpose. In this article, let us consider leveraging site-to-site (S2S) VPN to backup ExpressRoute private peering.
20+
21+
Unlike geo-redundant ExpressRoute circuits, you can use ExpressRoute-VPN combination only in active-passive mode. One of the major drawbacks of using any backup network connectivity in the passive mode is that the passive connection would often fail alongside the primary connection because of lack of active validation and maintance of the passive connection. Therefore, in this article let's focus on how to validate and actively maintain S2S VPN connectivity that is backing an ExpressRoute private peering.
22+
23+
>[!NOTE] When a given route is advertised via both ExpressRoute and VPN, Azure would prefer routing over ExpressRoute.
24+
>
25+
26+
In this article, let's see how to validate the connectivity both from the Azure perspective and from the perspective of the network equipments that peer with the Microsoft edge devices. This would enable you to do the validation irrespective of if you have a Layer 2 or Layer 3 network service provider.
27+
28+
## Example Topology
29+
30+
Let's consider the following topology for our discussion. In our setup, we have an on-premises network connected to an Azure hub Vnet and in turn to a spoke Vnet peered to the hub Vnet via both an ExpressRoute circuit and a S2S VPN connection.
31+
32+
[![1]][1]
33+
34+
In the setup, the ExpressRoute circuit is terminated on a pair of "Customer Edge" (CE) routers at the on-premises. The on-premises LAN is connected to the CE routers via a pair of firewalls that operate in master-slave mode. The S2S VPN is directly terminated on the firewalls.
35+
36+
## High availability and avoiding asymmetric traffic
37+
38+
### Configuring for high availability
39+
40+
[Configure ExpressRoute and Site-to-Site coexisting connections][Conf-CoExist] discusses how to configure the coexisting ExpressRoute circuit and S2S VPN connections. To improve high availability, as we discussed in [Designing for high availability with ExpressRoute][HA], our setup maintains the network redundancy (avoids single point of failure) all the way up to the endpoints. Also both the primary and secondary connections of the ExpressRoute circuits are configured to operate in active-active mode by advertsing the on-premises route prefixes the same way through both the connections.
41+
42+
The on-premises route advertisement of the primary CE router through the primary connection of the ExpressRoute circuit is show below (Junos commands):
43+
44+
rambala@SEA-MX03-01> show route advertising-protocol bgp 192.168.11.18
45+
46+
Cust11.inet.0: 8 destinations, 8 routes (7 active, 0 holddown, 1 hidden)
47+
Prefix Nexthop MED Lclpref AS path
48+
* 10.1.11.0/25 Self I
49+
50+
The on-premises route advertisement of the secondary CE router through the secondary connection of the ExpressRoute circuit is show below (Junos commands):
51+
52+
rambala@SEA-MX03-02> show route advertising-protocol bgp 192.168.11.22
53+
54+
Cust11.inet.0: 8 destinations, 8 routes (7 active, 0 holddown, 1 hidden)
55+
Prefix Nexthop MED Lclpref AS path
56+
* 10.1.11.0/25 Self I
57+
58+
To improve the high availability of the backup connection, the S2S VPN is also configured in the active-active mode. The Azure VPN gateway configuration is shown below. Note as part of the configuration VPN tunnels' BGP peer IP addresses of the gateway is also listed.
59+
60+
[![2]][2]
61+
62+
The on-premises route advertisement of the firewalls to the primary and secondary BGP peers of the VPN gateway is shown below (Junos):
63+
64+
rambala@SEA-SRX42-01> show route advertising-protocol bgp 10.17.11.76
65+
66+
Cust11.inet.0: 14 destinations, 21 routes (14 active, 0 holddown, 0 hidden)
67+
Prefix Nexthop MED Lclpref AS path
68+
* 10.1.11.0/25 Self I
69+
70+
{primary:node0}
71+
rambala@SEA-SRX42-01> show route advertising-protocol bgp 10.17.11.77
72+
73+
Cust11.inet.0: 14 destinations, 21 routes (14 active, 0 holddown, 0 hidden)
74+
Prefix Nexthop MED Lclpref AS path
75+
* 10.1.11.0/25 Self I
76+
77+
>[!NOTE] Configuring the S2S VPN in active-active mode not only provides high-availability to your disaster recovery backup network connectivity, but also provides higher throughput to the backup connectivity. In other words, configuring S2S VPN in active-active mode force create multiple underlying tunnels.
78+
>
79+
80+
### Configuring for symmetric traffic flow
81+
82+
We noted that when a given route is advertised via both ExpressRoute and VPN, Azure would prefer routing over ExpressRoute. To force Azure route over VPN instead of the coexisting ExpressRoute, you need to advertise more specic routes over the VPN connection. Our objective here is to use the VPN connections as back only. So, the default route selection behavior of Azure works for us.
83+
84+
It is our responsibility to ensure that the traffic destined to Azure also prefers ExpressRoute circuit over VPN. The default local preference of the CE routers and firewalls in our on-premises setup is 100. So, by configuring the local preference of the routes received through the ExpressRoute private peerings greater than 100 (say 150), we can make the traffic destined to Azure prefer ExpressRoute circuit.
85+
86+
The bgp configuration of the primary CE router that terminates the primary connection of the ExpressRoute circuit is shown below. Pay particular attention to local preference configured to 150. Similarly, we need to ensure the local preference of the secondary CE router that terminates the secondary connection of the ExpressRoute circuit is also configured to be 150.
87+
88+
rambala@SEA-MX03-01> show configuration routing-instances Cust11
89+
description "Customer 11 VRF";
90+
instance-type virtual-router;
91+
interface xe-0/0/0:0.110;
92+
interface ae0.11;
93+
protocols {
94+
bgp {
95+
group ibgp {
96+
type internal;
97+
local-preference 150;
98+
export nhs-vnet;
99+
neighbor 192.168.11.1;
100+
}
101+
group ebgp {
102+
peer-as 12076;
103+
bfd-liveness-detection {
104+
minimum-interval 300;
105+
multiplier 3;
106+
}
107+
neighbor 192.168.11.18;
108+
}
109+
}
110+
}
111+
112+
Let's ensure that per our configuration for Azure destined traffic ExpressRoute circuit is preferred over VPN by looking at the route table of the firewall.
113+
114+
rambala@SEA-SRX42-01> show route table Cust11.inet.0 10.17.11.0/24
115+
116+
Cust11.inet.0: 14 destinations, 21 routes (14 active, 0 holddown, 0 hidden)
117+
+ = Active Route, - = Last Active, * = Both
118+
119+
10.17.11.0/25 *[BGP/170] 2d 00:34:04, localpref 150
120+
AS path: 12076 I, validation-state: unverified
121+
> to 192.168.11.0 via reth1.11
122+
to 192.168.11.2 via reth2.11
123+
[BGP/170] 2d 00:34:01, localpref 150
124+
AS path: 12076 I, validation-state: unverified
125+
> to 192.168.11.2 via reth2.11
126+
[BGP/170] 2d 21:12:13, localpref 100, from 10.17.11.76
127+
AS path: 65515 I, validation-state: unverified
128+
> via st0.118
129+
[BGP/170] 2d 00:41:51, localpref 100, from 10.17.11.77
130+
AS path: 65515 I, validation-state: unverified
131+
> via st0.119
132+
10.17.11.76/32 *[Static/5] 2d 21:12:16
133+
> via st0.118
134+
10.17.11.77/32 *[Static/5] 2d 00:41:56
135+
> via st0.119
136+
10.17.11.128/26 *[BGP/170] 2d 00:34:04, localpref 150
137+
AS path: 12076 I, validation-state: unverified
138+
> to 192.168.11.0 via reth1.11
139+
to 192.168.11.2 via reth2.11
140+
[BGP/170] 2d 00:34:01, localpref 150
141+
AS path: 12076 I, validation-state: unverified
142+
> to 192.168.11.2 via reth2.11
143+
[BGP/170] 2d 21:12:13, localpref 100, from 10.17.11.76
144+
AS path: 65515 I, validation-state: unverified
145+
> via st0.118
146+
[BGP/170] 2d 00:41:51, localpref 100, from 10.17.11.77
147+
AS path: 65515 I, validation-state: unverified
148+
> via st0.119
149+
150+
In the above route table, for the hub and spoke Vnet routes--10.17.11.0/25 and 10.17.11.128/26--we see ExpressRoute circuit is prefered over VPN connections. The 192.168.11.0 and 192.168.11.2 are IPs on firewall interface towards CE routers.
151+
152+
## Validation and maintanence of backup VPN
153+
154+
Earlier in this article, we verified on-premises route advertisement of the firewalls to the primary and secondary BGP peers of the VPN gateway. Additionally, let's confirm Azure route received by the firewalls from the primary and secondary BGP peers of the VPN gateway.
155+
156+
rambala@SEA-SRX42-01> show route receive-protocol bgp 10.17.11.76 table Cust11.inet.0
157+
158+
Cust11.inet.0: 14 destinations, 21 routes (14 active, 0 holddown, 0 hidden)
159+
Prefix Nexthop MED Lclpref AS path
160+
10.17.11.0/25 10.17.11.76 65515 I
161+
10.17.11.128/26 10.17.11.76 65515 I
162+
163+
{primary:node0}
164+
rambala@SEA-SRX42-01> show route receive-protocol bgp 10.17.11.77 table Cust11.inet.0
165+
166+
Cust11.inet.0: 14 destinations, 21 routes (14 active, 0 holddown, 0 hidden)
167+
Prefix Nexthop MED Lclpref AS path
168+
10.17.11.0/25 10.17.11.77 65515 I
169+
10.17.11.128/26 10.17.11.77 65515 I
170+
171+
Similarly let's verify for on-premises route received by the Azure VPN gateway.
172+
173+
PS C:\Users\rambala> Get-AzVirtualNetworkGatewayLearnedRoute -ResourceGroupName SEA-Cust11 -VirtualNetworkGatewayName SEA-Cust11-VNet01-gw-vpn | where {$_.Network -eq "10.1.11.0/25"} | select Network, NextHop, AsPath, Weight
174+
175+
Network NextHop AsPath Weight
176+
------- ------- ------ ------
177+
10.1.11.0/25 192.168.11.88 65020 32768
178+
10.1.11.0/25 10.17.11.76 65020 32768
179+
10.1.11.0/25 10.17.11.69 12076-65020 32769
180+
10.1.11.0/25 10.17.11.69 12076-65020 32769
181+
10.1.11.0/25 192.168.11.88 65020 32768
182+
10.1.11.0/25 10.17.11.77 65020 32768
183+
10.1.11.0/25 10.17.11.69 12076-65020 32769
184+
10.1.11.0/25 10.17.11.69 12076-65020 32769
185+
186+
As seen above, the VPN gateway has routes received both by the primary and secondary BGP peers of the VPN gateway. It also has visibility over the routes received via primary and secondary ExpressRoute conections (the ones with AS-path prepended with 12076). To confirm the routes received via VPN connections, we need to know the on-premises BGP peer IP of the connections. In our setup under consideration it is 192.168.11.88 and we do see the routes received from it.
187+
188+
Next, let's verify the routes advertised to on-premises by the Azure VPN gateway.
189+
190+
PS C:\Users\rambala> Get-AzVirtualNetworkGatewayAdvertisedRoute -Peer 192.168.11.88 -ResourceGroupName SEA-Cust11 -VirtualNetworkGatewayName SEA-Cust11-VNet01-gw-vpn | select Network, NextHop, AsPath, Weight
191+
192+
Network NextHop AsPath Weight
193+
------- ------- ------ ------
194+
10.17.11.0/25 10.17.11.76 65515 0
195+
10.17.11.128/26 10.17.11.76 65515 0
196+
10.17.11.0/25 10.17.11.77 65515 0
197+
10.17.11.128/26 10.17.11.77 65515 0
198+
199+
200+
Failure to see route exchanges indicate connection failure. See [Troubleshooting: An Azure site-to-site VPN connection cannot connect and stops working][VPN Troubleshoot] for help with troubleshooting the VPN connection.
201+
202+
Now that we have confirmed successful route exchanges over the VPN connection, we are set to do a failover from the ExpressRoute connectivity and test the dataplane of the VPN connectivity.
203+
204+
>[!NOTE] In production environments failover testing has to be done during well notified network maintenance work-windows as they can be service disruptive.
205+
>
206+
207+
Prior to do the failover, let's trace route the current path in our setup from an on-premises server and a VM in the spoke Vnet.
208+
209+
C:\Users\PathLabUser>tracert 10.17.11.132
210+
211+
Tracing route to 10.17.11.132 over a maximum of 30 hops
212+
213+
1 <1 ms <1 ms <1 ms 10.1.11.1
214+
2 <1 ms <1 ms 11 ms 192.168.11.0
215+
3 <1 ms <1 ms <1 ms 192.168.11.18
216+
4 * * * Request timed out.
217+
5 6 ms 6 ms 5 ms 10.17.11.132
218+
219+
Trace complete.
220+
221+
The primary and secondary ExpressRoute connection subnets of our setup are, respectively, 192.168.11.16/30 and 192.168.11.20/30. In the above trace route, in step 3 we see that we are hitting 192.168.11.18, which is the interface IP of the primary MSEE. This confirm that as expected our current path is over the ExpressRoute.
222+
223+
Let's use the following set of commands to disable both the primary and secondary ExpressRoute connections.
224+
225+
$ckt = Get-AzExpressRouteCircuit -Name "expressroute name" -ResourceGroupName "SEA-Cust11"
226+
$ckt.Peerings[0].State = "Disabled"
227+
Set-AzExpressRouteCircuit -ExpressRouteCircuit $ckt
228+
229+
The failover time depends on the BGP convergence time. In our setup, the failover takes a few seconds (less than 10). Following, the failover, repeat of the traceroute shows the following:
230+
231+
C:\Users\PathLabUser>tracert 10.17.11.132
232+
233+
Tracing route to 10.17.11.132 over a maximum of 30 hops
234+
235+
1 <1 ms <1 ms <1 ms 10.1.11.1
236+
2 * * * Request timed out.
237+
3 6 ms 7 ms 9 ms 10.17.11.132
238+
239+
Trace complete.
240+
241+
This confirms that the backup connection is active and can provide service continuity in the event of both the primary and secondary ExpressRoute connections failure. Let's enable the ExpressRoute connections and switch the traffic over using the following set of commands.
242+
243+
$ckt = Get-AzExpressRouteCircuit -Name "expressroute name" -ResourceGroupName "SEA-Cust11"
244+
$ckt.Peerings[0].State = "Enabled"
245+
Set-AzExpressRouteCircuit -ExpressRouteCircuit $ckt
246+
247+
To confirm traffic it switched over, repeat the traceroute and ensure that it is going through one of the ExpressRoute connections.
248+
249+
## Next steps
250+
251+
The secret of maintaining the passive backup connection is not to let the VPN configuration stale and periodically (say every quarter) repeat the validation steps described in this article during maintenance window.
252+
253+
To enable monitoring and alerts based on VPN gateway metrics, see [Set up alerts on VPN Gateway metrics][VPN-alerts].
254+
255+
To expedite BGP convergence following an ExpressRoute failure, [Configure BFD over ExpressRoute][BFD].
256+
257+
<!--Image References-->
258+
[1]: ./media/active-validation-of-s2s-vpn-to-backup-expressroute-private-peering/topology.png "topology under consideration"
259+
[2]: ./media/active-validation-of-s2s-vpn-to-backup-expressroute-private-peering/vpn-gw-config.png "VPN GW configuration"
260+
261+
<!--Link References-->
262+
[DR-PP]: https://docs.microsoft.com/azure/expressroute/designing-for-disaster-recovery-with-expressroute-privatepeering
263+
[Conf-CoExist]: https://docs.microsoft.com/azure/expressroute/expressroute-howto-coexist-resource-manager
264+
[HA]: https://docs.microsoft.com/azure/expressroute/designing-for-high-availability-with-expressroute
265+
[VPN Troubleshoot]: https://docs.microsoft.com/azure/vpn-gateway/vpn-gateway-troubleshoot-site-to-site-cannot-connect
266+
[VPN-alerts]: https://docs.microsoft.com/azure/vpn-gateway/vpn-gateway-howto-setup-alerts-virtual-network-gateway-metric
267+
[BFD]: https://docs.microsoft.com/azure/expressroute/expressroute-bfd
268+
269+
270+

‎articles/expressroute/media/active-validation-of-s2s-vpn-to-backup-expressroute-private-peering/topology.png

21.7 KB
Loading

‎articles/expressroute/media/active-validation-of-s2s-vpn-to-backup-expressroute-private-peering/vpn-gw-config.PNG

42.8 KB
Loading

0 commit comments

Comments
 (0)