|
1 | 1 | --- |
2 | | -title: 'Azure ExpressRoute: Optimize routing' |
| 2 | +title: 'Optimize routing for Azure ExpressRoute' |
3 | 3 | description: This page provides details on how to optimize routing when you have more than one ExpressRoute circuits that connect between Microsoft and your corp network. |
4 | 4 | services: expressroute |
5 | 5 | author: duongau |
6 | 6 | ms.service: azure-expressroute |
7 | 7 | ms.topic: how-to |
8 | | -ms.date: 06/15/2023 |
| 8 | +ms.date: 01/31/2025 |
9 | 9 | ms.author: duau |
10 | 10 | --- |
11 | 11 |
|
12 | | -# Optimize ExpressRoute Routing |
13 | | - |
14 | | -When you have multiple ExpressRoute circuits, you have more than one path to connect to Microsoft. As a result, suboptimal routing may happen - that is, your traffic may take a longer path to reach Microsoft, and Microsoft to your network. The longer the network path, the higher the latency. Latency has direct effect on application performance and user experience. This article illustrates this problem and explain how to optimize routing using the standard routing technologies. |
| 12 | +# Optimize routing for Azure ExpressRoute |
| 13 | +When you have multiple ExpressRoute circuits, you have more than one path to connect to Microsoft. This can lead to suboptimal routing, where traffic takes a longer path, increasing latency and affecting application performance and user experience. This article explains how to optimize routing using standard routing technologies. |
15 | 14 |
|
16 | 15 | ## Path selection for Microsoft peering |
17 | 16 |
|
18 | | -It's important to ensure that when utilizing Microsoft that traffic flows over the desired path if you have one or more ExpressRoute circuits. You need to also ensure paths to the Internet use an Internet Exchange (IX) or Internet Service Provider (ISP). BGP utilizes a best path selection algorithm based on many factors including longest prefix match (LPM). To ensure that traffic destined for Azure through Microsoft traverses the ExpressRoute path, you must implement the *Local Preference* attribute. This setting ensures that the path is always preferred on ExpressRoute. |
| 17 | +To ensure traffic flows over the desired path with multiple ExpressRoute circuits, you need to manage paths to the Internet using an Internet Exchange (IX) or Internet Service Provider (ISP). BGP uses a best path selection algorithm based on factors like the longest prefix match (LPM). To ensure traffic destined for Azure through Microsoft uses the ExpressRoute path, implement the *Local Preference* attribute. This setting ensures the path is always preferred on ExpressRoute. |
19 | 18 |
|
20 | 19 | > [!NOTE] |
21 | | -> The default local preference is typically **100**. Higher local preferences are more preferred. |
22 | | -> |
| 20 | +> The default local preference is typically **100**. Higher local preferences are more preferred. |
23 | 21 |
|
24 | 22 | Consider the following example scenario: |
25 | 23 |
|
26 | | - |
| 24 | + |
27 | 25 |
|
28 | | -In the above example, to prefer ExpressRoute paths configure Local Preference as follows. |
| 26 | +To prefer ExpressRoute paths, configure Local Preference as follows: |
29 | 27 |
|
30 | 28 | **Cisco IOS-XE configuration from R1 perspective:** |
31 | 29 |
|
32 | | -- R1(config)#route-map prefer-ExR permit 10 |
33 | | -- R1(config-route-map)#set local-preference 150 |
| 30 | +``` |
| 31 | +R1(config)#route-map prefer-ExR permit 10 |
| 32 | +R1(config-route-map)#set local-preference 150 |
34 | 33 |
|
35 | | -- R1(config)#router BGP 345 |
36 | | -- R1(config-router)#neighbor 1.1.1.2 remote-as 12076 |
37 | | -- R1(config-router)#neighbor 1.1.1.2 activate |
38 | | -- R1(config-router)#neighbor 1.1.1.2 route-map prefer-ExR in |
| 34 | +R1(config)#router BGP 345 |
| 35 | +R1(config-router)#neighbor 1.1.1.2 remote-as 12076 |
| 36 | +R1(config-router)#neighbor 1.1.1.2 activate |
| 37 | +R1(config-router)#neighbor 1.1.1.2 route-map prefer-ExR in |
| 38 | +``` |
39 | 39 |
|
40 | 40 | **Junos configuration from R1 perspective:** |
41 | 41 |
|
42 | | -- user@R1# set protocols bgp group ibgp type internal |
43 | | -- user@R1# set protocols bgp group ibgp local-preference 150 |
44 | | - |
45 | | - |
| 42 | +``` |
| 43 | +user@R1# set protocols bgp group ibgp type internal |
| 44 | +user@R1# set protocols bgp group ibgp local-preference 150 |
| 45 | +``` |
46 | 46 |
|
47 | 47 | ## Suboptimal routing from customer to Microsoft |
48 | 48 |
|
49 | | -Let's take a close look at the routing problem by an example. Imagine you have two offices in the US, one in Los Angeles and one in New York. Your offices are connected on a Wide Area Network (WAN), which can be either your own backbone network or your service provider's IP VPN. You have two ExpressRoute circuits, one in US West and one in US East. Both are also connected on the WAN. Obviously, you have two paths to connect to the Microsoft network. |
50 | | - |
51 | | -Now imagine you have an Azure deployment, for example, an Azure App Service in both US West and US East. Your intention is to connect your users in Los Angeles to Azure US West and your users in New York to Azure US East. The reason for this set up is because your service admin advertises that users in each office access the nearby Azure services for optimal experiences. The plan works out well for the east coast users but not for the west coast users. |
52 | | - |
53 | | -The cause of the problem is on each ExpressRoute circuit, we advertise to on-premises both the prefix in Azure US East 23.100.0.0/16 and the prefix in Azure US West 13.100.0.0/16. If you don't know which prefix is from which region, you aren't able to treat it differently. Your WAN network may think both of the prefixes are closer to US East than US West and therefore route both office users to the ExpressRoute circuit in US East. In the end, you have many unhappy users in the Los Angeles office. |
| 49 | +Imagine you have two offices in the US, one in Los Angeles and one in New York, connected via a WAN. You have two ExpressRoute circuits, one in US West and one in US East, both connected on the WAN. You intend for Los Angeles users to connect to Azure US West and New York users to Azure US East. However, without knowing which prefix is from which region, your WAN may route both offices' traffic to the US East circuit, causing suboptimal routing for Los Angeles users. |
54 | 50 |
|
55 | | - |
| 51 | + |
56 | 52 |
|
57 | 53 | ### Solution: use BGP Communities |
58 | 54 |
|
59 | | -To optimize routing for both office users, you need to know which prefix is from Azure US West and which from Azure US East. We encode this information by using [BGP Community values](expressroute-routing.md). We've assigned a unique BGP Community value to each Azure region, for example `12076:51004` for US East, `12076:51006` for US West. Now that you know which prefix is from which Azure region, you can configure which ExpressRoute circuit should be preferred. Since we use the BGP to exchange routing info, you can use BGP's Local Preference to influence routing. |
| 55 | +To optimize routing, use [BGP Community values](expressroute-routing.md) to identify prefixes from each Azure region. For example, `12076:51004` for US East and `12076:51006` for US West. Configure higher local preference values for the appropriate prefixes in each region to ensure traffic takes the optimal path. |
60 | 56 |
|
61 | | -In our example, you can assign a higher local preference value to 13.100.0.0/16 in US West than in US East, and similarly, a higher local preference value to 23.100.0.0/16 in US East than in US West. This configuration makes sure that, when both paths to Microsoft are available, your users in Los Angeles takes the ExpressRoute circuit in US West to connect to Azure US West whereas your users in New York take the ExpressRoute in US East to Azure US East. Routing is optimized on both sides. |
62 | | - |
63 | | - |
| 57 | + |
64 | 58 |
|
65 | 59 | > [!NOTE] |
66 | | -> The same technique, using Local Preference, can be applied to routing from customer to Azure virtual network when using private peering. Microsoft doesn't tag BGP community values to the prefixes advertised from Azure to your network. However, since you know which of your virtual network deployment is close to which of your office, you can configure your routers accordingly to prefer one ExpressRoute circuit over another. |
67 | | -> |
| 60 | +> The same technique can be applied to routing from customer to Azure virtual network using private peering. Configure your routers to prefer one ExpressRoute circuit over another based on your virtual network deployments. |
68 | 61 |
|
69 | 62 | ## Suboptimal routing from Microsoft to customer |
70 | 63 |
|
71 | | -In this example, we have connections from Microsoft take a longer path to reach your network. In this case, you use on-premises Exchange servers and Exchange Online in a [hybrid environment](/exchange/exchange-hybrid). Your offices are connected to a WAN. You advertise the prefixes of your on-premises servers in both of your offices to Microsoft through two ExpressRoute circuits. |
72 | | - |
73 | | -Exchange Online initiates connections to the on-premises servers in cases such as mailbox migration. The connection to your Los Angeles office is routed to the ExpressRoute circuit in US East before traversing the entire continent back to the west coast. The cause of the problem is similar to the first one. Without any hint, the Microsoft network can't tell which on-premises prefix is close to US East and which one is close to US West. It happens to pick the wrong path to your office in Los Angeles. |
| 64 | +In this scenario, Microsoft connections take a longer path to reach your network. For example, Exchange Online connections to on-premises servers may route through the wrong ExpressRoute circuit. Without hints, Microsoft can't determine which on-premises prefix is closer to which circuit. |
74 | 65 |
|
75 | | - |
| 66 | + |
76 | 67 |
|
77 | 68 | ### Solution: use AS PATH prepending |
78 | 69 |
|
79 | | -There are two solutions to the problem. The first is you simply advertise your on-premises prefix for your Los Angeles office 177.2.0.0/31 on the ExpressRoute circuit in US West. Then you advertise your on-premises prefix for your New York office, 177.2.0.2/31 on the ExpressRoute circuit in US East. As a result, there's only one path for Microsoft to connect to each of your offices. There's no ambiguity and routing is optimized. With this design, you need to think about your failover strategy. If the path to Microsoft through ExpressRoute goes down, you need to make sure that Exchange Online can still connect to your on-premises servers. |
| 70 | +Advertise your on-premises prefixes on the appropriate ExpressRoute circuits or use AS PATH prepending to influence routing. Lengthen the AS PATH for prefixes in the less preferred region to ensure Microsoft prefers the optimal path. |
80 | 71 |
|
81 | | -The second solution is that you continue to advertise both of the prefixes on both ExpressRoute circuits, and in addition you give us a hint of which prefix is close to which one of your offices. Because we support BGP AS Path prepending, you can configure the AS Path for your prefix to influence routing. In this example, you can lengthen the AS PATH for 172.2.0.0/31 in US East so that we prefer the ExpressRoute circuit in US West for traffic destined for this prefix. Similarly you can lengthen the AS PATH for 172.2.0.2/31 in US West so that we prefer the ExpressRoute circuit in US East. Routing is optimized for both offices. With this design, if one ExpressRoute circuit is broken, Exchange Online can still reach you via another ExpressRoute circuit and your WAN. |
| 72 | + |
82 | 73 |
|
83 | 74 | > [!IMPORTANT] |
84 | | -> We remove private AS numbers in the AS PATH for the prefixes received on Microsoft Peering when peering using a private AS number. You need to peer with a public AS and append public AS numbers in the AS PATH to influence routing for Microsoft Peering. |
85 | | -> |
86 | | -> |
87 | | -
|
88 | | - |
89 | | - |
90 | | -> [!NOTE] |
91 | | -> While the examples given here are for Microsoft peering, we do support the same capabilities for the Private peering. Also, the AS Path prepending works within one single ExpressRoute circuit, to influence the selection of the primary and secondary paths. |
92 | | -> |
93 | | -> |
| 75 | +> We remove private AS numbers in the AS PATH for prefixes received on Microsoft Peering when using a private AS number. Peer with a public AS and append public AS numbers in the AS PATH to influence routing for Microsoft Peering. |
94 | 76 |
|
95 | 77 | ## Suboptimal routing between virtual networks |
96 | 78 |
|
97 | | -With ExpressRoute, you can enable Virtual Network to Virtual Network (which is also known as "VNet") communication by linking them to an ExpressRoute circuit. When you link them to multiple ExpressRoute circuits, suboptimal routing can happen between the VNets. Let's consider an example. You have two ExpressRoute circuits, one in US West and one in US East. In each region, you have two VNets. Your web servers are deployed in one VNet and application servers in the other. For redundancy, you link the two VNets in each region to both the local ExpressRoute circuit and the remote ExpressRoute circuit. As can be seen in the following diagram, from each VNet there are two paths to the other VNet. The VNets don't know which ExpressRoute circuit is local and which one is remote. Since Equal-Cost-Multi-Path (ECMP) routing is used to load-balance inter-VNet traffic, some traffic flows take the longer path and get routed at the remote ExpressRoute circuit. |
| 79 | +With ExpressRoute, you can enable VNet-to-VNet communication by linking them to an ExpressRoute circuit. Suboptimal routing can occur when VNets are linked to multiple circuits. For example, VNets in US West and US East may route traffic through the remote circuit due to ECMP routing. |
98 | 80 |
|
99 | | - |
| 81 | + |
100 | 82 |
|
101 | 83 | ### Solution: assign a high weight to local connection |
102 | 84 |
|
103 | | -The solution is simple. Since you know where the VNets and the circuits are, you can tell us which path each VNet should prefer. Specifically for this example, you assign a higher weight to the local connection than to the remote connection (see the configuration example [here](expressroute-howto-linkvnet-arm.md#modify-a-virtual-network-connection)). When a VNet receives the prefix of the other VNet on multiple connections, it prefers the connection with the highest weight to send traffic destined for that prefix. |
| 85 | +Assign a higher weight to the local connection to ensure VNets prefer the optimal path. This configuration ensures traffic takes the shortest path between VNets. |
104 | 86 |
|
105 | | - |
| 87 | + |
106 | 88 |
|
107 | 89 | > [!NOTE] |
108 | | -> You can also influence routing from VNet to your on-premises network, if you have multiple ExpressRoute circuits, by configuring the weight of a connection instead of applying AS PATH prepending, a technique described in the second scenario. For each prefix, we will always look at the connection weight before the AS Path length when deciding how to send traffic. |
109 | | -> |
| 90 | +> You can also influence routing from VNet to your on-premises network by configuring the connection weight instead of using AS PATH prepending. The connection weight is considered before the AS Path length when deciding how to send traffic. |
110 | 91 |
|
111 | 92 | ## Next steps |
112 | 93 |
|
|
0 commit comments