Remove intra-VRF EBGP sessions with other PE-devices (ipspace#2712)

An intra-VRF EBGP session should not be established between
PE-device running EVPN-over-EBGP when they're in the same EVPN
instance.

This commit adds a 'remove VRF EBGP PE-device neighbors' function
to the EVPN cleanup hook. That function performs a number of tests
before deciding to remove an intra-VRF EBGP session:

* Both devices must run EBGP session in the same VRF. This preserves
  loopback EBGP sessions between VRFs
* Only EBGP sessions between EVPN-enabled devices are removed
* The two EVPN-enabled devices must be in the same evpn.domain -- this
  check enables multi-pod/multi-site designs
* The EBGP session must run over an EVPN-controlled SVI interface --
  this check preserves the in-VRF EBGP sessions in Inter-AS Option A
  designs

Fixes ipspace#2477

Author: ipspace

docs/module/evpn.md

@@ -118,6 +118,7 @@ EVPN module supports these default/global/node parameters:
 * **evpn.as** (global parameter): Autonomous system number for VLAN and VRF route targets. Default value: **bgp.as** (when set globally) or **vrf.as**.
 * **evpn.start_transit_vni** (system default parameter) -- the first symmetric IRB transit VNI, range 4096..16777215
 * **evpn.start_transit_vlan** (device-dependent node parameter) -- the starting VLAN ID for VLANs used to map VXLAN transit VNIs
+* **evpn.domain** (node parameter) -- EVPN domain (site) name. This parameter might have to be used in multi-site designs when you want to have an in-VRF inter-site EBGP session across an EVPN-controlled VLAN (see [](evpn-rp) for more details).
 
 (evpn-vlan-service)=
 ### VLAN-Based Service Parameters
@@ -217,7 +218,15 @@ If you're using EVPN to implement symmetrical IRB (**evpn.transit_vni** is set i
 
 If you want to implement an EVPN L3VPN with CE-routers, use separate segments (not VXLAN-backed VLANs) for PE-CE connectivity. There is no way to block the establishment of PE-to-PE in-VRF IGP adjacencies if you expect to have the PE-CE IGP adjacencies on a VXLAN-backed VLAN.
 
-Finally, if you use EBGP with EVPN (simple EBGP, IBGP-over-EBGP, or EBGP-over-EBGP design), the PE devices have different AS numbers, and _netlab_ tries to establish VRF EBGP sessions between them whenever they share a VLAN. To prevent the formation of VRF EBGP sessions between the PE devices:
+Finally, if you use EBGP with EVPN (simple EBGP, IBGP-over-EBGP, or EBGP-over-EBGP design), the PE devices have different AS numbers. _netlab_ thus tries to establish VRF EBGP sessions between them whenever they share a VLAN. However, the EVPN module removes such PE-to-PE intra-VRF EBGP sessions when:
+
+* Both devices use the EVPN module
+* The EBGP session is established across an EVPN-controlled VLAN
+* The two devices use the same VRF name
+
+This algorithm might remove an inter-site PE-to-PE EBGP session when you use Inter-AS Option A (EBGP sessions in VRFs) over EVPN-controlled VLANs. If that happens, use a different value of the **evpn.domain** parameter on nodes belonging to different sites (all nodes within a site must have the same **evpn.domain** value).
+
+It might be even better to prevent the formation of VRF EBGP sessions between the PE devices:
 
 * Use EVPN as a pure L3VPN (no stretched VLAN-over-VXLAN segments);
 * Disable BGP in the EVPN-controlled VRFs with a setting similar to **vrfs._tenant_vrf_.bgp: False**;

netsim/defaults/hints.yml

@@ -12,6 +12,9 @@ evpn:
     route targets. 'bgp.as' specified on individual nodes or groups will not work. You
     can also specify the global AS used by EVPN in 'vrf.as' parameter if you use
     VRFs, or in 'evpn.as' parameter if you use EVPN in bridging-only scenarios.
+  ebgp_vlan: >
+    In most scenarios, you don't need EBGP sessions between PE-devices across an
+    EVPN-controlled VLAN. See https://netlab.tools/module/evpn/#evpn-rp for details.
 
 report:
   source: >

netsim/modules/evpn.py

@@ -6,6 +6,7 @@
 from ..augment import devices
 from ..data.types import must_be_dict, must_be_int, must_be_string
 from ..utils import log
+from ..utils import routing as _rp_utils
 from . import _dataplane, _Module, _routing
 
 VALID_TRANSPORTS = [ 'vxlan', 'mpls' ] # In order of preference
@@ -447,6 +448,56 @@ def check_vlan_rt_values(node: Box, topology: Box) -> None:
         category=log.IncorrectValue,
         module='evpn')
 
+"""
+Remove unneeded in-VRF EBGP sessions between EVPN routers
+
+In designs using "EBGP as a better IGP", we might get in-VRF EBGP sessions between PE-routers
+over IRB VLANs. This function removes those EBGP sessions if:
+
+* Both BGP routers are also running EVPN
+* Both BGP neighbors are in the same VRF
+* The "evpn.domain" attribute (when set) is the same
+"""
+def remove_vlan_ebgp_neighbors(node: Box,topology: Box) -> None:
+  cleanup_needed = False
+  for ngb in _rp_utils.neighbors(node, vrf=True,select=['ebgp']):
+    vrf = ngb.get('_src_vrf','default')
+    if ngb.get('_vrf','default') != vrf:                    # Are both neighbors in the same VRF?
+      continue                                              # ... if not, this could be a valid inter-VRF session
+    ngb_data = topology.nodes[ngb.name]                     # Get neighbor data
+    if 'evpn' not in ngb_data.get('module',[]):             # Is the neighbor running EVPN?
+      continue
+    if node.get('evpn.domain','default') != ngb_data.get('evpn.domain','default'):
+      continue                                              # EVPN PE-devices in different EVPN domains, probably OK
+
+    iflist = [ intf for intf in node.interfaces if intf.ifindex == ngb.ifindex ]
+    if len(iflist) < 1:                                     # Neighbor has incorrect ifindex. Weird.
+      continue                                              # we don't know what's going on, so keep it intact
+
+    ngb_intf = iflist[0]
+    if ngb_intf.type != 'svi':                              # We have to remove EVPN EBGP neighbors only over SVI interfaces
+      continue
+
+    ngb_vlan = ngb_intf.get('vlan.name')                    # Get the VLAN associated with the SVI interface
+    if not ngb_vlan:                                        # Can't get it? Weird, but let's move on
+      continue
+
+    if ngb_vlan not in node.get('evpn.vlans',[]):           # Is this an EVPN-enabled VLAN
+      continue                                              # No? We may need the EBGP session
+
+    ngb._must_remove = True
+    log.warning(
+      text=f'Removing an EBGP session between {node.name} and {ngb.name} running over an EVPN-enabled VLAN {ngb_vlan}',
+      flag='ebgp_vlan',
+      hint='ebgp_vlan')
+
+    cleanup_needed = True
+
+  if cleanup_needed:                                        # Finally, remove neighbors with '_must_remove' flag
+    for (bgp,_,_) in _rp_utils.rp_data(node,proto='bgp'):   # ... iterating over global and VRF BGP instances
+      if 'neighbors' in bgp:
+        bgp.neighbors = [ ngb for ngb in bgp.neighbors if '_must_remove' not in ngb ]
+
 class EVPN(_Module):
 
   def module_init(self, topology: Box) -> None:
@@ -488,3 +539,6 @@ def node_post_transform(self, node: Box, topology: Box) -> None:
     check_vlan_rt_values(node,topology)
     trim_node_evpn_lists(node)
     set_local_evpn_rd(node)
+
+  def node_cleanup(self, node: Box, topology: Box) -> None:
+    remove_vlan_ebgp_neighbors(node,topology)

netsim/modules/evpn.yml

@@ -19,6 +19,7 @@ attributes:
     session:
     vlans:
     vrfs:
+    domain: str
   vlan:
     evi: rd
     rd: rd