ESI Multi-Homing and ARP Suppression

Issue: Duplicate Packets & Flooding

When ESI is not functioning correctly, you may observe duplicate packets destined for hosts behind the ESI. Remote switches might flood traffic toward the ESI switches instead of handling it efficiently.

TCAM Carving for ARP Suppression

To use the suppress-arp feature effectively and prevent unnecessary flooding, you must allocate sufficient TCAM resources on the switch hardware.

1
2
3

# Configure TCAM carving for ARP suppression
hardware access-list tcam region racl 512 
hardware access-list tcam region arp-ether 256 double-wide

Warning: It was observed that enabling suppress-arp caused reachability and ARP resolution to break between hosts when attempting to use RFC 5549 (BGP Unnumbered / IPv6 underlay) between Spines and Leafs for IPv4 and EVPN. Note that an IPv6 underlay is generally not supported for Cisco Nexus 9300/9500v platforms in this context.

Fabric & VRF Configuration

Global Fabric Settings

fabric forwarding anycast-gateway-mac 0000.2222.1111
fabric forwarding admin-distance 19

VLAN & VN-Segment Mapping

vlan 111
  vn-segment 20111
vlan 2111
  name IRB_VLAN111
  vn-segment 30111

VRF Definition (Tenant BLUE)

vrf context BLUE
  vni 30001
  rd auto
  address-family ipv4 unicast
    route-target import 4244000000:30001
    route-target import 4244000000:30001 evpn
    route-target import 4244000000:30003
    route-target import 4244000000:30003 evpn
    route-target import 4244000000:30007 evpn
    route-target import 4244000000:30008 evpn
    route-target export 4244000000:30001
    route-target export 4244000000:30001 evpn

Interface Configuration

L3 SVIs (Anycast Gateway & Core Routing)

interface Vlan111
  no shutdown
  vrf member BLUE
  ip address 10.1.111.1/24
  fabric forwarding mode anycast-gateway

interface Vlan2111
  no shutdown
  vrf member BLUE
  ip forward

Loopback Interfaces

interface loopback0
  description #router-id
  ip address 10.44.1.1/32

interface loopback1
  description #vtep-source
  ip address 10.45.1.1/32

Downlink / Host Interfaces (ESI)

interface port-channel9
  switchport mode trunk
  switchport trunk allowed vlan 111-112,211-212,311-312
  ethernet-segment 9
    system-mac 0001.0102.0009
  spanning-tree port type edge trunk

interface Ethernet1/8
  switchport mode trunk
  switchport trunk allowed vlan 111-112,211-212,311-312
  spanning-tree port type edge trunk

interface Ethernet1/1
  description #SPINES
  no switchport
  evpn multihoming core-tracking
  ip forward
  ipv6 link-local use-bia
  ipv6 nd ra-interval 4 min 3
  no shutdown

interface Ethernet1/5
  no switchport
  evpn multihoming core-tracking
  ip forward
  ipv6 link-local use-bia
  ipv6 nd ra-interval 4 min 3
  no shutdown

interface Ethernet1/6
  no switchport
  evpn multihoming core-tracking
  ip forward
  ipv6 link-local use-bia
  ipv6 nd ra-interval 4 min 3
  no shutdown

NVE Interface (VTEP)

interface nve1
  no shutdown
  host-reachability protocol bgp
  source-interface loopback1
  member vni 20111
    ingress-replication protocol bgp
  member vni 30001 associate-vrf

BGP Routing Configuration

Limitation with RFC 5549 (BGP Unnumbered)

Note: It was determined that RFC 5549 cannot be used for advertising eBGP source IPv4 interfaces when combining BGP Unnumbered for IPv4 with eBGP peering using loopback interfaces.

BGP Setup using Peer Templates

This configuration utilizes peer templates to separate the IPv4 underlay peering from the L2VPN EVPN overlay peering.

router bgp 4244001001
  router-id 10.44.1.1
  log-neighbor-changes
  
  address-family ipv4 unicast
    network 10.44.1.1/32
    network 10.45.1.1/32
    maximum-paths 64
  
  address-family l2vpn evpn
    maximum-paths 64
    additional-paths receive
  
  # Underlay Template (IPv4)
  template peer SPINES_IPV4
    remote-as 4244001200
    address-family ipv4 unicast
      send-community
      send-community extended
      advertisement-interval 30
  
  # Overlay Template (EVPN)
  template peer SPINES_L2EVPN
    remote-as 4244001200
    update-source loopback0
    ebgp-multihop 2
    address-family l2vpn evpn
      send-community
      send-community extended
  
  # Apply Templates to Neighbors
  neighbor 10.44.1.201
    inherit peer SPINES_L2EVPN
  
  neighbor Ethernet1/1
    inherit peer SPINES_IPV4

  # EVPN Instance Configuration
  evpn
    vni 20111 l2
      rd auto
      route-target import auto
      route-target export auto

Troubleshooting Logs

Reachability Issue on EVPN Peering

The following BGP log excerpt highlights a failure to establish the EVPN overlay session with peer 10.44.1.201 due to underlying network unreachability. Because the loopback cannot be reached via the underlay, the TCP connection for the eBGP multihop session fails.

1
2
3
4
5
6
7
8

2024 May 8 07:46:18.899138 bgp: [491] (default) EVT: 10.44.1.201 remote iod 0 skip resolving l3 addr
2024 May 8 07:46:18.899217 bgp: [491] (default) EVT: 10.44.1.201 peer connection retry timer expired
2024 May 8 07:46:18.899747 bgp: [491] (default) EVT: 10.44.1.201 Triggered active open for peer
2024 May 8 07:46:18.901235 bgp: [491] (default) EVT: 10.44.1.201 went from Idle to Active (Active setup)
2024 May 8 07:46:18.901320 bgp: [491] (default) ADJ: bgp_tcp_connect: Peer 10.44.1.201 remote i/f (null)
2024 May 8 07:46:18.901640 bgp: [491] (default) ADJ: Local addr for peer 10.44.1.201 is 10.44.1.1
2024 May 8 07:46:18.901776 bgp: [491] (default) ADJ: set_local_port: Peer 10.44.1.201, remote iod (null)
2024 May 8 07:46:18.901955 bgp: [491] (default) ADJ: Cannot connect socket to peer 10.44.1.201: Network is unreachable

The content of this post was created by the author; AI was used for editing and restructuring.