Anycast Inside an EVPN/VXLAN Fabric with BGP Unnumbered (RFC 5549): A Hands-on Lab

Companion posts. This is the hands-on lab — real NX-OSv configs, the tests I ran, and where they broke. For the high-level summary see Anycast Inside the Data Center; for the deeper protocol write-up (overlay index, recursive resolution, why pure-IP fabrics lose per-host ECMP) see Anycast in EVPN/VXLAN Fabrics: The IPv6 Unnumbered and RFC 5549 Problem.

Why

Anycast is used for horizontal scaling and for better failure response. With modern orchestration and virtualization, a new instance of a service can be spawned on demand — by an engineer or by the system itself.

Anycast is a good option for network-level services such as DNS or load balancers, but whether it works end-to-end depends on the features available in the protocols you run in the fabric.

The Idea

In a modern data center, a service or VM can be spawned on any host. That dynamic placement requires dynamic routing all the way down to the server: we need a routing session between the fabric and the servers. BGP is the most practical choice here, with the most flexible deployment options for anycast.

A natural design is to run BGP between hosts and leaves over RFC 5549 — Advertising IPv4 NLRI with an IPv6 Next Hop (now obsoleted by RFC 8950). RFC 5549 is a point-to-point technique: it uses ICMPv6 RA/NA messages to discover the remote router, so neighbor discovery is based on IPv6 link-local addresses, and you may not even need IPv4 enabled on those links.

sequenceDiagram
    participant L as Leaf (IPv6 LL)
    participant H as Host (IPv6 LL)
    L->>H: ICMPv6 Router Solicitation / Advertisement
    H->>L: ICMPv6 Neighbor Advertisement
    Note over L,H: BGP session forms over IPv6 link-local
    H->>L: Advertise IPv4 prefix with IPv6 next hop (RFC 5549)

Because discovery is point-to-point over link-local, you can only build point-to-point relationships. We can use RFC 5549 both between the hosts and the leaves and for building the underlay. The type of underlay affects route distribution / BGP address families and introduces some knobs — which is where the trouble starts.

Let’s analyse RFC 5549 between host and leaf for the fabric below.

flowchart TB
    subgraph Spine
      S1["F1-S1<br/>AS 65201"]
    end
    subgraph Leaves
      L1["Leaf-1<br/>AS 65001<br/>VRF BLUE_TENANT"]
      L2["Leaf-2<br/>AS 65002<br/>VRF BLUE_TENANT"]
      L3["Leaf-3<br/>AS 65003<br/>VRF BLUE_TENANT"]
    end
    S1 --- L1
    S1 --- L2
    S1 --- L3
    L1 --- H1["Hosts AS 20 / AS 10<br/>advertise VIP prefix A"]
    L2 --- H2["Hosts AS 10<br/>advertise VIP prefix A"]
    L3 --- H3["Server<br/>advertise prefix A"]

The Type-5 Encapsulation Problem

In an EVPN/VXLAN fabric, the routes learned from the hosts over eBGP are carried as EVPN route-type 5 (IP Prefix routes). The question is how those routes get encapsulated into RT-5.

Normally the gateway IP field of the RT-5 route is all zeros. Even if we use IPv6 to advertise the IPv4 next hop (RFC 5549), this does not change anything from the EVPN perspective — the gateway IP is still all zeros, same mechanism.

The problem arises when we advertise a VIP (Virtual IP) from the hosts — a load balancer, DNS, or any virtual network service. VIPs require ECMP in the fabric. For an EVPN/VXLAN underlay using distributed routing and symmetric IRB, you generally need a VRF for the edge networks, so let’s focus on the ECMP problem inside that VRF’s routing table.

Take Leaf-3: it has a local server advertising prefix A, while remote servers behind Leaf-1 and Leaf-2 advertise the same prefix. Leaf-1 and Leaf-2 each have multiple servers advertising it, and they must advertise all of those paths into EVPN. The leaves with multiple servers must install every path into their routing table. That requires:

Multiple paths (maximum-paths) so each leaf installs all the local paths.
AS-path handling — bestpath as-path ignore, or relaxing it, depending on how the hosts use AS numbers.
Advertise all paths, not just best, via RT-5. If every path has the same AS and the same VRF, the routes look identical and only one is advertised. The only field left to differentiate them is the gateway IP, which is all zeros by default. Setting it to the remote BGP peer address of each host gives us distinct BGP paths for the same prefix even when AS and VRF are identical.
Per-vendor tuning — local-preference for selecting local routes, etc.

Otherwise Leaf-3 ends up with only 2 remote routes instead of the 5 it should have (3 from Leaf-1, 2 from Leaf-2).

Leaf-3 must install all remote routes and the local routes. You may need to ignore AS-path, because local paths have a shorter AS-path. On my tests with Cisco Nexus NX-OSv 9.3(11), the local leaf (e.g. Leaf-3) chose the remote prefixes as best path and advertised them back toward the spine, where they were dropped as an AS loop. NX-OSv seems to use the best path’s attributes — including AS-path — when constructing the multipaths with all-same-AS. From the remote leaves, all the local routes then look the same. I worked around it by setting the local-preference of the local routes so the local paths win best-path.

Leaf-1 also has a host advertising the VIP with a different AS (AS 20), and chose that local path as best. On the remote leaves, all Leaf-1 routes appear to come from AS 20.

Setting the Gateway IP — and Where RFC 5549 Breaks

The gateway IP is set with export gateway-ip on Cisco NX-OS. But because we are using IPv6 for the eBGP session with the hosts, the next hop is an IPv6 address. How does the switch encode that IPv6 address into the IPv4 gateway field of an RT-5 route?

It can’t, in any useful way. There is no solution for this. This means you cannot use RFC 5549 / RFC 8950 to advertise an IPv4 VIP through the data center: all the local routes on a single leaf end up with the same gateway IP, so only one of them is advertised into EVPN — and the receiving leaf may then drop the route because the gateway is invalid.

For example, configuring several host sessions on a leaf gives 6 local paths, 2 of them via IPv6 next hops. Looking at the route detail, the gateway IP for those is clearly wrong:

Test 1 — Leaf-1: four VIP paths, gateway-ip export with IPv6 next hops

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
router bgp 65001
  router-id 10.34.0.1
  log-neighbor-changes
  address-family ipv4 unicast
    redistribute direct route-map d2bgp
  address-family l2vpn evpn
    maximum-paths 64
    additional-paths send
    additional-paths receive
    additional-paths selection route-map pass_all
  neighbor 10.33.1.1
    remote-as 65201
    description spine
    address-family ipv4 unicast
      send-community
      send-community extended
    address-family l2vpn evpn
      send-community extended
  neighbor 10.101.5.1
    remote-as 20
    description HOST
    address-family ipv4 unicast
  neighbor 10.101.0.0/16
    remote-as 10
    description HOST
    address-family ipv4 unicast
  neighbor Ethernet1/1
    remote-as 65201
    description spine
    address-family ipv4 unicast
    address-family l2vpn evpn
      send-community extended
  vrf BLUE_TENANT
    bestpath as-path ignore
    address-family ipv4 unicast
      redistribute direct route-map permit_all
      export-gateway-ip
      maximum-paths mixed 64
      additional-paths receive
      additional-paths selection route-map pass_all
    neighbor 10.101.5.1
      remote-as 20
      description HOST
      address-family ipv4 unicast
    neighbor 10.101.0.0/16
      remote-as 10
      description HOST
      address-family ipv4 unicast

Test 2 — Leaf-2: RFC 5549 toward the spine invalidates the route

With RFC 5549 configured on the Leaf-2 ↔ spine session, Leaf-2 invalidates the route with an “invalid rnh resolve” error:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
router bgp 65002
  router-id 10.34.0.2
  log-neighbor-changes
  address-family ipv4 unicast
    redistribute direct route-map d2bgp
  address-family l2vpn evpn
    maximum-paths 64
    additional-paths send
    additional-paths receive
    additional-paths selection route-map pass_all
  neighbor 10.33.2.1
    remote-as 65201
    description spine
    shutdown
    address-family ipv4 unicast
      send-community
      send-community extended
    address-family l2vpn evpn
      send-community extended
  neighbor 10.102.0.0/16
    remote-as 10
    description HOST
    address-family ipv4 unicast
  neighbor Ethernet1/1
    remote-as 65201
    description spine
    address-family ipv4 unicast
    address-family l2vpn evpn
      send-community extended
  vrf BLUE_TENANT
    address-family ipv4 unicast
      redistribute direct route-map permit_all
      export-gateway-ip
      additional-paths receive
    neighbor 10.102.0.0/16
      remote-as 10
      description HOST
      address-family ipv4 unicast

When I changed the Leaf-2 ↔ spine session from RFC 5549 back to normal IPv4, Leaf-2 validated the route (note the shutdown moves to the Ethernet1/1 unnumbered session instead):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
router bgp 65002
  router-id 10.34.0.2
  log-neighbor-changes
  address-family ipv4 unicast
    redistribute direct route-map d2bgp
  address-family l2vpn evpn
    maximum-paths 64
    additional-paths send
    additional-paths receive
    additional-paths selection route-map pass_all
  neighbor 10.33.2.1
    remote-as 65201
    description spine
    address-family ipv4 unicast
      send-community
      send-community extended
    address-family l2vpn evpn
      send-community extended
  neighbor 10.102.0.0/16
    remote-as 10
    description HOST
    address-family ipv4 unicast
  neighbor Ethernet1/1
    remote-as 65201
    description spine
    shutdown
    address-family ipv4 unicast
    address-family l2vpn evpn
      send-community extended
  vrf BLUE_TENANT
    address-family ipv4 unicast
      redistribute direct route-map permit_all
      export-gateway-ip
      additional-paths receive
    neighbor 10.102.0.0/16
      remote-as 10
      description HOST
      address-family ipv4 unicast

What this means

We cannot use RFC 5549 for the host sessions when we need the gateway-IP trick. I could not find any way to make the leaf resolve the next hop when using export gateway-ip. I even tried advertising the next-hop /32 from Leaf-1 (network nh/32 under BGP) — it did not help.
We cannot use RFC 5549 for the underlay together with export-gateway-ip either.

The Working Solution: EVPN with IPv4 Host Sessions

Dropping RFC 5549 in favour of normal IPv4 eBGP sessions to the hosts, the anycast/ECMP design works. Full configs below; the inline annotations mark the knobs that matter.

Leaf config (Leaf-1; the other leaves are similar):

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
F1-R1-L1# sh run

nv overlay evpn
feature bgp
feature isis
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay

fabric forwarding anycast-gateway-mac 0000.2222.3333
vlan 1,10,1010
vlan 10
  vn-segment 900010
vlan 1010
  vn-segment 800010

ip prefix-list d2bgp seq 5 permit 10.34.0.1/32
ip prefix-list d2bgp seq 10 permit 10.35.0.1/32
route-map d2bgp permit 10
  match ip address prefix-list d2bgp
route-map d2bgp deny 100
route-map pass_all permit 10
  set path-selection all advertise
route-map test_in permit 10
  set local-preference 250
! ^ required for selecting local prefixes as best, used to construct the
!   multipath attributes. Mostly an NX-OS specific.

vrf context BLUE_TENANT
  vni 800010
  rd auto
  address-family ipv4 unicast
    route-target both auto
    route-target both auto evpn
    route-target import 65000:800010
    route-target import 65000:800010 evpn
    route-target export 65000:800010
    route-target export 65000:800010 evpn
vrf context management
  ip route 0.0.0.0/0 192.168.122.1

interface Vlan1

interface Vlan10
  no shutdown
  vrf member BLUE_TENANT
  ip address 10.10.10.1/24
  fabric forwarding mode anycast-gateway

interface Vlan1010
  no shutdown
  vrf member BLUE_TENANT
  ip forward

interface nve1
  no shutdown
  host-reachability protocol bgp
  source-interface loopback1
  member vni 800010 associate-vrf
  member vni 900010
    ingress-replication protocol bgp

interface Ethernet1/1
  no switchport
  ip address 10.33.1.0/31
  ipv6 link-local use-bia
  isis network point-to-point
  isis circuit-type level-2
  ip router isis underlay
  no shutdown

interface Ethernet1/2
interface Ethernet1/3
interface Ethernet1/4

interface Ethernet1/5
  no switchport
  vrf member BLUE_TENANT
  ip address 10.101.5.0/31
  ipv6 link-local use-bia
  no shutdown

interface Ethernet1/6
  no switchport
  vrf member BLUE_TENANT
  ip address 10.101.6.0/31
  no shutdown

interface Ethernet1/7
  no switchport
  vrf member BLUE_TENANT
  ip address 10.101.7.0/31
  no shutdown

interface Ethernet1/8
  no switchport
  vrf member BLUE_TENANT
  ip address 10.101.8.0/31
  ipv6 link-local use-bia
  no shutdown

interface loopback0
  ip address 10.34.0.1/32
  isis network point-to-point
  isis circuit-type level-2
  ip router isis underlay

interface loopback1
  ip address 10.35.0.1/32
  isis network point-to-point
  isis circuit-type level-2
  ip router isis underlay

router isis underlay
  net 49.0000.0000.0001.00
  is-type level-2

router bgp 65001
  router-id 10.34.0.1
  address-family ipv4 unicast
    redistribute direct route-map d2bgp
    maximum-paths 64
    additional-paths send
    additional-paths receive
    additional-paths selection route-map pass_all
  address-family l2vpn evpn
    maximum-paths 64                 ! required for installing multiple EVPN routes
    additional-paths send            ! required for sending multiple EVPN routes
    additional-paths receive         ! required for receiving multiple EVPN routes
    additional-paths selection route-map pass_all  ! select multiple EVPN routes to advertise
  template peer-policy host
    route-map test_in in
  template peer-session host
    remote-as 10
  neighbor 10.33.1.1
    remote-as 65201
    address-family ipv4 unicast
      disable-peer-as-check
      soft-reconfiguration inbound always
    address-family l2vpn evpn
      disable-peer-as-check
      send-community extended
  vrf BLUE_TENANT
    bestpath as-path ignore          ! remote-leaf and local routes have different AS-path lengths
    address-family ipv4 unicast
      redistribute direct route-map pass_all   ! with export-gateway-ip our routes use the remote
                                               ! BGP peer address as next hop, so advertise those peer IPs
      export-gateway-ip              ! differentiate local routes when building the EVPN NLRI
      maximum-paths mixed 64         ! mix local and remote prefixes for multipath
      additional-paths send          ! send multipath into EVPN
      additional-paths receive       ! receive multipath from hosts
      additional-paths selection route-map pass_all
    neighbor 10.101.5.1
      remote-as 20
      address-family ipv4 unicast
        route-map test_in in
    neighbor 10.101.0.0/16
      inherit peer-session host
      address-family ipv4 unicast
        inherit peer-policy host 1
    neighbor Ethernet1/5
      remote-as 20
      address-family ipv4 unicast
        route-map test_in in
    neighbor Ethernet1/8
      inherit peer-session host
      address-family ipv4 unicast
        inherit peer-policy host 1

evpn
  vni 900010 l2
    route-target import auto
    route-target export auto

logging monitor 7

Spine config:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
F1-S1# sh run

router isis underlay
  net 49.0000.0000.0201.00
  is-type level-2

router bgp 65201
  router-id 10.34.0.201
  bestpath as-path ignore     ! or relax the AS-path; at this level all routes come from
                              ! leaves and share the same AS-path length
  address-family ipv4 unicast
    maximum-paths 64
    retain route-target all
    additional-paths send
    additional-paths receive
    additional-paths selection route-map add_select   ! select prefixes for additional-path
  neighbor 10.33.1.0
    remote-as 65001
    address-family ipv4 unicast
      soft-reconfiguration inbound always
    address-family l2vpn evpn
      send-community extended
  neighbor 10.33.2.0
    remote-as 65002
    address-family ipv4 unicast
    address-family l2vpn evpn
      send-community extended
  neighbor 10.33.5.0
    remote-as 65003
    address-family ipv4 unicast
    address-family l2vpn evpn
      send-community extended

Conclusion

It is not yet feasible to use BGP unnumbered (RFC 5549 / RFC 8950) — either toward the hosts or for the underlay — for VNF/anycast services that depend on the Type-5 gateway-IP trick: the IPv6 next hop has nowhere to live in the IPv4 gateway field, the local routes collapse to a single gateway, and receiving leaves drop them as unresolvable. The same design works fine with normal IPv4-based eBGP sessions to the hosts, combined with export-gateway-ip, maximum-paths, and additional-paths on both the IPv4-unicast and L2VPN-EVPN address families.

References

The content of this post was created by the author; AI was used for editing and restructuring.

Why#

The Idea#

The Type-5 Encapsulation Problem#

Setting the Gateway IP — and Where RFC 5549 Breaks#

Test 1 — Leaf-1: four VIP paths, gateway-ip export with IPv6 next hops#

Test 2 — Leaf-2: RFC 5549 toward the spine invalidates the route#

What this means#

The Working Solution: EVPN with IPv4 Host Sessions#

Conclusion#

References#

Why