Why anycast inside the fabric is harder than it looks

An anycast service is advertised from multiple hosts into the data center fabric — the same prefix, the same service IP, originated from several places at once. The promise is appealing: multiple next hops for one service prefix gives you load balancing, fast failover, and horizontal scalability without any state in the network.

The catch is that for every leaf to actually use all of those origins, the fabric has to carry and install multiple equal-cost paths for a single prefix end to end. That is not BGP’s default behavior, and the moment you combine it with two things modern fabrics love — an IPv6 unnumbered underlay and IPv4-in-IPv6 next hops (RFC 5549) — one of the most common EVPN disambiguation tricks stops working. This post walks through why, and what the realistic options are.

This is a deeper rewrite of an earlier note of mine; the original screenshots of leaf/spine BGP tables are replaced here with described behavior and illustrative output, but the conclusions are unchanged.

The real requirement: ECMP for the anycast prefix

For all hosts to receive traffic, the fabric must build ECMP (Equal-Cost Multipath) toward the anycast prefix at every hop. Whether that is achievable depends entirely on the fabric type and the protocols you run, because plain BGP advertises only its single best path for a given NLRI. To carry more than one path for the same prefix you need a combination of:

  • BGP Additional Paths / add-path (RFC 7911) so a speaker can advertise (and a peer can receive) more than just the best path for the same NLRI.
  • maximum-paths (multipath) so the receiver actually installs several next hops instead of one.
  • bestpath as-path multipath-relax (and often as-path ignore) so that paths learned across different AS numbers in an eBGP leaf-spine can still be treated as multipath candidates.

Miss any one of these and the fabric silently collapses your N anycast origins back down to one.

EVPN / VXLAN fabrics

EVPN gives you the cleanest path to anycast, but it needs explicit tuning.

Advertise and receive multiple paths for the EVPN address family. Anycast requires not just the best path but the advertisement and reception of multiple paths for the same NLRI. That means maximum-paths and add-path on the EVPN address family — and, just as importantly, on the edge unicast address family of the leaf switches where the hosts are actually connected. It is easy to tune the overlay and forget the place where the host routes enter the fabric.

Relax best-path selection. Depending on whether your design is eBGP or iBGP, you will need bestpath as-path multipath-relax and/or AS-path ignore so the equal anycast paths survive best-path selection rather than being pruned to one.

The Type-5 disambiguation problem. External (host) prefixes are carried as EVPN route-type 5 (IP Prefix routes, RFC 9136). When several anycast origins hang off the same leaf, their re-originated Type-5 routes look identical once encapsulated — same originating VTEP next hop, same prefix — so a remote leaf cannot tell them apart and cannot ECMP across them. They appear as one path.

RFC 9136 solves this with the overlay index: the Type-5 route can carry a gateway IP (or a MAC/ESI) that points at the real next hop behind the advertising PE. The receiving NVE then does a recursive route resolution on that overlay index to find which egress actually reaches the host:

1
2
3
4

EVPN Type-5 (IP Prefix Route)
  Prefix:      10.0.0.0/32        <- the anycast service IP
  Next hop:    VTEP-Leaf1          <- same for every origin on Leaf1
  Gateway IP:  <host underlay IP>  <- overlay index, disambiguates + recursively resolves

Set the gateway IP to the host’s next hop and the previously-identical Type-5 routes become distinct and recursively resolvable, so the host actually advertising the prefix is reached. Note RFC 9136’s hard rule: if there is no route to resolve the overlay index, the RT-5 is not installed at all — even when everything else looks correct.

Where it breaks: IPv6 unnumbered + RFC 5549

Here is the collision the title is about.

To disambiguate same-leaf anycast origins you populate the gateway IP overlay index, and in practice that gateway IP has to be a resolvable IPv4 host address — something the receiving NVE can recursively resolve to a real egress.

But a modern underlay is very often built with BGP unnumbered: peers are discovered over IPv6 link-local addresses, and IPv4 prefixes are advertised with an IPv6 next hop using the Extended Next Hop Encoding of RFC 5549 (Advertising IPv4 NLRI with an IPv6 Next Hop; now obsoleted by RFC 8950). That is the whole point of unnumbered: there is no IPv4 next-hop address on the leaf↔host link — the host’s IPv4 prefix arrives carried over an IPv6 link-local next hop.

So the two mechanisms are mutually exclusive in the obvious deployment:

  • The Type-5 gateway-IP overlay index needs an IPv4 next hop to point at and resolve.
  • RFC 5549 / unnumbered deliberately removes the IPv4 next hop, leaving only an IPv6 link-local one.

There is nothing to put in the gateway IP, and even if you force an IPv6 gateway IP, there is no stable, recursively-resolvable IPv4 overlay index for the host. The conclusion from the original note still stands: you cannot use RFC 5549 between the switch and the host for anycast services that rely on the Type-5 gateway-IP trick. If you want this disambiguation, the leaf↔host link has to use real IPv4 next hops, not unnumbered/IPv4-in-IPv6.

Beyond this, expect additional per-design BGP tuning (eBGP vs iBGP), and tune the best-path algorithm to keep the equal paths.

Routed / pure IP fabrics

If you are not running EVPN, you are almost always on a BGP-routed fabric, and here the anycast story gets harder. The lab below is a multi-AS eBGP leaf-spine carrying the service prefix 172.16.100.100/32 in plain IPv4 unicast, with add-path and multipath already turned on everywhere — and it still cannot preserve the per-host weighting.

The leaf sees and advertises every origin. Leaf F1-R1-L1 is connected to several hosts advertising the same anycast /32, and with additional-paths + maximum-paths it installs all of them as multipath and advertises all of them to the spine:

Leaf BGP table — multiple local paths for the anycast prefix

Leaf advertising all of those paths to the spine

This is not a default-config accident — the leaf explicitly runs additional-paths send/receive, maximum-paths 64, bestpath as-path multipath-relax, and bestpath as-path ignore on the IPv4 unicast address family:

Leaf BGP configuration

So, to answer the obvious question directly: yes, additional-paths (add-path, RFC 7911) can be and is used on the IPv4 unicast address family. That is not where pure IPv4 falls down. (advertise best-external, by contrast, only ever adds a single extra path and is an iBGP fast-convergence feature — it is not a substitute for add-path here.)

The spine is where the diversity is lost. The spine receives all of the leaf’s paths — but every one of them arrives with the same BGP next hop: the leaf itself (10.33.1.0), because the leaf re-originates them with next-hop-self:

Spine receives all the paths, but all via the same leaf next hop

The consequence shows up in the spine’s table and FIB. For 172.16.100.100/32 the spine load-balances across leaves — three next hops, one per leaf (10.33.1.0, 10.33.2.0, 10.33.3.0) — even though it too runs additional-paths send/receive, maximum-paths 64, and additional-paths install backup:

Spine BGP table and routing table — three next hops, one per leaf

Spine BGP configuration

So the fabric does get ECMP, but only at the granularity of leaves, not hosts. The four origins behind a single leaf collapse into one next hop (that leaf), and their weight is gone. If two leaves each have four hosts, the spine balances 1:1 between the two leaves instead of 4:4 across the eight hosts. Add-path did its job carrying the NLRIs; the loss happens because next-hop-self hides the individual hosts behind the leaf’s address. There is no IPv4-unicast knob that re-exposes them.

Why a different address family fixes it. This is exactly what EVPN Type-5 with a gateway-IP overlay index restores: the per-host next hop is preserved as the overlay index and recovered by recursive resolution, so the spine/leaf can ECMP to the real hosts rather than to the leaf. Carrying the edge networks in VPNv4 or EVPN (where unique RDs also keep each origin a distinct NLRI) is therefore the practical answer — and it is the very mechanism that, as shown above, collides with RFC 5549 / unnumbered. Other routes out (iBGP, BGP PIC) tend toward eBGP-over-iBGP preference issues or complex, vendor-specific configuration.

There is also a topology that sidesteps all of this: dedicated switches for the anycast service — north-south appliances such as firewalls or external load balancers that are equally connected to multiple switches (same switches, same number of links). That produces ECMP equal to the number of switches, and each switch then does its own ECMP toward the hosts.

Summary

  • Anycast in the fabric is fundamentally an end-to-end ECMP problem: you need add-path (RFC 7911), maximum-paths, and multipath-relax on both the overlay and the host-facing edge address family.
  • EVPN Type-5 (RFC 9136) disambiguates same-leaf anycast origins with the gateway-IP overlay index and recursive resolution.
  • That gateway-IP trick needs a resolvable IPv4 next hop, which directly conflicts with BGP unnumbered + RFC 5549 (RFC 8950) IPv4-in-IPv6 next hops — so you cannot use RFC 5549 on the leaf↔host link for these anycast services.
  • Pure IPv4 unicast fabrics can carry the paths — additional-paths (RFC 7911) and maximum-paths work on the IPv4 unicast AF — but the leaf’s next-hop-self collapses all hosts behind a leaf into one next hop at the spine, so you get ECMP across leaves, not hosts, and lose the per-host weight. (advertise best-external does not help.) Moving the edge networks to VPNv4/EVPN restores per-host next hops, and dedicated, equally-connected service switches remain the simplest option for north-south anycast.

References

The content of this post was created by the author; AI was used for editing and restructuring.