The multi-homed node from the previous post has two equal-cost layer 3 uplinks. The kernel’s multipath hash never sees the flow — Cilium’s bpf_fib_lookup carries L3 only. Feeding it the flow label and the L4 ports makes ECMP real, on the default hash policy. With the help of AI coding with my stupid ideas, cilium distributes traffic to both uplinks.
Two L3 uplinks per node, address on dummy0, BGP unnumbered to the fabric, and Cilium peering with the node’s own FRR over a single ::1 session that carries both address families — so no Kubernetes manifest ever contains a TOR address.
How L3VPN affects Kubernetes and when it is meaningful?
An alternative Cilium Egress Gateway that removes state synchronization and conntrack overhead in active-active multi-gateway setups from the gateway
Dual-homed servers use IP-unnumbered physical interfaces. A dedicated dummy interface (not lo) is assigned to each server as its IPv4 node address. FRR runs between the servers and the ToR switches to provide node-level connectivity, advertising the server’s node IP (dummy interface address) and receiving the default route from the ToRs. Using a bond interface between ToRs can also be used for redundancy and failover for server connectivity. You may also configure Layer 3 interfaces. The main issue is about node identity and service continuity during the failure of a single link. Using a bond interface or a dummy interface seems like a more usable approach. ...
Overview This document describes a method for simulating multiple servers belonging to different network domains on a single Linux host. Each simulated server runs inside its own Linux network namespace and is connected to a dedicated VLAN. Training and demonstration environments Each namespace behaves as an independent server with: Its own IP address Its own MAC address An SSH service A simple HTTP server Network Topology The Linux host uses the following interfaces: ...
We run multiple teams on the same OpenStack platform, one project per team. Traffic between two projects always hairpins through the gateway nodes, even with DVR enabled. This is expected behavior, not a misconfiguration. Below is the technical summary of why. What DVR distributes East-west between networks on the same router: routed locally on the ingress chassis, tunneled to the destination chassis. Fully distributed. North-south with a floating IP: the chassis hosting the VM performs the 1:1 NAT and answers ARP for the FIP on the external network. Distributed, but requires the provider network to be attached to every compute node and distributed FIP to be enabled. North-south with SNAT: centralized. The router’s external address is shared by all VMs behind it; it can be ARP-owned at only one point of the external segment, and the NAT state must live in a single conntrack table. It is pinned to a gateway chassis by design. DVR distributes only what it can bind 1:1 to a single chassis. Everything based on a shared, stateful address is centralized. ...
Why anycast inside the fabric is harder than it looks An anycast service is advertised from multiple hosts into the data center fabric — the same prefix, the same service IP, originated from several places at once. The promise is appealing: multiple next hops for one service prefix gives you load balancing, fast failover, and horizontal scalability without any state in the network. The catch is that for every leaf to actually use all of those origins, the fabric has to carry and install multiple equal-cost paths for a single prefix end to end. That is not BGP’s default behavior, and the moment you combine it with two things modern fabrics love — an IPv6 unnumbered underlay and IPv4-in-IPv6 next hops (RFC 5549) — one of the most common EVPN disambiguation tricks stops working. This post walks through why, and what the realistic options are. ...
Companion posts. This is the hands-on lab — real NX-OSv configs, the tests I ran, and where they broke. For the high-level summary see Anycast Inside the Data Center; for the deeper protocol write-up (overlay index, recursive resolution, why pure-IP fabrics lose per-host ECMP) see Anycast in EVPN/VXLAN Fabrics: The IPv6 Unnumbered and RFC 5549 Problem. Why Anycast is used for horizontal scaling and for better failure response. With modern orchestration and virtualization, a new instance of a service can be spawned on demand — by an engineer or by the system itself. ...
Part of an anycast series. This is the high-level summary. For the hands-on NX-OS lab (configs, the RFC 5549 failure, the working EVPN solution) see Anycast Inside an EVPN/VXLAN Fabric with BGP Unnumbered (RFC 5549), and for the deeper protocol treatment see Anycast in EVPN/VXLAN Fabrics: The IPv6 Unnumbered and RFC 5549 Problem. Why Anycast services are advertised from multiple hosts into the data center. The same service prefix is originated from several places at once, which gives you multiple next hops for that service — and with it, load balancing, better failover, and scalability. ...