A Kubernetes node has one node IP. Giving it two uplinks normally means bonding plus MLAG or ESI on the switches — state shared between two boxes that would rather not share state.

This lab does it at layer 3 instead. Each node gets two independent routed links, the node address lives on dummy0, and BGP announces that address over both. Nothing is bonded and the TORs share nothing.

Design, in five decisions

The node address goes on dummy0. If Kubernetes takes its IP from a physical interface, losing that link means the node’s identity goes with it, and you end up building something complicated to paper over it. A dummy device never goes down, so from the kernel’s point of view there is one stable address and two ways to reach it.

The uplinks are unnumbered. No addressing plan, no IPAM for point-to-point links, no per-server switchport config. Plug the cable in, BGP comes up, the node announces its host route. IPv4 reachability rides IPv6 link-local next hops (RFC 8950), so there is no IPv4 addressing on the fabric links at all.

Cilium peers with the node’s own FRR, not with the TORs. This is the part that matters operationally. If Cilium peers with the switches, every cluster carries switch addresses and ASNs in its manifests, and every rack move is a manifest change. Peering with ::1 means one CiliumBGPClusterConfig works everywhere; the node’s FRR already knows the fabric.

One session, both families — no IPv4 session needed. A single ::1 peering carries IPv4 NLRI over the IPv6 session, the same way the fabric links do (RFC 8950). A 127.0.0.1 peer was tried and then removed: activating ::1 under the ipv4 unicast family makes it redundant.

Two TORs is a simulation, not a fabric design. The lab does not model a real spine/leaf AS layout — that part gets simpler once the logic works, not harder.

Everything is pre-staged. The nodes have no management address at all, which is a design requirement here rather than an inconvenience.

Topology

ext1 — tor1 · AS 65001 ============== tor2 · AS 65002 — ext2 | | | | | | +—-|—-|——————–+ | | | | (each node dual-homed, unnumbered) controller worker1 worker2 AS 65111 AS 65112 AS 65113 10.0.0.101 10.0.0.102 10.0.0.103 (dummy0) fd00:ff::101 fd00:ff::102 fd00:ff::103

Every node-to-TOR link is unnumbered eBGP with BFD. Solid double line between the TORs is their cross-link.

BGP adjacencies

graph LR
    subgraph node["worker1 (one node, two BGP speakers)"]
        CIL["Cilium<br/>AS 65999<br/>ListenPort -1"]
        FRR["FRR<br/>AS 65112"]
    end
    T1["tor1<br/>AS 65001"]
    T2["tor2<br/>AS 65002"]

    CIL -->|"::1 · IPv4 + IPv6"| FRR
    FRR -->|"unnumbered ens2 + BFD"| T1
    FRR -->|"unnumbered enp1s3 + BFD"| T2

One loopback session. FRR activates ::1 under both address families, so IPv4 prefixes ride the IPv6 session — no 127.0.0.1 peer exists.

Node FRR

frr defaults datacenter
hostname worker1
allow-reserved-ranges
service integrated-vtysh-config
!
interface dummy0
 ip address 10.0.0.102/32
 ipv6 address fd00:ff::102/128
exit
!
router bgp 65112
 bgp router-id 10.0.0.102
 no bgp default ipv4-unicast
 bgp bestpath as-path multipath-relax
 neighbor enp1s3 interface remote-as external
 neighbor enp1s3 description tor2
 neighbor enp1s3 bfd
 neighbor enp1s3 bfd profile fast
 neighbor ens2 interface remote-as external
 neighbor ens2 description tor1
 neighbor ens2 bfd
 neighbor ens2 bfd profile fast
 neighbor ::1 remote-as 65999
 neighbor ::1 description cilium
 bgp allow-martian-nexthop
 !
 address-family ipv4 unicast
  network 10.0.0.102/32
  neighbor enp1s3 activate
  neighbor ens2 activate
  neighbor ::1 activate
  neighbor ::1 route-map cilium-ipv6-nexthop in
  maximum-paths 8
 exit-address-family
 !
 address-family ipv6 unicast
  network fd00:ff::102/128
  neighbor enp1s3 activate
  neighbor ens2 activate
  neighbor ::1 activate
  neighbor ::1 route-map cilium-ipv6-nexthop in
  maximum-paths 8
 exit-address-family
exit
!
route-map cilium-ipv6-nexthop permit 10
 set ipv6 next-hop global fd00:ff::102
exit
!
ip nht resolve-via-default
!
ipv6 nht resolve-via-default
!
bfd
 profile fast
  transmit-interval 150
  receive-interval 150
 exit
exit

Four lines carry the whole loopback-peering trick:

  • allow-reserved-ranges and nht resolve-via-default are global commands, not router bgp ones. There is no bgp allow-reserved-ranges, and looking for it under router bgp makes FRR look older than it is.
  • bgp allow-martian-nexthop stops FRR answering Cilium’s first UPDATE with Invalid NEXT_HOP Attribute.
  • The inbound route-map rewrites Cilium’s loopback next hop to the node address. Without it FRR would re-advertise a loopback into the fabric.

Ubuntu 22.04 ships FRR 8.1, which does not have allow-reserved-ranges. FRR 9.1.3 from deb.frrouting.org is a hard requirement.

Install

kubeadm, dual stack, no kube-proxy:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

apiVersion: kubeadm.k8s.io/v1beta4
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 10.0.0.101
nodeRegistration:
  name: controller
  kubeletExtraArgs:
  - name: node-ip
    value: "10.0.0.101,fd00:ff::101"
---
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
controlPlaneEndpoint: "10.0.0.101:6443"
networking:
  podSubnet: "10.244.0.0/16,fd00:244::/56"
  serviceSubnet: "10.96.0.0/16,fd00:96::/112"

1
2
3
4
5
6
7

# kubeadm walks default routes to pick an address and cannot parse the
# BGP one (a nexthop group with IPv6 link-local members). metric 4000
# keeps BGP winning for actual forwarding; at metric 1 dummy0 blackholes
# and join hangs in preflight forever.
ip route replace default dev dummy0 src 10.0.0.101 metric 4000
kubeadm init --config init.yaml --skip-phases=addon/kube-proxy
ip route del default dev dummy0 metric 4000

Cilium — native routing, no masquerade, BGP control plane:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

# cilium-values.yaml
ipv6:
  enabled: true
routingMode: native
ipv4NativeRoutingCIDR: 10.244.0.0/16
ipv6NativeRoutingCIDR: fd00:244::/56
enableIPv4Masquerade: false
enableIPv6Masquerade: false
autoDirectNodeRoutes: false
kubeProxyReplacement: true
k8sServiceHost: 10.0.0.101
k8sServicePort: 6443
ipam:
  mode: cluster-pool
  operator:
    clusterPoolIPv4PodCIDRList: ["10.244.0.0/16"]
    clusterPoolIPv4MaskSize: 24
    clusterPoolIPv6PodCIDRList: ["fd00:244::/104"]
    clusterPoolIPv6MaskSize: 120
bgpControlPlane:
  enabled: true

1
2

helm upgrade --install cilium ./cilium-1.18.0.tgz -n kube-system -f cilium-values.yaml
helm get values cilium -n kube-system | grep useDigest   # verify it took them

Cilium’s cidrset allows at most 16 bits between the pool prefix and the per-node mask, so the IPv6 pool is /104 → /120. A /56 → /120 makes the operator die with the node CIDR size is too big.

BGP CRDs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64

apiVersion: cilium.io/v2
kind: CiliumBGPClusterConfig
metadata:
  name: worker1
spec:
  nodeSelector:
    matchLabels:
      kubernetes.io/hostname: worker1
  bgpInstances:
  - name: main
    localASN: 65999
    peers:
    - name: local-frr-v6
      peerASN: 65112
      peerAddress: "::1"
      peerConfigRef: {name: local-frr}
---
apiVersion: cilium.io/v2
kind: CiliumBGPPeerConfig
metadata:
  name: local-frr
spec:
  timers: {connectRetryTimeSeconds: 5, holdTimeSeconds: 9, keepAliveTimeSeconds: 3}
  families:
  - afi: ipv4
    safi: unicast
    advertisements: {matchLabels: {advertise: pods}}
  - afi: ipv6
    safi: unicast
    advertisements: {matchLabels: {advertise: pods}}
---
apiVersion: cilium.io/v2
kind: CiliumBGPAdvertisement
metadata:
  name: pods
  labels: {advertise: pods}
spec:
  advertisements:
  - advertisementType: PodCIDR
---
apiVersion: cilium.io/v2
kind: CiliumLoadBalancerIPPool
metadata:
  name: lab
spec:
  blocks:
  - cidr: 10.100.0.0/24
  - cidr: fd00:100::/120
---
apiVersion: cilium.io/v2
kind: CiliumBGPAdvertisement
metadata:
  name: services
  labels: {advertise: pods}
spec:
  advertisements:
  - advertisementType: Service
    service:
      addresses: [LoadBalancerIP]
    selector:
      matchExpressions:
      - key: bgp
        operator: In
        values: ["advertise"]

The Service must carry bgp: advertise or the VIP is allocated and never announced.

Workloads

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

apiVersion: apps/v1
kind: Deployment
metadata:
  name: http
  labels: {app: http}
spec:
  replicas: 2
  selector:
    matchLabels: {app: http}
  template:
    metadata:
      labels: {app: http}
    spec:
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels: {app: http}
      containers:
      - name: http
        image: dnzydn/go-simple-http-server:latest
        imagePullPolicy: IfNotPresent
        ports: [{containerPort: 8080}]
---
apiVersion: v1
kind: Service
metadata:
  name: http-lb
  labels: {bgp: advertise}
spec:
  type: LoadBalancer
  selector: {app: http}
  ipFamilyPolicy: PreferDualStack
  ports: [{port: 80, targetPort: 8080}]
---
apiVersion: v1
kind: Service
metadata:
  name: http-np
spec:
  type: NodePort
  selector: {app: http}
  ipFamilyPolicy: PreferDualStack
  ports: [{port: 80, targetPort: 8080, nodePort: 30080}]

Where everything lives

graph TB
    subgraph ext["outside the cluster"]
        E1["ext1 · 10.1.1.10 / fd00:1::10<br/>behind tor1"]
        E2["ext2 · 10.2.2.10 / fd00:2::10<br/>behind tor2"]
    end

    subgraph vip["advertised by Cilium via BGP"]
        LB["http-lb<br/>10.100.0.0 · fd00:100::<br/>port 80"]
        NP["http-np · NodePort 30080<br/>on every node address"]
    end

    subgraph c["controller · 10.0.0.101 / fd00:ff::101"]
        CP["control plane<br/>PodCIDR 10.244.1.0/24 · fd00:244::100/120"]
    end

    subgraph w1["worker1 · 10.0.0.102 / fd00:ff::102"]
        P1["http-...-xgwmp<br/>10.244.0.132 · fd00:244::b0"]
        N1["netshoot-gk7df<br/>10.244.0.110 · fd00:244::19"]
    end

    subgraph w2["worker2 · 10.0.0.103 / fd00:ff::103"]
        P2["http-...-bngvd<br/>10.244.2.118 · fd00:244::2c2"]
        N2["netshoot-krmgz<br/>10.244.2.147 · fd00:244::20f"]
    end

    E1 --> LB
    E2 --> LB
    E1 --> NP
    LB --> P1
    LB --> P2
    NP --> P1
    NP --> P2

$ kubectl get nodes -o wide
controller   Ready   10.0.0.101
worker1      Ready   10.0.0.102
worker2      Ready   10.0.0.103

$ kubectl get pods -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName,IP:.status.podIPs
NAME                   NODE      IP
http-9cfcf7db5-bngvd   worker2   10.244.2.118  fd00:244::2c2
http-9cfcf7db5-xgwmp   worker1   10.244.0.132  fd00:244::b0
netshoot-gk7df         worker1   10.244.0.110  fd00:244::19
netshoot-krmgz         worker2   10.244.2.147  fd00:244::20f

$ kubectl get svc
NAME         TYPE           CLUSTER-IP     EXTERNAL-IP             PORT(S)
http         ClusterIP      10.96.201.89   <none>                  80/TCP
http-lb      LoadBalancer   10.96.77.203   10.100.0.0,fd00:100::   80:31107/TCP
http-np      NodePort       10.96.22.243   <none>                  80:30080/TCP

Verification

Cilium and the node’s FRR — one session, both families on it:

worker1 $ cilium-dbg bgp peers
Local AS   Peer AS   Peer Address   Session       Uptime   Family         Received   Advertised
65999      65112     ::1:179        established   4m49s    ipv4/unicast   12         2
                                                           ipv6/unicast   12         2

worker1 $ vtysh -c 'show bgp summary'
IPv4 Unicast Summary:
Neighbor        V         AS   MsgRcvd   MsgSent  Up/Down State/PfxRcd  Desc
builder(enp1s3) 4      65002      3736      3736 00:51:11           10  tor2
builder(ens2)   4      65001      3853      3857 00:51:11           10  tor1
worker1(::1)    4      65999      1741      1844 00:04:50            2  cilium

IPv6 Unicast Summary:
builder(enp1s3) 4      65002      3736      3736 00:51:11           10  tor2
builder(ens2)   4      65001      3853      3857 00:51:11           10  tor1
worker1(::1)    4      65999      1741      1844 00:04:50            2  cilium

The same ::1 neighbour appears in both family summaries — that is the IPv4 session that does not exist, working.

What tor1 learned. The AS path is the whole design in one line — 65999 (Cilium) → 65112 (worker1’s own FRR) → 65002 (tor2):

tor1 $ vtysh -c 'show bgp ipv4 unicast'
 *= 10.100.0.0/32    ens4      0 65113 65999 i
 *  10.244.0.0/24    enp1s5    0 65002 65112 65999 i
 *  10.244.1.0/24    enp1s5    0 65002 65111 65999 i
 *  10.244.2.0/24    enp1s5    0 65002 65113 65999 i

tor1 $ vtysh -c 'show bgp ipv6 unicast'
 *  fd00:ff::101/128 enp1s5    0 65002 65111 i
 *  fd00:ff::102/128 enp1s5    0 65002 65112 i
 *  fd00:ff::103/128 enp1s5    0 65002 65113 i
 *= fd00:100::/128   ens4      0 65113 65999 i
 *  fd00:244::/120   enp1s5    0 65002 65112 65999 i
 *  fd00:244::100/120          0 65002 65111 65999 i
 *  fd00:244::200/120          0 65002 65113 65999 i

IPv4 carried over IPv6 link-local next hops, and the kernel installing it that way:

tor1 $ vtysh -c 'show bgp ipv4 unicast 10.0.0.101/32'
BGP routing table entry for 10.0.0.101/32
  65113 65002 65111
    fe80::a8c1:abff:fed1:5190(worker2) from worker2(ens4) (10.0.0.103)
      Origin IGP, valid, external

tor1 $ ip route get 10.0.0.101
10.0.0.101 via inet6 fe80::a8c1:abff:fecf:e014 dev ens2 src 10.255.0.1

Reachability from outside

ext1 sits behind tor1 and has no route to the cluster except through the fabric:

ext1 $ curl -s -o /dev/null -w '%{http_code}\n' http://10.244.0.132:8080       # pod v4
200
ext1 $ curl -s -o /dev/null -w '%{http_code}\n' "http://[fd00:244::b0]:8080"    # pod v6
200
ext1 $ curl -s -o /dev/null -w '%{http_code}\n' http://10.100.0.0/              # LB v4
200
ext1 $ curl -s -o /dev/null -w '%{http_code}\n' "http://[fd00:100::]/"          # LB v6
200
ext1 $ curl -s -o /dev/null -w '%{http_code}\n' http://10.0.0.101:30080/        # NodePort v4, controller
200
ext1 $ curl -s -o /dev/null -w '%{http_code}\n' http://10.0.0.102:30080/        # NodePort v4, worker1
200
ext1 $ curl -s -o /dev/null -w '%{http_code}\n' "http://[fd00:ff::102]:30080/"  # NodePort v6, worker1
200
ext1 $ curl -s -o /dev/null -w '%{http_code}\n' "http://[fd00:ff::103]:30080/"  # NodePort v6, worker2
200

ext1 $ ip route get 10.100.0.0
10.100.0.0 via 10.1.1.1 dev eth1 src 10.1.1.10

NodePort needs no advertisement at all — the node host routes are already in the fabric, which is the point of the whole exercise.

Failover

Carrier does not propagate across the qemu tap→veth path, so link-down is invisible to the guest and BFD is not optional here. With it:

loss
without BFD~6150 ms
with BFD (150 ms × 3)~450 ms

Notes

  • rp_filter is the reason node-to-node traffic silently disappears. The effective value is max(all, iface), so all=0 changes nothing while the interfaces carry 2. With an ECMP return path over two unnumbered uplinks, loose RPF cannot validate and drops without a counter or log. Set it per interface.

Lab and scripts: github.com/denizaydin