It is not about SRv6, it is about VPN

I started this by looking at SRv6 L3VPN for Kubernetes. We have different workload on different VRF’s and also external resources. I was able to put PODS into different VRF’es but the control plane problems seems more complex than I thought.

Kubernetes’ basic network model requires

  • Every pod can communicate with every other pod across all nodes without NAT.
  • Every node can communicate with every pod on any node without NAT.
  • A pod sees its own IP address exactly as other pods and nodes see it — there is no hidden address translation between them.

Kubernetes components such as egress gateway, cluster services, node port etc. do not have a concept of VPN or tenancy.

What happens if we put pods into VPNs

We can extend L3VPN into Kubernetes and put pods into different VPNs. However;

  • Nodes must connect to each other freely for Kubernetes services.
  • Kubernetes services must talk to the pods directly. kubelet probes, the API server and CNI health checks are all direct pod access.

Thus the nodes themselves sit in the underlay, or default VRF, whatever you call it. If pods are in a different forwarding data path, it must leak into this default VRF. And the moment you leak, the two things a VPN promises — isolation and overlapping addresses — are both gone.

As we said, Kubernetes must access the pods. Pods cannot have overlapped addresses. We must either use secondary interfaces (Multus style) towards the host route table where the nodes and kubelet live, or we can NAT pod IP addresses when they leave the VPN domain towards the host route table and give the pods unique IP addresses, or vice versa.

Overlap is not only a routing problem. In Cilium, security identity, ipcache, conntrack and Hubble all assume cluster-wide unique IPs. NAT can fix the address on the wire, but the identity layer cannot follow an overlapped IP. You lose policy and observability for exactly the traffic you put into the VPN. The other solution can be using different conntrack, fib tables or feed them with vpn-id.

There is also a simpler point: tenancy inside the cluster is already a solved problem. Namespaces plus NetworkPolicy give identity-based isolation without any VRF. What a VRF would buy you inside the cluster already exists there, in a form the rest of Kubernetes understands.

Because of this, it is better to use VPN outside the Kubernetes cluster, for extending external VPNs into the pod networks. The pod keeps its normal cluster IP and its cluster connectivity; the VRF applies only to traffic that crosses the cluster border. This is also the design point the commercial implementations have landed on — the VRF selection happens at the edge, for external destinations, while the node and the cluster stay in the default VRF. But the problem still sames do not use overlapping ip addresses or a lot work must be done on the CNI e.t.c.

The VPN belongs at the border. Inside, Kubernetes already has its own answer.