Highly available Kubernetes on BGP. Part 1: Design and plan
All parts of the series “Highly available Kubernetes on BGP”This is the first article of a series on building from scratch a highly available Kubernetes cluster in which both the API server and LoadBalancer services are published over BGP. I run such a cluster on Hyper-V virtual machines, and among other things it hosts this blog.
The series turned out practical: every part has commands, configuration and the things I managed to trip over.
What we'll end up with
- three masters (control plane) and five worker nodes;
- the API server address (VIP) lives on all three masters at once, and each of them announces it to the router over BGP — that's done by kube-vip;
LoadBalancerservice addresses are assigned and announced by Calico — no separate MetalLB needed;- the router (Linux + FRR) accepts the routes and spreads traffic across the nodes via ECMP;
- the CNI is Calico with the eBPF dataplane, and there's no
kube-proxyin the cluster; - incoming traffic is accepted by Traefik, which sits behind an address from BGP.
No virtual addresses "on the interface of one of the nodes" and no ARP: an address is simply a route the router has as long as there's at least one live node able to serve the request.
Why BGP when there's ARP
Both "classic" ways to publish an address — kube-vip in ARP mode and MetalLB in L2 mode — share the same trait: at any given moment the address belongs to one node.
- In kube-vip's ARP mode, the nodes elect a leader through a lease in the Kubernetes API, and the leader puts the VIP on its interface and sends gratuitous ARP. While the leader is alive, all traffic goes to it. If it dies, you have to wait for the lease to expire, elect a new leader and update the neighbors' ARP tables.
- MetalLB in L2 mode works the same way: one node answers ARP for the service address and takes all the traffic to it.
Both approaches require clients and nodes to be in the same L2 segment.
With BGP the picture is different:
- all nodes announce at once, the router sees several equal-cost routes to the same address and spreads traffic across them (ECMP);
- a node failure means its route disappears: either immediately (the session dropped), or by the BGP hold timer, or because the node withdrew the announcement itself;
- the address doesn't need an L2 segment: it can live in a subnet that doesn't physically exist.
The price
To be honest about the downsides:
- You need a router that speaks BGP. Mine is an ordinary Linux box with FRR, but MikroTik, pfSense/OPNsense with the FRR plugin and "grown-up" hardware will all do.
- The address subnet must be purely routed. In BGP mode kube-vip doesn't answer ARP, so if the VIP is in the same subnet as the nodes, neighbors will look for it with broadcasts and won't find it. I set up a separate
192.168.20.0/24that has no L2 segment. - Traffic within the node subnet becomes asymmetric (a node goes to the VIP through the router, while the reply comes back directly). There's a separate part about this: without a firewall change on the router, the setup simply won't work.
- The router has become part of the control plane. Worker nodes reach the API through it, so rebooting the router puts them into
NotReady. Previously, when the VIP was in the same subnet, the router wasn't on this path.
Design
internet / clients
│
┌─────────────┴──────────────┐
│ router.example.com │
│ Linux + FRR, AS 65010 │
│ LAN: 192.168.10.1/24 │
└─────────────┬──────────────┘
│ node network 192.168.10.0/24
┌──────────────────────┼──────────────────────┐
│ │ │
k8s-master-1..3 k8s-node-1..5 (routed subnet
192.168.10.17-19 192.168.10.20-24 without L2)
kube-vip, AS 65020 Calico (BIRD), AS 65030 192.168.20.0/24
announces announces /32s of ├─ .1 API VIP
192.168.20.1/32 LoadBalancer addresses └─ .64/26 LoadBalancer pool
There are eight BGP sessions: one between the router and each node. Masters and workers don't talk BGP to each other.
Address plan
| What | Value |
|---|---|
| Node subnet | 192.168.10.0/24, router 192.168.10.1 |
Masters k8s-master-1…3 |
192.168.10.17–19 |
Worker nodes k8s-node-1…5 |
192.168.10.20–24 |
| API server VIP | 192.168.20.1 = control.k8s.example.com |
| LoadBalancer address pool | 192.168.20.64/26 |
| Pod network | 172.20.0.0/16, /26 blocks |
| Service network | 10.96.0.0/12 |
The LoadBalancer pool starts at .64 so as not to overlap with the VIP, which lives in the same subnet.
AS numbers
| Who | AS |
|---|---|
| Router | 65010 |
| Masters (kube-vip) | 65020 |
| Worker nodes (Calico) | 65030 |
All numbers are from the private range 64512–65534, and they're different — a deliberate decision:
- the router and the nodes are in different ASes, which means eBGP sessions; with identical ASes it would be iBGP, where ECMP is enabled by a separate command (
maximum-paths ibgp) that's easy to forget, leaving you without load balancing; - masters and workers have different ASes because they have different roles, and it's more convenient to apply policy to them separately; besides, the same AS on a node and the router would make it impossible to pass routes back into the cluster: the node would drop a route on seeing its own number in the AS_PATH.
The example in the kube-vip documentation uses identical values only because the defaults were left in place.
Versions
| Component | Version |
|---|---|
| OS | Ubuntu 24.04 |
| Kubernetes (kubeadm) | 1.36 |
| containerd | 2.x from the Docker repository |
| kube-vip | 1.2.4 |
| Calico | 3.32 (via tigera-operator) |
| FRR on the router | 8.4 |
| Traefik | 3.7 |
Versions are as of the time of writing; the principle doesn't depend on them, but field and flag names may differ.
What you'll need
- eight machines with Ubuntu 24.04 (or fewer — three masters and at least one worker): mine are virtual, with 4 vCPUs, 6 GB of memory and a 60 GB disk each;
- a router running Linux (or another BGP router) that sits on the path between the nodes and the other networks;
- a free subnet that nobody on the network uses: the VIP and service addresses will live in it;
- a DNS name for the API (
control.k8s.example.comin my case).
Let's move on: in part 2 we'll prepare the nodes.
Only registered users can leave a comment.
Sign in or register to leave a comment.