Highly available Kubernetes on BGP
A series about building, from scratch, a highly available Kubernetes cluster in which both the API server and the LoadBalancer services are announced over BGP. Such a cluster runs on my Hyper-V virtual machines, and it hosts, among other things, this blog.
The series is hands-on: every part has the commands, the configuration and the things I managed to step on. The order is:
- the cluster design and the plan;
- preparing the Ubuntu 24.04 nodes — network, containerd, kubeadm, time;
- BGP on the router with FRR — filters, ECMP and the firewall rule for hairpin traffic;
- kube-vip in BGP mode and the first master, then the other masters;
- Calico with eBPF instead of
kube-proxy, and the worker nodes; LoadBalancerservice addresses from Calico over BGP;- Traefik behind BGP and why
externalTrafficPolicy: Localmatters; - failover testing, a troubleshooting cheat sheet and a review of the mistakes.
Design and plan
The first article on a Kubernetes cluster that announces its API address (kube-vip) and LoadBalancer addresses (Calico) over BGP: why, design, addresses, ASNs.
Preparing Ubuntu 24.04 nodes
Preparing Kubernetes nodes on Ubuntu 24.04: static address, swap, modules and sysctl, containerd, kubeadm, time, auto-updates and nr_cpus for Hyper-V.
BGP on the router (FRR)
An FRR BGP router on Linux for a Kubernetes cluster: peer groups for masters and workers, route filters, ECMP, timers and a firewall rule.
kube-vip in BGP mode and the first master
A VIP for the Kubernetes API server via kube-vip in BGP mode: a static pod, health checks, kubeadm init with controlPlaneEndpoint, second and third masters.
Calico with eBPF and worker nodes
Calico via tigera-operator in eBPF mode without kube-proxy: a ConfigMap with the API address, the pod pool, worker nodes, dataplane checks and operator pitfalls.
LoadBalancer addresses from Calico over BGP
LoadBalancer service addresses from Calico over BGP: BGPConfiguration, a BGPPeer for workers only, a pool with allowedUses, the loadBalancerIPs annotation, a test.
Traefik behind BGP and externalTrafficPolicy Local
Traefik behind a LoadBalancer service with an address from Calico over BGP: three replicas with anti-affinity, a PodDisruptionBudget and why Local matters.
Failover testing and lessons learned
Which failures the BGP design of a Kubernetes cluster covers and how fast, how to test them, a BGP troubleshooting cheat sheet, maintenance rules and pitfalls.