4 October 2026, 18:50Виталий Лещенко

Highly available Kubernetes on BGP. Part 8: Failover testing and lessons learned

All parts of the series “Highly available Kubernetes on BGP”

Everything is built; time to put it to the test. This part covers which failures are noticed by whom and how fast, how to reproduce them, a BGP troubleshooting cheat sheet, and maintenance rules I derived from my own mistakes.

Which failure is covered by whom

What happened	Who notices	How fast
`kube-apiserver` hung or crashed on a master, the node itself is alive	the kube-vip health check	about 15 s (3 checks every 5 s)
The kube-vip pod crashed or the node was shut down cleanly	the BGP TCP session drops	almost immediately
A master or worker died "hard" (power, network)	the BGP hold timer on the router	up to 30 s
A Traefik pod was deleted or fails readiness	Calico withdraws the `/32` from that node	per readiness
`calico-node` crashed on a worker	the session drops, then the hold timer	immediately or up to 30 s

Two general remarks.

During the detection window some requests are lost. Until the router knows a node is dead, it keeps sending some flows to it (ECMP). For the API this is tolerable: kubectl clients and kubelet retry requests. For websites it means a few seconds of errors for some visitors. If that's not acceptable, look toward BFD (Bidirectional Forwarding Detection); it works with FRR, but I haven't tried it yet.

Changing hold isn't free. I chose the 10 30 timers (keepalive/hold) as a compromise. Any lower, and the session will drop on brief packet loss.

Testing

Below are the scenarios in the form worth running, each with the result you should expect. Run them before you rely on the setup.

Before experimenting, open a continuous probe in a separate window:

while true; do
  printf '%s ' "$(date +%T)"
  curl -sk -o /dev/null -m 2 -w '%{http_code}\n' https://192.168.20.1:6443/readyz
  sleep 1
done

And a second one on the router, to watch ECMP:

watch -n1 'ip route show 192.168.20.1; ip route show 192.168.20.65'

1. The apiserver died, the node is alive

The trickiest scenario: the BGP session is intact, but the API isn't. This is exactly what the health check is for.

# on k8s-master-2
sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml /root/

Expected: in about 15 seconds kube-vip on this master stops announcing the route (the announcing route lines disappear from its log), and the router's table keeps two nexthops for 192.168.20.1 instead of three. The probe may show a few errors during this time (a third of the flows were directed to the dead node), then 200 again.

sudo mv /root/kube-apiserver.yaml /etc/kubernetes/manifests/

When the apiserver comes back up and the health check passes, the master returns to ECMP.

This is a check you absolutely must do before relying on the setup: without it you won't know that the black hole protection really works.

2. Restarting kube-vip

sudo crictl stop $(sudo crictl ps --name kube-vip -q)

kubelet will start a new container within seconds. The session drops for that time, the router removes the master from ECMP and brings it back as soon as the session comes up. Meanwhile the other two masters keep serving the VIP.

3. Hard power-off of a master

Turn off the virtual machine without shutting down the OS (in Hyper-V, Stop-VM -TurnOff).

Expected: the session lives until the hold timer expires — up to 30 seconds — then the router removes the node. etcd remains with two members out of three, and the cluster keeps working. Don't turn off two masters at once: losing quorum stops the cluster.

4. A worker with Traefik

kubectl drain k8s-node-1.example.com --ignore-daemonsets --delete-emptydir-data

Because of the PodDisruptionBudget, Traefik gives up no more than one pod, and the /32 disappears from the node together with the pod. Try this check with a continuous probe against the site:

while true; do curl -s -o /dev/null -m 3 -w '%{http_code} %{time_total}\n' https://www.example.com/; sleep 0.5; done

Afterwards, kubectl uncordon.

Troubleshooting cheat sheet

On the router (via vtysh)

sudo vtysh -c 'show bgp summary'                                   # neighbors and their state
sudo vtysh -c 'show bgp ipv4 unicast'                              # what was received
sudo vtysh -c 'show bgp ipv4 unicast 192.168.20.65'               # who announces the address
sudo vtysh -c 'show bgp ipv4 unicast neighbors 192.168.10.20 received-routes'   # what the neighbor sent
sudo vtysh -c 'show bgp ipv4 unicast neighbors 192.168.10.20 filtered-routes'   # what we threw away
sudo vtysh -c 'show bgp ipv4 unicast neighbors 192.168.10.20 advertised-routes' # what we send (empty)
sudo vtysh -c 'show route-map FROM-CLUSTER'                        # the Invoked counter
ip route show 192.168.20.65                                       # what made it into the kernel
sudo vtysh -b                                                      # apply frr.conf without a restart
sudo vtysh -c 'clear bgp 192.168.10.20 soft in'                   # re-request without dropping the session
sudo vtysh -c 'show bgp neighbor 192.168.10.20' | grep -A4 'Last reset'
sudo journalctl -u frr -n 50

How to read show bgp summary: a number in the State/PfxRcd column means the session is up and that many prefixes were received; a word (Active, Connect, Idle) means there's no session. Up/Down shows the session's age: if it was reset recently, it went down. In the show bgp ipv4 unicast table, a *> line is the best path, and *= is an ECMP member.

A full clear bgp * without soft tears down all sessions, and the API VIP will feel it immediately. Use soft in.

On the cluster side

kubectl get tigerastatus
kubectl -n calico-system exec ds/calico-node -c calico-node -- birdcl show protocols
kubectl -n calico-system exec ds/calico-node -c calico-node -- birdcl show route export Node_192_168_10_1
kubectl get bgpconfiguration default -o yaml
kubectl get bgppeers.projectcalico.org -o yaml
kubectl -n kube-system logs kube-vip-k8s-master-1.example.com --tail=20

The key lines in the kube-vip log are BGP health check passed, announcing route and BGP_FSM_ESTABLISHED.

Typical scenarios

A service doesn't respond from outside. First, show bgp ipv4 unicast <address> on the router: is there a path at all? If there is, check the port forwarding and ip route show. If not, go to the cluster: is the pod alive, is the session up on its node?

It responds intermittently. For services with Local this looks like ECMP to nodes without a pod. Check that serviceLoadBalancerAggregation: Disabled is set, and compare the list of paths in show bgp with where the pods actually live.

show bgp is empty while sessions are up. A typo in a route-map name (with bgp ebgp-requires-policy on) — routes are silently dropped. Look at the route-map's Invoked counter.

A service is stuck in <pending>. Check the calico-kube-controllers logs: most likely the loadBalancerIPs annotation is written as a string rather than a JSON array.

From a node, the service address "doesn't ping" and ip route get complains. That's normal, see part 6: check with curl.

An address moved, but some nodes don't know about it. Services are handled by eBPF; check calico-node -bpf nat dump on the node — the new address should be among the frontends.

A name resolves to the old address. Besides DNS, check /etc/hosts on the host itself: on one of my servers the address was hardcoded in /etc/hosts, and half an hour after the service address changed, an agent on the router started spewing "No route to host" errors, even though everything answered by IP. If you change an address, look for it not only in DNS.

Maintenance rules

The router is part of the control plane. Rebooting the router puts the worker nodes into NotReady, because they reach the API through it. Plan router reboots as work that affects the cluster.

Reboot nodes strictly one at a time. drain → reboot → wait for Ready → uncordon. The cluster won't survive two masters at once (etcd quorum is lost). First the three masters (between them, check etcdctl endpoint health --cluster — it should be 3/3), then the workers. Before starting, make sure every web service has replicas on different nodes.

A new node. One neighbor <address> peer-group WORKERS line on the router (and vtysh -b). Nothing needs to be done on the Calico side: BGPPeer selects nodes by a selector.

Upgrade kube-vip one master at a time. Bump the tag in the manifest, wait for Ready and announcing route, check curl -sk -o /dev/null -w '%{http_code}' https://192.168.20.1:6443/readyz and move on to the next.

Don't update nodes automatically. I've already told the story of needrestart, which once stopped systemd-networkd and left a master without an address for 17 hours. Cluster nodes are updated by hand in a maintenance window.

Independence from DNS. /etc/hosts with the API address on every node is a small price to pay so that a DNS failure doesn't take down the cluster. If the VIP address changes, don't forget to update it.

kube-vip metrics are available on the node at :2112/metrics. Keep an eye on memory: on version 1.2.4 and newer it's flat.

What broke for me along the way

So you don't have to repeat it:

A memory leak in kube-vip (v1.2.1–v1.2.3): 18 MB per hour on each master, until I found a defer inside the infinite health check loop. Fixed in 1.2.4 (kube-vip#1778).
A lost DHCP address after a glibc update and a restart of systemd-networkd — 17 hours of an unreachable master. Now static addresses and a ban on needrestart restarts.
A single DNS server brought down the whole cluster: the kubelets reached the API by name. Now /etc/hosts.
A dangling /etc/resolv.conf link to /run/systemd/resolve/ — after a reboot the node couldn't create a single pod.
Hyper-V per-CPU memory (240 possible processors): 1.3–1.5 GB per node wasted on eBPF maps. Fixed with nr_cpus.
Page loads hanging with Cluster — losses in the eBPF NAT tunnel between Hyper-V hosts. Fixed with externalTrafficPolicy: Local.
Hairpinned traffic and conntrack: ping works, the TCP handshake works, data doesn't flow.
The loadBalancerIPs annotation as a string instead of a JSON array: the service silently hangs in <pending>.

Summary

What we ended up with:

the cluster API answers at a single address announced by three masters, and disappears from a dead node within seconds;
services get addresses assigned and announced by Calico — without MetalLB, without ARP and without an L2 segment;
load balancing across nodes is done by the router (ECMP), and a node failure is simply a route disappearing.

The price is that the router has become part of the control plane, and that without the hairpin rule in the firewall the setup silently doesn't work.

This concludes the series. If you've read this far — thank you. Leave your questions and comments below.