4 October 2026, 18:50Виталий Лещенко

Highly available Kubernetes on BGP. Part 3: BGP on the router (FRR)

All parts of the series “Highly available Kubernetes on BGP”

In the third part we configure the router side. There's no cluster yet, but the router can be prepared in advance: it knows all its future neighbors up front and will wait for them to show up.

In my case the router is an ordinary Ubuntu 24.04 machine that already handles routing and the firewall (iptables). The node network is connected to its eth1 interface (address 192.168.10.1/24). If you have a different router, the ideas are the same; only the syntax changes.

Installing FRR

sudo apt-get install -y frr frr-pythontools
sudo sed -i 's/^bgpd=no/bgpd=yes/' /etc/frr/daemons
sudo systemctl enable --now frr

Of all the FRR daemons, only zebra is enabled by default — it talks to the kernel about routes. The bgpd=yes line enables BGP itself. The frr-pythontools package is needed for vtysh -b: this command applies frr.conf to the running daemons without restarting the service.

Configuration

Here's the entire /etc/frr/frr.conf:

frr version 8.4
frr defaults traditional
hostname router.example.com
log syslog informational
service integrated-vtysh-config
!
router bgp 65010
 bgp router-id 192.168.10.1
 no bgp default ipv4-unicast
 bgp graceful-restart
 !
 neighbor MASTERS peer-group
 neighbor MASTERS remote-as 65020
 neighbor MASTERS description Kubernetes control plane
 neighbor MASTERS timers 10 30
 !
 neighbor WORKERS peer-group
 neighbor WORKERS remote-as 65030
 neighbor WORKERS description Kubernetes workers
 neighbor WORKERS timers 10 30
 !
 neighbor 192.168.10.17 peer-group MASTERS
 neighbor 192.168.10.18 peer-group MASTERS
 neighbor 192.168.10.19 peer-group MASTERS
 !
 neighbor 192.168.10.20 peer-group WORKERS
 neighbor 192.168.10.21 peer-group WORKERS
 neighbor 192.168.10.22 peer-group WORKERS
 neighbor 192.168.10.23 peer-group WORKERS
 neighbor 192.168.10.24 peer-group WORKERS
 !
 address-family ipv4 unicast
  neighbor MASTERS activate
  neighbor MASTERS soft-reconfiguration inbound
  neighbor MASTERS route-map FROM-CLUSTER in
  neighbor MASTERS route-map NOTHING-OUT out
  neighbor WORKERS activate
  neighbor WORKERS soft-reconfiguration inbound
  neighbor WORKERS route-map FROM-CLUSTER in
  neighbor WORKERS route-map NOTHING-OUT out
  maximum-paths 8
 exit-address-family
!
ip prefix-list SERVICES seq 10 permit 192.168.20.0/24 le 32
!
route-map FROM-CLUSTER permit 10
 match ip address prefix-list SERVICES
!
route-map NOTHING-OUT deny 10
!
line vty

Apply it:

sudo install -o frr -g frr -m 640 frr.conf /etc/frr/frr.conf
sudo vtysh -b

What everything here means

router bgp 65010 and bgp router-id — our AS and the router identifier.

no bgp default ipv4-unicast — neighbors don't get IPv4 route exchange automatically; each is enabled explicitly with activate in the address-family block. A safeguard against exchanging routes you don't expect.

peer-group MASTERS and WORKERS — neighbor groups. Masters (AS 65020, kube-vip) and workers (AS 65030, Calico) play different roles, and it's more convenient to apply policies to them separately rather than repeat the same lines eight times. The neighbors themselves are separate neighbor <address> peer-group ... lines. A new node is one line.

timers 10 30 — keepalive of 10 seconds and hold of 30 seconds instead of the standard 60/180. The hold timer is the maximum time it takes the router to detect a hard-dead neighbor (power cut, cable pulled) and remove its routes. 180 seconds is far too long for the API.

maximum-paths 8 — enables ECMP. The same prefix arrives from several nodes at once: the VIP from three masters, and later a service address from several workers. Without this line the router picks one route out of the equal ones and keeps the rest in reserve. It works because we use eBGP: with matching ASes we'd have to write maximum-paths ibgp.

soft-reconfiguration inbound — the router stores received routes before the filter. This matters for diagnostics (received-routes and filtered-routes, see below); without it these commands print nothing, and it's easy to conclude the neighbor isn't sending anything.

ip prefix-list SERVICES + route-map FROM-CLUSTER in — we accept only 192.168.20.0/24 and the subnets within it. This is a fuse: if Calico one day starts telling the router about the pod network 172.20.0.0/16 (and at some point it did happen to me), the router will drop those routes and they won't reach the kernel.

route-map NOTHING-OUT out — the router gives the cluster nothing. The nodes have a default route via the router, and they don't need anything more.

The bgp ebgp-requires-policy trap

By default (with frr defaults traditional), bgp ebgp-requires-policy is on: an eBGP neighbor without an inbound and outbound policy doesn't exchange routes at all. We meet this requirement (route-map ... in/out on both groups), but the flip side is this: a typo in a route-map name will cause all routes to be silently dropped, without a single error in the log. If show bgp is empty, look here first.

Firewall: the BGP port

BGP runs over TCP port 179. If, like me, your INPUT policy is DROP, it has to be opened for the neighbors:

sudo iptables -A INPUT -i eth1 -s 192.168.10.0/24 -p tcp --dport 179 -j ACCEPT

The rule must be kept among the persisted rules (iptables-persistent, netfilter-persistent save) — otherwise the sessions won't come up after the router reboots.

The least obvious part: hairpinning traffic within the node subnet

Now for the thing without which the whole setup simply doesn't work. If you skip this section, at first glance everything will work, but not a single connection with a payload will get through.

The nodes live in 192.168.10.0/24, and the VIP is in 192.168.20.0/24, which doesn't exist at the link layer. When a worker node contacts the VIP, it sends the packet, as it should, via the gateway — the router. The router finds 192.168.20.1/32 via a master in its routing table and sends the packet back out the same interface eth1.

The master receives the packet, builds a reply and sees that the destination (the worker) is in its own subnet. So it replies directly, bypassing the router.

As a result, the router sees only half of the conversation. Its conntrack creates an entry on the SYN, never sees the SYN-ACK and stays in the SYN_SENT state. The very next packet, now with data, hits a rule that exists in almost every firewall:

-A FORWARD -m conntrack --ctstate INVALID -j DROP

It looks like a mockery:

ping works;
the TCP handshake works (the reply bypasses the router and arrives anyway);
but the very first data packet disappears and is retransmitted forever;
in tcpdump on the router the SYN is seen twice (in and out), while the data packet is seen once.

The fix is a rule that allows the hairpin and sits before the INVALID check:

sudo iptables -I FORWARD 1 -i eth1 -o eth1 -d 192.168.20.0/24 -j ACCEPT

The rule is deliberately narrow: only the hairpin within the node subnet toward the service subnet. No other clients need it: their return path goes through the router, the traffic is symmetric, and conntrack works as usual.

Note that I wrote the rule for the whole 192.168.20.0/24 subnet, not for the single VIP address. The LoadBalancer addresses that will appear later end up in exactly the same situation, and then the rule won't need to be extended.

If the router's firewall is managed by some agent or rule generator (as it is in my case), add the rule through its mechanism for "custom" rules at the start of the FORWARD chain: otherwise it will disappear at the next generation.

Verification

While there are no cluster nodes yet, the neighbors will be in the Active state (the router tries to connect, but nobody is listening on the other side). That's normal for BGP.

sudo vtysh -c 'show bgp summary'

The important column here is State/PfxRcd: a number means the session is up and that many prefixes have been received; a word (Active, Connect, Idle) means there's no session.

Once the cluster is up, a few more commands will come in handy for checks, and I'll save them for later:

sudo vtysh -c 'show bgp ipv4 unicast'                # what was received
sudo vtysh -c 'show bgp ipv4 unicast 192.168.20.1'     # who announces a particular address
ip route show 192.168.20.1                             # what made it into the kernel
sudo vtysh -c 'show bgp ipv4 unicast neighbors 192.168.10.20 filtered-routes'   # what was dropped

Part 8 will have a complete cheat sheet, but for now the router is ready: in part 4 we'll bring up the VIP for the API server with kube-vip and initialize the first master.