Kubernetes for beginners. Part 17: Setting up a LoadBalancer without a cloud using MetalLB
All parts of the series “Kubernetes for beginners”In part 15 the LoadBalancer service was left with EXTERNAL-IP <pending>: kube-proxy did its part, but nobody was there to hand out an external address. Today we close that gap. We look at what the cloud controller does for us in a cloud and why on your own hardware someone else has to take over that role; install MetalLB, the most common load balancer for clusters without a cloud; give it the 192.168.10.200–192.168.10.220 pool and watch how in L2 mode one of the nodes starts answering ARP for the service address, how the address moves to another node, and at the end we switch MetalLB to BGP mode and look at the sessions and routes to the service address on a real router running FRR. The cluster DNS from the previous part is not needed today: the work happens at the level of ARP, routes and node addresses in the 192.168.10.0/24 network.
The demo namespace with the three-pod hello Deployment is rebuilt from part 11: two pods on k8s-node-2, one on k8s-node-1.
What a cloud load balancer does
type: LoadBalancer in Kubernetes is only a request. kube-apiserver itself allocates a ClusterIP and a NodePort for the service, kube-proxy sets up rules on the nodes, and the external address has to appear in status.loadBalancer.ingress thanks to some third party. In a cloud that is the cloud controller manager, a component that knows the cloud provider's API. When it sees a new LoadBalancer service, it:
- creates a load balancer at the provider and gets a public address from it;
- adds the cluster nodes (their addresses and the service NodePort) and a health check to the load balancer;
- writes the address into
status.loadBalancer.ingress; - watches for changes: nodes added — it updates the load balancer, service deleted — it deletes the load balancer.
The traffic itself goes through the provider's infrastructure: the address belongs to its load balancer, not to a node.
In a cluster on your own hardware there is no provider and no provider API. Someone has to, first, pick an address from the range the administrator set aside for services and, second, make the network deliver packets for that address to the cluster nodes. MetalLB does both.
MetalLB: controller and speaker
MetalLB is a CNCF project, an implementation of LoadBalancer services for clusters without a cloud. It consists of two components and a set of CRDs:
| Component | Kind | What it does |
|---|---|---|
controller |
Deployment, one replica | watches LoadBalancer services, hands out addresses from pools, writes them into status.loadBalancer.ingress; validates MetalLB resources through a webhook |
speaker |
DaemonSet, in the node network | announces the assigned addresses to the network: in L2 mode it answers ARP, in BGP mode it advertises routes to a router |
It is important to understand what MetalLB is not: it does not sit in the packet path. It does not proxy traffic and does not distribute it. Its job is to make a packet addressed to 192.168.10.200 reach one of the cluster nodes. After that the packet hits the kube-proxy rules: kube-proxy adds the address from status.loadBalancer.ingress to the same service-ips map as the ClusterIP, and DNAT sends the packet to a pod.
laptop 192.168.10.50 k8s-node-2 192.168.10.21
│ who has 192.168.10.200? (ARP) ──────► speaker: "me, MAC 52:54:00:3a:10:21"
│ TCP 192.168.10.200:80 ──────────────► ens18 → kube-proxy nftables:
│ service-ips: 192.168.10.200 . tcp . 80
│ → external-WRGIWGFD-demo/hello-lb/tcp/http
│ → DNAT to a hello pod (on any node)
◄──────────────────────── reply ───────┘
The address 192.168.10.200 is not on any interface, just like a ClusterIP: the speaker only answers ARP requests for it with its node's MAC address.
L2 mode
In L2 mode one node answers for each service address — the leader of that address. Its speaker answers ARP requests "who has 192.168.10.200?" with the MAC address of its ens18. ARP (RFC 826) is a broadcast question "who has this IP address?" within the local network and the owner's answer with its MAC address; the protocol is covered in part 7 of the networking series. The router and the clients in 192.168.10.0/24 put that MAC into their ARP tables and send packets for 192.168.10.200 to the leader node.
Leader election needs no voting and no storage. Every speaker knows which speakers are alive right now, and for each address computes the same thing: a list of eligible nodes sorted by a hash of the "node name + address" pair. The first in the list is the leader, and only its speaker answers ARP. Since everyone computes the same way, there is nothing to agree on. Different addresses get different leaders, so the load spreads across nodes per service, but the traffic of one address always goes through one node. A node that leaves the list does not change the leader of other addresses, and a node that comes back takes an address back only if it is first by hash again.
Only nodes where the following holds are eligible:
- a live speaker is running;
- there is no
node.kubernetes.io/exclude-from-external-load-balancerslabel — kubeadm puts it on the control plane, sok8s-master-1does not announce addresses (the speaker's--ignore-exclude-lbflag turns this off, but for the control plane this is how it should be); - the node matches
nodeSelectorsin the L2Advertisement, if they are set; - with
externalTrafficPolicy: Local— there is a ready pod of the service (more on this below).
Speakers learn who is alive through memberlist, a HashiCorp library implementing a gossip protocol: each member periodically probes random peers over UDP and TCP on port 7946, and news about a failed member spreads through the group within seconds. A speaker that stops cleanly tells the group itself that it is leaving, and then the address moves almost instantly. Without memberlist (it is turned off by removing the METALLB_ML_BIND_ADDR variable from the DaemonSet) the speakers would learn about a node failure only from the Kubernetes API, where the node status changes after 50 seconds or more (part 13). So port 7946 must be open between the nodes. The key that encrypts memberlist traffic is generated by the controller on first start and stored in the memberlist Secret in the metallb-system namespace.
Failover. When an address gets a new leader, the new leader sends gratuitous ARP — ARP packets nobody asked for — so that the router and the clients update the MAC address for 192.168.10.200 right away instead of waiting for their ARP cache to expire. On the lab the speaker sent them five times 1.1 seconds apart, each time in two forms: a broadcast ARP request where the sender and the target address are the same (such a packet is called an ARP Announcement in RFC 5227) and a broadcast ARP reply. Most systems accept such packets, and failover takes a few seconds; the MetalLB documentation treats failover slower than 10 seconds as a bug. Clients that ignore gratuitous ARP will keep sending packets to the old MAC until their cache expires.
During planned node maintenance (kubectl cordon, kubectl drain) the speaker keeps running: a DaemonSet is not evicted, and the address does not leave the node. To move addresses off a node in advance, put the node.kubernetes.io/exclude-from-external-load-balancers=true label on it — the speakers recompute the leaders, and the new leader announces itself while the old one is still alive. The documentation advises keeping the old node running for a couple more minutes after that, so it can serve clients with a stale cache.
Limitations of L2 mode to accept consciously:
- all incoming traffic of an address goes through one node. The service bandwidth is limited by the leader's network card. This is failover, not load balancing: distribution across pods is done by kube-proxy on the leader;
- all nodes and pool addresses are in one L2 segment. ARP does not cross routers, so the pool must be in the node subnet (
192.168.10.0/24for us), and nodes from another subnet cannot announce these addresses; - failover time depends on the clients and how they handle gratuitous ARP;
- some networks forbid "foreign" ARP replies. Virtual switches of hypervisors and clouds with MAC spoofing protection may drop replies for an IP address not bound to the virtual machine. Then it has to be allowed in the hypervisor settings.
A similar mechanism is VRRP (Virtual Router Redundancy Protocol, RFC 5798), which Keepalived — a Linux daemon for highly available addresses — is built on: several machines elect a master by exchanging VRRP messages, and the master holds the shared address. MetalLB does the same without VRRP — the choice is computed from the list of live speakers, so there is no limit of 255 groups and no virtual router IDs to hand out.
BGP mode
In BGP mode the speaker on every eligible node sets up a session with a router and advertises the service address as a /32 route via itself. BGP (RFC 4271) is the protocol routers use to exchange information about reachable networks (part 22 of the networking series). The router gets several equal routes to the service address — one via each node — and uses ECMP (Equal-Cost Multi-Path): it spreads traffic over all those nodes, picking a node by a hash of packet fields. By default Linux hashes only the source and destination addresses, so all connections of one client go through one node, and different clients spread across nodes; with ports added to the hash, even the connections of one client spread. This is load balancing at the network level: no single-node bottleneck. How fast failover is depends on how the node went away. A speaker that stops cleanly closes its TCP connection to the router, and the router removes the route via that node at once. When a node fails suddenly, the router waits for the session's hold time to expire — 90 seconds by default in MetalLB. BFD (RFC 5880) speeds this up: it is a protocol for fast liveness checks between routing neighbors, which exchange short packets every few tens or hundreds of milliseconds and declare the neighbor down after several misses. Pool addresses do not have to be in the node subnet, and it is better when they are outside it — why, you will see in "BGP mode on the lab".
The resources of this mode are BGPPeer and BGPAdvertisement (the full configuration, checked against a real router, is at the end of the article). BGPPeer describes the neighbor: the autonomous system numbers (an AS is a network under a single administration, the number a BGP speaker identifies itself with) of our side (myASN) and of the router (peerASN), and the router address. BGPAdvertisement says which pools to advertise and with which attributes: route aggregation, local preference (route preference inside our AS, which is why it is not passed to a neighbor in another AS), communities (tags the router uses to apply its policies to routes). The downside of ECMP: when the set of nodes changes (a node leaves or joins), the hash moves some connections to other nodes, and they break unless the router supports resilient hashing, where only the connections of the departed node are redistributed.
MetalLB can speak BGP in three ways. FRR-K8s is recommended and the default since 0.16: every node runs FRR (a free routing suite with daemons for BGP, BFD and more), managed by a separate controller through its own API; this brings BFD, and sessions can be shared with other FRR users on the node. native is MetalLB's own lightweight BGP implementation in the speaker, without BFD; it comes with the metallb-native.yaml manifest we install MetalLB with below. The FRR mode with FRR built into the speaker is deprecated and will be removed.
BGP next to Calico. Every node of the lab already runs BIRD, Calico's BGP daemon from part 14, in the node network, and it listens on BGP port 179. A speaker in native mode does not compete with it for the port: the speaker opens the connection to the router itself from a random port and does not listen on 179. So the router has to wait for connections from the nodes rather than call them itself — otherwise it would reach BIRD. The real conflict is a different one, and the MetalLB documentation warns about it: BGP allows only one session per "node — router" pair, and if Calico already talks to the same router, the router rejects MetalLB's session as a duplicate. On the lab BIRD exchanges routes only between the nodes and has no sessions with the router, so MetalLB brings up its own without trouble. In clusters where Calico already talks BGP to the router, the MetalLB documentation recommends leaving the advertisement of service addresses to Calico: MetalLB assigns the addresses (a pool without a BGPAdvertisement), Calico announces them. That is exactly how it is done in the series "Highly available Kubernetes on BGP".
Installation
We install MetalLB with the official manifest in native mode: L2 does not need FRR, and in this mode the speaker speaks BGP itself. The current version at the time of checking is 0.16.1. Best practice is not to apply the manifest straight from the internet but to download it, read it and keep it next to the other cluster manifests, so the version is pinned:
$ curl -fsSLO https://raw.githubusercontent.com/metallb/metallb/v0.16.1/config/manifests/metallb-native.yaml
$ grep -E '^kind:' metallb-native.yaml | sort | uniq -c
2 kind: ClusterRole
2 kind: ClusterRoleBinding
1 kind: ConfigMap
9 kind: CustomResourceDefinition
1 kind: DaemonSet
1 kind: Deployment
1 kind: Namespace
2 kind: Role
2 kind: RoleBinding
1 kind: Secret
1 kind: Service
2 kind: ServiceAccount
1 kind: ValidatingWebhookConfiguration
$ grep -n 'image:' metallb-native.yaml
2136: image: quay.io/metallb/controller:v0.16.1
2233: image: quay.io/metallb/speaker:v0.16.1
$ kubectl apply -f metallb-native.yaml
namespace/metallb-system created
customresourcedefinition.apiextensions.k8s.io/bfdprofiles.metallb.io created
customresourcedefinition.apiextensions.k8s.io/bgpadvertisements.metallb.io created
customresourcedefinition.apiextensions.k8s.io/bgppeers.metallb.io created
customresourcedefinition.apiextensions.k8s.io/communities.metallb.io created
customresourcedefinition.apiextensions.k8s.io/configurationstates.metallb.io created
customresourcedefinition.apiextensions.k8s.io/ipaddresspools.metallb.io created
customresourcedefinition.apiextensions.k8s.io/l2advertisements.metallb.io created
customresourcedefinition.apiextensions.k8s.io/servicebgpstatuses.metallb.io created
customresourcedefinition.apiextensions.k8s.io/servicel2statuses.metallb.io created
serviceaccount/controller created
serviceaccount/speaker created
role.rbac.authorization.k8s.io/controller created
role.rbac.authorization.k8s.io/pod-lister created
clusterrole.rbac.authorization.k8s.io/metallb-system:controller created
clusterrole.rbac.authorization.k8s.io/metallb-system:speaker created
rolebinding.rbac.authorization.k8s.io/controller created
rolebinding.rbac.authorization.k8s.io/pod-lister created
clusterrolebinding.rbac.authorization.k8s.io/metallb-system:controller created
clusterrolebinding.rbac.authorization.k8s.io/metallb-system:speaker created
configmap/metallb-excludel2 created
secret/metallb-webhook-cert created
service/metallb-webhook-service created
deployment.apps/controller created
daemonset.apps/speaker created
validatingwebhookconfiguration.admissionregistration.k8s.io/metallb-webhook-configuration created
For the first half minute the speaker pods sit in ContainerCreating with FailedMount events — MountVolume.SetUp failed for volume "memberlist" : secret "memberlist" not found: the Secret with the memberlist key is created by the controller once it starts. Another trap: kubectl wait -l app=metallb right after apply may not wait for the speakers — it takes the list of pods at start, and the DaemonSet creates them a moment later. So we wait with a separate command once the pods exist. On the lab the controller was ready after 34 seconds, the speakers after about a minute, image pulls included:
$ kubectl -n metallb-system wait --for=condition=Ready pod -l app=metallb --timeout=180s
pod/controller-658745d67-mlpqs condition met
pod/speaker-4ld9k condition met
pod/speaker-7zhm6 condition met
pod/speaker-lc7m7 condition met
$ kubectl -n metallb-system get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
controller-658745d67-mlpqs 1/1 Running 0 104s 172.20.172.26 k8s-node-2 <none> <none>
speaker-4ld9k 1/1 Running 0 103s 192.168.10.17 k8s-master-1 <none> <none>
speaker-7zhm6 1/1 Running 0 103s 192.168.10.21 k8s-node-2 <none> <none>
speaker-lc7m7 1/1 Running 0 103s 192.168.10.20 k8s-node-1 <none> <none>
$ kubectl get ns metallb-system --show-labels
NAME STATUS AGE LABELS
metallb-system Active 108s kubernetes.io/metadata.name=metallb-system,pod-security.kubernetes.io/audit=privileged,pod-security.kubernetes.io/enforce=privileged,pod-security.kubernetes.io/warn=privileged
What appeared:
- the
metallb-systemnamespace with Pod Security Admission labels at theprivilegedlevel (the mechanism that restricts what pods in a namespace may do, part 30): the speaker needs the node network and theNET_RAWcapability to send and receive ARP directly, and such rights are not acceptable in a namespace with ordinary applications; - CRDs: pools (
IPAddressPool), advertisements (L2Advertisement,BGPAdvertisement), BGP peers, communities and BFD profiles, plus the status resourcesServiceL2Status,ServiceBGPStatusandConfigurationState, where MetalLB reports who announces a service and whether the configuration has errors; - controller — an ordinary pod in the pod network, running as user
65534(nobody), with no Linux capabilities; speaker — one pod per node, the master included (it runs there but announces nothing because of the exclusion label), in the node network. For all its node network rights the speaker is reasonably restricted: a read-only container filesystem, privilege escalation forbidden, and the only Linux capability left isNET_RAW; - the webhook — the
metallb-webhook-serviceService and a ValidatingWebhookConfiguration: before saving a MetalLB resource, kube-apiserver sends it to the controller for validation, and a pool with bad addresses is rejected right away instead of being silently ignored. The controller issues the webhook certificate itself and writes its CA into the ValidatingWebhookConfiguration — no cert-manager needed for that.
The exclusion label has been on the master since kubeadm init (the mark-control-plane phase), with an empty value:
$ kubectl get node k8s-master-1 -o jsonpath='{.metadata.labels}'
{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/os":"linux","kubernetes.io/arch":"amd64","kubernetes.io/hostname":"k8s-master-1","kubernetes.io/os":"linux","node-role.kubernetes.io/control-plane":"","node.kubernetes.io/exclude-from-external-load-balancers":""}
The speaker listens for memberlist on the node address, and for metrics (over HTTPS since 0.16) and health checks on its own ports:
user@k8s-master-1:~$ sudo ss -lntup | grep speaker
udp UNCONN 0 0 192.168.10.17:7946 0.0.0.0:* users:(("speaker",pid=1119538,fd=9))
tcp LISTEN 0 4096 192.168.10.17:7946 0.0.0.0:* users:(("speaker",pid=1119538,fd=8))
tcp LISTEN 0 4096 127.0.0.1:17472 0.0.0.0:* users:(("speaker",pid=1119538,fd=15))
tcp LISTEN 0 4096 *:9120 *:* users:(("speaker",pid=1119538,fd=16))
kube-proxy needs special preparation for MetalLB only in IPVS mode: there all service addresses sit on the kube-ipvs0 interface, and the nodes themselves would start answering ARP for them, so the MetalLB documentation requires strictARP: true. That is an IPVS setting. In our nftables mode there are no service addresses on interfaces, and nothing needs changing — strictARP stays at its default:
$ kubectl -n kube-system get configmap kube-proxy -o jsonpath='{.data.config\.conf}' | grep -E '^mode:|strictARP'
strictARP: false
mode: nftables
Without configuration MetalLB does nothing. We give it a pool and allow it to announce the pool in L2 mode:
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: default
namespace: metallb-system
spec:
addresses:
- 192.168.10.200-192.168.10.220
autoAssign: true
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
name: default
namespace: metallb-system
spec:
ipAddressPools:
- default
interfaces:
- ens18
addresses— a range or a CIDR. These addresses must be excluded from the DHCP pool and not assigned manually to any machine: otherwise the DHCP server will one day hand192.168.10.205to a laptop, MetalLB will give it to a service, and two devices will answer ARP for one address. Before creating the pool, make sure once more that the addresses are free:pingandarpingto them must get no answer.autoAssign: true(the default) — hand out addresses from this pool to services that did not ask for a specific one. For a pool of "special" addresses it is set tofalse, and they are given out only on explicit request. TheavoidBuggyIPsfield (falseby default) skips addresses ending in.0and.255, which some old devices dislike.interfacesin the L2Advertisement — which interfaces to answer ARP on. Without it the speaker answers on all node interfaces except those listed in themetallb-excludel2ConfigMap (lo,cali*,docker*,kube-ipvs*and others).vxlan.calicois not in that list: on the lab the speaker created ARP responders on bothens18andvxlan.calicoat start. On nodes with several networks the field prevents answers going the wrong way.
$ kubectl apply -f metallb-config.yaml
ipaddresspool.metallb.io/default created
l2advertisement.metallb.io/default created
$ kubectl -n metallb-system get ipaddresspools,l2advertisements
NAME AUTO ASSIGN AVOID BUGGY IPS ADDRESSES
ipaddresspool.metallb.io/default true false ["192.168.10.200-192.168.10.220"]
NAME IPADDRESSPOOLS IPADDRESSPOOL SELECTORS INTERFACES
l2advertisement.metallb.io/default ["default"] ["ens18"]
In the pool's status the controller keeps the assignedIPv4 and availableIPv4 counters — how many addresses are assigned and how many are left. The webhook does not let a broken pool through:
$ kubectl apply -f bad-pool.yaml
Error from server (Forbidden): error when creating "bad-pool.yaml": admission webhook "ipaddresspoolvalidationwebhook.metallb.io" denied the request: parsing address pool bad: invalid CIDR "192.168.10.210-192.168.10.200" in pool "bad": invalid IP range "192.168.10.210-192.168.10.200": start IP "192.168.10.210" is after the end IP "192.168.10.200"
A pool overlapping an existing one is rejected the same way (… overlaps with already defined CIDR …).
A service for hello
apiVersion: v1
kind: Service
metadata:
name: hello-lb
namespace: demo
spec:
type: LoadBalancer
selector:
app.kubernetes.io/name: hello
ports:
- name: http
port: 80
targetPort: http
$ kubectl apply -f hello-lb.yaml && kubectl -n demo get svc hello-lb
service/hello-lb created
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hello-lb LoadBalancer 10.110.13.185 <pending> 80:30165/TCP 0s
$ kubectl -n demo get svc hello-lb
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hello-lb LoadBalancer 10.110.13.185 192.168.10.200 80:30165/TCP 4s
$ curl -s http://192.168.10.200/
hello from hello-6d5c6db6b4-k4mn2
The address is assigned within seconds, and hello answers from the laptop. The service events tell who did what:
$ kubectl -n demo describe svc hello-lb | sed -n '/Events:/,$p'
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal IPAllocated 4s metallb-controller Assigned IP ["192.168.10.200"]
Normal nodeAssigned 4s metallb-speaker announcing from node "k8s-node-2" with protocol "layer2"
$ kubectl -n demo get svc hello-lb -o jsonpath='{.status}'
{"loadBalancer":{"ingress":[{"ip":"192.168.10.200","ipMode":"VIP"}]}}
The controller assigned the address, and the speaker on k8s-node-2 became its leader. ipMode: VIP means that packets arrive at the node with the destination 192.168.10.200, and kube-proxy treats it as a virtual service address (which is why it is in the service-ips map). Cloud load balancers that proxy the traffic themselves set Proxy: then kube-proxy does not intercept packets to the load balancer address from inside the cluster, and they go through the load balancer.
A specific address, a shared address and allocation errors
If a service needs a particular address (a DNS record already points to it), it is requested with the metallb.io/loadBalancerIPs annotation. MetalLB still understands the old spec.loadBalancerIP field (on the lab a service with it got its address, and kube-apiserver 1.36 did not even warn), but Kubernetes deprecated it in 1.24. The pool to allocate from is chosen with the metallb.io/address-pool annotation. The metallb.io/allow-shared-ip annotation with the same key on two services lets them share one address — if their ports differ and both use the Cluster policy (or both select exactly the same pods):
apiVersion: v1
kind: Service
metadata:
name: hello-a
namespace: demo
annotations:
metallb.io/loadBalancerIPs: 192.168.10.210
metallb.io/allow-shared-ip: hello-shared
spec:
type: LoadBalancer
selector:
app.kubernetes.io/name: hello
ports:
- name: http
port: 8080
targetPort: http
The second service, hello-b, is the same with port 8081:
$ kubectl apply -f lb-extra.yaml
service/hello-a created
service/hello-b created
$ kubectl -n demo get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hello-a LoadBalancer 10.102.148.205 192.168.10.210 8080:30548/TCP 5s
hello-b LoadBalancer 10.109.27.181 192.168.10.210 8081:30273/TCP 5s
hello-lb LoadBalancer 10.110.13.185 192.168.10.200 80:30165/TCP 8m55s
$ curl -s http://192.168.10.210:8080/
hello from hello-6d5c6db6b4-vlh27
$ curl -s http://192.168.10.210:8081/
hello from hello-6d5c6db6b4-vlh27
Nobody answers on port 80 of 192.168.10.210 — curl times out: that port is not described in any service. If an address cannot be assigned, the service stays <pending>, and the controller emits a Warning AllocationFailed event. The texts on the lab:
- the pool is exhausted —
Failed to allocate IP for "demo/hello-c": no available IPs; - an address outside all pools was requested —
Failed to allocate IP for "demo/hello-d": ["192.168.10.230"] is not allowed in config; - a nonexistent pool was requested —
Failed to allocate IP for "demo/hello-e": unknown pool "special".
A service with spec.loadBalancerClass: example.com/other-lb is not touched by MetalLB at all: it hangs in <pending> without a single event. The field is for clusters with several LoadBalancer implementations: each handles only its own class, and MetalLB without the --lb-class flag handles only services without a class.
The hello-lb service is a training one: in the next part the address 192.168.10.200 goes to a reverse proxy that becomes the single entry point for all HTTP applications, and a separate address per application is bad practice once there are more than two applications.
externalTrafficPolicy: Local with MetalLB
With the default Cluster policy (from part 15) the leader accepts the traffic and spreads it over pods on all nodes, masquerading the client address with its own. The pod sees in its log not 192.168.10.50 but the leader node's address.
With externalTrafficPolicy: Local both kube-proxy and leader election change:
- kube-proxy on a node sends external traffic only to pods of that node and does not change the source address — the pod sees the real client address;
- MetalLB in L2 mode picks the leader only among nodes that have a ready pod of the service. If the pod on the leader node disappears, leadership moves to a node that has one.
The price in L2: all of the service's traffic is served by the pods of one node — the hello pods on k8s-node-1 stay idle while k8s-node-2 is the leader. For services where the client address matters (a reverse proxy with logs and rate limiting) this is acceptable, and the replicas on other nodes serve as a hot standby. In BGP mode with Local, only nodes with pods advertise the route, and the router spreads traffic between them — without double load balancing and with the client address preserved; it stays uneven if the nodes have different numbers of pods (the path weight on the router does not depend on the number of pods), so the pods of such a service should be spread across nodes (part 13). How this looks on the router — in "BGP mode on the lab".
The policy is turned on by patching the service. On the lab we did it after the leader failover experiments in "Hands-on", and for now the service stays with Cluster:
$ kubectl -n demo patch svc hello-lb -p '{"spec":{"externalTrafficPolicy":"Local"}}'
service/hello-lb patched
$ kubectl -n demo get svc hello-lb -o jsonpath='{.spec.healthCheckNodePort}{"\n"}'
31655
kube-apiserver allocated a healthCheckNodePort — the port on which kube-proxy on every node reports whether the node has ready pods of the service. MetalLB in L2 does not poll it, it learns about pods from EndpointSlices, but external load balancers use exactly this port. How it looks live — at the end of "Hands-on".
Alternatives
- kube-vip — a program that holds a virtual address for the control plane (that is how it is used in the BGP series) and can do the same for LoadBalancer services: addresses are assigned by its cloud controller, kube-vip-cloud-provider, and announced by kube-vip itself over ARP or BGP. Convenient if kube-vip is already there for the control plane.
- Cilium LB IPAM — in clusters with Cilium (a network plugin built on eBPF, programs loaded into the Linux kernel that process packets) LoadBalancer addresses are assigned by Cilium itself from
CiliumLoadBalancerIPPoolresources and announced with its own L2 announcements or BGP — no separate MetalLB needed. - Calico LoadBalancer IPAM — Calico kube-controllers assign addresses to services from an IP pool with
allowedUses: LoadBalancer, and Calico itself advertises them over BGP (theserviceLoadBalancerIPsfield in BGPConfiguration); together with a BGP router this solves the task without MetalLB.
Hands-on
Who is the leader for an address — MetalLB's status resource:
$ kubectl -n metallb-system get servicel2statuses
NAME ALLOCATED NODE SERVICE NAME SERVICE NAMESPACE
l2-2rvxt k8s-node-2 hello-lb demo
The resource is created by the leader speaker, and its owner (ownerReferences) is the speaker pod: kind: Pod, name: speaker-7zhm6. Its status holds the node and the interfaces the answers go from (interfaces: [{name: ens18}]).
ARP from the network. arping sends ARP requests for a given address and prints who answered; the nodes do not have it, so we run it in a debug pod in the network of node k8s-node-1 (kubectl debug node from part 15):
$ kubectl debug node/k8s-node-1 -n demo --profile=sysadmin --image=nicolaka/netshoot:v0.16 -- sleep 3600
Creating debugging pod node-debugger-k8s-node-1-d29qs with container debugger on node k8s-node-1.
$ kubectl -n demo exec node-debugger-k8s-node-1-d29qs -- arping -I ens18 -c 3 192.168.10.200
ARPING 192.168.10.200 from 192.168.10.20 ens18
Unicast reply from 192.168.10.200 [52:54:00:3A:10:21] 1.817ms
Unicast reply from 192.168.10.200 [52:54:00:3A:10:21] 0.870ms
Unicast reply from 192.168.10.200 [52:54:00:3A:10:21] 0.873ms
Sent 3 probes (1 broadcast(s))
Received 3 response(s)
$ kubectl -n demo exec node-debugger-k8s-node-1-d29qs -- arping -I ens18 -c 2 192.168.10.210
ARPING 192.168.10.210 from 192.168.10.20 ens18
Sent 2 probes (2 broadcast(s))
Received 0 response(s)
command terminated with exit code 1
52:54:00:3a:10:21 is the MAC of ens18 on k8s-node-2, the same as for 192.168.10.21. A pool address not assigned to anyone yet stays silent: the speaker answers only for assigned addresses. If there are answers from several MACs, two nodes consider themselves leaders (usually because port 7946 is closed between them) or the pool address is taken by someone else in the network. The same exchange on the leader:
user@k8s-node-2:~$ sudo timeout 8 tcpdump -eni ens18 -c 4 arp and host 192.168.10.200
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ens18, link-type EN10MB (Ethernet), snapshot length 262144 bytes
00:36:06.770359 52:54:00:3a:10:20 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.10.200 (ff:ff:ff:ff:ff:ff) tell 192.168.10.20, length 28
00:36:06.770494 52:54:00:3a:10:21 > 52:54:00:3a:10:20, ethertype ARP (0x0806), length 60: Reply 192.168.10.200 is-at 52:54:00:3a:10:21, length 46
00:36:07.770377 52:54:00:3a:10:20 > 52:54:00:3a:10:21, ethertype ARP (0x0806), length 42: Request who-has 192.168.10.200 (52:54:00:3a:10:21) tell 192.168.10.20, length 28
00:36:07.770483 52:54:00:3a:10:21 > 52:54:00:3a:10:20, ethertype ARP (0x0806), length 60: Reply 192.168.10.200 is-at 52:54:00:3a:10:21, length 46
4 packets captured
4 packets received by filter
0 packets dropped by kernel
The reply left 0.13 ms after the request. The first arping request is broadcast, the following ones go straight to the MAC it found. The reply is 60 bytes against 42 for the request: the reply is built not by the kernel but by the speaker itself, and it pads the frame to the minimum Ethernet size. The laptop's ARP table (it runs Windows, hence PowerShell):
PS> Get-NetNeighbor -IPAddress 192.168.10.20,192.168.10.21,192.168.10.200 | Format-Table -AutoSize
ifIndex IPAddress LinkLayerAddress State PolicyStore
------- --------- ---------------- ----- -----------
3 192.168.10.200 52-54-00-3A-10-21 Reachable ActiveStore
3 192.168.10.20 52-54-00-3A-10-20 Reachable ActiveStore
3 192.168.10.21 52-54-00-3A-10-21 Reachable ActiveStore
The service address and k8s-node-2 have the same MAC. The speaker log on k8s-node-2:
$ kubectl -n metallb-system logs speaker-7zhm6 | grep announc
{"caller":"announcer.go:126","event":"createARPResponder","interface":"ens18","level":"info","msg":"created ARP responder for interface","ts":"2026-10-28T21:32:36Z"}
…
{"caller":"announcer.go:126","event":"createARPResponder","interface":"vxlan.calico","level":"info","msg":"created ARP responder for interface","ts":"2026-10-28T21:32:36Z"}
…
{"caller":"main.go:481","event":"serviceAnnounced","ips":["192.168.10.200"],"level":"info","msg":"service has IP, announcing","pool":"default","protocol":"layer2","ts":"2026-10-28T21:34:06Z"}
ARP responders were created on both ens18 and vxlan.calico — this is what the interfaces field of the L2Advertisement is for. The speaker logs the ARP requests themselves only at the debug level. Let us turn it on for a while by replacing the --log-level=info argument in the DaemonSet (the speakers restart one by one, and meanwhile the addresses move between nodes):
$ kubectl -n metallb-system patch ds speaker --type=json -p '[{"op":"replace","path":"/spec/template/spec/containers/0/args/1","value":"--log-level=debug"}]'
daemonset.apps/speaker patched
$ kubectl -n metallb-system logs speaker-hjdtc | grep 'got ARP'
{"caller":"arp.go:110","interface":"ens18","ip":"192.168.10.200","level":"debug","msg":"got ARP request for service IP, sending response","responseMAC":"52:54:00:3a:10:20","senderIP":"192.168.10.17","senderMAC":"52:54:00:3a:10:17","ts":"2026-10-28T21:45:31Z"}
Here the leader is already k8s-node-1 (why — see below), and the master was asking. The debug level is not for permanent use: besides the ARP lines, on every reconciliation the speaker writes its whole configuration into the log — pools, advertisements and Secret metadata — several times a second.
The address in the kube-proxy rules on any node:
user@k8s-node-1:~$ sudo nft list map ip kube-proxy service-ips | grep hello-lb
10.110.13.185 . tcp . 80 : goto service-WRGIWGFD-demo/hello-lb/tcp/http,
192.168.10.200 . tcp . 80 : goto external-WRGIWGFD-demo/hello-lb/tcp/http,
A packet for 192.168.10.200:80 arriving at the leader's ens18 hits the same services chain as packets to the ClusterIP and goes through external-… to DNAT onto a pod. The address itself is absent from the interfaces:
user@k8s-node-2:~$ ip -br addr | grep -c 192.168.10.200
0
Failover when a speaker leaves. The service policy is still Cluster. On the laptop we start requests to the service every half second; on the master, tcpdump of ARP packets for the service address and arping from a debug pod in the master's network (the master itself will not become a leader). Then in another terminal we delete the speaker pod on the leader, k8s-node-2:
$ for i in $(seq 1 70); do echo "$(date -u +%T.%3N) $(curl -s -m1 http://192.168.10.200/ || echo FAIL)"; sleep 0.5; done > curl.log
$ kubectl -n metallb-system delete pod speaker-7zhm6
pod "speaker-7zhm6" deleted from metallb-system namespace
The pod was deleted at 21:37:45.96 UTC. On the master (node time is Moscow time):
user@k8s-master-1:~$ sudo tcpdump -tttt -eni ens18 arp and host 192.168.10.200
…
2026-10-29 00:37:45.540076 52:54:00:3a:10:17 > 52:54:00:3a:10:21, ethertype ARP (0x0806), length 42: Request who-has 192.168.10.200 (52:54:00:3a:10:21) tell 192.168.10.17, length 28
2026-10-29 00:37:45.540310 52:54:00:3a:10:21 > 52:54:00:3a:10:17, ethertype ARP (0x0806), length 60: Reply 192.168.10.200 is-at 52:54:00:3a:10:21, length 46
2026-10-29 00:37:46.251298 52:54:00:3a:10:20 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.10.200 (ff:ff:ff:ff:ff:ff) tell 192.168.10.200, length 46
2026-10-29 00:37:46.251298 52:54:00:3a:10:20 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Reply 192.168.10.200 is-at 52:54:00:3a:10:20, length 46
2026-10-29 00:37:46.540077 52:54:00:3a:10:17 > 52:54:00:3a:10:21, ethertype ARP (0x0806), length 42: Request who-has 192.168.10.200 (52:54:00:3a:10:21) tell 192.168.10.17, length 28
2026-10-29 00:37:47.351672 52:54:00:3a:10:20 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.10.200 (ff:ff:ff:ff:ff:ff) tell 192.168.10.200, length 46
2026-10-29 00:37:47.351702 52:54:00:3a:10:20 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Reply 192.168.10.200 is-at 52:54:00:3a:10:20, length 46
2026-10-29 00:37:47.540075 52:54:00:3a:10:17 > 52:54:00:3a:10:21, ethertype ARP (0x0806), length 42: Request who-has 192.168.10.200 (52:54:00:3a:10:21) tell 192.168.10.17, length 28
…
2026-10-29 00:37:51.540065 52:54:00:3a:10:17 > 52:54:00:3a:10:21, ethertype ARP (0x0806), length 42: Request who-has 192.168.10.200 (52:54:00:3a:10:21) tell 192.168.10.17, length 28
2026-10-29 00:37:51.781789 52:54:00:3a:10:21 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.10.200 (ff:ff:ff:ff:ff:ff) tell 192.168.10.200, length 46
2026-10-29 00:37:51.781789 52:54:00:3a:10:21 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Reply 192.168.10.200 is-at 52:54:00:3a:10:21, length 46
2026-10-29 00:37:52.540076 52:54:00:3a:10:17 > 52:54:00:3a:10:21, ethertype ARP (0x0806), length 42: Request who-has 192.168.10.200 (52:54:00:3a:10:21) tell 192.168.10.17, length 28
2026-10-29 00:37:52.540295 52:54:00:3a:10:21 > 52:54:00:3a:10:17, ethertype ARP (0x0806), length 60: Reply 192.168.10.200 is-at 52:54:00:3a:10:21, length 46
0.3 seconds after the pod was deleted, k8s-node-1 (MAC …:20) sent gratuitous ARP — "ARP Announcement + broadcast reply" pairs every 1.1 seconds. Five and a half seconds later a new speaker pod came up on k8s-node-2, and the address went back to it with the same packets from …:21: by hash, k8s-node-2 is first for this address. The service events and the speaker log on k8s-node-1:
$ kubectl -n demo get events --field-selector involvedObject.name=hello-lb
LAST SEEN TYPE REASON OBJECT MESSAGE
4m38s Normal IPAllocated service/hello-lb Assigned IP ["192.168.10.200"]
4m38s Normal nodeAssigned service/hello-lb announcing from node "k8s-node-2" with protocol "layer2"
58s Normal nodeAssigned service/hello-lb announcing from node "k8s-node-1" with protocol "layer2"
53s Normal nodeAssigned service/hello-lb announcing from node "k8s-node-2" with protocol "layer2"
$ kubectl -n metallb-system logs speaker-lc7m7 --since=5m | grep -E 'NodeLeave|NodeJoin|serviceAnnounced|serviceWithdrawn'
{"caller":"speakerlist.go:343","level":"info","msg":"node event - forcing sync","node addr":"192.168.10.21","node event":"NodeLeave","node name":"k8s-node-2","ts":"2026-10-28T21:37:46Z"}
{"caller":"main.go:481","event":"serviceAnnounced","ips":["192.168.10.200"],"level":"info","msg":"service has IP, announcing","pool":"default","protocol":"layer2","ts":"2026-10-28T21:37:46Z"}
{"caller":"speakerlist.go:343","level":"info","msg":"node event - forcing sync","node addr":"192.168.10.21","node event":"NodeJoin","node name":"k8s-node-2","ts":"2026-10-28T21:37:51Z"}
{"caller":"main.go:525","event":"serviceWithdrawn","ip":["192.168.10.200"],"ips":["192.168.10.200"],"level":"info","msg":"withdrawing service announcement","pool":"default","protocol":"layer2","reason":"notOwner","ts":"2026-10-28T21:37:51Z"}
NodeLeave is a clean departure from memberlist: the speaker being deleted managed to say goodbye. In curl.log, out of 70 requests from the laptop there is not a single FAIL: while k8s-node-2 is alive, its kube-proxy handles packets for 192.168.10.200 regardless of the speaker, so a client with the old MAC still gets an answer. The arping on the master, though, got no answer seven times out of 39 (Sent 39 probes (1 broadcast(s)), Received 32 response(s)): after the first reply it asks the old MAC …:21 directly, and without the speaker the k8s-node-2 kernel does not answer ARP for the service address — in nftables mode the address is not on the interface. This is what a client that missed the move looks like.
A speaker killed without a goodbye. If the speaker process is killed with kill -9, it has no time to say goodbye, and its peers find out through the memberlist protocol. On the lab, a second later the log on k8s-node-1 showed:
{"caller":"state.go:521","component":"Memberlist","level":"info","msg":"memberlist: Suspect k8s-node-2 has failed, no acks received","ts":"2026-10-28T21:46:36Z"}
But kubelet restarted the container two seconds later, the speaker rejoined the group before it was declared dead, and the address stayed on k8s-node-2 (which sent gratuitous ARP again). A whole-node failure — a powered-off machine or a broken network — was not reproduced on the lab; according to the documentation the address moves as soon as memberlist declares the node dead, within a few seconds. With the Cluster policy there is a subtlety: for a while the new leader keeps sending some connections to pods of the dead node — for Kubernetes those pods are still ready until the node lifecycle controller marks the node unreachable (about 50 seconds). Some requests in that window hang until a timeout. A readiness probe does not help: it is run by kubelet on the same dead node. The window can be shortened by lowering --node-monitor-grace-period, but the real cure is spare replicas and client-side retries.
Planned node maintenance. kubectl cordon k8s-node-2 does not change the leader: all 28 answers to arping -b (with -b it asks by broadcast every time and sees a new owner immediately) came from the MAC of k8s-node-2. Neither does drain: the speaker is not evicted, and with the Cluster policy the node stays the leader even when no hello pods are left on it:
$ kubectl drain k8s-node-2 --ignore-daemonsets --delete-emptydir-data
node/k8s-node-2 already cordoned
Warning: ignoring DaemonSet-managed Pods: calico-system/calico-node-7mx4w, calico-system/csi-node-driver-6nzck, kube-system/kube-proxy-xjhjp, metallb-system/speaker-sqk47
evicting pod metallb-system/controller-658745d67-mlpqs
evicting pod demo/hello-6d5c6db6b4-fgzhx
evicting pod demo/hello-6d5c6db6b4-k4mn2
pod/controller-658745d67-mlpqs evicted
pod/hello-6d5c6db6b4-fgzhx evicted
pod/hello-6d5c6db6b4-k4mn2 evicted
node/k8s-node-2 drained
One request from the laptop timed out at the moment the pods were evicted — it hit a terminating hello pod, MetalLB has nothing to do with it. The address can be moved away in advance with the exclusion label:
$ kubectl label node k8s-node-2 node.kubernetes.io/exclude-from-external-load-balancers=true
node/k8s-node-2 labeled
$ kubectl -n metallb-system get servicel2statuses
NAME ALLOCATED NODE SERVICE NAME SERVICE NAMESPACE
l2-hmxxn k8s-node-1 hello-lb demo
k8s-node-1 sent gratuitous ARP 0.1 seconds after the command, the laptop's ARP table switched the MAC within the same second, and not a single request was lost. After kubectl label node k8s-node-2 node.kubernetes.io/exclude-from-external-load-balancers- and kubectl uncordon k8s-node-2 the address went back to k8s-node-2.
Cluster versus Local. Now all three hello pods are on k8s-node-1 (drain evicted them there), and the leader is k8s-node-2. With the Cluster policy the pods see the address of the node the request came through:
$ kubectl -n demo logs -l app.kubernetes.io/name=hello --tail=2 --prefix
[pod/hello-6d5c6db6b4-vx7c4/hello] 2026/10/28 21:41:27 192.168.10.20:64880 GET /
[pod/hello-6d5c6db6b4-vx7c4/hello] 2026/10/28 21:41:29 192.168.10.20:57760 GET /
[pod/hello-6d5c6db6b4-8pzrk/hello] 2026/10/28 21:41:28 192.168.10.20:24832 GET /
[pod/hello-6d5c6db6b4-8pzrk/hello] 2026/10/28 21:41:42 192.168.10.21:62072 GET /
[pod/hello-6d5c6db6b4-vlh27/hello] 2026/10/28 21:41:42 192.168.10.21:61896 GET /
[pod/hello-6d5c6db6b4-vlh27/hello] 2026/10/28 21:41:42 192.168.10.21:1269 GET /
The earlier requests came through k8s-node-1 while it was the leader, the later ones through k8s-node-2; the laptop's address is nowhere. We switch the service to Local (the command is in the section above): k8s-node-2 has no pods, so leadership moves to k8s-node-1 within the same second, and the pods start seeing the real client address:
$ kubectl -n metallb-system get servicel2statuses
NAME ALLOCATED NODE SERVICE NAME SERVICE NAMESPACE
l2-2n2z2 k8s-node-1 hello-lb demo
$ kubectl -n demo logs -l app.kubernetes.io/name=hello --tail=1 --prefix
[pod/hello-6d5c6db6b4-vlh27/hello] 2026/10/28 21:42:12 192.168.10.50:60634 GET /
[pod/hello-6d5c6db6b4-vx7c4/hello] 2026/10/28 21:42:15 192.168.10.50:51651 GET /
[pod/hello-6d5c6db6b4-8pzrk/hello] 2026/10/28 21:42:14 192.168.10.50:51649 GET /
The speaker log on k8s-node-2 meanwhile shows serviceWithdrawn with the same reason, notOwner. The kube-proxy health check on healthCheckNodePort from the laptop:
$ curl -si http://192.168.10.20:31655/ | head -4
HTTP/1.1 200 OK
Content-Type: application/json
X-Content-Type-Options: nosniff
X-Load-Balancing-Endpoint-Weight: 3
$ curl -si http://192.168.10.21:31655/ | head -4
HTTP/1.1 503 Service Unavailable
Content-Type: application/json
X-Content-Type-Options: nosniff
X-Load-Balancing-Endpoint-Weight: 0
An external load balancer would use these answers to send traffic only to k8s-node-1, which has three local pods; MetalLB reaches the same conclusion on its own, from EndpointSlices.
Configuration errors. Overlapping pools and invalid ranges are stopped by the webhook. Whatever passes the webhook but cannot be applied, the controller and every speaker record in their own ConfigurationState resource — the first thing to look at when an address is not assigned or not announced:
$ kubectl -n metallb-system get configurationstates
NAME RESULT ERRORSUMMARY AGE
controller Valid 3m45s
speaker-k8s-master-1 Valid 11m
speaker-k8s-node-1 Valid 11m
speaker-k8s-node-2 Valid 6m16s
This output was taken right after creating an L2Advertisement referring to a nonexistent pool nope: the webhook accepted it, and MetalLB does not consider such a reference an error — the advertisement simply announces nothing. A typo in a pool name has to be found by eye.
We keep the training service until the next part and delete it when the reverse proxy needs the address: kubectl -n demo delete svc hello-lb; the hello-a and hello-b services go right away, and so do the debug pods. If MetalLB is no longer needed in the cluster, delete the configuration first (kubectl delete -f metallb-config.yaml), then the manifest itself (kubectl delete -f metallb-native.yaml): the CRDs, roles and the webhook go together with the namespace, and after that nobody in the network answers ARP for the pool addresses.
BGP mode on the lab
We checked BGP mode in a separate run, from a clean install of MetalLB with the same manifest and without an L2Advertisement, so that nobody answered ARP for the service addresses. The network's router, 192.168.10.1, is a Linux machine running FRR 8.4. BGP gets its own pool, 192.168.20.100–192.168.20.103, outside the node subnet: a client looks up an address in its own subnet with an ARP request, and in BGP mode nobody answers ARP, while for an address in another subnet the client goes through its gateway — the router — anyway. demo is rebuilt from part 11 again: two hello pods on k8s-node-1, one on k8s-node-2. MetalLB allows L2 and BGP at the same time for different pools, but we did not run them together on the lab.
The FRR router
The FRR configuration on the router (/etc/frr/frr.conf; the bgpd daemon is enabled with the bgpd=yes line in /etc/frr/daemons):
frr version 8.4.4
frr defaults traditional
!
router bgp 64513
bgp router-id 192.168.10.1
no bgp default ipv4-unicast
neighbor METALLB peer-group
neighbor METALLB remote-as 64512
neighbor METALLB passive
neighbor 192.168.10.17 peer-group METALLB
neighbor 192.168.10.20 peer-group METALLB
neighbor 192.168.10.21 peer-group METALLB
!
address-family ipv4 unicast
neighbor METALLB activate
neighbor METALLB route-map METALLB-IN in
neighbor METALLB route-map DENY-ALL out
exit-address-family
exit
!
ip prefix-list METALLB-IN seq 10 permit 192.168.20.100/30 ge 32
!
route-map METALLB-IN permit 10
match ip address prefix-list METALLB-IN
exit
!
route-map DENY-ALL deny 10
exit
router bgp 64513— the router's AS number, the verypeerASNin the BGPPeer; the cluster nodes come from AS 64512 (remote-as), so the sessions are external — eBGP (external BGP, between different ASes).peer-group METALLB— common settings for all three nodes: each node is a separate neighbor with its own session, the master included.passive— the router does not open the connection itself and only waits for incoming ones. This matters on nodes with Calico: port 179 there is taken by BIRD (you will see it inssbelow), and the router's attempt to connect to a node would reach BIRD, not the speaker.no bgp default ipv4-unicastandactivate— the IPv4 address family is enabled for the neighbors explicitly.- Filters. With
frr defaults traditional, FRR hasbgp ebgp-requires-policyenabled (an RFC 8212 requirement): without an inbound filter no routes from an eBGP neighbor are accepted, without an outbound one nothing is sent to it. The inboundroute-map METALLB-INlets through only/32routes from192.168.20.100/30— the nodes cannot advertise someone else's network to the router, not even by mistake. The outboundDENY-ALLdenies everything: the nodes do not need the router's routes, they have a default route.
MetalLB configuration
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: bgp
namespace: metallb-system
spec:
addresses:
- 192.168.20.100-192.168.20.103
autoAssign: true
---
apiVersion: metallb.io/v1beta2
kind: BGPPeer
metadata:
name: router
namespace: metallb-system
spec:
myASN: 64512
peerASN: 64513
peerAddress: 192.168.10.1
---
apiVersion: metallb.io/v1beta1
kind: BGPAdvertisement
metadata:
name: bgp
namespace: metallb-system
spec:
ipAddressPools:
- bgp
$ kubectl apply -f metallb-bgp.yaml
ipaddresspool.metallb.io/bgp created
bgppeer.metallb.io/router created
bgpadvertisement.metallb.io/bgp created
The first trap: in a cluster with Calico, kubectl get bgppeers shows the wrong thing:
$ kubectl -n metallb-system get bgppeers
No resources found
$ kubectl get bgppeers -v=6 2>&1 | grep -E "GET|No resources" | head -3
I1029 06:10:14.276583 1181365 round_trippers.go:632] "Response" verb="GET" url="https://k8s-api.example.com:6443/apis/projectcalico.org/v3/bgppeers?limit=500" status="200 OK" milliseconds=7
No resources found
$ kubectl -n metallb-system get bgppeers.metallb.io -o wide
NAME ADDRESS ASN BFD PROFILE MULTI HOPS
router 192.168.10.1 64513
The bgppeers resource exists in three API groups — MetalLB's (metallb.io), Calico's (crd.projectcalico.org) and the Calico API server's (projectcalico.org) — and kubectl resolved the short name to Calico. With MetalLB next to Calico, refer to bgppeers by the full name bgppeers.metallb.io.
Sessions
15 seconds after apply, on the router:
user@router:~$ sudo vtysh -c "show bgp summary"
IPv4 Unicast Summary (VRF default):
BGP router identifier 192.168.10.1, local AS number 64513 vrf-id 0
BGP table version 0
RIB entries 0, using 0 bytes of memory
Peers 3, using 2172 KiB of memory
Peer groups 1, using 64 bytes of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
192.168.10.17 4 64512 2 2 0 0 0 00:00:15 0 0 N/A
192.168.10.21 4 64512 2 2 0 0 0 00:00:15 0 0 N/A
192.168.10.20 4 64512 2 2 0 0 0 00:00:15 0 0 N/A
Total number of neighbors 3
All three sessions are up: a number in State/PfxRcd instead of a state word means Established, and the number is the count of accepted routes — zero while there are no services. Before apply this column showed Active with Up/Down never: the router was waiting for connections. Who calls whom — a capture on the router at the moment of apply (the router's time, like the nodes', is Moscow time):
user@router:~$ sudo timeout 40 tcpdump -nni eth1 -c 40 "tcp port 179 and host 192.168.10.20"
…
06:09:04.268224 IP 192.168.10.20.59761 > 192.168.10.1.179: Flags [S], seq 2857158583, win 64240, options [mss 1460,sackOK,TS val 821419571 ecr 0,nop,wscale 9], length 0
06:09:04.268253 IP 192.168.10.1.179 > 192.168.10.20.59761: Flags [S.], seq 3676335871, ack 2857158584, win 65160, options [mss 1460,sackOK,TS val 2375575134 ecr 821419571,nop,wscale 7], length 0
06:09:04.268665 IP 192.168.10.20.59761 > 192.168.10.1.179: Flags [.], ack 1, win 126, options [nop,nop,TS val 821419571 ecr 2375575134], length 0
06:09:04.268701 IP 192.168.10.20.59761 > 192.168.10.1.179: Flags [P.], seq 1:50, ack 1, win 126, options [nop,nop,TS val 821419571 ecr 2375575134], length 49: BGP
06:09:04.268729 IP 192.168.10.1.179 > 192.168.10.20.59761: Flags [.], ack 50, win 509, options [nop,nop,TS val 2375575135 ecr 821419571], length 0
06:09:04.269311 IP 192.168.10.1.179 > 192.168.10.20.59761: Flags [P.], seq 1:111, ack 50, win 509, options [nop,nop,TS val 2375575135 ecr 821419571], length 110: BGP
06:09:04.269517 IP 192.168.10.20.59761 > 192.168.10.1.179: Flags [.], ack 111, win 126, options [nop,nop,TS val 821419572 ecr 2375575135], length 0
06:09:04.269539 IP 192.168.10.1.179 > 192.168.10.20.59761: Flags [P.], seq 111:130, ack 50, win 509, options [nop,nop,TS val 2375575135 ecr 821419572], length 19: BGP
06:09:04.269663 IP 192.168.10.20.59761 > 192.168.10.1.179: Flags [P.], seq 50:69, ack 111, win 126, options [nop,nop,TS val 821419572 ecr 2375575135], length 19: BGP
…
06:09:34.270734 IP 192.168.10.20.59761 > 192.168.10.1.179: Flags [P.], seq 69:88, ack 130, win 126, options [nop,nop,TS val 821449573 ecr 2375590427], length 19: BGP
06:09:34.271455 IP 192.168.10.1.179 > 192.168.10.20.59761: Flags [P.], seq 130:149, ack 88, win 509, options [nop,nop,TS val 2375605137 ecr 821449573], length 19: BGP
…
The speaker on k8s-node-1 opens a TCP connection from the random port 59761 to the router's port 179. BGP runs over TCP, and its sequence follows: both neighbors send OPEN (49 bytes from the speaker, 110 from FRR, which lists more capabilities), then 19-byte KEEPALIVEs — the confirmation after which the session counts as established. Further KEEPALIVEs go every 30 seconds. On the node this connection belongs to the speaker process, and only BIRD listens on port 179:
user@k8s-node-1:~$ sudo ss -tnp | grep -E ":179\b"
ESTAB 0 0 192.168.10.20:42671 192.168.10.21:179 users:(("bird",pid=5097,fd=9))
ESTAB 0 0 192.168.10.20:179 192.168.10.17:43615 users:(("bird",pid=5097,fd=8))
ESTAB 0 0 192.168.10.20:59761 192.168.10.1:179 users:(("speaker",pid=716828,fd=19))
user@k8s-node-1:~$ sudo ss -lntp | grep -E "speaker|:179"
LISTEN 0 8 0.0.0.0:179 0.0.0.0:* users:(("bird",pid=5097,fd=7))
LISTEN 0 4096 127.0.0.1:17472 0.0.0.0:* users:(("speaker",pid=716828,fd=15))
LISTEN 0 4096 192.168.10.20:7946 0.0.0.0:* users:(("speaker",pid=716828,fd=8))
LISTEN 0 4096 *:9120 *:* users:(("speaker",pid=716828,fd=16))
The first two lines are BIRD's mesh between the nodes from part 14, the third is the speaker's session with the router. That is how MetalLB's BGP and Calico's BGP get along on one node: a speaker in native mode only calls out and does not take port 179, and the router, configured as passive, does not connect to the nodes. In the speaker log:
$ kubectl -n metallb-system logs speaker-dw7l2 --since=15m | grep -iE "bgp|session|peer"
…
{"caller":"bgp_controller.go:235","event":"peerAdded","level":"info","msg":"peer configured, starting BGP session","peer":"192.168.10.1","ts":"2026-10-29T03:09:04Z"}
{"caller":"native.go:123","event":"sessionUp","level":"info","localASN":64512,"msg":"BGP session established","peer":"192.168.10.1","peerASN":64513,"ts":"2026-10-29T03:09:04Z"}
…
The native manifest has no separate resource with session state — it is read from the speaker log and on the router. Session details on the FRR side:
user@router:~$ sudo vtysh -c "show bgp neighbors 192.168.10.20"
BGP neighbor is 192.168.10.20, remote AS 64512, local AS 64513, external link
…
Member of peer-group METALLB for session parameters
BGP version 4, remote router ID 192.168.10.20, local router ID 192.168.10.1
BGP state = Established, up for 00:00:48
Last read 00:00:18, Last write 00:00:18
Hold time is 90 seconds, keepalive interval is 30 seconds
Configured hold time is 180 seconds, keepalive interval is 60 seconds
…
Route map for incoming advertisements is *METALLB-IN
Route map for outgoing advertisements is *DENY-ALL
0 accepted prefixes
…
Local host: 192.168.10.1, Local port: 179
Foreign host: 192.168.10.20, Foreign port: 59761
…
Hold time is how long a neighbor waits for messages before declaring the session down. The sides propose their values in OPEN and take the smaller one: the speaker proposed 90 seconds (the MetalLB default), FRR 180, and they settled on 90 with KEEPALIVEs every 30 seconds. A node's router ID is its address.
The service and the routes
$ kubectl apply -f hello-lb.yaml && kubectl -n demo get svc hello-lb
service/hello-lb created
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hello-lb LoadBalancer 10.104.217.188 <pending> 80:30277/TCP 1s
$ kubectl -n demo get svc hello-lb
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hello-lb LoadBalancer 10.104.217.188 192.168.20.100 80:30277/TCP 5s
$ kubectl -n demo describe svc hello-lb | sed -n '/Events:/,$p'
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal IPAllocated 5s metallb-controller Assigned IP ["192.168.20.100"]
Normal nodeAssigned 4s metallb-speaker announcing from node "k8s-node-2" with protocol "bgp"
Normal nodeAssigned 4s metallb-speaker announcing from node "k8s-node-1" with protocol "bgp"
$ kubectl -n metallb-system get servicebgpstatuses -o wide
NAME NODE SERVICE NAME SERVICE NAMESPACE
bgp-xpjhn k8s-node-1 hello-lb demo
bgp-ztrkj k8s-node-2 hello-lb demo
In this run the bgp pool is the only one, and hello-lb (the same manifest as above) got its first address. Unlike L2, where one leader announces, here both workers announce the address at once: there are two nodeAssigned events, and each node has its own ServiceBGPStatus, owned by that node's speaker pod, with the names of the BGPPeers the route went to in status.peers ([router]). The master with the exclusion label keeps its session but does not announce the address. On the router:
user@router:~$ sudo vtysh -c "show bgp summary" -c "show bgp ipv4 unicast" -c "show ip route bgp"
…
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
192.168.10.17 4 64512 4 4 0 0 0 00:01:24 0 0 N/A
192.168.10.21 4 64512 5 4 0 0 0 00:01:24 1 0 N/A
192.168.10.20 4 64512 5 4 0 0 0 00:01:24 1 0 N/A
Total number of neighbors 3
BGP table version is 1, local router ID is 192.168.10.1, vrf id 0
Default local pref 100, local AS 64513
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*= 192.168.20.100/32 192.168.10.20 0 64512 i
*> 192.168.10.21 0 64512 i
Displayed 1 routes and 2 total paths
…
B>* 192.168.20.100/32 [20/0] via 192.168.10.21, eth1, weight 1, 00:00:05
* via 192.168.10.20, eth1, weight 1, 00:00:05
user@router:~$ ip route | grep -A4 192.168.20.10
192.168.20.100 nhid 30 proto bgp metric 20
nexthop via 192.168.10.21 dev eth1 weight 1
nexthop via 192.168.10.20 dev eth1 weight 1
…
One route, 192.168.20.100/32, arrived over two paths — via k8s-node-1 and k8s-node-2, with AS path 64512 and origin i. Both paths are equal: > marks the best one, = one taking part in multipath, and FRR installed the route into the routing table with two next hops ([20/0] is the eBGP administrative distance 20 and metric 0). maximum-paths is not set in the configuration; FRR enables ECMP for eBGP by itself. In the Linux kernel this is one route with two nexthops — this is what ECMP looks like. Requests from the router itself:
user@router:~$ for i in 1 2 3 4 5 6; do curl -s -m2 http://192.168.20.100/; done
hello from hello-6d5c6db6b4-5d5p4
hello from hello-6d5c6db6b4-x9lg9
hello from hello-6d5c6db6b4-x9lg9
hello from hello-6d5c6db6b4-5d5p4
hello from hello-6d5c6db6b4-x9lg9
hello from hello-6d5c6db6b4-x9lg9
user@router:~$ ip route get 192.168.20.100
192.168.20.100 via 192.168.10.20 dev eth1 src 192.168.10.1 uid 1000
cache
The answers come from pods on both nodes, but all six connections entered the cluster through one node, k8s-node-1; kube-proxy with the Cluster policy spread them over the pods. The router sends packets for 192.168.20.100 to the node's MAC address — by routing, with no ARP for the service address at all:
user@k8s-node-1:~$ sudo timeout 15 tcpdump -eni ens18 -c 3 "tcp port 80 and host 192.168.20.100"
…
06:16:29.422611 52:54:00:3a:10:01 > 52:54:00:3a:10:20, ethertype IPv4 (0x0800), length 74: 192.168.10.1.42288 > 192.168.20.100.80: Flags [S], seq 3527674648, win 64240, options [mss 1460,sackOK,TS val 1845417792 ecr 0,nop,wscale 7], length 0
06:16:29.422672 52:54:00:3a:10:20 > 52:54:00:3a:10:01, ethertype IPv4 (0x0800), length 74: 192.168.20.100.80 > 192.168.10.1.42288: Flags [S.], seq 2226412369, ack 3527674649, win 64308, options [mss 1410,sackOK,TS val 2203233316 ecr 1845417792,nop,wscale 9], length 0
06:16:29.423070 52:54:00:3a:10:01 > 52:54:00:3a:10:20, ethertype IPv4 (0x0800), length 66: 192.168.10.1.42288 > 192.168.20.100.80: Flags [.], ack 1, win 502, options [nop,nop,TS val 1845417792 ecr 2203233316], length 0
…
How the kernel picks a path. The ECMP hash in Linux is set by net.ipv4.fib_multipath_hash_policy, which is 0 by default — only the source and destination addresses, no ports. All connections of one client go through one node, and different clients spread across nodes:
user@router:~$ sysctl net.ipv4.fib_multipath_hash_policy
net.ipv4.fib_multipath_hash_policy = 0
user@router:~$ for p in 40001 40002 40003 40004; do ip route get 192.168.20.100 ipproto tcp sport $p dport 80 | head -1; done
192.168.20.100 via 192.168.10.20 dev eth1 src 192.168.10.1 uid 1000
192.168.20.100 via 192.168.10.20 dev eth1 src 192.168.10.1 uid 1000
192.168.20.100 via 192.168.10.20 dev eth1 src 192.168.10.1 uid 1000
192.168.20.100 via 192.168.10.20 dev eth1 src 192.168.10.1 uid 1000
user@router:~$ for s in 192.168.30.10 192.168.30.11 192.168.30.12 192.168.30.13; do ip route get 192.168.20.100 from $s iif eth2 2>&1 | head -1; done
192.168.20.100 from 192.168.30.10 via 192.168.10.20 dev eth1
192.168.20.100 from 192.168.30.11 via 192.168.10.21 dev eth1
192.168.20.100 from 192.168.30.12 via 192.168.10.21 dev eth1
192.168.20.100 from 192.168.30.13 via 192.168.10.21 dev eth1
Different source ports with one address give the same node; client addresses from the 192.168.30.0/24 network behind the router's eth2 interface (the command only looks up a route, it sends no packets) already give different nodes. The value 1 adds the protocol and ports to the hash, and then even the connections of one client spread across nodes; we did not reconfigure the router on the lab.
Advertisement attributes
Let us add a community and local preference to the BGPAdvertisement:
apiVersion: metallb.io/v1beta1
kind: BGPAdvertisement
metadata:
name: bgp
namespace: metallb-system
spec:
ipAddressPools:
- bgp
localPref: 200
communities:
- 64512:100
A community (RFC 1997) is an optional route attribute, four bytes written as AS:number; by itself it changes nothing, but the router can choose a policy by it (lower the priority, not pass the route on). What the router got — in the UPDATE (the BGP message with new and withdrawn routes) and in the table:
user@router:~$ sudo timeout 25 tcpdump -vv -nni eth1 -c 6 "tcp port 179 and host 192.168.10.20 and greater 80"
…
Update Message (2), length: 55
Origin (1), length: 1, Flags [T]: IGP
0x0000: 00
AS Path (2), length: 6, Flags [T]: 64512
0x0000: 0201 0000 fc00
Next Hop (3), length: 4, Flags [T]: 192.168.10.20
0x0000: c0a8 0a14
Community (8), length: 4, Flags [OT]: 64512:100
0x0000: fc00 0064
Updated routes:
192.168.20.100/32
…
user@router:~$ sudo vtysh -c "show bgp ipv4 unicast 192.168.20.100/32"
BGP routing table entry for 192.168.20.100/32, version 2
Paths: (2 available, best #2, table default)
Not advertised to any peer
64512
192.168.10.20 from 192.168.10.20 (192.168.10.20)
Origin IGP, valid, external, multipath
Community: 64512:100
Last update: Thu Oct 29 06:11:00 2026
64512
192.168.10.21 from 192.168.10.21 (192.168.10.21)
Origin IGP, valid, external, multipath, best (Older Path)
Community: 64512:100
Last update: Thu Oct 29 06:11:00 2026
The community arrived, but there is no local preference in the UPDATE: this attribute is passed only inside one's own AS (iBGP), and MetalLB does not send it to a neighbor in another AS. localPref in a BGPAdvertisement makes sense when the router is in the same AS as the cluster.
aggregationLength rolls the service addresses up into one shorter prefix. Let us set aggregationLength: 30:
user@router:~$ sudo timeout 20 tcpdump -vv -nni eth1 -c 4 "tcp port 179 and src host 192.168.10.20 and greater 80"
…
Update Message (2), length: 55
Origin (1), length: 1, Flags [T]: IGP
0x0000: 00
AS Path (2), length: 6, Flags [T]: 64512
0x0000: 0201 0000 fc00
Next Hop (3), length: 4, Flags [T]: 192.168.10.20
0x0000: c0a8 0a14
Community (8), length: 4, Flags [OT]: 64512:100
0x0000: fc00 0064
Updated routes:
192.168.20.100/30
…
Update Message (2), length: 28
Withdrawn routes:
192.168.20.100/32
End-of-Rib Marker (empty NLRI)
…
Eight seconds after the change, on the router:
user@router:~$ sudo vtysh -c "show bgp summary" -c "show bgp ipv4 unicast" -c "show ip route bgp"
IPv4 Unicast Summary (VRF default):
BGP router identifier 192.168.10.1, local AS number 64513 vrf-id 0
BGP table version 3
RIB entries 0, using 0 bytes of memory
Peers 3, using 2172 KiB of memory
Peer groups 1, using 64 bytes of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
192.168.10.17 4 64512 7 7 0 0 0 00:02:45 0 0 N/A
192.168.10.21 4 64512 11 7 0 0 0 00:02:45 0 0 N/A
192.168.10.20 4 64512 11 7 0 0 0 00:02:45 0 0 N/A
Total number of neighbors 3
No BGP prefixes displayed, 0 exist
show ip route bgp printed nothing. The speaker advertised 192.168.20.100/30 and withdrew the /32, and the router did not accept the /30: the inbound filter lets through only /32 (ge 32), and no route to the service was left. That is the router-side protection at work. We return the BGPAdvertisement without attributes (kubectl apply -f metallb-bgp.yaml), and the /32 route with two paths comes back.
externalTrafficPolicy: Local in BGP mode
With the Cluster policy all requests from the router entered through k8s-node-1, and the pods saw its address — including the pod on k8s-node-2, to which kube-proxy forwarded the request from k8s-node-1:
$ kubectl -n demo logs -l app.kubernetes.io/name=hello --tail=2 --prefix
[pod/hello-6d5c6db6b4-wl6kn/hello] 2026/10/29 03:12:23 192.168.10.20:41369 GET /
[pod/hello-6d5c6db6b4-wl6kn/hello] 2026/10/29 03:12:23 192.168.10.20:38250 GET /
[pod/hello-6d5c6db6b4-x9lg9/hello] 2026/10/29 03:10:35 192.168.10.20:37347 GET /
[pod/hello-6d5c6db6b4-x9lg9/hello] 2026/10/29 03:12:23 192.168.10.20:27460 GET /
[pod/hello-6d5c6db6b4-5d5p4/hello] 2026/10/29 03:10:35 192.168.10.20:31866 GET /
[pod/hello-6d5c6db6b4-5d5p4/hello] 2026/10/29 03:12:23 192.168.10.20:61935 GET /
After kubectl -n demo patch svc hello-lb -p '{"spec":{"externalTrafficPolicy":"Local"}}' both nodes keep announcing the address — both have pods — and the pods see the client address, the router's 192.168.10.1. Only the pods on k8s-node-1 get requests: the hash sends all connections from the router there, and with Local kube-proxy does not forward them to other nodes:
$ kubectl -n demo logs -l app.kubernetes.io/name=hello --tail=1 --prefix
[pod/hello-6d5c6db6b4-5d5p4/hello] 2026/10/29 03:12:27 192.168.10.1:37826 GET /
[pod/hello-6d5c6db6b4-wl6kn/hello] 2026/10/29 03:12:27 192.168.10.1:37814 GET /
[pod/hello-6d5c6db6b4-x9lg9/hello] 2026/10/29 03:12:23 192.168.10.20:27460 GET /
Let us scale the Deployment down to one pod — it stayed on k8s-node-2, and k8s-node-1 withdraws the route:
$ kubectl -n demo scale deploy hello --replicas=1
deployment.apps/hello scaled
$ kubectl -n demo get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
hello-6d5c6db6b4-x9lg9 1/1 Running 0 5m46s 172.20.172.29 k8s-node-2 <none> <none>
$ kubectl -n metallb-system get servicebgpstatuses
NAME NODE SERVICE NAME SERVICE NAMESPACE
bgp-ztrkj k8s-node-2 hello-lb demo
$ kubectl -n metallb-system logs speaker-dw7l2 --since=4m | grep serviceWithdrawn
{"caller":"main.go:525","event":"serviceWithdrawn","ip":["192.168.20.100"],"ips":["192.168.20.100"],"level":"info","msg":"withdrawing service announcement","pool":"bgp","protocol":"bgp","reason":"noLocalEndpoints","ts":"2026-10-29T03:12:43Z"}
user@router:~$ sudo vtysh -c "show bgp summary" | tail -6; sudo vtysh -c "show ip route bgp" | tail -2
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
192.168.10.17 4 64512 9 9 0 0 0 00:03:41 0 0 N/A
192.168.10.21 4 64512 15 9 0 0 0 00:03:41 1 0 N/A
192.168.10.20 4 64512 16 9 0 0 0 00:03:41 0 0 N/A
Total number of neighbors 3
B>* 192.168.20.100/32 [20/0] via 192.168.10.21, eth1, weight 1, 00:00:02
The withdrawal reason is noLocalEndpoints (in L2 the same situation gave notOwner). The session of k8s-node-1 stayed up, it just carries zero routes. The path weight does not depend on the number of pods: with two pods on k8s-node-1 and one on k8s-node-2 both paths had weight 1. We bring back three replicas and the Cluster policy.
A speaker leaves, a node is excluded
Let us delete the speaker pod on k8s-node-1 — the very node the router's requests go through — while the router sends requests to the service every half second and captures BGP with that node:
$ kubectl -n metallb-system delete pod speaker-dw7l2
pod "speaker-dw7l2" deleted from metallb-system namespace
user@router:~$ sudo timeout 40 tcpdump -tttt -vv -nni eth1 -c 30 "tcp port 179 and host 192.168.10.20"
…
2026-10-29 06:13:26.611327 IP (tos 0x0, ttl 64, id 36076, offset 0, flags [DF], proto TCP (6), length 52)
192.168.10.20.59761 > 192.168.10.1.179: Flags [F.], cksum 0x5c00 (correct), seq 2857159142, ack 3676336153, win 126, options [nop,nop,TS val 821681914 ecr 2375830555], length 0
2026-10-29 06:13:26.622212 IP (tos 0xc0, ttl 1, id 24389, offset 0, flags [DF], proto TCP (6), length 52)
192.168.10.1.179 > 192.168.10.20.59761: Flags [F.], cksum 0x8726 (incorrect -> 0x3f6b), seq 1, ack 1, win 509, options [nop,nop,TS val 2375837488 ecr 821681914], length 0
2026-10-29 06:13:26.622882 IP (tos 0x0, ttl 64, id 36077, offset 0, flags [DF], proto TCP (6), length 52)
192.168.10.20.59761 > 192.168.10.1.179: Flags [.], cksum 0x40de (correct), seq 1, ack 2, win 126, options [nop,nop,TS val 821681926 ecr 2375837488], length 0
2026-10-29 06:13:31.100566 IP (tos 0x0, ttl 64, id 59586, offset 0, flags [DF], proto TCP (6), length 60)
192.168.10.20.59749 > 192.168.10.1.179: Flags [S], cksum 0x2431 (correct), seq 1175485718, win 64240, options [mss 1460,sackOK,TS val 2739229015 ecr 0,nop,wscale 9], length 0
…
Updated routes:
192.168.20.100/32
…
The speaker being deleted closed the TCP connection (F. is FIN) without a BGP NOTIFICATION message, and the router removed the path via k8s-node-1 at once, without waiting for the hold time. 4.5 seconds later the new speaker pod on the same node opened a new connection from another port and advertised the route again. The router state three seconds after the deletion and fifteen seconds after it:
user@router:~$ date -u +%T; sudo vtysh -c "show bgp summary" | tail -6; sudo vtysh -c "show ip route bgp" | tail -2
03:13:29
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
192.168.10.17 4 64512 10 10 0 0 0 00:04:25 0 0 N/A
192.168.10.21 4 64512 16 10 0 0 0 00:04:25 1 0 N/A
192.168.10.20 4 64512 18 10 0 0 0 00:00:03 Active 0 N/A
Total number of neighbors 3
B>* 192.168.20.100/32 [20/0] via 192.168.10.21, eth1, weight 1, 00:00:03
user@router:~$ date -u +%T; sudo vtysh -c "show bgp summary" | tail -6; sudo vtysh -c "show ip route bgp" | tail -2
03:13:41
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
192.168.10.17 4 64512 11 11 0 0 0 00:04:37 0 0 N/A
192.168.10.21 4 64512 17 11 0 0 0 00:04:37 1 0 N/A
192.168.10.20 4 64512 21 12 0 0 0 00:00:11 1 0 N/A
Total number of neighbors 3
B>* 192.168.20.100/32 [20/0] via 192.168.10.21, eth1, weight 1, 00:00:10
* via 192.168.10.20, eth1, weight 1, 00:00:10
Out of 60 requests from the router during this time not a single one failed: the connections the hash now sent to k8s-node-2 were served by its kube-proxy, and connections through k8s-node-1 are served by that node's kube-proxy even without a speaker. A whole-node failure was not reproduced on the lab: then nobody sends a FIN, and the router removes the path only when the hold time expires — up to 90 seconds with MetalLB defaults, and all that time some connections keep going to the dead node.
The exclusion label takes the address off a node without touching the session:
$ kubectl label node k8s-node-2 node.kubernetes.io/exclude-from-external-load-balancers=true
node/k8s-node-2 labeled
$ kubectl -n metallb-system get servicebgpstatuses
NAME NODE SERVICE NAME SERVICE NAMESPACE
bgp-vhz7r k8s-node-1 hello-lb demo
$ kubectl -n metallb-system logs -l component=speaker --since=30s --prefix | grep -E "serviceWithdrawn|nodeLabels"
[pod/speaker-s8swn/speaker] {"caller":"main.go:525","event":"serviceWithdrawn","ip":["192.168.20.100"],"ips":["192.168.20.100"],"level":"info","msg":"withdrawing service announcement","pool":"bgp","protocol":"bgp","reason":"nodeLabeledExcludeBalancers","ts":"2026-10-29T03:14:34Z"}
user@router:~$ sudo timeout 15 tcpdump -tttt -vv -nni eth1 -c 3 "tcp port 179 and src host 192.168.10.21 and greater 60"
…
2026-10-29 06:14:34.129944 IP (tos 0x0, ttl 64, id 59604, offset 0, flags [DF], proto TCP (6), length 80)
192.168.10.21.50677 > 192.168.10.1.179: Flags [P.], cksum 0x7dbb (correct), seq 353385230:353385258, ack 3835031817, win 126, options [nop,nop,TS val 310078514 ecr 3639304966], length 28: BGP
Update Message (2), length: 28
Withdrawn routes:
192.168.20.100/32
End-of-Rib Marker (empty NLRI)
…
user@router:~$ sudo vtysh -c "show bgp summary" | tail -6; sudo vtysh -c "show ip route bgp" | tail -2
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
192.168.10.17 4 64512 13 13 0 0 0 00:05:33 0 0 N/A
192.168.10.21 4 64512 20 13 0 0 0 00:05:33 0 0 N/A
192.168.10.20 4 64512 23 14 0 0 0 00:01:07 1 0 N/A
Total number of neighbors 3
B>* 192.168.20.100/32 [20/0] via 192.168.10.20, eth1, weight 1, 00:00:03
The speaker on k8s-node-2 sent an UPDATE with the withdrawn route, and the session stayed Established with zero routes — like the master's. This is planned node maintenance in BGP mode: the router stops sending new connections to the node while the node is still working. After removing the label (kubectl label node k8s-node-2 node.kubernetes.io/exclude-from-external-load-balancers-) the path via k8s-node-2 came back.
BFD in native mode
BFD could speed up detecting a node failure: a BFDProfile and a reference to it in the BGPPeer (bfdProfile). The webhook accepts such a configuration, but speakers in native mode reject it:
$ kubectl apply -f bfd.yaml
bfdprofile.metallb.io/fast created
bgppeer.metallb.io/router configured
$ kubectl -n metallb-system get configurationstates
NAME RESULT ERRORSUMMARY AGE
controller Valid 8m9s
speaker-k8s-master-1 Invalid configuration error: bfd profiles section set 7m6s
speaker-k8s-node-1 Invalid configuration error: bfd profiles section set 2m13s
speaker-k8s-node-2 Invalid configuration error: bfd profiles section set 7m12s
A second later the speakers' errorSummary changed to configuration error: peer 192.168.10.1 has bfd-profile set on native bgp mode. The sessions and routes did not suffer — the speakers kept working with the previous configuration — but no BFD appeared. BFD is available only in the modes with FRR (FRR-K8s and the deprecated FRR mode), and the router needs the bfdd daemon as well. Without BFD what remains is lowering holdTime and keepaliveTime in the BGPPeer.
Cleanup
After the experiments we deleted demo, the BGP configuration (kubectl delete -f metallb-bgp.yaml) and MetalLB itself on the lab. The router is back to waiting for connections and knows no routes to services:
user@router:~$ sudo vtysh -c "show bgp summary" -c "show ip route bgp" -c "show bgp ipv4 unicast"
…
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
192.168.10.17 4 64512 17 17 0 0 0 00:00:20 Active 0 N/A
192.168.10.21 4 64512 26 17 0 0 0 00:00:20 Active 0 N/A
192.168.10.20 4 64512 29 19 0 0 0 00:00:20 Active 0 N/A
Total number of neighbors 3
No BGP prefixes displayed, 0 exist
Sources
- MetalLB: Concepts — controller and speaker, address assignment and announcement.
- MetalLB: Layer 2 mode — leader election by hash, memberlist, gratuitous ARP, failover time and limitations, comparison with Keepalived and VRRP.
- MetalLB: BGP mode — ECMP, connections breaking when the node set changes, the native, FRR and FRR-K8s modes.
- MetalLB: Installation — installing with a manifest,
strictARPfor IPVS, choosing the BGP implementation,--lb-class. - MetalLB: Configuration — IPAddressPool and L2Advertisement; Advanced L2 configuration —
nodeSelectorsandinterfaces; Advanced IPAddressPool configuration —autoAssign,avoidBuggyIPs, pool selection; Advanced BGP configuration —localPref, communities,aggregationLength, selecting nodes and peers for an advertisement. - MetalLB: Usage — the
metallb.io/loadBalancerIPs,address-pool,allow-shared-ipannotations,externalTrafficPolicyin L2 and BGP modes. - MetalLB: Issues with Calico — one BGP session per pair of neighbors when Calico and MetalLB talk to the same router, and advertising service addresses through Calico.
- MetalLB: Troubleshooting —
arping, several MACs for one address, thedebuglog level. - MetalLB: Release notes — version 0.16.1, FRR-K8s as the default since 0.16.0, ConfigurationState since 0.15.3.
- FRR-K8s — the FRR controller for Kubernetes that MetalLB's BGP runs on by default.
- FRR: BGP —
router bgp, peer groups, address families, a route map on a neighbor,bgp ebgp-requires-policy,maximum-paths(FRR 8.4 documentation); IP Prefix Lists and Route Maps — filtering accepted routes on the router. - Linux: IP Sysctl —
fib_multipath_hash_policy, which fields the kernel uses to pick an ECMP path. - hashicorp/memberlist — the group membership and failure detection library.
- Kubernetes: Service, type LoadBalancer —
loadBalancerClass, the deprecation ofspec.loadBalancerIP,ipMode. - Kubernetes: Cloud Controller Manager — what the cloud controller does, including for load balancers.
- Kubernetes: Create an External Load Balancer — preserving the client address and
healthCheckNodePort. - Kubernetes: Well-Known Labels, Annotations and Taints — the
node.kubernetes.io/exclude-from-external-load-balancerslabel. - Kubernetes: Virtual IPs and Service Proxies — kube-proxy in nftables and IPVS modes.
- Calico: Advertise Kubernetes service IP addresses and LoadBalancer IP address management — advertising service addresses over BGP and
allowedUses: LoadBalancerpools. - kube-vip: Cloud Provider and Cilium: LB IPAM — alternatives to MetalLB.
- RFC 826 — ARP; RFC 5227 — ARP Announcement; RFC 4271 — BGP-4; RFC 1997 — communities; RFC 8212 — eBGP without policies accepts and sends nothing; RFC 5880 — BFD; RFC 5798 — VRRP version 3.
Next — Ingress and Gateway API: one address, 192.168.10.200, for all HTTP applications, the Traefik reverse proxy that routes requests to services by host name and path, and two APIs for describing routes — the familiar Ingress and the newer Gateway API.
Only registered users can leave a comment.
Sign in or register to leave a comment.