4 October 2026, 18:50Виталий Лещенко

Highly available Kubernetes on BGP. Part 6: LoadBalancer addresses from Calico over BGP

All parts of the series “Highly available Kubernetes on BGP”

The cluster works, and the API is highly available. Now let's teach Calico to assign addresses to type: LoadBalancer services and announce them to the router. No separate MetalLB is needed: Calico can do both.

What we're going to do

Tell Calico its AS (65030) and turn off the full BGP mesh between nodes.
Set up BGP sessions from the workers to the router. The masters don't take part in this.
Create an address pool from which addresses can only be assigned to services.
Enable BGP in the Installation.
Test it with a test service.

BGPConfiguration

apiVersion: projectcalico.org/v3
kind: BGPConfiguration
metadata:
  name: default
spec:
  asNumber: 65030
  logSeverityScreen: Info
  nodeToNodeMeshEnabled: false
  serviceLoadBalancerAggregation: Disabled
  serviceLoadBalancerIPs:
    - cidr: 192.168.20.64/26

Let's go through the fields.

nodeToNodeMeshEnabled: false. By default, when BGP is enabled, Calico builds a full mesh: every node would peer with every other. That makes no sense for us: pod traffic goes over VXLAN, and BGP is needed only to tell the router about service addresses. Enabling BGP without this setting means getting a mesh of eight nodes. Apply the configuration before enabling BGP.

asNumber: 65030. The default is 64512, so we set it explicitly: the router is 65010, the masters (kube-vip) 65020, the workers (Calico) 65030.

serviceLoadBalancerIPs. Which addresses are allowed to be announced at all. Matches the pool.

serviceLoadBalancerAggregation: Disabled. The most important parameter; let's dwell on it.

Why disable aggregation

When aggregation is enabled (the default), Calico announces the whole pool block (192.168.20.64/26) from all nodes at once, regardless of where the service's endpoints live. The router builds ECMP across all workers. For services with externalTrafficPolicy: Cluster this is acceptable (any node will forward the packet further), but for Local it's a disaster: a packet arriving at a node without a service pod is simply lost.

With Disabled, Calico announces a separate /32 for each address, and for services with Local — only from the nodes that have a ready endpoint (Calico looks at the ready addresses in Endpoints). This protects against three things:

A pod crashed or fails readiness. The session is alive, but the route from the node is withdrawn by itself. With aggregation enabled, the router would keep sending some new connections into nowhere.
A new worker node. calico-node brings up the session before the pod has pulled its image and become ready; with aggregation, the node would immediately join ECMP empty.
A service with one or two replicas. With aggregation enabled it would break right away.

The cost is one /32 per node, that is, nothing. The setting is global.

BGPPeer: workers only

apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: router
spec:
  nodeSelector: "!has(node-role.kubernetes.io/control-plane)"
  peerIP: 192.168.10.1
  asNumber: 65010

nodeSelector excludes the masters on purpose. On the masters, kube-vip already holds the session with the router, and two BGP daemons on one node can't both keep a session with the same neighbor: a session is identified by a pair of addresses, and the router will accept only one from 192.168.10.x. As a result, BIRD (Calico's BGP daemon) runs on the masters but has no peers at all and announces nothing. The correct answer from calicoctl node status on a master is No IPv4 peers found.

Calico's own AS is set globally in BGPConfiguration; here it's the neighbor's AS.

The address pool for services

apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
  name: loadbalancer-pool
spec:
  cidr: 192.168.20.64/26
  allowedUses:
    - LoadBalancer
  disableBGPExport: false
  natOutgoing: false
  nodeSelector: "all()"

allowedUses: [LoadBalancer] — an address from this pool can't be given to a pod, only to a service.
nodeSelector: all() — otherwise Calico rejects the pool with the message "IP Pool with AllowedUse LoadBalancer must have node selector set to all()". For a service pool the selector doesn't mean anything anyway: who announces an address is decided by where the endpoints are, not by the pool.
The pool starts at .64 and doesn't overlap with the API VIP (192.168.20.1).

Enabling BGP

kubectl apply -f bgpconfiguration.yaml
kubectl apply -f bgppeer.yaml
kubectl apply -f loadbalancer-ippool.yaml
kubectl patch installation default --type=merge -p '{"spec":{"calicoNetwork":{"bgp":"Enabled"}}}'

The order matters: BGPConfiguration first, then enabling. The encapsulation of pod traffic doesn't change — VXLAN CrossSubnet stays.

Check from a worker:

kubectl -n calico-system exec ds/calico-node -c calico-node -- birdcl show protocols

The output should show a protocol Node_192_168_10_1 in the Established state (the name is Node_ plus the neighbor's address with underscores). Or, if calicoctl is on the node: sudo calicoctl node status.

On the router:

sudo vtysh -c 'show bgp summary'

Now there should be eight neighbors: .17–.19 from AS 65020 and .20–.24 from AS 65030. The masters show 1 prefix each in PfxRcd (the VIP). The workers show 0 for now — and that's correct: we don't have a single LoadBalancer service yet, and the pod network isn't exported.

How to request an address

The Calico controller can assign addresses in two ways: to all services indiscriminately (AllServices) or only to those that ask explicitly (RequestedServicesOnly). I chose the second — addresses shouldn't be handed out "just because".

The mode is set in KubeControllersConfiguration:

kubectl patch kubecontrollersconfiguration default --type=merge \
  -p '{"spec":{"controllers":{"loadBalancer":{"assignIPs":"RequestedServicesOnly"}}}}'

An address is requested with an annotation on the service:

metadata:
  annotations:
    projectcalico.org/loadBalancerIPs: '["192.168.20.100"]'

The annotation is a JSON array, not a string

This is the most common source of confusion. The correct form is '["192.168.20.100"]' (with square brackets and quotes). With a plain string 192.168.20.100, the controller fails with failed to parse annotations, the service silently stays <pending>, and nothing on the service itself shows it — the error is only in the calico-kube-controllers logs:

kubectl -n calico-system logs -l k8s-app=calico-kube-controllers --tail=50 | grep -i loadbalancer

A test service

apiVersion: apps/v1
kind: Deployment
metadata:
  name: whoami
spec:
  replicas: 2
  selector:
    matchLabels: {app: whoami}
  template:
    metadata:
      labels: {app: whoami}
    spec:
      containers:
        - name: whoami
          image: traefik/whoami:latest
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: whoami
  annotations:
    projectcalico.org/loadBalancerIPs: '["192.168.20.100"]'
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
  selector:
    app: whoami
  ports:
    - port: 80
      targetPort: 80

kubectl apply -f whoami.yaml
kubectl get svc whoami          # EXTERNAL-IP 192.168.20.100
kubectl get pods -l app=whoami -o wide

On the router:

sudo vtysh -c 'show bgp ipv4 unicast 192.168.20.100'
ip route show 192.168.20.100

The route should come from exactly the nodes where the whoami pods live, not from all five. From a client in another subnet (not the node subnet):

curl http://192.168.20.100/

whoami prints the pod name, and repeated requests show that the router spreads connections across different nodes.

Remove the test:

kubectl delete -f whoami.yaml

What else you need to know

The service address is reachable from the node itself, but check it with curl. BIRD installs a blackhole 192.168.20.100 route on every announcing node, and ip route get for this address answers Invalid argument. Yet connections from the node's own processes work: eBPF rewrites connect() to the pod before routing happens. Check with curl -k https://<address>, not ip route get.

A risk for services with Local. Calico has a known race (calico#11523): when a pod appears on a node, BGP may announce the /32 before eBPF has programmed the necessary maps. The window is short and occurs only when a pod restarts or is rolled out.

The fuse on the router. When I enabled BGP, the disableBGPExport: true value on the pod pool didn't stick at first: the operator reverted it within seconds, and Calico offered the router nine 172.20.x.x/26 blocks. All nine were dropped by prefix-list SERVICES — received-routes showed 10 (9 filtered). The fuse we installed in part 3 turned out to be no formality. The reason was that the operator reconciles the pool against the list in the Installation, not against the IPPool itself — that's why in part 5 we set the flag there right away.

In part 7 we'll hang a real service on this mechanism — Traefik — and go through why it needs externalTrafficPolicy: Local.