Highly available Kubernetes on BGP. Part 5: Calico with eBPF and worker nodes

Three masters are assembled and the VIP works, but the cluster has no network. In this part we install Calico, switch it to eBPF instead of kube-proxy and join the worker nodes.

---

Why Calico, and why eBPF

In my cluster Calico does three jobs at once:

1. CNI — the pod network (VXLAN in CrossSubnet mode: between nodes in the same subnet, packets travel without encapsulation);
2. services — in eBPF mode Calico handles ClusterIP, NodePort and LoadBalancer by itself, and kube-proxy isn't needed;
3. LoadBalancer addresses over BGP (part 6).

I chose eBPF mode for two reasons. First, a whole layer of iptables rules that something has to keep updated goes away: once, kube-proxy didn't pick up a service address change on some of the nodes, and that became the main delay of a migration. Second, eBPF preserves the client address for connections accepted by another node.

Requirements: kernel 5.10+ (Ubuntu 24.04 has 6.8), cgroup v2 and a mounted bpffs — Ubuntu 24.04 has all of this out of the box.

---

Installing the operator

Calico is installed via the tigera-operator: it deploys and maintains all the components, and configuration boils down to a few resources.

kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.32.2/manifests/operator-crds.yaml
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.32.2/manifests/tigera-operator.yaml

(The operator-crds.yaml file exists in versions 3.30 and newer; for older ones tigera-operator.yaml alone is enough. It's create, not apply: the CRDs have too large an annotation.)

The API address without kube-proxy

There's a chicken-and-egg problem here. The operator and calico-node are hostNetwork pods, and to find the API they go to the kubernetes service (10.96.0.1). But that address turns into a master's address only thanks to kube-proxy — which we don't have. And Calico's eBPF dataplane, which is supposed to replace it, itself needs access to the API in order to start.

The solution from the Calico documentation is to tell the operator the real API address via a ConfigMap. We specify the VIP name:

apiVersion: v1
kind: ConfigMap
metadata:
  name: kubernetes-services-endpoint
  namespace: tigera-operator
data:
  KUBERNETES_SERVICE_HOST: control.k8s.example.com
  KUBERNETES_SERVICE_PORT: "6443"

kubectl apply -f kubernetes-services-endpoint.yaml
kubectl -n tigera-operator rollout restart deploy/tigera-operator

The restart is needed so the pod rereads its environment variables.

---

Installation

apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
  name: default
spec:
  calicoNetwork:
    linuxDataplane: BPF
    bgp: Disabled
    ipPools:
      - name: default-ipv4-ippool
        cidr: 172.20.0.0/16
        blockSize: 26
        encapsulation: VXLANCrossSubnet
        natOutgoing: Enabled
        nodeSelector: all()
        disableBGPExport: true
---
apiVersion: operator.tigera.io/v1
kind: APIServer
metadata:
  name: default
spec: {}

What matters here:

• The pod pool is set explicitly. If you don't specify it, the operator creates the stock 192.168.0.0/16, which can easily overlap with your home network. Our kubeadm didn't set podSubnet, so the pod network is known only to Calico.
• bgp: Disabled for now: first I want to bring up the pod network, and we'll enable BGP in part 6 together with the rest of the configuration. Otherwise, "out of the box" Calico will start building a full BGP mesh between all nodes, which we don't need.
• disableBGPExport: true on the pod pool: once BGP is enabled, Calico won't tell the router about the 172.20.x.x/26 blocks. Pod traffic goes over VXLAN, and the router has no reason to know about them. (The router would drop these routes by the prefix-list anyway, but it's better not to create extra work for it.)
• APIServer deploys calico-apiserver — the aggregated projectcalico.org/v3 API. Without it kubectl can't work with BGPConfiguration, BGPPeer and regular IPPool in human-readable form. Admittedly, in Calico 3.32 this component is declared deprecated in favor of native CRDs, but at the time of writing it's needed.

kubectl apply -f installation.yaml
kubectl get tigerastatus -w

After a couple of minutes all components (calico, apiserver, ippools) should become Available=True, and the three masters Ready. CoreDNS will go to Running.

---

Worker nodes

We saved the join command from the kubeadm init output. If the token has already expired (it lives for 24 hours), create a new one on any master:

kubeadm token create --print-join-command

On each of the five workers:

sudo kubeadm join control.k8s.example.com:6443 --token <token> --discovery-token-ca-cert-hash sha256:<hash>

Workers reach the API through the router: the name leads to the VIP, the VIP to a subnet that isn't nearby, and the route goes via 192.168.10.1. So the hairpin rule from part 3 must be in place by now: without it, join will hang or break off.

kubectl get nodes -o wide
kubectl get pods -A -o wide

---

Checking eBPF

kubectl -n kube-system get ds                  # no kube-proxy
kubectl -n calico-system get pods -o wide
kubectl get installation default -o jsonpath='{.status.computed.calicoNetwork.linuxDataplane}'; echo   # BPF

What the dataplane sees:

kubectl -n calico-system exec ds/calico-node -c calico-node -- calico-node -bpf nat dump | head -20

Here are all services with their backends. Plain iptables-save will no longer tell you anything about services.

Check that pods have working DNS and internet access:

kubectl run -it --rm dnscheck --image=busybox:1.37 --restart=Never -- nslookup kubernetes.default
kubectl run -it --rm nettest --image=busybox:1.37 --restart=Never -- wget -qO- -T 5 http://example.com | head -3

---

How to manage Calico with the operator

Newcomers to operator mode usually have to survive two surprises.

The operator puts everything back. It continuously drives the cluster toward the described state. Editing daemonset/calico-node or deployment/calico-typha in calico-system by hand is pointless: the operator will silently revert them to its own version within seconds, with no error and no trace in the logs. Network parameters (MTU, encapsulation, pools) are changed only through the Installation.

Two API groups.

Group	What it's about	Examples
operator.tigera.io/v1	how Calico is deployed	Installation, APIServer, TigerStatus
projectcalico.org/v3	how Calico is configured	IPPool, BGPConfiguration, BGPPeer, FelixConfiguration

projectcalico.org/v3 resources are edited directly. The exception is the default-ipv4-ippool pod pool: it's owned by the operator, which keeps its copy in the Installation and reverts edits to the IPPool itself. Pool parameters are changed only in the Installation, and the patch must be JSON, not merge: a merge would replace the whole list of pools.

kubectl patch installation default --type=json \
  -p '[{"op":"replace","path":"/spec/calicoNetwork/ipPools/0/disableBGPExport","value":true}]'

One more trap: IPPool is visible through two groups at once — ippools.projectcalico.org (via the aggregated API, with validation) and the "raw" ippools.crd.projectcalico.org. The data is the same, but you should write to the first.

Diagnostics. Health is the first thing to check:

kubectl get tigerastatus
kubectl -n tigera-operator logs deploy/tigera-operator --tail=50

tigerastatus says something vague like "Waiting for Tigera API server", while the operator's logs give the real reason.

The calico-apiserver logs will contain lots of level=error entries marked "Timeout or abort while handling" on GET /apis/projectcalico.org/v3 — this is not a malfunction. Every 30 seconds kube-aggregator checks the availability of the APIService and closes the stream without reading the response to the end. If the APIService is Available=True and the errors are only on discovery, everything is fine.

---

Memory

calico-node in eBPF mode takes 200–300 MB — BPF maps are accounted in its cgroup. If you're on Hyper-V and didn't add nr_cpus (part 2), gigabytes of per-CPU memory will be added on top.

---

In part 6 we enable BGP in Calico and set up LoadBalancer address assignment.