4 October 2026, 18:50Виталий Лещенко

Highly available Kubernetes on BGP. Part 7: Traefik behind BGP and externalTrafficPolicy Local

All parts of the series “Highly available Kubernetes on BGP”

The address mechanism is ready. What's left is to hang on it the workload all this was for: incoming HTTP(S) traffic. In this part we install Traefik and figure out why one line matters so much for it — externalTrafficPolicy: Local.

I've already written about installing Traefik in a cluster (How to install Traefik as an Ingress Controller in Kubernetes); here is only what relates to BGP.

Installation

helm repo add traefik https://traefik.github.io/charts
helm repo update
helm upgrade --install traefik traefik/traefik \
  --namespace traefik --create-namespace \
  -f traefik-values.yaml

The values relevant to the topic:

# traefik-values.yaml
deployment:
  kind: Deployment
  replicas: 3

# A hard rule: a second Traefik pod won't be scheduled on a node,
# an extra replica will stay Pending.
affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - topologyKey: kubernetes.io/hostname
        labelSelector:
          matchLabels:
            app.kubernetes.io/name: traefik
            app.kubernetes.io/instance: traefik-traefik

# Draining a node won't take down more than one pod at a time.
podDisruptionBudget:
  enabled: true
  minAvailable: 2

service:
  annotations:
    # The address is assigned and announced by Calico. The value is a JSON array of strings.
    projectcalico.org/loadBalancerIPs: '["192.168.20.65"]'
  spec:
    externalTrafficPolicy: Local
    internalTrafficPolicy: Cluster

In the Traefik chart, the app.kubernetes.io/instance label is built from the release name and the namespace, so for the traefik release in the traefik namespace it's traefik-traefik. Check it on your pod: kubectl -n traefik get pod --show-labels.

After installation:

kubectl -n traefik get svc traefik      # EXTERNAL-IP 192.168.20.65
kubectl -n traefik get pods -o wide     # three pods on three different workers

The masters don't accept Traefik pods because they have the NoSchedule taint (we didn't remove it). That's by design: the masters don't accept incoming traffic, and Calico doesn't peer with the router on them.

Cluster or Local

externalTrafficPolicy determines what happens when a packet addressed to a service arrives at a node.

Cluster (the default). The receiving node may hand the connection to any pod of the service, including one on another node. In that case the packet is tunneled to the node with the pod, and the reply comes back through the same receiving node. In eBPF mode this is a NAT tunnel (VXLAN, VNI 0xCA11C0) that preserves the client address. The upside is that any node can serve the service, and the address can be announced from all workers.

Local. A node serves a connection only with its local pod. If there's no pod on the node, the packet is lost. That's why only nodes with a ready pod should announce the address — and that's exactly what the disabled aggregation from part 6 ensures.

Why I switched to Local

At first my Traefik ran with Cluster and worked quite decently. But after a while a nuisance turned up: roughly one in ten external page loads hung for seconds. In the browser it looked like a page that was "thinking" (Stalled in the network tools) and then suddenly opened.

The analysis found two factors. The first: the router had no MSS clamping for traffic coming from IPsec tunnels (that's a separate story). The second: the eBPF NAT tunnel between nodes — between Hyper-V hosts, batches of response segments (GSO) were being lost in it, and instead of an instant response the client waited for a retransmission.

Switching to Local removed the inter-node hop for incoming connections: the router picks a node with a ready Traefik pod via ECMP, and that node serves the connection itself — without a tunnel. After the switch, 50 page loads in a row from home went through without a single delay, at most 0.16 s; before that, 5 out of 40 timed out.

I never fully proved the root cause of the tunnel problem (hv_netvsc offloads, vSwitch settings) — that remains a task, but the sites no longer depend on it.

What you need to keep in mind

1. Pods on the nodes that announce the address. With Local, every node in ECMP must have its own ready pod. Anti-affinity with required guarantees that two pods won't land on one node (otherwise the point is lost: three replicas on two nodes means two paths, not three). If there are more pods than nodes, the extra one stays Pending rather than landing on an extra node.

2. Spare nodes for updates. Three replicas, five workers: during an update, maxSurge brings up the new pod before the old one goes away. If there were as many replicas as nodes, the new pod would simply have nowhere to go.

3. PodDisruptionBudget. minAvailable: 2 — draining a node won't take down more than one Traefik pod at a time, and at least two paths always remain in ECMP.

4. Deployment instead of DaemonSet. Traefik used to be a DaemonSet on all five workers. Three replicas give the same three paths but use less memory, and the number of nodes with a pod can be changed with the replicas value.

What the router shows

sudo vtysh -c 'show bgp ipv4 unicast 192.168.20.65'
ip route show 192.168.20.65

Three paths — one for each worker with a Traefik pod:

192.168.20.65 proto bgp metric 20
	nexthop via 192.168.10.20 dev eth1 weight 1
	nexthop via 192.168.10.22 dev eth1 weight 1
	nexthop via 192.168.10.24 dev eth1 weight 1

Compare these addresses with kubectl -n traefik get pods -o wide: the list of nodes should match. A node that's among the paths but has no ready Traefik pod is a reason to look first at the pod, then at calico-node.

Now you can watch how the route reacts to events:

kubectl -n traefik delete pod <one-of-the-pods>
watch -n1 "ip route show 192.168.20.65"

While the pod is being recreated (and the PDB and anti-affinity tell the scheduler where it should go), the route from that node disappears and reappears when the pod passes readiness.

How traffic gets in from outside

In my setup all public names in external DNS point to the router's external address, and the router forwards ports 80 and 443 (DNAT) to 192.168.20.65. Inside the network, split-horizon DNS returns 192.168.20.65 directly for the same names.

The path of a client from the internet is symmetric: the packet arrives at the router, the router sends it to a worker via ECMP, and the worker replies to the client address (which doesn't belong to the node subnet), and therefore via the router. So ordinary clients need no special rules.

A special case is a client in the same subnet as the nodes (another machine in the node subnet, not part of the cluster). It ends up in the situation discussed in part 3: the request goes through the router, the reply goes directly. The hairpin rule for 192.168.20.0/24 covers the LoadBalancer addresses too, because I wrote it for the whole subnet from the start, and it didn't need extending when Traefik appeared.

And for the nodes and pods themselves it isn't a problem at all: eBPF turns a connection to a service address into a connection to a pod before the packet reaches routing. From any pod in the cluster (including on the masters), the site's external name opens as usual.

Small but unpleasant things

The BGP/eBPF race. With Local there's a known race (calico#11523): when a pod appears on a node, the /32 may be announced before eBPF has programmed the maps. The window is short and occurs only when a Traefik pod restarts or is rolled out. It's worth knowing about, especially if you roll out Traefik during peak hours.

A new worker doesn't get traffic right away. Only once a Traefik pod on it becomes ready. That's normal and exactly what we want.

With Local there's no inter-node hop, but it remains for other services. If other LoadBalancer or NodePort services appear with Cluster, they'll keep using the eBPF tunnel.

In the last, eighth part, we break nodes and see how highly available all of this really is, and put together a troubleshooting cheat sheet.