Vitaliy Leschenko’s blog
ru
Виталий Лещенко

Highly available Kubernetes on BGP. Part 4: kube-vip in BGP mode and the first master

All parts of the series “Highly available Kubernetes on BGP”

The router is ready to wait for neighbors, the nodes are prepared, and it's time to create the cluster. In this part we'll bring up the VIP for the API server with kube-vip in BGP mode, initialize the first master and join the other two.

How kube-vip works in BGP mode

kube-vip runs as a static pod on every master: the manifest lives in /etc/kubernetes/manifests/kube-vip.yaml, and kubelet starts it by itself — before and independently of the apiserver. Neither Helm nor kubectl apply is involved.

In BGP mode:

  1. kube-vip puts the VIP (192.168.20.1/32) on the lo interface — not on eth0. The address must not answer ARP, and loopback is the standard place for that.
  2. Inside kube-vip runs a BGP speaker (based on GoBGP), which opens an eBGP session with the router.
  3. All three masters announce at the same time. No leader is elected, no lease is needed.
  4. Instead of moving the address, a health check does the work: kube-vip polls the local apiserver, and if it doesn't respond, withdraws the announcement. The router removes the master from ECMP.

The health check deserves special mention. Since everyone announces, the router spreads requests across the masters. If the apiserver dies on one node while kubelet and the kube-vip pod stay alive, the node keeps announcing the route, and a third of all requests will go into a black hole. That's why the health check is enabled in the manifest: after three failed checks in a row, the announcement is withdrawn.

DNS for the API

Create an A record control.k8s.example.com → 192.168.20.1 in DNS and, as agreed in part 2, duplicate it in /etc/hosts on every node.

The kube-vip manifest

The first master. On the other masters exactly two values differ: bgp_routerid (the node's own address) and control_plane_health_check_address (the node's name).

apiVersion: v1
kind: Pod
metadata:
  name: kube-vip
  namespace: kube-system
spec:
  containers:
  - args:
    - manager
    env:
    - name: address
      value: "192.168.20.1"
    - name: port
      value: "6443"
    - name: vip_subnet
      value: "32"
    - name: vip_interface
      value: lo
    - name: vip_arp
      value: "false"
    - name: vip_nodename
      valueFrom:
        fieldRef:
          fieldPath: spec.nodeName
    - name: cp_enable
      value: "true"
    - name: cp_namespace
      value: kube-system
    - name: bgp_enable
      value: "true"
    - name: bgp_as
      value: "65020"
    - name: bgp_routerid
      value: "192.168.10.17"
    - name: bgp_peeraddress
      value: "192.168.10.1"
    - name: bgp_peeras
      value: "65010"
    - name: control_plane_health_check_address
      value: "https://k8s-master-1.example.com:6443/livez"
    - name: control_plane_health_check_ca_path
      value: /etc/kubernetes/pki/ca.crt
    - name: control_plane_health_check_period_seconds
      value: "5"
    - name: control_plane_health_check_timeout_seconds
      value: "3"
    - name: control_plane_health_check_failure_threshold
      value: "3"
    - name: prometheus_server
      value: :2112
    image: ghcr.io/kube-vip/kube-vip:v1.2.4
    imagePullPolicy: IfNotPresent
    name: kube-vip
    resources:
      requests:
        memory: 64Mi
      limits:
        memory: 512Mi
    securityContext:
      capabilities:
        add:
        - NET_ADMIN
        - NET_RAW
        drop:
        - ALL
    volumeMounts:
    - mountPath: /etc/kubernetes/admin.conf
      name: kubeconfig
    - mountPath: /etc/kubernetes/pki/ca.crt
      name: ca-cert
      readOnly: true
  hostAliases:
  - hostnames:
    - kubernetes
    ip: 127.0.0.1
  hostNetwork: true
  volumes:
  - hostPath:
      path: /etc/kubernetes/super-admin.conf     # only for the duration of kubeadm init, see below
    name: kubeconfig
  - hostPath:
      path: /etc/kubernetes/pki/ca.crt
      type: File
    name: ca-cert

Parameters

Parameter Value Meaning
address 192.168.20.1 The VIP. Lives in a subnet that doesn't physically exist
vip_interface lo in BGP mode the address must not answer ARP
vip_arp "false" ARP announcements are off
cp_enable "true" control plane mode
bgp_as "65020" our AS (the masters' AS)
bgp_routerid the node's address Router ID of the local speaker, each has its own
bgp_peeraddress, bgp_peeras 192.168.10.1, 65010 the router, an eBGP session
control_plane_health_check_* protection against a black hole
prometheus_server :2112 kube-vip metrics

What's not in the manifest. If you start from an ARP-mode example, remove vip_leaderelection, vip_leasename, vip_leaseduration, vip_renewdeadline and vip_retryperiod: that's the mechanism for electing a single address owner, and BGP doesn't need it. Same goes for dhcp_mode and dns_mode.

The health check address is the node name, not localhost. The apiserver certificate's SANs don't include localhost, and TLS verification against it will fail. By node name, anonymous /livez returns 200: kubeadm sets up the system:public-info-viewer role for this. The CA is mounted via a separate hostPath volume.

The memory limit. In kube-vip versions v1.2.1–v1.2.3, BGP mode leaked memory — about 18 MB per hour on each master, because defer resp.Body.Close() in the health check loop never fired. I found the cause and described it in kube-vip#1778; the fix shipped in v1.2.4. Use that version or newer. I kept the 512Mi limit as insurance: on the fixed version it's unreachable (growth of 0.03 MB per hour), but if the leak comes back, the container will hit the limit, kubelet will restart it within seconds, and meanwhile the other two masters will announce the VIP.

The first master's snag: super-admin.conf

kube-vip needs a kubeconfig with administrator rights. In the manifest that's /etc/kubernetes/admin.conf. But on the first master it doesn't exist yet: kubeadm init creates it at the very end. And since version 1.29, kubeadm also creates super-admin.conf, which appears already at the kubeconfig generation stage.

So for the duration of kubeadm init the manifest points to super-admin.conf (as shown above), and after initialization we'll switch back to admin.conf. This is needed only on the first master.

Put the manifest in place before running kubeadm init:

sudo mkdir -p /etc/kubernetes/manifests
sudo cp kube-vip.yaml /etc/kubernetes/manifests/kube-vip.yaml

kubelet doesn't see it yet: it's not configured or running yet, and it will pick up the manifest as soon as kubeadm starts it.

kubeadm init

The initialization configuration:

# kubeadm-init.yaml
apiVersion: kubeadm.k8s.io/v1beta4
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.10.17
nodeRegistration:
  criSocket: unix:///run/containerd/containerd.sock
  ignorePreflightErrors:
    - DirAvailable--etc-kubernetes-manifests   # the kube-vip manifest is already there
---
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
kubernetesVersion: v1.36.4
controlPlaneEndpoint: control.k8s.example.com:6443
networking:
  serviceSubnet: 10.96.0.0/12
proxy:
  disabled: true

A few remarks:

  • controlPlaneEndpoint is a name, not an IP address. It goes into the certificates and kubeconfigs. The VIP itself isn't in the SANs, and that gives freedom: the address can be changed without reissuing certificates (though /etc/hosts and DNS will have to be fixed).
  • proxy.disabled: true — there will be no kube-proxy in the cluster: services are handled by Calico in eBPF mode (part 5). The field also works with kubeadm upgrade: it will report Skipping the addon/kube-proxy phase and won't bring kube-proxy back.
  • podSubnet is deliberately not set. Calico uses its own IPAM, and we'll set the pod pool in the Installation (otherwise Calico creates the stock 192.168.0.0/16, which will overlap with a home network).
  • advertiseAddress is the node's real address, not the VIP.

Run it:

sudo kubeadm init --config kubeadm-init.yaml --upload-certs

--upload-certs uploads the control plane certificates into a secret so the other masters can fetch them on join. The encryption key (certificate-key) is valid for 2 hours; after that, kubeadm init phase upload-certs --upload-certs will issue a new one.

If initialization gets stuck for a long time waiting for the control plane, look at the kube-vip log: sudo crictl logs $(sudo crictl ps -a --name kube-vip -q | head -1). Some kubeadm steps access the cluster by the name control.k8s.example.com, that is, via the VIP, and the VIP doesn't appear immediately: first the apiserver has to come up and pass the health check, then the BGP session with the router has to be established. It usually takes a few seconds.

When it finishes, kubeadm prints the join commands for masters and workers — save them.

Switching back to admin.conf

mkdir -p $HOME/.kube
sudo cp /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

sudo sed -i 's#/etc/kubernetes/super-admin.conf#/etc/kubernetes/admin.conf#' /etc/kubernetes/manifests/kube-vip.yaml

Editing the manifest is the deployment: kubelet watches the directory and recreates the pod by itself. A few seconds later:

ip -brief addr show lo          # 192.168.20.1/32 on lo
sudo crictl ps --name kube-vip

The kube-vip log should contain the lines BGP health check passed, announcing route (with cidr=192.168.20.1/32) and the session state BGP_FSM_ESTABLISHED. On the router:

sudo vtysh -c 'show bgp summary'
ip route show 192.168.20.1

The neighbor 192.168.10.17 should show a number in the State/PfxRcd column (1), and the kernel table should get a route 192.168.20.1 via 192.168.10.17. Check access from any machine on the network:

curl -sk https://192.168.20.1:6443/livez ; echo

The second and third masters

On the first master (or wherever the command from the kubeadm init output is):

sudo kubeadm join control.k8s.example.com:6443 \
  --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash> \
  --control-plane --certificate-key <key> \
  --apiserver-advertise-address 192.168.10.18

Here --apiserver-advertise-address is the address of the node being joined.

kubeadm join --control-plane creates admin.conf on the new master at the very end of the join, so here we put the kube-vip manifest in place after join. While kube-vip isn't running on the new master, the others serve the VIP just fine: there's no leader, and nobody waits for anybody.

Take the first master's manifest in its final form (with hostPath pointing to admin.conf, after the sed edit above), copy it to the new master and change two values: the address in bgp_routerid and the node name in the health check address:

sudo sed -e 's/192.168.10.17/192.168.10.18/' -e 's/k8s-master-1/k8s-master-2/' kube-vip-master-1.yaml \
  | sudo tee /etc/kubernetes/manifests/kube-vip.yaml

Repeat for the third (.19, k8s-master-3). The result on the router:

ip route show 192.168.20.1

192.168.20.1 proto bgp metric 20
	nexthop via 192.168.10.17 dev eth1 weight 1
	nexthop via 192.168.10.18 dev eth1 weight 1
	nexthop via 192.168.10.19 dev eth1 weight 1

Three nexthops is ECMP: the router spreads API requests across the three masters.

What's next

The nodes are still NotReady, and the CoreDNS pods are Pending: the cluster has no network. The masters have the NoSchedule taint by default, and I don't remove it — user workloads live only on the workers.

kubectl get nodes

Check that etcd has formed with three members:

kubectl -n kube-system get pods -l component=etcd -o wide

What's important to know

  • kubectl delete pod doesn't restart a static pod. kubelet only recreates the "mirror" object, while the container keeps running. Restarting is done only on the node: sudo crictl stop $(sudo crictl ps --name kube-vip -q).
  • A static pod can't reference ConfigMaps or Secrets, only the downward API and hostPath.
  • Upgrading kube-vip means bumping the tag in the manifest on the node, one master at a time, and waiting until the pod is Ready and announcing route and Peer Up reappear in the log. The VIP doesn't disappear between masters: the remaining two announce it.

In part 5 we'll install Calico with eBPF so the cluster gets a network, and join the worker nodes.

Only registered users can leave a comment.

Sign in or register to leave a comment.

© 2006-2026 Vitaliy Leschenko