Highly available Kubernetes on BGP. Part 2: Preparing Ubuntu 24.04 nodes
All parts of the series “Highly available Kubernetes on BGP”In the second part of the series we'll prepare the nodes. Everything in this article is done on each of the eight nodes (both masters and workers) unless stated otherwise.
Node name and address
A node's name in Kubernetes comes from its hostname, so we make it fully qualified (FQDN) and matching DNS right away:
sudo hostnamectl set-hostname k8s-master-1.example.com
Static address
In my case the addresses were initially assigned by DHCP, and that turned into trouble. After a glibc update, the needrestart package restarted systemd-networkd, the master lost its lease and didn't get it back — the node stayed unreachable for 17 hours. Since then all addresses in the cluster are static.
# /etc/netplan/50-cloud-init.yaml
network:
version: 2
ethernets:
eth0:
dhcp4: false
dhcp6: false
addresses: [192.168.10.17/24] # each node has its own
routes:
- to: default
via: 192.168.10.1
metric: 100
If you apply the file on a live node over SSH, do it with a rollback, or you risk losing access:
sudo cp /etc/netplan/50-cloud-init.yaml /root/netplan-backup.yaml
sudo netplan generate
sudo bash -c 'nohup bash -c "netplan apply; sleep 6; ping -c 2 -W 2 192.168.10.1 >/dev/null || { cp /root/netplan-backup.yaml /etc/netplan/50-cloud-init.yaml; netplan apply; }" >/root/netplan-apply.log 2>&1 &'
DNS
The nodes should have two DNS servers in /etc/resolv.conf, and the file itself should be a regular file, not a link into /run/systemd/resolve/. If systemd-resolved is disabled (which is necessary if CoreDNS forwards external names to the node's resolv.conf — the 127.0.0.53 stub would create a loop), the link will dangle after a reboot, and kubelet won't be able to create a single pod: Failed to create pod sandbox: open /etc/resolv.conf: no such file. I stepped on this on one of the nodes during a planned reboot.
sudo systemctl disable --now systemd-resolved
sudo rm /etc/resolv.conf
printf 'nameserver 192.168.10.15\nnameserver 192.168.10.10\n' | sudo tee /etc/resolv.conf
The control plane name in /etc/hosts
Worker nodes connect to the API by the name control.k8s.example.com. If DNS goes away, kubelet won't find the API, and the nodes will go NotReady — even though the cluster itself is intact. This actually happened to me once: the site's only DNS server went down, and seven nodes out of eight became NotReady. The fix is one line on each node:
echo '192.168.20.1 control.k8s.example.com' | sudo tee -a /etc/hosts
echo '192.168.10.17 k8s-master-1.example.com k8s-master-1' | sudo tee -a /etc/hosts # the node's own name
There's only one price to pay: if the VIP address changes, the entry has to be fixed on all nodes.
Swap, kernel modules, sysctl
Since version 1.32 kubelet won't start with swap enabled, so we turn it off right away and for good:
sudo swapoff -a
sudo sed -i '/\sswap\s/ s/^/#/' /etc/fstab
The modules and parameters that kubeadm checks during its preflight checks:
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
sudo modprobe overlay
sudo modprobe br_netfilter
cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF
sudo sysctl --system
containerd
I install containerd.io from the Docker repository (the one in Ubuntu is noticeably older):
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" \
| sudo tee /etc/apt/sources.list.d/docker.list
sudo apt-get update
sudo apt-get install -y containerd.io
The default configuration, in which we enable systemd as the cgroup driver:
sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
sudo systemctl restart containerd
The pause (sandbox) image in the configuration must match the one kubeadm expects; we check it after installing the Kubernetes packages:
kubeadm config images list | grep pause
grep -n sandbox /etc/containerd/config.toml
I also have earlier notes on how I used to install containerd: Preparing a clean Ubuntu Server to be a Kubernetes cluster node with CRI: ContainerD.
kubeadm, kubelet, kubectl
The Kubernetes repository is tied to a minor version, so the version number is right in the URL:
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.36/deb/Release.key \
| sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.36/deb/ /' \
| sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl containerd.io
sudo systemctl enable --now kubelet
apt-mark hold isn't decoration here. When containerd.io isn't on hold, it gets upgraded along with the other packages at the very first apt upgrade, and the versions on the nodes drift apart. The cluster should be upgraded according to plan, one minor version at a time: apt-mark unhold before each step, and hold again after it.
Time
etcd and certificates are sensitive to time, and in an Active Directory domain clock skew also breaks Kerberos. I decided not to rely on external servers: the three masters are the cluster's NTP servers, and the workers get time from them.
On the masters it's chrony, in a separate file; the stock chrony.conf stays untouched:
# /etc/chrony/conf.d/cluster.conf on k8s-master-1
peer 192.168.10.18
peer 192.168.10.19
allow 192.168.10.0/24
local stratum 10 orphan
local stratum 10 orphan keeps the service useful if the internet is lost: the masters agree among themselves instead of drifting apart. One more detail: the stock chrony unit has no Restart=, and if the OOM killer kills chronyd, it won't come back by itself. Let's add it:
# /etc/systemd/system/chrony.service.d/override.conf
[Service]
Restart=always
On the workers systemd-timesyncd is enough:
# /etc/systemd/timesyncd.conf.d/cluster.conf
[Time]
NTP=192.168.10.17 192.168.10.18 192.168.10.19
FallbackNTP=ntp.ubuntu.com
Automatic updates: package lists only
A cluster node shouldn't update itself. I've already mentioned needrestart, which restarts services after library updates. Let's forbid it from touching what the node depends on:
# /etc/needrestart/conf.d/50-kubernetes-needrestart.conf
$nrconf{override_rc}{qr(^systemd-networkd)} = 0;
$nrconf{override_rc}{qr(^kubelet)} = 0;
$nrconf{override_rc}{qr(^containerd)} = 0;
And leave apt with only updating the lists, without installing updates:
# /etc/apt/apt.conf.d/20auto-upgrades
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "0";
Once a month, in a maintenance window, we update the nodes by hand one at a time: apt-get update && apt-get upgrade, needrestart -r l shows what's waiting for a restart, then a reboot with drain and uncordon.
A Hyper-V quirk: nr_cpus
If your nodes are Hyper-V virtual machines, add nr_cpus to the kernel parameters right away. The guest is told there are 240 "possible" processors (smpboot: Allowing 240 CPUs, 236 hotplug CPUs) with only four real ones, and the kernel allocates per-CPU memory for all 240. An ordinary node doesn't care, but Calico in eBPF mode has maps with per-CPU values, and on my setup 1.3–1.5 GB per node was being wasted.
sudo sed -i 's/^GRUB_CMDLINE_LINUX_DEFAULT="/&nr_cpus=4 /' /etc/default/grub
sudo update-grub
sudo reboot
Check after the reboot:
grep Percpu /proc/meminfo # tens of megabytes, not gigabytes
When adding virtual processors, the number in GRUB has to be changed.
Checks before kubeadm
On each node:
swapon --show # empty
lsmod | grep -E 'overlay|br_netfilter'
sysctl net.ipv4.ip_forward # = 1
systemctl is-active containerd # active
kubeadm version -o short # v1.36.x
getent hosts control.k8s.example.com
The nodes are ready. But before creating the cluster, we need a router that can accept routes: in part 3 we'll configure FRR.
Only registered users can leave a comment.
Sign in or register to leave a comment.