24 October 2026, 14:16Виталий Лещенко

Kubernetes for beginners. Part 4: Installing a cluster with kubeadm

All parts of the series “Kubernetes for beginners”

In the first part we looked at why an orchestrator is needed, in the second what components a cluster consists of, and in the third how its API works. Now we'll build the lab cluster that the whole series will run on: three Ubuntu Server 26.04 virtual machines, Kubernetes 1.36, containerd, kube-proxy in nftables mode and Calico. We'll install it with kubeadm, the project's official tool, which does exactly one thing: it turns prepared machines into a cluster following best practices, without hiding a single component. We'll go through every command: what it changes on the node and in the API.

The cluster will have a single master. A highly available control plane with three masters and a shared API address is covered in the series "Highly available Kubernetes on BGP"; here we only lay the groundwork for moving to it later.

The lab setup

Node	Address	Role	Resources
`k8s-master-1.example.com`	192.168.10.17	control plane	2 vCPU, 4 GB, 32 GB
`k8s-node-1.example.com`	192.168.10.20	worker node	2 vCPU, 4 GB, 32 GB
`k8s-node-2.example.com`	192.168.10.21	worker node	2 vCPU, 4 GB, 32 GB

Network	Value
node network	`192.168.10.0/24`, gateway `192.168.10.1` (`router.example.com`)
pod network	`172.20.0.0/16` (Calico hands it out in /26 blocks, one per node)
service network	`10.96.0.0/12`, the API address inside the cluster is `10.96.0.1`, DNS is `10.96.0.10`
API address	`k8s-api.example.com:6443` → 192.168.10.17

The kubeadm minimum is 2 GB of memory per machine and 2 processors on the master; with 4 GB on the nodes there is room for both the lab applications and the monitoring from part 32. The pod and service networks must not overlap with each other, with the node network, or with the networks the pods will connect to: otherwise a packet to an external address will end up in a pod.

A separate name for the API, k8s-api.example.com, is the most important decision in this part. For now it is an A record in DNS pointing to the single master. All the certificates, kubeconfigs and nodes will know the API under this name, and when a second and a third master are needed, it is enough to point the name at the virtual address of a load balancer, for example kube-vip, as in the BGP series. If you install the cluster against the master's address instead, kubeadm cannot turn it into a highly available one: the address is baked into the certificates and configurations of all the nodes, and the cluster would have to be built again from scratch.

DNS needs records for all three nodes and for k8s-api.example.com. Without DNS, /etc/hosts on every node and on the workstation will do, but that is one more place to maintain.

Preparing the nodes

Everything in this section is done on all three machines. Ubuntu is installed in the minimal server variant with OpenSSH, and the node's address does not change: it is set statically in netplan (Ubuntu's YAML-based network configuration system) or reserved for the machine in DHCP.

Names and uniqueness

user@k8s-node-1:~$ hostnamectl --static
k8s-node-1
user@k8s-node-1:~$ ip -4 -br addr show ens18; sudo cat /sys/class/dmi/id/product_uuid; cat /etc/machine-id
ens18            UP             192.168.10.20/24 metric 100
aea5dde7-527d-4a66-bea4-feb9594806aa
07300e61e2dd497d9a97580fba176afb

By default, a node's name in the cluster is taken from its hostname, so it must be unique and stable. The Ubuntu installer asks for the name during installation; you can change it later with sudo hostnamectl set-hostname k8s-node-1. kubeadm also checks that product_uuid (an identifier the hypervisor gives to the virtual machine) is unique: for clones of a virtual machine it is sometimes the same. For clones it is also worth checking /etc/machine-id, the identifier from which systemd builds, among other things, the DUID for DHCP; an identical machine-id on two nodes is a source of hard-to-trace problems.

The API name

All the nodes must be able to find k8s-api.example.com. If there is no DNS record yet, a line in /etc/hosts on every node will do:

user@k8s-node-1:~$ echo '192.168.10.17 k8s-api.example.com' | sudo tee -a /etc/hosts
192.168.10.17 k8s-api.example.com
user@k8s-node-1:~$ getent hosts k8s-api.example.com
192.168.10.17   k8s-api.example.com

getent hosts checks name resolution the way programs look names up (through /etc/hosts and DNS), so it shows exactly what kubeadm will see.

swap

user@k8s-node-1:~$ sudo swapoff -a
user@k8s-node-1:~$ sudo sed -i '/\sswap\s/ s/^/#/' /etc/fstab
user@k8s-node-1:~$ sudo rm -f /swap.img
user@k8s-node-1:~$ swapon --show; free -h | grep -i swap
Swap:             0B          0B          0B

The Ubuntu Server installer creates a swap file /swap.img and a line for it in /etc/fstab. By default kubelet refuses to start if it sees swap (failSwapOn: true): its log shows running with swap on is not supported, please disable swap or set --fail-swap-on flag to false, and kubeadm init or join waits for a healthy kubelet for up to four minutes and then fails. The kubeadm 1.36 preflight check is only a soft warning — [WARNING Swap]: swap is supported for cgroup v2 only. The kubelet must be properly configured to use swap… — and it will not stop the installation in advance. Kubernetes does support swap: kubelet can be allowed to work with it, and pods can be allowed to use it in the LimitedSwap mode. But swap is turned off for a lab cluster and for most production ones: the scheduler and the memory limits (part 10) count only physical memory, and a pod that has gone into swap, instead of a quick and clear OOMKilled (forced termination of a process when it exceeds its memory limit), slowly degrades together with the whole node.

Kernel modules and sysctl

user@k8s-node-1:~$ cat <<'EOF' | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
overlay
br_netfilter
user@k8s-node-1:~$ sudo modprobe overlay
user@k8s-node-1:~$ sudo modprobe br_netfilter

user@k8s-node-1:~$ cat <<'EOF' | sudo tee /etc/sysctl.d/99-kubernetes.conf
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
user@k8s-node-1:~$ sudo sysctl --system | grep -E 'ip_forward|bridge-nf'
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1

(The tee command prints what it writes to the file, so each block is repeated: first the contents of the file, then the applied values.)

overlay — the kernel module for the overlay file system, on which containerd assembles a container's root from the image layers (the overlayfs snapshotter).
br_netfilter — makes frames that cross a Linux bridge pass through the netfilter hooks; the net.bridge.bridge-nf-call-* variables appear only after it is loaded. This is needed by network plugins that connect pods with a bridge (the simplest bridge plugin, kind — a cluster in Docker containers, which the bonus part at the end of the series is devoted to): without the module, traffic between pods on one node bypasses the kube-proxy rules. Calico builds the network without a bridge, with routes through veth pairs, so it does not require the module, but having it loaded breaks nothing, and changing the plugin or experimenting will not turn into a hunt for vanishing packets.
net.ipv4.ip_forward = 1 — routing between interfaces. The node forwards packets between pods and between a pod and the node network; without this parameter the pod network does not work at all. It is the only setting that the official documentation calls mandatory.

Time

Certificates, service account tokens, node and leader leases — everything in the cluster relies on time. A node whose clock has drifted by a couple of minutes rejects certificates that are "not yet valid" and looks dead to the control plane. In Ubuntu 26.04 time is synchronized by chrony, and an already installed system is configured correctly:

user@k8s-node-1:~$ sudo timedatectl set-timezone Europe/Minsk
user@k8s-node-1:~$ timedatectl | grep -E 'Time zone|synchronized'
                Time zone: Europe/Minsk (+03, +0300)
System clock synchronized: yes
user@k8s-node-1:~$ chronyc tracking | grep -E 'Reference|System time'
Reference ID    : B97DBE7B (ntp-nts-3.ps5.canonical.com)
System time     : 0.000031123 seconds slow of NTP time
user@k8s-node-1:~$ chronyc sources
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^+ ntp-nts-2.ps5.canonical.>     2   6   377    98    +47us[+2951ns] +/-   18ms
^* ntp-nts-3.ps5.canonical.>     2   6   377    33    +49us[  +69us] +/-   17ms
^- ntp-nts-2.ps6.canonical.>     2   6   377    33  +8835us[+8835us] +/-   85ms
^- ntp-nts-3.ps6.canonical.>     2   6   377    31  -2591us[-2591us] +/-   86ms

The time zone affects only how time looks in date and in the logs: certificates and Kubernetes objects store time in UTC. chrony takes its time sources from the /etc/chrony/sources.d directory: by default it contains the file ubuntu-ntp-pools.sources with Canonical's servers, and they are connected with NTS (Network Time Security, RFC 8915), an extension of NTP that exchanges keys with the server over TLS and then authenticates every response, so a rogue "time server" on the network cannot shift the node's clock. In the chronyc sources output, the asterisk ^* marks the source the node is currently synchronized with, ^+ marks usable backup sources, and ^- marks healthy sources that were not selected.

In a closed network it is more convenient for the nodes to synchronize with one local server than for each of them to go to the internet: add the file /etc/chrony/sources.d/local.sources with the line server router.example.com iburst and run sudo chronyc reload sources.

Firewall

On the lab setup ufw is off (that is how it ships in Ubuntu Server: sudo ufw status answers Status: inactive), and the node network is closed off from the outside world by the router. In a production cluster filtering is turned on at the nodes, and these are the ports that are needed:

Nodes	Port	From	Purpose
master	6443/TCP	all nodes, administrators, API clients	kube-apiserver
master	2379–2380/TCP	other masters	etcd
all	10250/TCP	masters, metrics collectors	kubelet API
all	10256/TCP	external load balancers	kube-proxy health check
workers	30000–32767/TCP and UDP	clients	NodePort services
all	179/TCP	other nodes	BGP between nodes in Calico
all	4789/UDP	other nodes	Calico VXLAN (between subnets)
nodes with Typha	5473/TCP	Calico agents on other nodes	Calico Typha, an intermediary between the API and the Calico agents on the nodes

Ports 10257 and 10259 do not need to be opened: kubeadm binds the controller manager and the scheduler to 127.0.0.1. The firewall rules on the cluster nodes should be coordinated with Calico network policies, which can protect the nodes themselves too (part 20).

containerd

user@k8s-node-1:~$ sudo apt-get update
user@k8s-node-1:~$ sudo apt-get install -y containerd runc
user@k8s-node-1:~$ containerd --version
containerd github.com/containerd/containerd/v2 2.2.2

The containerd package from the Ubuntu repository is containerd 2.2 with security updates from Canonical. The alternative is the containerd.io package from the Docker repository; it is newer, but it is updated separately from the system. Predictability matters more for a cluster, so we take the distribution's package.

Let's generate the full default configuration and adjust two parameters in it:

user@k8s-node-1:~$ sudo mkdir -p /etc/containerd
user@k8s-node-1:~$ containerd config default | sudo tee /etc/containerd/config.toml > /dev/null
user@k8s-node-1:~$ sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
user@k8s-node-1:~$ sudo sed -i "s|sandbox = 'registry.k8s.io/pause:.*'|sandbox = 'registry.k8s.io/pause:3.10.2'|" /etc/containerd/config.toml
user@k8s-node-1:~$ sudo systemctl restart containerd
user@k8s-node-1:~$ sudo containerd config dump | grep -E "^version|SystemdCgroup|sandbox = "
version = 3
      sandbox = 'registry.k8s.io/pause:3.10.2'
            SystemdCgroup = true

version = 3 is the containerd 2.x configuration format. The paths to the CRI settings in it differ from those in 1.x: plugins.'io.containerd.cri.v1.runtime' instead of plugins."io.containerd.grpc.v1.cri", so recipes for older versions silently do not work — a parameter with the wrong path is simply ignored. The result has to be checked with containerd config dump, which shows the effective configuration. If the generated file turns out to have no SystemdCgroup line (the template changed between versions), it is added to the table [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc.options].

SystemdCgroup = true — runc creates container cgroups through systemd rather than directly in /sys/fs/cgroup (the cgroupfs driver). On a machine with systemd and cgroup v2 (and Ubuntu 26.04 has only v2) there must be one managing cgroup — systemd; two managers with different views of the hierarchy lead to node instability under load. kubelet must use the same driver; modern versions of kubelet learn it from containerd themselves through CRI (the RuntimeConfig call), and kubeadm writes cgroupDriver: systemd by default anyway.

sandbox is the pause container image. kubeadm 1.36 pulls registry.k8s.io/pause:3.10.2, while containerd by default creates pods from 3.10.1. Without the fix there would be two pause images on the node, and the first pod would have to wait while containerd pulls the one named in its own settings: kubeadm 1.36 does not complain about the mismatch, so it is easy not to notice it. containerd does not delete a "pinned" image when it cleans up unused images.

kubeadm, kubelet and kubectl

The Kubernetes packages live in the project's repository pkgs.k8s.io, with a separate repository for each minor version. The old apt.kubernetes.io has been frozen since 2023.

user@k8s-node-1:~$ sudo apt-get install -y ca-certificates curl gpg
user@k8s-node-1:~$ curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.36/deb/Release.key \
    | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
user@k8s-node-1:~$ echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.36/deb/ /' \
    | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.36/deb/ /
user@k8s-node-1:~$ sudo apt-get update
user@k8s-node-1:~$ sudo apt-get install -y kubelet kubeadm kubectl cri-tools
user@k8s-node-1:~$ sudo apt-mark hold kubelet kubeadm kubectl cri-tools
kubelet set on hold.
kubeadm set on hold.
kubectl set on hold.
cri-tools set on hold.
user@k8s-node-1:~$ sudo systemctl enable --now kubelet
user@k8s-node-1:~$ kubeadm version -o short
v1.36.5

signed-by ties the key to a single repository: this key cannot be used to sign packages coming from other sources. apt-mark hold forbids updating the packages with a regular apt upgrade: the versions of the cluster components are linked by an allowed skew (kubelet cannot be newer than kube-apiserver), and the upgrade goes in a strict order through kubeadm upgrade — that is part 35. The cri-tools package with the crictl utility is separate: kubeadm 1.36 does not pull it in as a dependency (it depends only on iptables, kubernetes-cni, mount and util-linux), but crictl will be needed for diagnostics right on the node, so we install it in the same command. So that crictl knows where to connect and does not try obsolete sockets, we point it at the containerd socket:

user@k8s-node-1:~$ cat <<'EOF' | sudo tee /etc/crictl.yaml
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
EOF
user@k8s-node-1:~$ sudo crictl version | grep -E 'RuntimeName|RuntimeVersion'
RuntimeName:  containerd
RuntimeVersion:  2.2.2

After installation kubelet restarts every few seconds — this is normal:

user@k8s-node-1:~$ systemctl status kubelet --no-pager | grep Active
     Active: activating (auto-restart) (Result: exit-code) since Sat 2026-10-24 13:29:38 +03; 3s ago
user@k8s-node-1:~$ journalctl -u kubelet -n 3 --no-pager | grep -o 'failed to load kubelet config file.*'
failed to load kubelet config file, path: /var/lib/kubelet/config.yaml, error: failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file "/var/lib/kubelet/config.yaml", error: open /var/lib/kubelet/config.yaml: no such file or directory"

Its configuration will be written by kubeadm during init or join.

Cluster configuration

The cluster parameters can be passed to kubeadm init as command-line flags, but a file is better: it lives in Git, it can be reviewed and repeated, and flags cannot set the kubelet and kube-proxy parameters. On the master we create kubeadm-config.yaml:

apiVersion: kubeadm.k8s.io/v1beta4
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.10.17
  bindPort: 6443
nodeRegistration:
  name: k8s-master-1
  criSocket: unix:///run/containerd/containerd.sock
---
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
kubernetesVersion: v1.36.5
controlPlaneEndpoint: k8s-api.example.com:6443
networking:
  podSubnet: 172.20.0.0/16
  serviceSubnet: 10.96.0.0/12
  dnsDomain: cluster.local
apiServer:
  certSANs:
    - k8s-api.example.com
    - k8s-master-1.example.com
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd
serverTLSBootstrap: true
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: nftables

Four documents are four configuration objects:

InitConfiguration — whatever applies only to this run and this node: which address the master's kube-apiserver listens on, what the node is called, where the CRI socket is. A bootstrap token can also go here; if it doesn't, kubeadm generates one itself.
ClusterConfiguration — the parameters of the whole cluster; kubeadm saves them in the ConfigMap kubeadm-config and reads them during join and upgrade. controlPlaneEndpoint is that shared API name. podSubnet is the pod network: Calico hands out addresses with its own IPAM (the address allocation module) and does not use the podCIDR ranges allocated to nodes, but the value is still needed — kube-proxy uses it to tell pod traffic apart, and the Calico operator checks its pool against it. serviceSubnet is the range of virtual service addresses; the first address, 10.96.0.1, goes to the kubernetes service — the API address from inside the cluster. certSANs are additional names in the kube-apiserver certificate (kubeadm adds the controlPlaneEndpoint name itself, but listing it explicitly is easier to read).
KubeletConfiguration — the common kubelet configuration for all nodes; kubeadm puts it in the ConfigMap kubelet-config, and each node gets it from there when it joins. serverTLSBootstrap: true — kubelet will request a certificate for its HTTPS port 10250 from the cluster CA instead of generating a self-signed one. Why — we'll see below.
KubeProxyConfiguration — mode: nftables: kube-proxy writes the service rules into nftables. This mode has been stable since version 1.33 and scales better than iptables; the default mode in 1.36 is still iptables, so it is set explicitly. More in part 15.

The apiVersion values differ because these are different APIs: the kubeadm configuration format is currently v1beta4, kubelet's is v1beta1, kube-proxy's is v1alpha1 (despite the "alpha", it has long been a working configuration format, not an experimental feature).

Let's check the file and pull the images in advance, so that init does not stumble on a slow network:

user@k8s-master-1:~$ sudo kubeadm config validate --config kubeadm-config.yaml
ok
user@k8s-master-1:~$ sudo kubeadm config images pull --config kubeadm-config.yaml
[config/images] Pulled registry.k8s.io/kube-apiserver:v1.36.5
[config/images] Pulled registry.k8s.io/kube-controller-manager:v1.36.5
[config/images] Pulled registry.k8s.io/kube-scheduler:v1.36.5
[config/images] Pulled registry.k8s.io/kube-proxy:v1.36.5
[config/images] Pulled registry.k8s.io/coredns/coredns:v1.14.2
[config/images] Pulled registry.k8s.io/pause:3.10.2
[config/images] Pulled registry.k8s.io/etcd:3.6.8-0

kubeadm init and its phases

user@k8s-master-1:~$ sudo kubeadm init --config kubeadm-config.yaml
[init] Using Kubernetes version: v1.36.5
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-api.example.com k8s-master-1 k8s-master-1.example.com kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.10.17]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master-1 localhost] and IPs [192.168.10.17 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master-1 localhost] and IPs [192.168.10.17 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "super-admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/instance-config.yaml"
[patches] Applied patch of type "application/strategic-merge-patch+json" to target "kubeletconfiguration"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests"
[kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s
[kubelet-check] The kubelet is healthy after 746.117µs
[control-plane-check] Waiting for healthy control plane components. This can take up to 4m0s
[control-plane-check] Checking kube-apiserver at https://192.168.10.17:6443/livez
[control-plane-check] Checking kube-controller-manager at https://127.0.0.1:10257/healthz
[control-plane-check] Checking kube-scheduler at https://127.0.0.1:10259/livez
[control-plane-check] kube-controller-manager is healthy after 4.779062ms
[control-plane-check] kube-scheduler is healthy after 5.14416ms
[control-plane-check] kube-apiserver is healthy after 16.501302316s
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8s-master-1 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node k8s-master-1 as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: 7k2m9x.q4w8e1r5t6y3u0i2
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Configured RBAC rules to allow the API server kubelet client certificate to access the kubelet API
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

...

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join k8s-api.example.com:6443 --token 7k2m9x.q4w8e1r5t6y3u0i2 \
	--discovery-token-ca-cert-hash sha256:3f9a5c2e8b1d4f7a0c6e9b2d5f8a1c4e7b0d3f6a9c2e5b8d1f4a7c0e3b6d9f2a

In under a minute the control plane is running on the master (kube-apiserver responds sixteen seconds after kubelet starts). Every line of the output belongs to one of the phases; any phase can be run on its own (kubeadm init phase <name>) or skipped (--skip-phases), and knowing the phases means knowing where things are in the cluster.

preflight — checks: kernel version, free ports 6443, 2379, 2380, 10250, 10257, 10259, availability of the CRI socket, an empty /etc/kubernetes/manifests directory, routing turned on (net.ipv4.ip_forward). kubeadm 1.36 only flags enabled swap with a warning ([WARNING Swap]), while missing routing is an error. Errors stop the installation, warnings do not; skip checks with --ignore-preflight-errors only if you understand exactly what you are skipping.

certs — the cluster's public key infrastructure in /etc/kubernetes/pki. Three independent certificate authorities, each with its own area of responsibility:

CA	What it signs
`ca`	the kube-apiserver certificate, the client certificates of components, administrators and nodes; its public key is the root of trust for the whole cluster
`etcd/ca`	the etcd certificates (server, peers) and the etcd clients' certificates — kube-apiserver and health checks. A separate CA means that a certificate issued by the common CA will not be let into etcd
`front-proxy-ca`	the certificate with which kube-apiserver introduces itself to aggregated API servers (such as metrics-server), passing them the user name in headers

Plus the sa.key/sa.pub key pair — not a certificate but the key for signing service account tokens. The kube-apiserver certificate contains all the names by which it is addressed: k8s-api.example.com, the master's name, the names of the kubernetes service inside the cluster and the address 10.96.0.1. Component certificates are valid for a year, CAs for ten years.

kubeconfig — API connection files in /etc/kubernetes; each component has its own identity:

File	Who	Identity in the certificate
`admin.conf`	administrator, kubeadm	`CN=kubernetes-admin`, `O=kubeadm:cluster-admins` — a group bound to the `cluster-admin` role through RBAC
`super-admin.conf`	emergency access	`CN=kubernetes-super-admin`, `O=system:masters` — a group that bypasses RBAC entirely
`controller-manager.conf`	kube-controller-manager	`CN=system:kube-controller-manager`
`scheduler.conf`	kube-scheduler	`CN=system:kube-scheduler`
`kubelet.conf`	the master's kubelet	`CN=system:node:k8s-master-1`, `O=system:nodes`

The difference between the two administrator files is fundamental. The rights of admin.conf are granted through RBAC and can be revoked by deleting the role binding. The rights of super-admin.conf cannot be revoked by anything except replacing the CA: the system:masters group is checked before RBAC. That is why after installation super-admin.conf is moved off the master to a safe place (a password vault, an encrypted archive) and used only if RBAC is broken.

etcd and control-plane — the static pod manifests in /etc/kubernetes/manifests: etcd.yaml, kube-apiserver.yaml, kube-controller-manager.yaml, kube-scheduler.yaml. They contain all the component flags assembled from ClusterConfiguration: for example, kube-apiserver gets --authorization-mode=Node,RBAC, --enable-admission-plugins=NodeRestriction, --service-cluster-ip-range=10.96.0.0/12, and kube-controller-manager gets --cluster-cidr=172.20.0.0/16 and --use-service-account-credentials=true.

kubelet-start — kubelet receives its configuration and restarts. The parameters live in /var/lib/kubelet/config.yaml (our KubeletConfiguration plus whatever kubeadm added itself), the CRI socket is also in /var/lib/kubelet/instance-config.yaml, and the flags file /var/lib/kubelet/kubeadm-flags.env is empty in 1.36 (KUBELET_KUBEADM_ARGS=""): kubeadm has moved all the settings from command-line flags into the configuration file. The [patches] line means kubeadm applied "patches" to the kubelet configuration (a directory of fixes that can be specified with the --patches flag; we have none, so the patch is empty). Now kubelet sees the manifests directory and starts the control plane.

wait-control-plane — kubeadm waits for kubelet to answer on /healthz, and then for all three control plane components to answer on their /livez and /healthz. If something has not come up within four minutes, init stops; the cause is looked for in journalctl -u kubelet and in crictl logs of the failed container.

upload-config — ClusterConfiguration goes into the ConfigMap kubeadm-config, and KubeletConfiguration into kubelet-config (both in kube-system). join and upgrade will take them from there; changing the cluster configuration later means editing these ConfigMaps.

upload-certs — skipped. With the --upload-certs flag kubeadm would have put the CA certificates into a Secret encrypted with a separate key, so that further masters could download them during join --control-plane; the Secret lives for two hours.

mark-control-plane — the label node-role.kubernetes.io/control-plane (this is where ROLES in kubectl get nodes comes from) and the taint node-role.kubernetes.io/control-plane:NoSchedule: ordinary pods will not land on the master. Removing the taint on a working cluster is not advisable — an application that eats the master's memory will bring down the control plane.

bootstrap-token — the token for joining nodes (it lives for 24 hours by default) and everything needed for joining: RBAC rules, automatic approval of certificate requests and the public ConfigMap cluster-info. How this works we'll see with join.

kubelet-finalize — the master's kubelet switches from the certificate embedded in kubelet.conf to the file /var/lib/kubelet/pki/kubelet-client-current.pem, which it renews itself before it expires.

addon — CoreDNS, the cluster's DNS server (a Deployment and the kube-dns service at 10.96.0.10), and kube-proxy (a DaemonSet with our configuration from the kube-proxy ConfigMap) are created as ordinary objects through the API.

show-join-command — the hint with the join command.

Access from the workstation

On the master kubectl can already be set up with the commands from the hint, but it is more convenient to work with the cluster from a workstation. The admin.conf file is readable only by root, so we copy it through a temporary file:

user@k8s-master-1:~$ sudo install -m 600 -o user -g user /etc/kubernetes/admin.conf ~/admin.conf

$ mkdir -p ~/.kube
$ scp user@k8s-master-1.example.com:admin.conf ~/.kube/config
$ chmod 600 ~/.kube/config
$ ssh user@k8s-master-1.example.com rm admin.conf
$ kubectl config get-contexts
CURRENT   NAME                          CLUSTER      AUTHINFO           NAMESPACE
*         kubernetes-admin@kubernetes   kubernetes   kubernetes-admin
$ kubectl cluster-info
Kubernetes control plane is running at https://k8s-api.example.com:6443
CoreDNS is running at https://k8s-api.example.com:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

The name k8s-api.example.com must resolve on the workstation too — in DNS or in its hosts file: the address is taken from the kubeconfig, and the kube-apiserver certificate contains it only under this name and under the master's address.

A kubeconfig is YAML with three lists: clusters (the API address and the CA to verify its certificate with), users (credentials: here a client certificate and key), contexts (cluster–user pairs with a default namespace). current-context selects the active pair, and kubectl config use-context switches it.

The file contains a private key with cluster administrator rights, valid for a year. It must not end up in Git, in shared directories or in correspondence; the 600 permissions are mandatory. Personal accounts for administrators instead of the shared admin.conf — through separate certificates or sign-in via OIDC (single sign-on through an external identity server) — are the topic of part 29.

The pod network: Calico

The node is still NotReady, and CoreDNS is Pending:

$ kubectl get nodes
NAME           STATUS     ROLES           AGE   VERSION
k8s-master-1   NotReady   control-plane   2m    v1.36.5

$ kubectl describe node k8s-master-1 | grep -m1 'NetworkReady'
  Ready            False   Fri, 09 Oct 2026 11:04:52 +0300   Fri, 09 Oct 2026 11:02:39 +0300   KubeletNotReady              container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

kubelet honestly reports that the runtime cannot connect pods to the network because there is no CNI configuration in /etc/cni/net.d. We install Calico. First the custom resource definitions (CRDs) and the Tigera operator, a controller that deploys and updates all the Calico components from a single Installation object:

$ kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.32.2/manifests/v1_crd_projectcalico_org.yaml
customresourcedefinition.apiextensions.k8s.io/apiservers.operator.tigera.io created
customresourcedefinition.apiextensions.k8s.io/gatewayapis.operator.tigera.io created
customresourcedefinition.apiextensions.k8s.io/goldmanes.operator.tigera.io created
...
customresourcedefinition.apiextensions.k8s.io/tiers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusternetworkpolicies.policy.networking.k8s.io created

$ kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.32.2/manifests/tigera-operator.yaml
namespace/tigera-operator created
serviceaccount/tigera-operator created
clusterrole.rbac.authorization.k8s.io/tigera-operator-secrets created
clusterrole.rbac.authorization.k8s.io/tigera-operator created
clusterrolebinding.rbac.authorization.k8s.io/tigera-operator created
rolebinding.rbac.authorization.k8s.io/tigera-operator-secrets created
deployment.apps/tigera-operator created

The first file creates 32 definitions: the operator's own types (*.operator.tigera.io), the Calico types (*.crd.projectcalico.org: address pools, BGP, policies) and one type from the Kubernetes API, clusternetworkpolicies.policy.networking.k8s.io. It takes a while to create because it is large — do not interrupt kubectl create halfway (for example with a pipe into head): some of the types will be left uncreated.

It is kubectl create here and not apply, and this is not carelessness: the Calico CRD definitions are so large that a client-side apply cannot fit them into the last-applied-configuration annotation (the limit is 256 KiB, which we mentioned in part 3): kubectl apply on this file answers The CustomResourceDefinition "installations.operator.tigera.io" is invalid: metadata.annotations: Too long: may not be more than 262144 bytes. We will update them later with kubectl replace or server-side apply.

Now the configuration object itself. The standard custom-resources.yaml file from the Calico repository assumes the pool 192.168.0.0/16, and we have our own network, so we write calico-installation.yaml:

apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
  name: default
spec:
  calicoNetwork:
    linuxDataplane: Nftables
    ipPools:
      - name: default-ipv4-ippool
        cidr: 172.20.0.0/16
        blockSize: 26
        encapsulation: VXLANCrossSubnet
        natOutgoing: Enabled
        nodeSelector: all()
---
apiVersion: operator.tigera.io/v1
kind: APIServer
metadata:
  name: default
spec: {}

linuxDataplane: Nftables — Calico writes its rules (network policies, NAT, packet marking) into nftables, just as kube-proxy does. The Calico documentation requires kube-proxy to run in nftables mode as well when this mode is used, and that is how ours is set up. A single filtering mechanism on the node is easier to diagnose than a mix of iptables and nftables. The nftables mode in Calico 3.32 is stable.
cidr: 172.20.0.0/16 — matches podSubnet from the kubeadm configuration.
blockSize: 26 — the pool is divided into blocks of 64 addresses; a node takes a whole block and hands out addresses to its pods. When the block runs out, the node takes another one. The route to a neighboring node is one per block, not one per pod.
encapsulation: VXLANCrossSubnet — packets between pods on different nodes of the same subnet go as they are, by ordinary routing: the node sends a packet for 172.20.140.0/26 straight to 192.168.10.21. Between nodes in different subnets, where the router in between knows nothing about the pod network, the packet is wrapped in VXLAN — an L2 frame inside a UDP datagram to port 4789 (the protocol is covered in detail in part 18 of the networking series). On our lab setup all the nodes are in 192.168.10.0/24, so there will be no encapsulation at all, and none of its overhead either.
natOutgoing: Enabled — packets from pods to addresses outside the Calico pools (to the internet, to servers in the node network) leave with the node's address: the router 192.168.10.1 has no route to 172.20.0.0/16 and could not answer. In the IPPool object this is the field natOutgoing: true. A consequence worth remembering: an external server sees the node, not the pod, and access restricted "by node addresses" lets in any pod of the cluster. In part 22 this becomes the main topic of the discussion of the NFS server.
nodeSelector: all() — the pool is available to all nodes.
APIServer — the Calico API server, through which kubectl works with projectcalico.org/v3 resources (policies, pools) with validation and defaulting.

The standard file creates two more objects — Goldmane (an aggregator of network flow logs) and Whisker (a web interface for them). They are in technology preview status, and we do not install them for the lab cluster.

$ kubectl create -f calico-installation.yaml
installation.operator.tigera.io/default created
apiserver.operator.tigera.io/default created

$ kubectl get tigerastatus
NAME        AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
apiserver   True        False         False      5s      All objects available
calico      True        False         False      55s     All objects available
ippools     True        False         False      3m5s    All objects available
tiers                                 True               Waiting for Tigera API server to be ready

$ kubectl get nodes
NAME           STATUS   ROLES           AGE     VERSION
k8s-master-1   Ready    control-plane   5m11s   v1.36.5

In the first seconds the tiers line shows a wait: the operator creates the built-in policy "tiers" (more about them in part 20) once the Calico API server starts; in a minute it will become True as well.

The operator created the calico-system namespace with calico-node, calico-typha, calico-kube-controllers, calico-apiserver (the API server from the APIServer object) and csi-node-driver; calico-node placed the CNI plugin /opt/cni/bin/calico and the configuration /etc/cni/net.d/10-calico.conflist on the node, kubelet saw the network — the node is Ready, and CoreDNS has started.

Worker nodes: kubeadm join

k8s-node-1 and k8s-node-2 have already been prepared, with containerd and the Kubernetes packages installed. We take the join command from the init output or, if more than a day has passed and the token has expired, generate a new one:

user@k8s-master-1:~$ sudo kubeadm token create --print-join-command
kubeadm join k8s-api.example.com:6443 --token 1p8v3d.h6n2k9w4x7q0z5m1 --discovery-token-ca-cert-hash sha256:3f9a5c2e8b1d4f7a0c6e9b2d5f8a1c4e7b0d3f6a9c2e5b8d1f4a7c0e3b6d9f2a

user@k8s-node-1:~$ sudo kubeadm join k8s-api.example.com:6443 --token 7k2m9x.q4w8e1r5t6y3u0i2 \
    --discovery-token-ca-cert-hash sha256:3f9a5c2e8b1d4f7a0c6e9b2d5f8a1c4e7b0d3f6a9c2e5b8d1f4a7c0e3b6d9f2a
[preflight] Running pre-flight checks
[preflight] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[preflight] Use 'kubeadm init phase upload-config kubeadm --config your-config-file' to re-upload it.
W1024 13:37:17.660286    3729 utils.go:69] The recommended value for "bindAddress" in "KubeProxyConfiguration" is: ::; the provided value is: 0.0.0.0
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/instance-config.yaml"
[patches] Applied patch of type "application/strategic-merge-patch+json" to target "kubeletconfiguration"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s
[kubelet-check] The kubelet is healthy after 501.234746ms
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

The W… line about bindAddress is a kubeadm warning about the kube-proxy settings: for clusters with IPv6 it recommends the address ::, while our configuration (an IPv4 cluster) has 0.0.0.0; the warning can be ignored.

The same on k8s-node-2. The token in this command is secret: with it and network access to the API, anyone can join their own machine to the cluster as a node. That is why its lifetime is a day, and unneeded tokens are deleted: kubeadm token list, kubeadm token delete <id>.

What happens during join

 kubeadm              kubelet          kube-apiserver      controller-manager
    │                    │                    │                     │
    ├── 1. cluster-info ─┼───────────────────►│                     │
    │ anonymous; then    │                    │                     │
    │ JWS check          │                    │                     │
    │ and CA hash        │                    │                     │
    ├── 2. start ───────►│                    │                     │
    │                    │ bootstrap-         │                     │
    │                    │ kubelet.conf       │                     │
    │                    ├── 3. POST CSR ────►│                     │
    │                    │                    ├── 4. watch CSR ────►│
    │                    │                    │                     │ approve,
    │                    │                    │                     │ sign with CA
    │                    │                    │◄─ 5. certificate ───┤
    │                    │◄─ 6. watch ────────┤                     │
    │                    │ kubelet.conf,      │                     │
    │                    │ own certificate    │                     │
    │                    ├── 7. POST Node ───►│                     │
    │                    │                    │                     │

A new node has nothing: no certificate, no trust in the server. It has only the token and the hash. The task is to obtain both safely.

The token looks like 7k2m9x.q4w8e1r5t6y3u0i2: a public identifier (6 characters) and a secret (16 characters). On the cluster side it is the Secret bootstrap-token-7k2m9x in kube-system with an expiration time and a purpose.

1. Discovery. kubeadm reads the ConfigMap cluster-info in the kube-public namespace anonymously — a kubeconfig with the API address and the cluster CA certificate. Anonymously, because the node cannot trust the server yet. The authenticity of the response is checked in two ways. The bootstrapsigner controller in the controller manager puts into the same ConfigMap a JWS (JSON Web Signature) of its contents made with the token's secret: only someone who knows the token can forge it. And the sha256:… hash from the command is a fingerprint of the CA's public key: kubeadm compares it with the CA from the ConfigMap, and so protects itself even against someone who stole the token and is posing as the cluster. The hash can be computed on the master and compared:

user@k8s-master-1:~$ openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt \
    | openssl pkey -pubin -outform der | openssl dgst -sha256 -hex | sed 's/^.* //'
3f9a5c2e8b1d4f7a0c6e9b2d5f8a1c4e7b0d3f6a9c2e5b8d1f4a7c0e3b6d9f2a

The --discovery-token-unsafe-skip-ca-verification flag disables the hash check; its name honestly says why it is not used.

2. Starting kubelet. kubeadm reads kubeadm-config and kubelet-config from the cluster, writes /var/lib/kubelet/config.yaml and a temporary /etc/kubernetes/bootstrap-kubelet.conf in which the token stands in for a certificate, and starts kubelet.

3. TLS bootstrapping. kubelet authenticates with the token as the user system:bootstrap:7k2m9x from the group system:bootstrappers:kubeadm:default-node-token. The role binding kubeadm:kubelet-bootstrap allows this group exactly one significant action — to create a CertificateSigningRequest (CSR, an object meaning "please sign a certificate for me"): kubelet generates a private key and asks for a client certificate with CN=system:node:k8s-node-1, O=system:nodes from the signer kubernetes.io/kube-apiserver-client-kubelet.

4–5. Approval and signing. The csrapproving controller approves the request automatically — this is permitted by the binding kubeadm:node-autoapprove-bootstrap — and the csrsigning controller signs it with the cluster CA key (/etc/kubernetes/pki/ca.key on the master) and puts the certificate into the CSR's status.

6. Permanent credentials. kubelet fetches the certificate, saves it in /var/lib/kubelet/pki/kubelet-client-<date>.pem (with the symbolic link kubelet-client-current.pem), writes /etc/kubernetes/kubelet.conf and deletes bootstrap-kubelet.conf. The token is no longer needed.

7. Registration. Under its new identity kubelet creates the Node object. The NodeRestriction admission plugin makes sure that system:node:k8s-node-1 can create and modify only a Node named k8s-node-1.

From here everything is familiar: the DaemonSet controllers create kube-proxy and calico-node pods on the new node, calico-node lays down the CNI plugin, and the node becomes Ready. kubelet renews its client certificate itself, with the same CSR mechanism, and those requests are approved automatically by the binding kubeadm:node-autoapprove-certificate-rotation.

The certificate for the kubelet HTTPS port

The kubelet client certificate is what it needs to talk to the API. But kubelet is also a server: on port 10250 it is contacted by kube-apiserver (logs, exec) and metrics-server (resource metrics). By default kubelet generates a self-signed certificate for this port, which nobody can verify — and in such a cluster metrics-server has to be run with verification turned off (--kubelet-insecure-tls), which opens the way to node impersonation.

We have turned on serverTLSBootstrap: true, and kubelet asks the cluster CA for a serving certificate — also through a CSR, but with the signer kubernetes.io/kubelet-serving. kubeadm does not approve such requests automatically, and rightly so: in the request kubelet itself lists the names and addresses it wants the certificate for, and automatic approval would let a compromised node obtain a certificate for someone else's address. Let's look at the requests:

$ kubectl get csr
NAME        AGE     SIGNERNAME                                    REQUESTOR                  REQUESTEDDURATION   CONDITION
csr-2whjl   5m53s   kubernetes.io/kube-apiserver-client-kubelet   system:node:k8s-master-1   <none>              Approved,Issued
csr-6x6hd   7s      kubernetes.io/kubelet-serving                 system:node:k8s-node-2     <none>              Pending
csr-mdcst   5m53s   kubernetes.io/kubelet-serving                 system:node:k8s-master-1   <none>              Pending
csr-nj5kg   17s     kubernetes.io/kubelet-serving                 system:node:k8s-node-1     <none>              Pending
csr-q4fzv   11s     kubernetes.io/kube-apiserver-client-kubelet   system:bootstrap:7k2m9x    <none>              Approved,Issued
csr-w7sgm   19s     kubernetes.io/kube-apiserver-client-kubelet   system:bootstrap:7k2m9x    <none>              Approved,Issued
csr-xj59f   5m33s   kubernetes.io/kubelet-serving                 system:node:k8s-master-1   <none>              Pending

$ kubectl get csr csr-nj5kg -o jsonpath='{.spec.request}' | base64 -d | openssl req -noout -subject -text | grep -E 'Subject:|DNS:'
        Subject: O=system:nodes, CN=system:node:k8s-node-1
                    DNS:k8s-node-1, IP Address:192.168.10.20

$ kubectl certificate approve csr-mdcst csr-xj59f csr-nj5kg csr-6x6hd
certificatesigningrequest.certificates.k8s.io/csr-mdcst approved
certificatesigningrequest.certificates.k8s.io/csr-xj59f approved
certificatesigningrequest.certificates.k8s.io/csr-nj5kg approved
certificatesigningrequest.certificates.k8s.io/csr-6x6hd approved

The workers' client requests (csr-q4fzv, csr-w7sgm) came from the token and were approved automatically; csr-2whjl is the master's own request to renew its client certificate. The serving requests are waiting for us, and the master has two of them: kubelet sent the same request twice, twenty seconds apart, and both need to be approved.

Before approving, we check that the request contains the name and address of exactly this node. Until the serving request is approved, kubelet cannot accept HTTPS connections, and kubectl logs and kubectl exec to pods on that node fail with a TLS error — this is the first thing to check if they do not work on a fresh node:

$ kubectl logs -n kube-system etcd-k8s-master-1 --tail=1
Error from server: Get "https://192.168.10.17:10250/containerLogs/kube-system/etcd-k8s-master-1/etcd?tailLines=1": remote error: tls: internal error
$ kubectl exec -n kube-system etcd-k8s-master-1 -- true
error: unable to upgrade connection: error dialing backend: remote error: tls: internal error

The serving certificate is also valid for a year, and renewing it creates another CSR that has to be approved; in large clusters a separate approver controller is installed for this, which checks names and addresses against the Node object.

Verification

$ kubectl get nodes -o wide
NAME           STATUS   ROLES           AGE     VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION             CONTAINER-RUNTIME
k8s-master-1   Ready    control-plane   7m38s   v1.36.5   192.168.10.17   <none>        Ubuntu 26.04.1 LTS   7.0.0-38-generic (amd64)   containerd://2.2.2
k8s-node-1     Ready    <none>          2m3s    v1.36.5   192.168.10.20   <none>        Ubuntu 26.04.1 LTS   7.0.0-38-generic (amd64)   containerd://2.2.2
k8s-node-2     Ready    <none>          115s    v1.36.5   192.168.10.21   <none>        Ubuntu 26.04.1 LTS   7.0.0-38-generic (amd64)   containerd://2.2.2

$ kubectl get pods -A | grep -v Running
NAMESPACE   NAME   READY   STATUS   RESTARTS   AGE

$ kubectl get pods -n kube-system
NAME                                   READY   STATUS    RESTARTS   AGE
coredns-589f44dc88-bzxpn               1/1     Running   0          7m11s
coredns-589f44dc88-lv5bl               1/1     Running   0          7m11s
etcd-k8s-master-1                      1/1     Running   0          7m29s
kube-apiserver-k8s-master-1            1/1     Running   0          7m34s
kube-controller-manager-k8s-master-1   1/1     Running   0          7m30s
kube-proxy-7rbtd                       1/1     Running   0          7m12s
kube-proxy-sf5g2                       1/1     Running   0          114s
kube-proxy-xmjz2                       1/1     Running   0          2m3s
kube-scheduler-k8s-master-1            1/1     Running   0          7m32s

$ kubectl get pods -n calico-system
NAME                                       READY   STATUS    RESTARTS   AGE
calico-apiserver-7c56559b65-dm42w          1/1     Running   0          6m20s
calico-apiserver-7c56559b65-phtkx          1/1     Running   0          6m20s
calico-kube-controllers-6879897867-8jnzf   1/1     Running   0          6m15s
calico-node-6d2qb                          1/1     Running   0          2m45s
calico-node-l9g44                          1/1     Running   0          5m54s
calico-node-q62j9                          1/1     Running   0          2m36s
calico-typha-55d594d885-7cb8k              1/1     Running   0          2m32s
calico-typha-55d594d885-dsh7d              1/1     Running   0          6m19s
csi-node-driver-7jlkp                      2/2     Running   0          6m17s
csi-node-driver-q6t25                      2/2     Running   0          2m44s
csi-node-driver-s546r                      2/2     Running   0          2m36s

While the workers were joining, some of the Calico pods on them were still starting (Init, ContainerCreating) — this takes about a minute, after which the list of problem pods is empty.

calico-apiserver is two replicas of the Calico API server (from the APIServer object), calico-typha has one replica per two nodes, calico-node runs on every node, and csi-node-driver is the Calico CSI driver, which mounts a service socket for application-layer (L7) policies into pods; the operator always installs it, and without such policies it sits idle. kubectl get pods -A | grep -v Running is a quick way to see the cluster's problem pods: only the header remains here, which means all the pods are running.

The first pod

$ kubectl run hello --image=nginx:1.29 --overrides='{"spec":{"nodeName":"k8s-node-2"}}'
pod/hello created

$ kubectl get pod hello -o wide
NAME    READY   STATUS    RESTARTS   AGE   IP             NODE         NOMINATED NODE   READINESS GATES
hello   1/1     Running   0          6s    172.20.172.4   k8s-node-2   <none>           <none>

$ kubectl run -it --rm check --image=busybox:1.37 --restart=Never \
    --overrides='{"spec":{"nodeName":"k8s-node-1"}}' -- \
    sh -c 'wget -qO- http://172.20.172.4 | grep title; nslookup kubernetes.default.svc.cluster.local'
<title>Welcome to nginx!</title>
Server:    10.96.0.10
Address:   10.96.0.10:53

Name:   kubernetes.default.svc.cluster.local
Address: 10.96.0.1

pod "check" deleted from default namespace

The --overrides flag replaces part of the pod specification: we explicitly set the nodes (nodeName) so that hello and check end up on different ones — otherwise the scheduler might put them together, and the check would not exercise the network between nodes (how it chooses a node by itself and how to control that is in part 13). The check pod on k8s-node-1 reached nginx on k8s-node-2 by the pod's address, and the name of the kubernetes service from the default namespace turned into the address of the API service, 10.96.0.1, through CoreDNS at 10.96.0.10. So the pod network between nodes, kube-proxy and DNS all work — the cluster is ready. kubectl delete pod hello.

Hands-on

What kubeadm left on the master:

user@k8s-master-1:~$ sudo ls /etc/kubernetes /etc/kubernetes/pki /etc/kubernetes/pki/etcd
/etc/kubernetes:
admin.conf  controller-manager.conf  kubelet.conf  manifests  pki  scheduler.conf  super-admin.conf

/etc/kubernetes/pki:
apiserver.crt              apiserver-etcd-client.key  apiserver-kubelet-client.crt  ca.crt  ca.key  front-proxy-ca.crt  front-proxy-client.crt  sa.key
apiserver-etcd-client.crt  apiserver.key              apiserver-kubelet-client.key  etcd    front-proxy-ca.key  front-proxy-client.key  sa.pub

/etc/kubernetes/pki/etcd:
ca.crt  ca.key  healthcheck-client.crt  healthcheck-client.key  peer.crt  peer.key  server.crt  server.key

super-admin.conf is still in place — now is the time to move it off the master. The certificate expiration dates:

user@k8s-master-1:~$ sudo kubeadm certs check-expiration
CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Oct 24, 2027 10:31 UTC   364d            ca                      no
apiserver                  Oct 24, 2027 10:31 UTC   364d            ca                      no
apiserver-etcd-client      Oct 24, 2027 10:31 UTC   364d            etcd-ca                 no
apiserver-kubelet-client   Oct 24, 2027 10:31 UTC   364d            ca                      no
controller-manager.conf    Oct 24, 2027 10:31 UTC   364d            ca                      no
etcd-healthcheck-client    Oct 24, 2027 10:31 UTC   364d            etcd-ca                 no
etcd-peer                  Oct 24, 2027 10:31 UTC   364d            etcd-ca                 no
etcd-server                Oct 24, 2027 10:31 UTC   364d            etcd-ca                 no
front-proxy-client         Oct 24, 2027 10:31 UTC   364d            front-proxy-ca          no
scheduler.conf             Oct 24, 2027 10:31 UTC   364d            ca                      no
super-admin.conf           Oct 24, 2027 10:31 UTC   364d            ca                      no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Oct 21, 2036 10:31 UTC   9y              no
etcd-ca                 Oct 21, 2036 10:31 UTC   9y              no
front-proxy-ca          Oct 21, 2036 10:31 UTC   9y              no

In a year the component certificates will expire and the cluster will stop working. kubeadm upgrade renews them at every version upgrade, and upgrading once a year is the minimum to stay on a supported version; more about this, and about manual renewal, in part 35.

What kubeadm put into the cluster for joining nodes:

$ kubectl get configmap cluster-info -n kube-public -o yaml | grep -E 'jws|server:'
        server: https://k8s-api.example.com:6443
  jws-kubeconfig-7k2m9x: eyJhbGciOiJIUzI1NiIsImtpZCI6IjdrMm05eCJ9..Wq3b7Tg2m9vJ0K4xRZp1cYfH8sLd6NuE5oAiQwMkV2s

user@k8s-master-1:~$ sudo kubeadm token list
TOKEN                     TTL   EXPIRES                USAGES                   DESCRIPTION   EXTRA GROUPS
7k2m9x.q4w8e1r5t6y3u0i2   23h   2026-10-25T10:32:01Z   authentication,signing   <none>        system:bootstrappers:kubeadm:default-node-token

The JWS signature is in "detached" form — the header, an empty spot instead of the payload, and the signature, separated by dots: the payload is the kubeconfig itself, right next to it. The header (eyJhbGciOiJIUzI1NiIsImtpZCI6IjdrMm05eCJ9 is {"alg":"HS256","kid":"7k2m9x"} in base64) names the identifier of the token that signed it. When a token is deleted or expires, the tokencleaner controller deletes the Secret, and bootstrapsigner deletes the signature.

On a worker node — the kubelet certificates and the Calico routes:

user@k8s-node-1:~$ sudo ls -l /var/lib/kubelet/pki/
total 8
-rw------- 1 root root 1118 Oct 24 13:37 kubelet-client-2026-10-24-13-37-18.pem
lrwxrwxrwx 1 root root   59 Oct 24 13:37 kubelet-client-current.pem -> /var/lib/kubelet/pki/kubelet-client-2026-10-24-13-37-18.pem
-rw------- 1 root root 1167 Oct 24 13:37 kubelet-server-2026-10-24-13-37-49.pem
lrwxrwxrwx 1 root root   59 Oct 24 13:37 kubelet-server-current.pem -> /var/lib/kubelet/pki/kubelet-server-2026-10-24-13-37-49.pem

user@k8s-node-1:~$ ip route | grep 172.20
172.20.70.192/26 via 192.168.10.17 dev ens18 proto 80 onlink
172.20.172.0/26 via 192.168.10.21 dev ens18 proto 80 onlink
blackhole 172.20.211.0/26 proto 80
172.20.211.1 dev cali01e594701c5 scope link metric 1024
172.20.211.3 dev calie9f7c1b68b7 scope link metric 1024

k8s-node-1 got the block 172.20.211.0/26; the blackhole route for the whole block drops packets to nonexistent pods of the block, and each real pod gets its own /32 route to its cali… veth (172.20.211.3 is our check, while it was alive). The blocks of the master and of k8s-node-2 are reachable directly through their addresses in the shared subnet — without VXLAN, as intended by the VXLANCrossSubnet mode; the onlink flag tells the kernel that the next hop is reachable on this interface. proto 80 is the protocol number with which Calico marks routes in the kernel: they are installed by BIRD, the BGP daemon inside calico-node, through which the nodes exchange routes to the blocks by default. A router with BGP is not needed for this: the nodes establish sessions with each other (a "full mesh", node-to-node mesh), and BGP does not leave the cluster. The sessions are visible right in calico-node:

user@k8s-node-1:~$ sudo crictl exec $(sudo crictl ps --name calico-node -q) birdcl show protocols
BIRD v0.3.3+birdv1.6.8 ready.
name     proto    table    state  since       info
static1  Static   master   up     10:39:27
kernel1  Kernel   master   up     10:39:27
device1  Device   master   up     10:39:27
direct1  Direct   master   up     10:39:27
Mesh_192_168_10_21 BGP      master   up     10:39:29    Established
Mesh_192_168_10_17 BGP      master   up     10:39:28    Established

How BGP works, and how Calico announces the pod networks and service addresses outward, to the router, is covered in the BGP series.

And the nftables tables:

user@k8s-node-1:~$ sudo nft list tables
table ip mangle
table ip filter
table ip6 mangle
table ip nat
table ip6 nat
table ip6 filter
table ip kube-proxy
table ip6 kube-proxy
table ip calico
table arp calico-arp

kube-proxy holds the service rules, and calico and calico-arp hold the Calico rules (policies, NAT, packet marking). The mangle, filter and nat tables are created by the iptables-nft compatibility layer: through it kubelet writes the familiar iptables commands into the right chains (its service chains KUBE-KUBELET-CANARY and KUBE-FIREWALL are visible there). What lies in the kube-proxy tables and how a packet addressed to a service becomes a packet addressed to a pod is the topic of part 15.

If something goes wrong

While the cluster holds no valuable data, it is easier to start a failed installation over than to repair it:

user@k8s-node-2:~$ sudo kubeadm reset -f
[preflight] Running pre-flight checks
W1024 13:43:30.795284    6560 removeetcdmember.go:105] [reset] No kubeadm config, using etcd pod spec to get data directory
[reset] Deleted contents of the etcd data directory: /var/lib/etcd
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Deleting contents of directories: [/etc/kubernetes/manifests /var/lib/kubelet /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/super-admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]

The reset process does not perform cleanup of CNI plugin configuration,
network filtering rules and kubeconfig files.

For information on how to perform this cleanup manually, please see:
    https://k8s.io/docs/reference/setup-tools/kubeadm/kubeadm-reset/

kubeadm reset stops kubelet and deletes the containers, manifests, certificates, /var/lib/etcd (even on a worker node, where there is none) and the kubelet configuration, but it does not touch what other programs have done, as it honestly warns. After reset the node still has the CNI configuration and the Calico data, the vxlan.calico interface, the routes to the pod blocks, and the Calico and kube-proxy nftables tables:

user@k8s-node-2:~$ sudo ls /etc/cni/net.d /var/lib/calico
/etc/cni/net.d:
10-calico.conflist  calico-kubeconfig

/var/lib/calico:
mtu  nodename
user@k8s-node-2:~$ ip -br link | grep -E 'cali|vxlan'; ip route | grep 172.20; sudo nft list tables | grep -E 'calico|kube-proxy'
vxlan.calico     UNKNOWN        66:05:38:46:ab:ea <BROADCAST,MULTICAST,UP,LOWER_UP>
172.20.70.192/26 via 192.168.10.17 dev ens18 proto 80 onlink
blackhole 172.20.172.0/26 proto 80
172.20.211.0/26 via 192.168.10.20 dev ens18 proto 80 onlink
table ip kube-proxy
table ip6 kube-proxy
table ip calico
table arp calico-arp

The CNI files and the Calico data are deleted by hand, while the interfaces, routes and nftables rules are easiest to remove with a reboot — they do not survive a restart:

user@k8s-node-2:~$ sudo rm -rf /etc/cni/net.d /var/lib/cni /var/lib/calico
user@k8s-node-2:~$ sudo reboot

After the reboot the node has no vxlan.calico, no 172.20 routes and no calico or kube-proxy tables, and it can be joined again. For a worker node, the node object is then deleted on the master: kubectl delete node k8s-node-2, and for the master $HOME/.kube is deleted as well.

The most frequent causes of failure and what is visible in each case:

swap turned on — kubeadm only warns ([WARNING Swap]), and kubelet does not start: running with swap on is not supported, please disable swap or set --fail-swap-on flag to false;
net.ipv4.ip_forward = 0 — kubeadm init stops at the check [ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1 (kubeadm 1.36 does not check this during join, so the setting has to be made on all nodes beforehand);
the name k8s-api.example.com not resolving on the nodes — join fails;
clock drift — errors like x509: certificate has expired or is not yet valid;
a forgotten SystemdCgroup = true — the control plane pods start restarting every few minutes.

Sources

Installing kubeadm and Creating a cluster with kubeadm — the official procedure this part is built on.
The kubeadm reference: init and its phases, join, reset, certs and the v1beta4 configuration.
TLS bootstrapping for kubelet, CertificateSigningRequests and kubelet certificate rotation.
Ports and Protocols, container runtimes and swap on a node; kube-proxy.
containerd: getting started and CRI configuration; the crictl utility.
Calico: installing on self-managed on-premises clusters, VXLAN and IP-in-IP and nftables mode.
chrony and RFC 8915 — Network Time Security for NTP.

Next — part 5: how do you build an image and where do you keep it. How to build the image of the hello application, put it in a private registry and connect the registry to the cluster.