7 November 2026, 10:39Виталий Лещенко

Kubernetes for beginners. Part 35: Upgrading the cluster

All parts of the series “Kubernetes for beginners”

The cluster built in part 4 cannot stay on its version forever: each Kubernetes minor version is supported for a little over a year, kubeadm certificates live exactly one year, and the kernel and packages on the nodes get security fixes every week. Today is about cluster maintenance: how versions work and how much skew between components is allowed, what to check before an upgrade, how kubeadm upgrade changes the master and the workers, how certificates are renewed, how to update the nodes' OS and how to replace a node entirely. All of it on the example of upgrading the lab from 1.36 to 1.37. Before you start you need a fresh etcd snapshot and a copy of /etc/kubernetes/pki from the previous part, and drain, cordon and PodDisruptionBudget, which no node maintenance can do without, are covered in part 13.

Versions and support periods

A Kubernetes version is 1.MINOR.PATCH. A minor version (1.36, 1.37) brings new features, moves features from alpha to beta and GA, removes deprecated things. A patch version (1.36.5 → 1.36.6) contains only bug and vulnerability fixes, with no behaviour changes; patches come out roughly once a month.

About three minor versions come out per year. The project supports the latest three: each gets patches for roughly 14 months — a year of regular support and two months during which only critical bugs and vulnerabilities are fixed. At the time of writing:

Version	Latest patch	End of support
1.37	1.37.2	28 October 2027
1.36	1.36.6	28 June 2027
1.35	1.35.10	28 February 2027

Version 1.34 stopped receiving patches on 27 October 2026. Hence the rhythm: upgrade the cluster to the next minor version roughly every 4–8 months and apply patches of the current one as they come out. A cluster that skipped a year ends up on an unsupported version and has to be upgraded several times in a row: minor versions cannot be skipped. From 1.35 to 1.37 you go through 1.36 — project policy requires this even for a single-node control plane, because the migrations of stored objects and component configuration are designed for a step of exactly one version.

Version skew

Cluster components are not upgraded at the same time, and the version skew policy says how far apart their versions may be:

Component	Relative to kube-apiserver
kube-apiserver in HA	all instances within one minor version of each other
kube-controller-manager, kube-scheduler	not newer than the apiserver, may lag by one minor version
kubelet	not newer than the apiserver, may lag by up to three minor versions
kube-proxy	not newer than the apiserver, may lag by up to three minor versions, and no more than three in either direction from the kubelet on its node
kubectl	one minor version older or newer than the apiserver

Consequences:

The order is always the same: control plane first, then the nodes. A kubelet newer than the apiserver is not supported, so a worker cannot be upgraded before the master.
Workers don't have to be upgraded right away. After the master moves to 1.37, a kubelet 1.36 and even 1.34 keeps working. That lets you stretch the upgrade of a large cluster over days. But you can't stay like that for long: if a kubelet lags by three versions, the next control plane version can't be installed until the nodes are upgraded.
The kubelet minor version is changed only on a drained node. The skew policy says so directly: an in-place minor version upgrade of the kubelet, with pods running, is not supported. The new version may handle the state in /var/lib/kubelet and the containers started by the old one differently, and the project does not test such a transition.
kubectl on the workstation is upgraded together with the cluster: kubectl 1.36 works with apiserver 1.37, kubectl 1.35 no longer does.

Preparation

An upgrade changes the whole cluster at once, and most of the work is done before the first command.

Release notes. Start with the "Urgent Upgrade Notes" section of the new version's CHANGELOG: it lists what needs action before the upgrade. 1.37 has four such items, and none of them affects our lab: SELinux volume mounting (SELinuxMount) became stable and is enabled by default — that matters only for nodes with SELinux; the alpha API version scheduling.k8s.io/v1alpha2 was removed; eventRecordQPS: 0 in the kubelet configuration now means "no limit"; the kubelet writes its effective configuration to its log, so it is worth checking who can read node logs through the API. Next, the kubeadm changes: in 1.37 it no longer understands the kubeadm.k8s.io/v1beta3 configuration format. We have written the lab configuration in v1beta4 since part 4, and an older one is converted with kubeadm config migrate from kubeadm 1.35 or 1.36 — before the upgrade.

It also helps to remember what changed earlier and is easy to miss when you jump through a version. Since 1.35 the kubelet by default refuses to start on a node with cgroup v1 (failCgroupV1: true). The lab nodes have run on cgroup v2 with SystemdCgroup = true in containerd since part 4, but every node is worth checking:

user@k8s-node-1:~$ stat -fc %T /sys/fs/cgroup
cgroup2fs

Removed APIs. Kubernetes removes beta API versions a few releases after a stable replacement appears. After the removal a manifest with apiVersion: flowcontrol.apiserver.k8s.io/v1beta3, for example, simply won't apply, and a Helm release that contains it will stop upgrading. How to find such places:

the apiserver metric apiserver_requested_deprecated_apis — it shows which deprecated API versions someone has called, and the removed_release label says in which release they will be removed. This catches clients that are running now: controllers, scripts, CI;
the apiserver audit log (part 33): requests to deprecated APIs carry the k8s.io/deprecated annotation, which leads to the specific client;
manifests and charts — with pluto (checks files and Helm releases for deprecated and removed apiVersions for a given Kubernetes version) or kubent (kube-no-trouble, the same for objects in the cluster).

On the lab the metric shows one call:

$ kubectl get --raw /metrics | grep '^apiserver_requested_deprecated_apis'
apiserver_requested_deprecated_apis{group="",removed_release="",resource="endpoints",subresource="",version="v1"} 1

That's the Endpoints object: since 1.33 it is deprecated in favour of EndpointSlice, but there are no plans to remove it — removed_release is empty. It would be a reason to worry if the label held the version we are moving to.

pluto checks a directory of manifests and the objects in the cluster. For the example, a FlowSchema with an old API version was put into the directory next to the hello manifest:

$ pluto detect-files -d ./manifests --target-versions k8s=v1.37.0 -o wide
NAME    NAMESPACE   KIND         VERSION                                REPLACEMENT                       DEPRECATED   DEPRECATED IN   REMOVED   REMOVED IN   REPL AVAIL   REPL AVAIL IN
batch   <UNKNOWN>   FlowSchema   flowcontrol.apiserver.k8s.io/v1beta3   flowcontrol.apiserver.k8s.io/v1   true         v1.24.0         true      v1.32.0      false
$ pluto detect-all-in-cluster --target-versions k8s=v1.37.0 -o wide
There were no resources found with known deprecated apiVersions.

detect-helm does the same for the manifests stored in Helm releases. Whatever is found is rewritten to the new API version — by hand or with the kubectl convert plugin (installed separately from kubectl), which converts a manifest between versions of the same resource. Objects already stored in etcd don't need converting: the apiserver keeps them in its storage version and serves them in any supported one.

The latest removals in the deprecated API migration guide are for version 1.32: it lists only beta versions, and removals of alpha versions, such as scheduling.k8s.io/v1alpha2 in 1.37, are found in the CHANGELOG. Our lab is clean at this step. But the habit of checking is worth building: every removal breaks exactly those clusters where nobody checked.

Component compatibility. Everything running in the cluster on top of Kubernetes has its own matrix of supported versions:

Calico 3.32 is tested with Kubernetes 1.34–1.36, and Calico 3.33 with 1.35–1.37. So Calico is upgraded to 3.33 first, on the 1.36 cluster (both versions are within its range), and only then Kubernetes. Upgrading Calico with the Tigera operator is two kubectl apply commands from its guide: the CRDs and the operator manifest of the new version, after which the operator upgrades calico-node node by node itself:

$ curl -fsSLO https://raw.githubusercontent.com/projectcalico/calico/v3.33.0/manifests/v1_crd_projectcalico_org.yaml
$ curl -fsSLO https://raw.githubusercontent.com/projectcalico/calico/v3.33.0/manifests/tigera-operator.yaml
$ kubectl apply --server-side --force-conflicts -f v1_crd_projectcalico_org.yaml
...
customresourcedefinition.apiextensions.k8s.io/tiers.crd.projectcalico.org serverside-applied
$ kubectl apply --server-side --force-conflicts -f tigera-operator.yaml
...
deployment.apps/tigera-operator serverside-applied

The operator restarts the Calico pods one node at a time. On the lab it took five and a half minutes; application pods were not restarted. Readiness is visible from the version in the Installation object and from tigerastatus:

$ kubectl get installation default -o jsonpath='{.status.calicoVersion}{"\n"}'
v3.33.0
$ kubectl get tigerastatus
NAME        AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
apiserver   True        False         False      85s     All objects available
calico      True        False         False      1s      All objects available
ippools     True        False         False      12d     All objects available
tiers       True        False         False      12d     All objects available

MetalLB, Traefik, cert-manager, csi-driver-nfs and csi-driver-smb, metrics-server, kube-prometheus-stack — every project has a table of supported Kubernetes versions in its documentation. If a component doesn't support the new version yet, the cluster upgrade waits for it, not the other way round.

Health and a backup. Only a healthy cluster should be upgraded: all nodes Ready, no pods in CrashLoopBackOff, etcdctl endpoint health answers. And right before the upgrade — a manual etcd snapshot and a copy of /etc/kubernetes/pki as in part 34: when moving to 1.37 kubeadm also upgrades etcd itself from 3.6 to 3.7, and a snapshot taken on the old version is the most reliable way back.

Upgrade order

 1. workstation       Calico 3.32 → 3.33, etcd snapshot, pki copy
                            │
                            ▼
 2. k8s-master-1      repository v1.37 → kubeadm 1.37.2
                      kubeadm upgrade plan
                      kubeadm upgrade apply v1.37.2
                        static pods: etcd, apiserver,
                        controller-manager, scheduler
                        certificates, CoreDNS, kube-proxy
                      drain → kubelet, kubectl 1.37.2 → uncordon
                            │
                            ▼
 3. k8s-node-1        drain → kubeadm 1.37.2 → kubeadm upgrade node
                      → kubelet 1.37.2 → uncordon
                            │
                            ▼
 4. k8s-node-2        the same
                            │
                            ▼
 5. workstation       kubectl 1.37.2, verification

Nodes are upgraded one at a time: while a node is drained, the others carry its load, and with two workers on the lab that means the whole application lives on one node for a while. In a bigger cluster nodes are upgraded in small groups, but so that the remaining capacity is enough for all pods.

The master

Repository. Kubernetes packages on pkgs.k8s.io are split into a separate repository per minor version: the v1.36 repository doesn't contain 1.37 at all. This is deliberate — apt upgrade can't accidentally jump to a new minor version. Moving to a new one is a one-line edit:

user@k8s-master-1:~$ cat /etc/apt/sources.list.d/kubernetes.list
deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.36/deb/ /
user@k8s-master-1:~$ sudo sed -i 's#/v1.36/#/v1.37/#' /etc/apt/sources.list.d/kubernetes.list
user@k8s-master-1:~$ sudo apt-get update
user@k8s-master-1:~$ apt-cache madison kubeadm
   kubeadm | 1.37.2-1.1 | https://pkgs.k8s.io/core:/stable:/v1.37/deb  Packages
   kubeadm | 1.37.1-1.1 | https://pkgs.k8s.io/core:/stable:/v1.37/deb  Packages
   kubeadm | 1.37.0-1.1 | https://pkgs.k8s.io/core:/stable:/v1.37/deb  Packages

kubeadm. After installation in part 4 the kubeadm, kubelet and kubectl packages are frozen with apt-mark hold, so that neither apt upgrade nor automatic updates touch them. For the upgrade the hold is lifted and set again:

user@k8s-master-1:~$ sudo apt-mark unhold kubeadm && \
    sudo apt-get install -y kubeadm='1.37.2-*' && \
    sudo apt-mark hold kubeadm
Canceled hold on kubeadm.
...
Setting up kubeadm (1.37.2-1.1)…
...
kubeadm set on hold.
user@k8s-master-1:~$ kubeadm version -o short
v1.37.2

Plan. kubeadm upgrade plan checks that the cluster is healthy and upgradable and shows what will turn into what:

user@k8s-master-1:~$ sudo kubeadm upgrade plan
[preflight] Running pre-flight checks.
[upgrade/config] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[upgrade/config] Use 'kubeadm init phase upload-config kubeadm --config your-config-file' to re-upload it.
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: 1.36.5
[upgrade/versions] kubeadm version: v1.37.2
[upgrade/versions] Target version: v1.37.2
[upgrade/versions] Latest version in the v1.36 series: v1.36.6

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   NODE           CURRENT   TARGET
kubelet     k8s-master-1   v1.36.5   v1.36.6
kubelet     k8s-node-1     v1.36.5   v1.36.6
kubelet     k8s-node-2     v1.36.5   v1.36.6

Upgrade to the latest version in the v1.36 series:

COMPONENT                 NODE           CURRENT   TARGET
kube-apiserver            k8s-master-1   v1.36.5   v1.36.6
kube-controller-manager   k8s-master-1   v1.36.5   v1.36.6
kube-scheduler            k8s-master-1   v1.36.5   v1.36.6
kube-proxy                               1.36.5    v1.36.6
CoreDNS                                  v1.14.2   v1.14.6
etcd                      k8s-master-1   3.6.8-0   3.7.0-0

You can now apply the upgrade by executing the following command:

	kubeadm upgrade apply v1.36.6

_____________________________________________________________________

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   NODE           CURRENT   TARGET
kubelet     k8s-master-1   v1.36.5   v1.37.2
kubelet     k8s-node-1     v1.36.5   v1.37.2
kubelet     k8s-node-2     v1.36.5   v1.37.2

Upgrade to the latest stable version:

COMPONENT                 NODE           CURRENT   TARGET
kube-apiserver            k8s-master-1   v1.36.5   v1.37.2
kube-controller-manager   k8s-master-1   v1.36.5   v1.37.2
kube-scheduler            k8s-master-1   v1.36.5   v1.37.2
kube-proxy                               1.36.5    v1.37.2
CoreDNS                                  v1.14.2   v1.14.6
etcd                      k8s-master-1   3.6.8-0   3.7.0-0

You can now apply the upgrade by executing the following command:

	kubeadm upgrade apply v1.37.2

_____________________________________________________________________


The table below shows the current state of component configs as understood by this version of kubeadm.
Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or
resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually
upgrade to is denoted in the "PREFERRED VERSION" column.

API GROUP                 CURRENT VERSION   PREFERRED VERSION   MANUAL UPGRADE REQUIRED
kubeproxy.config.k8s.io   v1alpha1          v1alpha1            no
kubelet.config.k8s.io     v1beta1           v1beta1             no
_____________________________________________________________________

There are two options. Patch 1.36.6 is out for 1.36, and kubeadm offers it first and then the move to 1.37.2. Skipping patches is fine: the "one at a time" rule applies only to minor versions, so we go straight to 1.37.2. Note etcd: in both options it is 3.7.0 — the etcd version is chosen by the kubeadm that performs the upgrade, and kubeadm 1.37 knows only that one. Whoever wants to stay on 1.36 with its etcd installs the kubeadm 1.36.6 patch from the v1.36 repository.

The last table shows the component configuration versions (KubeProxyConfiguration, where we set the nftables mode, and KubeletConfiguration). A "yes" in the last column would mean that the configuration format changed and it has to be converted by hand before the upgrade.

Apply. kubeadm upgrade apply does the following:

checks that the apiserver is reachable, all nodes are Ready, the control plane is healthy, and that the target version doesn't violate the skew policy. For the health check it creates a Job upgrade-health-check with a single pause container in kube-system and waits a minute for it to complete — this checks the whole chain: apiserver, controller-manager, scheduler, kubelet;
pulls the images of the new components in advance, so as not to wait for them at the moment of replacement;
upgrades the component configurations if their format needs a migration;
replaces the static pods one by one: writes new manifests to /etc/kubernetes/manifests, waits for the kubelet to start the new pod and for it to become healthy, and rolls the component back if it didn't come up. The old manifests and a copy of the etcd data go to /etc/kubernetes/tmp;
renews all control plane certificates on this node for another year (disabled with --certificate-renewal=false, but there's no reason to disable it);
applies the new addon manifests: the CoreDNS Deployment and the kube-proxy DaemonSet with the new version's images, together with the RBAC rules they need. In a cluster with several masters the addons are upgraded only after the last master has been upgraded; we have one master, so it happens right away.

etcd is upgraded together with the control plane, and there's a subtlety: while etcd restarts, requests to kube-apiserver hang. The kubeadm documentation suggests a workaround — gracefully stop the apiserver a few seconds before the upgrade, so that in-flight requests can finish:

user@k8s-master-1:~$ sudo killall -s SIGTERM kube-apiserver; sleep 20
user@k8s-master-1:~$ sudo kubeadm upgrade apply v1.37.2 -y
[upgrade] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[upgrade] Use 'kubeadm init phase upload-config kubeadm --config your-config-file' to re-upload it.
[upgrade/preflight] Running preflight checks
[upgrade] Running cluster health checks
[preflight] Some fatal errors occurred:
	[ERROR CreateJob]: Job "upgrade-health-check-1793951130060" in the namespace "kube-system" did not complete in 1m0s: no condition of type Complete
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
error: error execution phase preflight: preflight checks failed
To see the stack trace of this error execute with --v=5 or higher

On a single master this trick didn't work. The kubelet restarted the stopped apiserver only after a minute, and by then the controller-manager and the scheduler had lost their leader leases. The health check Job was created, but its pod was scheduled to a node only after 42 seconds, and the minute kubeadm waits wasn't enough. Nothing broke: the Job completed by itself a little later, and the cluster stayed on 1.36. The takeaway for a single-node control plane: either don't stop the apiserver at all, or after stopping it wait until kubectl get --raw /readyz answers ok again and give it another minute. Running kubeadm upgrade apply again on the settled cluster succeeded:

user@k8s-master-1:~$ sudo kubeadm upgrade apply v1.37.2 -y
[upgrade] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[upgrade] Use 'kubeadm init phase upload-config kubeadm --config your-config-file' to re-upload it.
[upgrade/preflight] Running preflight checks
[upgrade] Running cluster health checks
[upgrade/preflight] You have chosen to upgrade the cluster version to "v1.37.2"
[upgrade/versions] Cluster version: v1.36.5
[upgrade/versions] kubeadm version: v1.37.2
[upgrade/preflight] Pulling images required for setting up a Kubernetes cluster
[upgrade/preflight] This might take a minute or two, depending on the speed of your internet connection
[upgrade/preflight] You can also perform this action beforehand using 'kubeadm config images pull'
[upgrade/control-plane] Upgrading your static Pod-hosted control plane to version "v1.37.2" (timeout: 5m0s)...
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests2526878674"
[upgrade/staticpods] Preparing for "etcd" upgrade
[upgrade/staticpods] Renewing etcd-server certificate
[upgrade/staticpods] Renewing etcd-peer certificate
[upgrade/staticpods] Renewing etcd-healthcheck-client certificate
[upgrade/staticpods] Moving new manifest to "/etc/kubernetes/manifests/etcd.yaml" and backing up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2026-11-06-07-48-09/etcd.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This can take up to 5m0s
E1106 07:48:44.339960 3287212 request.go:1198] "Unexpected error when reading response body" err="net/http: request canceled (Client.Timeout or context cancellation while reading body)"
[apiclient] Found 1 Pods for label selector component=etcd
[upgrade/staticpods] Component "etcd" upgraded successfully!
[upgrade/etcd] Waiting for etcd to become available
[upgrade/staticpods] Preparing for "kube-apiserver" upgrade
[upgrade/staticpods] Renewing apiserver certificate
[upgrade/staticpods] Renewing apiserver-kubelet-client certificate
[upgrade/staticpods] Renewing front-proxy-client certificate
[upgrade/staticpods] Renewing apiserver-etcd-client certificate
[upgrade/staticpods] Moving new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backing up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2026-11-06-07-48-09/kube-apiserver.yaml"
...
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-controller-manager" upgrade
[upgrade/staticpods] Renewing controller-manager.conf certificate
...
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-scheduler" upgrade
[upgrade/staticpods] Renewing scheduler.conf certificate
...
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[upgrade/control-plane] The control plane instance for this node was successfully upgraded!
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upgrade/kubeconfig] The kubeconfig files for this node were successfully upgraded!
W1106 07:51:15.059802 3287212 postupgrade.go:105] Using temporary directory /etc/kubernetes/tmp/kubeadm-kubelet-config-2026-11-06-07-51-15 for kubelet config. To override it set the environment variable KUBEADM_UPGRADE_DRYRUN_DIR
[upgrade] Backing up kubelet config file to /etc/kubernetes/tmp/kubeadm-kubelet-config-2026-11-06-07-51-15/config.yaml
[patches] Applied patch of type "application/strategic-merge-patch+json" to target "kubeletconfiguration"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade/kubelet-config] The kubelet configuration for this node was successfully upgraded!
[upgrade/bootstrap-token] Configuring bootstrap token and cluster-info RBAC rules
...
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

[upgrade] SUCCESS! A control plane node of your cluster was upgraded to "v1.37.2".

[upgrade] Now please proceed with upgrading the rest of the nodes by following the right order.

The whole control plane upgrade took four and a half minutes. The line E1106 … request canceled is exactly that request hanging during the etcd restart: kubeadm's own request to the apiserver didn't get an answer, and kubeadm simply retried it. The -y flag answers "yes" to the Are you sure you want to proceed? [y/N] question that is asked before the upgrade otherwise.

If the command failed halfway — say, the session dropped — run it again: kubeadm upgrade is idempotent and brings the state to the target. If a component itself didn't come up and the automatic rollback didn't work, the old manifests are in /etc/kubernetes/tmp/kubeadm-backup-manifests-* and the copy of the etcd data is in /etc/kubernetes/tmp/kubeadm-backup-etcd-*; they are put back into /etc/kubernetes/manifests and /var/lib/etcd by hand. The tmp directory itself is not removed after the upgrade; it is cleaned up by hand once it's clear that everything works.

kubeadm rewrites the etcd manifest too, and the data directory in it comes from etcd.local.dataDir of the kubeadm configuration — /var/lib/etcd in our case. That's why in the previous part the snapshot was restored into the original directory: an edit of hostPath in the manifest would have been silently lost here.

kubelet and kubectl on the master. The control plane is already 1.37, but the kubelet on the master is still 1.36. It is upgraded on a drained node — even if there's almost nothing to drain:

$ kubectl drain k8s-master-1 --ignore-daemonsets
node/k8s-master-1 cordoned
Warning: ignoring DaemonSet-managed Pods: calico-system/calico-node-n6sv7, calico-system/csi-node-driver-r4ltf, kube-system/kube-proxy-wrtc9
node/k8s-master-1 drained

On the lab, CoreDNS and the Calico pods were already on the workers by this point: the Calico upgrade and the CoreDNS replacement recreated them, and the scheduler placed the new pods on other nodes. But on another cluster anything with a matching toleration may be running on the master, and drain will evict it.

user@k8s-master-1:~$ sudo apt-mark unhold kubelet kubectl && \
    sudo apt-get install -y kubelet='1.37.2-*' kubectl='1.37.2-*' && \
    sudo apt-mark hold kubelet kubectl
user@k8s-master-1:~$ sudo systemctl daemon-reload
user@k8s-master-1:~$ sudo systemctl restart kubelet

$ kubectl uncordon k8s-master-1
node/k8s-master-1 uncordoned
$ kubectl get nodes
NAME           STATUS   ROLES           AGE   VERSION
k8s-master-1   Ready    control-plane   12d   v1.37.2
k8s-node-1     Ready    <none>          12d   v1.36.5
k8s-node-2     Ready    <none>          12d   v1.36.5

The VERSION column is the kubelet version, not the control plane's. The cluster is now in an allowed mixed state: apiserver 1.37, workers 1.36.

Workers

The order for each worker: drain it, upgrade kubeadm and through it the kubelet configuration, upgrade the kubelet, bring the node back into service.

Draining. kubectl drain marks the node unschedulable (cordon) and evicts pods through the Eviction API rather than deleting them directly. The difference is that an eviction checks the PodDisruptionBudget: if after evicting a pod its application would have fewer available replicas than the PDB requires, the apiserver answers 429, and drain retries until a replacement comes up on another node. That way maintenance doesn't bring the application down, as long as there are enough replicas and they are spread across nodes. For hello we have two replicas spread across nodes (topology spread from part 13) and a PDB with minAvailable: 1:

$ kubectl drain k8s-node-1 --ignore-daemonsets --delete-emptydir-data --timeout=10m
node/k8s-node-1 cordoned
Warning: ignoring DaemonSet-managed Pods: calico-system/calico-node-gx9mh, calico-system/csi-node-driver-x6xtp, kube-system/kube-proxy-t78zm
evicting pod tigera-operator/tigera-operator-686489b657-5vwv8
evicting pod calico-system/calico-apiserver-78bf74db6b-fhcb8
evicting pod demo/hello-7f74d68455-wlwkw
evicting pod kube-system/coredns-559f6c778d-rw859
evicting pod calico-system/calico-typha-757df7544f-b6h8m
pod/tigera-operator-686489b657-5vwv8 evicted
pod/calico-apiserver-78bf74db6b-fhcb8 evicted
pod/calico-typha-757df7544f-b6h8m evicted
pod/hello-7f74d68455-wlwkw evicted
pod/coredns-559f6c778d-rw859 evicted
node/k8s-node-1 drained

The PDB allowed the eviction: the second hello replica is running and ready on k8s-node-2, so after the eviction one of two is available, exactly what minAvailable: 1 requires. The evicted replica is recreated by the ReplicaSet controller and, since k8s-node-1 is closed for scheduling, also lands on k8s-node-2. Calico has its own PDBs too (calico-typha, calico-apiserver, each with maxUnavailable: 1), and they work the same way.

A refusal looks like this — the next node, k8s-node-2, showed it, since by then it held both hello replicas and both calico-typha pods:

$ kubectl drain k8s-node-2 --ignore-daemonsets --delete-emptydir-data --timeout=10m
node/k8s-node-2 cordoned
...
evicting pod demo/hello-7f74d68455-t4kjd
evicting pod demo/hello-7f74d68455-jlc2t
evicting pod kube-system/coredns-559f6c778d-kl4bj
error when evicting pods/"calico-typha-757df7544f-b2vk5" -n "calico-system" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
error when evicting pods/"hello-7f74d68455-jlc2t" -n "demo" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
...
pod/hello-7f74d68455-t4kjd evicted
evicting pod demo/hello-7f74d68455-jlc2t
...
pod/calico-typha-757df7544f-b2vk5 evicted
pod/hello-7f74d68455-jlc2t evicted
node/k8s-node-2 drained

The PDB let the first replica go, but not the second: until the replacement became ready on k8s-node-1, evicting it would have left the application without a single available replica. drain retried every five seconds and finished half a minute later.

Flags:

--ignore-daemonsets — DaemonSet pods are not evicted: the DaemonSet controller would put them back on the node anyway, and they (calico-node, kube-proxy, the MetalLB speaker, the CSI node plugin) are exactly what the node needs to work;
--delete-emptydir-data — permission to evict pods with emptyDir; the data in it disappears with the pod. Without the flag drain stops at such a pod and asks for confirmation;
--timeout — how long to wait in total. Without it, a drain stuck on a PDB will try forever.

Two cases where drain won't get through on its own:

A PDB that can't be satisfied: one replica and minAvailable: 1 (or maxUnavailable: 0). Such a pod can never be evicted — that's a configuration error, not protection. An application that has to survive node maintenance needs at least two replicas on different nodes;
not enough room: the evicted pods don't fit on the remaining nodes by requests and hang in Pending, and the PDB won't let the next ones be evicted. On a lab with two workers this is a real risk: for the duration of the maintenance everything has to fit on one node. kubectl describe node k8s-node-2 helps check it in advance — the "Allocated resources" section shows the sum of requests.

kubeadm and the kubelet configuration. On a worker the repository is changed and the new kubeadm installed as well, after which kubeadm upgrade node fetches the ClusterConfiguration from the cluster and updates this node's kubelet configuration (/var/lib/kubelet/config.yaml) from the kubelet-config ConfigMap:

user@k8s-node-1:~$ sudo sed -i 's#/v1.36/#/v1.37/#' /etc/apt/sources.list.d/kubernetes.list
user@k8s-node-1:~$ sudo apt-get update
user@k8s-node-1:~$ sudo apt-mark unhold kubeadm && \
    sudo apt-get install -y kubeadm='1.37.2-*' && \
    sudo apt-mark hold kubeadm
user@k8s-node-1:~$ sudo kubeadm upgrade node
[upgrade] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[upgrade] Use 'kubeadm init phase upload-config kubeadm --config your-config-file' to re-upload it.
[upgrade/preflight] Running pre-flight checks
[upgrade/preflight] Skipping prepull. Not a control plane node.
[upgrade/control-plane] Skipping phase. Not a control plane node.
[upgrade/kubeconfig] Skipping phase. Not a control plane node.
W1106 07:55:52.696863 3025021 postupgrade.go:105] Using temporary directory /etc/kubernetes/tmp/kubeadm-kubelet-config-2026-11-06-07-55-52 for kubelet config. To override it set the environment variable KUBEADM_UPGRADE_DRYRUN_DIR
[upgrade] Backing up kubelet config file to /etc/kubernetes/tmp/kubeadm-kubelet-config-2026-11-06-07-55-52/config.yaml
[patches] Applied patch of type "application/strategic-merge-patch+json" to target "kubeletconfiguration"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade/kubelet-config] The kubelet configuration for this node was successfully upgraded!
[upgrade/addon] Skipping the addon/coredns phase. Not a control plane node.
[upgrade/addon] Skipping the addon/kube-proxy phase. Not a control plane node.

The old kubelet configuration stays in /etc/kubernetes/tmp on the worker too — the directory is cleaned up the same way as on the master.

kubelet — same as on the master:

user@k8s-node-1:~$ sudo apt-mark unhold kubelet kubectl && \
    sudo apt-get install -y kubelet='1.37.2-*' kubectl='1.37.2-*' && \
    sudo apt-mark hold kubelet kubectl
user@k8s-node-1:~$ sudo systemctl daemon-reload
user@k8s-node-1:~$ sudo systemctl restart kubelet

$ kubectl uncordon k8s-node-1
node/k8s-node-1 uncordoned

After uncordon the node accepts new pods again, but pods that are already running won't move back by themselves: the scheduler doesn't rebalance the cluster. The pods evicted from k8s-node-1 live on k8s-node-2, and when k8s-node-2 is drained next, they move back — that's usually how it works out. If the load needs to be spread right away, kubectl rollout restart deployment/hello -n demo helps: the new pods are placed anew with topology spread taken into account. What matters here is matchLabelKeys: [pod-template-hash] from part 13: during a rollout the constraint counts only the pods of the new revision, not the old ones still finishing on a single node.

Then the same for k8s-node-2. And last — kubectl on the workstation, the same version 1.37.

Looking at it by hand

Versions of all components:

$ kubectl version -o json | jq -r '.clientVersion.gitVersion, .serverVersion.gitVersion'
v1.37.2
v1.37.2
$ kubectl get nodes -o custom-columns=NAME:.metadata.name,KUBELET:.status.nodeInfo.kubeletVersion
NAME           KUBELET
k8s-master-1   v1.37.2
k8s-node-1     v1.37.2
k8s-node-2     v1.37.2

Images of the static pods and addons:

$ kubectl -n kube-system get pods -o custom-columns=NAME:.metadata.name,IMAGE:.spec.containers[0].image | grep -E 'etcd|kube-|coredns'
coredns-559f6c778d-gqkbp               registry.k8s.io/coredns/coredns:v1.14.6
coredns-559f6c778d-kjsll               registry.k8s.io/coredns/coredns:v1.14.6
etcd-k8s-master-1                      registry.k8s.io/etcd:3.7.0-0
kube-apiserver-k8s-master-1            registry.k8s.io/kube-apiserver:v1.37.2
kube-controller-manager-k8s-master-1   registry.k8s.io/kube-controller-manager:v1.37.2
kube-proxy-t78zm                       registry.k8s.io/kube-proxy:v1.37.2
kube-proxy-t7zlp                       registry.k8s.io/kube-proxy:v1.37.2
kube-proxy-wrtc9                       registry.k8s.io/kube-proxy:v1.37.2
kube-scheduler-k8s-master-1            registry.k8s.io/kube-scheduler:v1.37.2

The cluster version from kubeadm's point of view is kept in the kubeadm-config ConfigMap — that's how kubeadm upgrade node on the workers learns what to install:

$ kubectl -n kube-system get configmap kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' | grep -E 'kubernetesVersion|imageRepository'
imageRepository: registry.k8s.io
kubernetesVersion: v1.37.2

The backups left by kubeadm upgrade apply:

user@k8s-master-1:~$ sudo ls /etc/kubernetes/tmp/
kubeadm-backup-etcd-2026-11-06-07-48-09
kubeadm-backup-manifests-2026-11-06-07-48-09
kubeadm-kubelet-config-2026-11-06-07-51-15
user@k8s-master-1:~$ sudo ls /etc/kubernetes/tmp/kubeadm-backup-manifests-2026-11-06-07-48-09/
etcd.yaml
kube-apiserver.yaml
kube-controller-manager.yaml
kube-scheduler.yaml
user@k8s-master-1:~$ sudo du -sh /etc/kubernetes/tmp/kubeadm-backup-etcd-2026-11-06-07-48-09/
278M	/etc/kubernetes/tmp/kubeadm-backup-etcd-2026-11-06-07-48-09/

The copy of the etcd data in tmp is the cluster's Secrets in an unencrypted directory on the master's disk, just like the snapshot from the previous part, only whole, with the entire revision history: 278 MB against the snapshot's 35 MB. Only root has access to the directory, but it shouldn't be kept longer than needed for a rollback:

user@k8s-master-1:~$ sudo rm -rf /etc/kubernetes/tmp

kubeadm certificates

During init kubeadm issued a set of certificates in /etc/kubernetes/pki and embedded client certificates into the components' kubeconfig files. Client and server certificates are valid for 1 year, the three CAs (cluster, etcd and front-proxy) for 10 years. Both are set in ClusterConfiguration version v1beta4 by the certificateValidityPeriod and caCertificateValidityPeriod fields, but extending the lifetime of regular certificates isn't worth it: a short lifetime limits the damage from a leaked key, and renewal is automatic.

user@k8s-master-1:~$ sudo kubeadm certs check-expiration
[check-expiration] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[check-expiration] Use 'kubeadm init phase upload-config kubeadm --config your-config-file' to re-upload it.

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Nov 06, 2027 07:47 UTC   364d            ca                      no
apiserver                  Nov 06, 2027 07:47 UTC   364d            ca                      no
apiserver-etcd-client      Nov 06, 2027 07:47 UTC   364d            etcd-ca                 no
apiserver-kubelet-client   Nov 06, 2027 07:47 UTC   364d            ca                      no
controller-manager.conf    Nov 06, 2027 07:47 UTC   364d            ca                      no
etcd-healthcheck-client    Nov 06, 2027 07:47 UTC   364d            etcd-ca                 no
etcd-peer                  Nov 06, 2027 07:47 UTC   364d            etcd-ca                 no
etcd-server                Nov 06, 2027 07:47 UTC   364d            etcd-ca                 no
front-proxy-client         Nov 06, 2027 07:47 UTC   364d            front-proxy-ca          no
scheduler.conf             Nov 06, 2027 07:47 UTC   364d            ca                      no
super-admin.conf           Nov 06, 2027 07:47 UTC   364d            ca                      no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Oct 21, 2036 10:31 UTC   9y              no
etcd-ca                 Oct 21, 2036 10:31 UTC   9y              no
front-proxy-ca          Oct 21, 2036 10:31 UTC   9y              no

All certificates are fresh: kubeadm upgrade apply renewed them a few minutes ago. This is the main rule: a cluster that is upgraded at least once a year never runs into expired certificates. And a cluster left untouched for a year stops answering one day: the apiserver rejects the controller-manager's expired certificate, kubectl with admin.conf gets refused.

What the rows are:

Certificate	Who uses it
`apiserver`	the API server certificate on 6443 with the names `k8s-api.example.com`, `k8s-master-1`, `k8s-master-1.example.com`, `kubernetes`, `kubernetes.default.svc.cluster.local` and the addresses `10.96.0.1`, `192.168.10.17`
`apiserver-kubelet-client`	the apiserver when it goes to the kubelet for logs and `exec`
`apiserver-etcd-client`	the apiserver when it goes to etcd
`front-proxy-client`	the apiserver when it proxies requests to API extensions (metrics-server)
`etcd-*`	etcd and its clients, see the previous part
`controller-manager.conf`, `scheduler.conf`	client certificates inside the components' kubeconfigs
`admin.conf`	the cluster administrator, group `kubeadm:cluster-admins` with permissions via RBAC
`super-admin.conf`	break-glass access, group `system:masters`, which bypasses RBAC entirely

admin.conf, and even more so super-admin.conf, are keys to the whole cluster, which are not shared and not used every day: people get their own access for daily work as in part 29, and these files stay on the master for emergencies.

Renewing by hand — if the cluster hasn't been upgraded for a long time or a certificate is needed urgently:

user@k8s-master-1:~$ sudo kubeadm certs renew all
[renew] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[renew] Use 'kubeadm init phase upload-config kubeadm --config your-config-file' to re-upload it.

certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed
certificate embedded in the kubeconfig file for the super-admin renewed

Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.

The last line is what matters. Not all components reload certificates on the fly: the apiserver picked up its serving certificate by itself — on the lab openssl s_client against port 6443 showed the new certificate even before the restart — but the client certificates inside the controller-manager's and scheduler's kubeconfigs are read only at startup. That's why all four static pods are restarted after renew. systemctl restart kubelet won't do it: the kubelet doesn't restart a static pod if its manifest hasn't changed. You have to move the manifest out of the directory, wait for the kubelet to stop the pod (it checks the directory every 20 seconds), and move it back:

user@k8s-master-1:~$ for c in etcd kube-apiserver kube-controller-manager kube-scheduler; do
    sudo mv /etc/kubernetes/manifests/$c.yaml /etc/kubernetes/
    sleep 25
    sudo mv /etc/kubernetes/$c.yaml /etc/kubernetes/manifests/
    sleep 25
done

The loop runs for a little over three minutes, and while etcd and the apiserver restart the API is unavailable — on a single-node control plane that's a minute or two of API downtime, which running pods don't notice. After renewal admin.conf contains the new certificate, and the copy in ~/.kube/config on the master is updated: sudo cp /etc/kubernetes/admin.conf ~/.kube/config. The old certificate is not revoked and keeps working until it expires.

The kubelet's certificates are not renewed by kubeadm, and that's not needed: the kubelet does it itself. It has two, both in /var/lib/kubelet/pki/, and each is a *-current.pem symlink to a file named with its issue date:

user@k8s-node-1:~$ sudo ls -l /var/lib/kubelet/pki/
-rw------- 1 root root 1118 Oct 24 10:37 kubelet-client-2026-10-24-10-37-18.pem
lrwxrwxrwx 1 root root   59 Oct 24 10:37 kubelet-client-current.pem -> /var/lib/kubelet/pki/kubelet-client-2026-10-24-10-37-18.pem
-rw------- 1 root root 1167 Oct 24 10:37 kubelet-server-2026-10-24-10-37-49.pem
lrwxrwxrwx 1 root root   59 Oct 24 10:37 kubelet-server-current.pem -> /var/lib/kubelet/pki/kubelet-server-2026-10-24-10-37-49.pem
user@k8s-node-1:~$ sudo openssl x509 -noout -subject -enddate -in /var/lib/kubelet/pki/kubelet-client-current.pem
subject=O=system:nodes, CN=system:node:k8s-node-1
notAfter=Oct 24 10:32:18 2027 GMT
user@k8s-node-1:~$ sudo openssl x509 -noout -issuer -enddate -in /var/lib/kubelet/pki/kubelet-server-current.pem
issuer=CN=kubernetes
notAfter=Oct 24 10:32:48 2027 GMT

The client certificate — the kubelet uses it to talk to the apiserver. The rotateCertificates: true parameter, which kubeadm sets in KubeletConfiguration, makes the kubelet create a new certificate request (CertificateSigningRequest) in advance, once 70–90 % of the lifetime has passed. The controller-manager approves and signs such requests from nodes automatically: kubeadm created the RBAC bindings for that — the very ones it sets up again in the [bootstrap-token] lines of the kubeadm upgrade apply output. The kubelet switches the symlink to the new file without a restart.
The serving certificate — on port 10250, where the apiserver goes for logs and exec and metrics-server for metrics. In part 4 we enabled serverTLSBootstrap: true, so it is signed by the cluster CA (issuer=CN=kubernetes) rather than self-signed — and that's why metrics-server in part 32 verifies the kubelet without --kubelet-insecure-tls. The kubelet renews it the same way, with one difference: the controller-manager does not approve requests for the kubernetes.io/kubelet-serving signer automatically, because it can't verify that the names and addresses in the request really belong to the node. A year from now, once 70–90 % of the lifetime has passed, every kubelet will send a new request, and it has to be approved — by hand or by a separate approver controller that does such a check. Otherwise the serving certificate expires, and kubectl logs, exec and metrics-server stop working with that node. What such a request and its approval look like is shown below, in the section on replacing a node.

The CA lifetime is 10 years, and kubeadm doesn't automate its renewal: it's a manual procedure of issuing a new CA and moving all components and nodes to it, described in the kubeadm documentation. A cluster that lives ten years without a reinstall is rare, but the CA expiry date is worth writing down wherever the other dates are kept.

Updating the nodes' OS

Ubuntu Server installs security fixes by itself: the unattended-upgrades package is enabled by default and installs updates from the -security repository once a day. That's good for a Kubernetes node, with two caveats.

What not to update automatically. kubelet, kubeadm and kubectl are frozen with apt-mark hold, and unattended-upgrades leaves them alone — their version changes only through the procedure above. containerd is worth freezing the same way (or adding to Unattended-Upgrade::Package-Blacklist in /etc/apt/apt.conf.d/50unattended-upgrades): restarting containerd doesn't kill running containers — the containerd-shim processes keep them — but changing the runtime version on a running node without draining it is an unnecessary risk. It's upgraded like the kubelet: drain, upgrade, check, uncordon.

Reboot. A new kernel, libraries such as glibc and OpenSSL, systemd itself take effect only after a reboot. By default unattended-upgrades doesn't reboot the node, it only creates the /var/run/reboot-required file with the list of reasons in /var/run/reboot-required.pkgs. A cluster node must not be rebooted without draining — its pods would die without warning, and the PDB wouldn't help. So automatic reboot in unattended-upgrades is left off, and nodes are rebooted one at a time.

Waiting for the node to be ready after the reboot needs care. Right after systemctl reboot the node is still Ready in the API: the kubelet no longer answers, but the node controller doesn't notice right away — on the lab it took 47 seconds. A plain kubectl wait --for=condition=Ready returns immediately at that moment — that's what happened on the lab, and the node was put back into service in the middle of its reboot. The reliable sign is a change of bootID, the boot identifier the kubelet reports in the node status:

user@k8s-node-1:~$ cat /var/run/reboot-required.pkgs
linux-image-7.0.0-42-generic
linux-base
$ boot=$(kubectl get node k8s-node-1 -o jsonpath='{.status.nodeInfo.bootID}')
$ kubectl drain k8s-node-1 --ignore-daemonsets --delete-emptydir-data --timeout=10m
user@k8s-node-1:~$ sudo systemctl reboot
$ until [ "$(kubectl get node k8s-node-1 -o jsonpath='{.status.nodeInfo.bootID}')" != "$boot" ]; do sleep 5; done
$ kubectl wait --for=condition=Ready node/k8s-node-1 --timeout=10m
node/k8s-node-1 condition met
$ kubectl uncordon k8s-node-1

The new bootID appeared a minute after the reboot command, while the node was still NotReady, and a second later the node became Ready. After the reboot uname -r shows the new kernel 7.0.0-42-generic, and the /var/run/reboot-required file is gone.

The master is rebooted the same way, but on a single-node control plane that's several minutes of API unavailability — three on the lab: running pods on the workers carry on, but changes, scaling and restarts of failed pods wait. This is done in a maintenance window.

Exactly this cycle is automated by kured (Kubernetes Reboot Daemon) — a DaemonSet that watches /var/run/reboot-required on every node, takes a cluster-wide lock so that only one node reboots at a time, drains the node, reboots it and brings it back into service, and only within a configured time window.

Replacing a node

A node is taken out of the cluster when hardware is replaced, the OS is reinstalled or the node simply broke.

Removal. First drain, then delete the Node object from the API, then clean up the machine itself:

$ kubectl drain k8s-node-2 --ignore-daemonsets --delete-emptydir-data --timeout=10m
node/k8s-node-2 drained
$ kubectl delete node k8s-node-2
node "k8s-node-2" deleted

The node's Lease in kube-node-lease disappears immediately — the garbage collector deletes it, because the Lease is owned by the Node object. The block of pod addresses that Calico had assigned to the node (172.20.172.0/26) was released by the Calico controller by itself about two minutes later. On the machine itself kubeadm reset reverts what kubeadm join did: stops the kubelet, removes containers, cleans /etc/kubernetes and /var/lib/kubelet:

user@k8s-node-2:~$ sudo kubeadm reset
W1106 08:06:42.031356 2401991 preflight.go:55] [reset] WARNING: Changes made to this host by 'kubeadm init' or 'kubeadm join' will be reverted.
[reset] Are you sure you want to proceed? [y/N]: y
[preflight] Running pre-flight checks
...
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in "/var/lib/kubelet"
E1106 08:06:46.513050 2401991 remote_runtime.go:355] "StopPodSandbox from runtime service failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" podSandboxID="81e0dc0d8ab6ec997a5d65c45a31b54fbe115b5a74033aa8a9de7826ed34a77d"
[reset] Deleting contents of directories: [/etc/kubernetes/manifests /var/lib/kubelet /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/super-admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]

The reset process does not perform cleanup of CNI plugin configuration,
network filtering rules and kubeconfig files.

For information on how to perform this cleanup manually, please see:
    https://k8s.io/docs/reference/setup-tools/kubeadm/kubeadm-reset/

The StopPodSandbox error means the pod network didn't get torn down cleanly within the allotted time; reset removes the containers anyway, and crictl pods is empty afterwards. What it honestly leaves alone is the CNI configuration in /etc/cni/net.d (in our case 10-calico.conflist and calico-kubeconfig), the kube-proxy and calico nftables tables, the vxlan.calico interface and the /etc/kubernetes/tmp directory. If the machine leaves the cluster for good or comes back into it, the most reliable option is to reinstall the OS: that guarantees no old rules and no leftover pod data remain. If reinstalling isn't possible — delete the CNI configuration and the kubeadm leftovers and reboot the node: the nftables rules and Calico interfaces live only in memory and disappear by themselves.

user@k8s-node-2:~$ sudo rm -rf /etc/cni/net.d/* /etc/kubernetes/tmp
user@k8s-node-2:~$ sudo systemctl reboot

After the reboot nft list tables shows nothing, and there are no cali* or vxlan.calico interfaces.

Adding the new one. The new machine — k8s-node-2 with the same name and address after a reinstall — is prepared as in part 4: the OS, kernel modules and sysctl, containerd with SystemdCgroup = true, the kubeadm, kubelet, kubectl packages of the cluster's current version, that is, already 1.37.2 from the v1.37 repository. The join command is issued by the master:

user@k8s-master-1:~$ sudo kubeadm token create --ttl 1h --print-join-command
kubeadm join k8s-api.example.com:6443 --token 7q3k2m.x9w4p8v1n6c5b2z0 --discovery-token-ca-cert-hash sha256:5f1b8c3e9a7d2f4b6c8e0a1d3f5b7c9e2a4d6f8b0c2e4a6d8f0b2c4e6a8d0f2b

--token is a bootstrap token the new kubelet uses once to introduce itself to the apiserver and get its client certificate; --ttl 1h limits its life to an hour instead of the default day — the token lets anyone join any machine to the cluster, and it has no use for extra time. --discovery-token-ca-cert-hash is the hash of the cluster CA's public key: the new node uses it to verify that it's talking to the real master and not an impostor. The address in the command is the controlPlaneEndpoint from the part 4 configuration, the API name rather than the master's address.

user@k8s-node-2:~$ sudo kubeadm join k8s-api.example.com:6443 --token 7q3k2m.x9w4p8v1n6c5b2z0 \
    --discovery-token-ca-cert-hash sha256:5f1b8c3e9a7d2f4b6c8e0a1d3f5b7c9e2a4d6f8b0c2e4a6d8f0b2c4e6a8d0f2b
[preflight] Running pre-flight checks
[preflight] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[preflight] Use 'kubeadm init phase upload-config kubeadm --config your-config-file' to re-upload it.
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/instance-config.yaml"
[patches] Applied patch of type "application/strategic-merge-patch+json" to target "kubeletconfiguration"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s
[kubelet-check] The kubelet is healthy after 3.000766747s
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

The new kubelet sent two certificate requests right away. The client one, on behalf of the bootstrap token, was approved by the controller-manager itself; the serving one waits for the administrator:

$ kubectl get csr
NAME        AGE   SIGNERNAME                                    REQUESTOR                    REQUESTEDDURATION   CONDITION
csr-fwhcg   19s   kubernetes.io/kubelet-serving                 system:node:k8s-node-2       <none>              Pending
csr-ktp7w   22s   kubernetes.io/kube-apiserver-client-kubelet   system:bootstrap:7q3k2m      <none>              Approved,Issued
$ kubectl get csr csr-fwhcg -o jsonpath='{.spec.request}' | base64 -d | openssl req -noout -text | grep -A1 'Subject Alternative'
                X509v3 Subject Alternative Name:
                    DNS:k8s-node-2, IP Address:192.168.10.21
$ kubectl certificate approve csr-fwhcg
certificatesigningrequest.certificates.k8s.io/csr-fwhcg approved
$ kubectl get nodes
NAME           STATUS   ROLES           AGE   VERSION
k8s-master-1   Ready    control-plane   12d   v1.37.2
k8s-node-1     Ready    <none>          12d   v1.37.2
k8s-node-2     Ready    <none>          26s   v1.37.2

The name and address in the request match the node, and the request came from the node's own account — we approve it. Until the serving certificate is issued, the node is already Ready and accepts pods, but kubectl logs and exec to its pods don't work. DaemonSets deploy calico-node, kube-proxy, the MetalLB speaker and the CSI node plugins on the new node by themselves; regular pods come to it as they restart or after a rollout restart.

Replacing the master when there's only one is restoring the control plane from a backup as in the previous part: a new machine with the same name and address, the pki directory from the copy, etcd data from the snapshot and kubeadm init with the same configuration on top. In a highly available control plane the master is replaced like a regular member: remove it, delete it from etcd with etcdctl member remove, join a new one with kubeadm join --control-plane; that procedure is covered in the series about Kubernetes on BGP.

The lab is on 1.37.2, Calico on 3.33, the certificates are renewed for a year, nodes are upgraded and rebooted one at a time without application downtime.

Sources

Kubernetes: Releases and Patch Releases — versions, patches and support periods; Version Skew Policy — allowed skew and upgrading the kubelet only on a drained node.
CHANGELOG 1.37 — Urgent Upgrade Notes and kubeadm changes.
Deprecated API Migration Guide — removed API versions by release.
Upgrading kubeadm clusters — upgrade order, stopping the apiserver before replacing etcd, backups in /etc/kubernetes/tmp; Changing The Kubernetes Package Repository — pkgs.k8s.io repositories per minor version.
Certificate Management with kubeadm — lifetimes, check-expiration, renew, restarting static pods, kubelet-serving CSRs.
Configure Certificate Rotation for the Kubelet and TLS bootstrapping — kubelet certificates and their renewal.
Safely Drain a Node and Specifying a Disruption Budget — drain, the Eviction API and PDB.
kubeadm reset and kubeadm join — removing and adding a node.
Calico: requirements with supported Kubernetes versions and upgrading an operator-based installation.
pluto — finding deprecated API versions in files, Helm releases and the cluster.
Ubuntu: Automatic updates — unattended-upgrades; kured — rebooting nodes one at a time.

Next — part 36, Windows nodes in the cluster: it opens the block about a hybrid cluster of Linux and Windows workers. How a Windows node works: why the control plane stays on Linux only, how the kubelet and containerd run as Windows services, what HCS and HNS do, why with process isolation the image version must match the host version, how the kubernetes.io/os label, taints and RuntimeClass send pods to their own nodes, what HostProcess containers are and what Windows doesn't have.