6 November 2026, 10:27Виталий Лещенко

Kubernetes for beginners. Part 34: How does etcd work?

All parts of the series “Kubernetes for beginners”

Everything a cluster knows about itself — every Deployment, Secret, node, leader lease and event — is kept in one place: etcd. In part 2 it appeared as one line in the list of control plane components, in part 3 we saw its trace in resourceVersion and watch, in part 4 kubeadm brought it up as a static pod on k8s-master-1, and in part 12 we enabled encryption of Secrets before they are written to it. Today we look at etcd itself: how keys and values are organised, what revisions are and why the database grows, how Raft agrees on a write and what losing quorum means, how to inspect etcd with etcdctl, why it needs a fast disk, and most importantly, how to take snapshots, where to keep them and how to restore a cluster from them.

Our cluster has one master, and therefore one etcd member. This is a learning setup: losing the disk of k8s-master-1 without a backup means losing the whole cluster. That is why half of this part is about backups.

What is stored in etcd

etcd is a distributed key-value store with strong consistency: any read sees all writes confirmed before it, whichever cluster member serves it. Clients talk to it over gRPC (a remote call protocol on top of HTTP/2 with protobuf messages) on port 2379. Keys are arbitrary byte strings ordered lexicographically, so a query for "all keys with a prefix" is cheap: it is a walk over a sorted range.

The only etcd client in the cluster is kube-apiserver. Neither the kubelet, nor the controllers, nor the scheduler talk to etcd: they work with the API, and apiserver translates objects into keys. The key scheme:

/registry/<resource>/<namespace>/<name>          namespaced objects
/registry/<resource>/<name>                      cluster-scoped objects
/registry/<group>/<resource>/<namespace>/<name>  CRD resources and some API groups

For example:

Key	Object
`/registry/pods/demo/hello-599d6c557d-97s8z`	Pod
`/registry/deployments/demo/hello`	Deployment (group `apps`, but the prefix omits it for historical reasons)
`/registry/services/specs/demo/hello`	Service
`/registry/endpointslices/demo/hello-w68dr`	EndpointSlice
`/registry/secrets/demo/hello-tls`	Secret
`/registry/minions/k8s-node-1`	Node (minion is the old name for nodes in the API)
`/registry/leases/kube-node-lease/k8s-node-1`	Lease: kubelet heartbeat
`/registry/leases/kube-system/kube-controller-manager`	Lease: who is the controller-manager leader right now
`/registry/crd.projectcalico.org/bgpconfigurations/default`	a CRD resource, Calico in this case

Values of built-in types are stored in protobuf, Google's binary serialization format, where fields are encoded by numbers and lengths rather than names. The message starts with the magic prefix k8s\x00, followed by a runtime.Unknown wrapper with apiVersion and kind, and then the object itself. CRD resources are stored as JSON: apiserver has no compiled protobuf schemas for them. A Secret, if encryption from part 12 is enabled, is stored with the prefix k8s:enc:<provider>:v1:<key name>: followed by ciphertext; without encryption it is the same protobuf, where Secret values are visible after base64 decoding, that is, in plain text.

Hence the first security rule: access to etcd equals full access to the cluster. Whoever can read keys reads all Secrets, including service account tokens with administrator rights; whoever can write can create a ClusterRoleBinding for themselves, bypassing RBAC and the admission checks apiserver performs before a write. That is why etcd in kubeadm accepts only connections with verified client certificates, and those certificates are issued by its own CA.

Revisions and MVCC

etcd stores not only the current value of a key but its history. This is MVCC (multi-version concurrency control): each write creates a new version, and old ones stay until they are explicitly removed. The version counter, the revision, is shared by the whole store and grows monotonically with each transaction. Every key has three numbers:

create_revision — the revision at which the key appeared;
mod_revision — the revision of the last change;
version — how many times the key has changed since it was created.

The metadata.resourceVersion of a Kubernetes object is exactly the mod_revision of its key, passed to the client as an opaque string. This explains the mechanism from part 3: a watch with resourceVersion=N turns into an etcd watch "from revision N", and etcd returns all changes after it from the history. Optimistic locking also relies on revisions: apiserver performs an object update as a transaction "if the mod_revision of the key is still the one the client sent, write", and on a mismatch the client gets 409 Conflict.

On disk the MVCC store is the file member/snap/db, a bbolt database (an embedded key-value store on a B+ tree in a single file, a fork of BoltDB). The key in it is a "revision, sub-number" pair, and the value is the original etcd key and the data. A separate in-memory index maps an etcd key to the list of its revisions.

The history grows forever unless it is trimmed. Trimming is compaction: "forget all versions older than revision N, except the latest version of each key". In a Kubernetes cluster it is done not by etcd itself but by kube-apiserver: every 5 minutes (the --etcd-compaction-interval flag) it calls compact up to the revision that was current five minutes earlier. That is why a watch with a very old resourceVersion gets the error 410 Gone (too old resource version): the required revisions no longer exist in etcd, and the client must read the list again. In practice the error comes even earlier: watches are served by apiserver's in-memory cache, which holds only recent changes, so a 410 can arrive while the old revision can still be read in etcd itself.

Compaction frees space inside the bbolt file but does not shrink the file: the freed pages simply go to new writes. Only defragmentation can return space to the file system: etcdctl defrag makes etcd rewrite the database into a new file without empty pages. While it runs, the member serves neither reads nor writes, and on a single-node etcd this means the whole API pauses for seconds or tens of seconds. It should be done rarely and in a maintenance window; the Kubernetes documentation recommends the etcd-defrag utility for this, which defragments members one at a time and only when specified conditions are met (for example, if more than half of the file is free).

Quota. The database size is limited by the --quota-backend-bytes flag; the default is 2 GiB, and the recommended ceiling is 8 GiB. kubeadm does not set the quota, so our cluster runs with 2 GiB. When the database reaches the quota, etcd raises a cluster-wide NOSPACE alarm and enters maintenance mode: it accepts only reads and deletes and answers writes with etcdserver: mvcc: database space exceeded. For Kubernetes this is an outage: no object can be created or updated, the kubelet cannot renew its node Lease, and after a few dozen seconds the node controller starts treating the nodes as NotReady. The way out is to free space (compaction up to the current revision, then defrag) and clear the alarm with etcdctl alarm disarm: it does not go away by itself, even when there is plenty of space again, and writes stay forbidden until this command runs. A normal cluster with hundreds of pods takes tens, at most hundreds, of megabytes, so only a leak can hit the quota: an operator creating thousands of objects, giant ConfigMaps, events with no expiry. The database size metrics are shown below, and it is worth putting a warning on them at 70–80% of the quota. If the quota has to be raised, it is set in the kubeadm configuration:

apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
etcd:
  local:
    extraArgs:
    - name: quota-backend-bytes
      value: "8589934592"   # 8 GiB

Raft: how several members reach agreement

Strong consistency across several members rests on the Raft consensus protocol. Its idea: cluster members keep an identical ordered log of operations, and an operation counts as done only when it has been written to the log of a majority. Each member applies the log to its copy of bbolt in the same order and therefore arrives at the same state.

The roles:

leader — the only one that accepts writes. The other members forward client writes to it;
followers — receive new log entries from the leader and confirm them;
candidate — a follower that has stopped hearing the leader and is trying to become the leader itself.

 kube-apiserver
      │ 1. Put /registry/pods/demo/hello-...
      ▼
 ┌───────────┐ 2. AppendEntries: entry 1843322  ┌─────────────┐
 │ etcd-1    │─────────────────────────────────►│ etcd-2      │
 │ (leader)  │──────────────────┐               │ (follower)  │
 └───────────┘                  │               └─────────────┘
      ▲                         │               ┌─────────────┐
      │                         └──────────────►│ etcd-3      │
      │                                         │ (follower)  │
      │ 3. etcd-2: written to WAL               └─────────────┘
      │ 4. majority (2 of 3) reached: entry committed,
      │    applied to bbolt, reply to apiserver

The order of a write: the leader appends the operation to its log and sends it to the followers in an AppendEntries message; each follower writes it to disk and confirms. As soon as a majority has the entry (counting the leader itself), it is committed: the leader applies it and answers the client, and the followers apply it when they learn from the next message up to which number the log is committed.

The log on disk is the WAL (write-ahead log) in member/wal/. A member confirms an entry to the leader only after it has been appended to the WAL and flushed to disk with an fdatasync call. This is the source of etcd's sensitivity to the disk, covered below. From time to time etcd takes a snapshot of its state (by default every 10,000 entries, the --snapshot-count flag) and cuts off the old part of the log: a lagging follower gets the whole snapshot rather than the log from the beginning of time.

Elections. The leader sends a heartbeat every 100 ms. A follower that has not heard the leader for longer than the election timeout (1000 ms by default, plus a random addition so that candidates do not appear simultaneously) increments the term number and asks the others for votes. A member gives its vote only if the candidate's log is not behind its own: a member without committed entries cannot become leader. A candidate with the votes of a majority becomes the leader of the new term. Before a real election etcd runs a pre-vote: a member cut off from the network first asks whether the others are ready to vote for it, and so does not disrupt the current leader by raising the term when the network comes back.

Quorum is a majority of members: ⌊n/2⌋ + 1.

Members	Quorum	Tolerated failures
1	1	0
2	2	0
3	2	1
4	3	1
5	3	2
7	4	3

The table shows why the number of members is odd: a fourth member adds no resilience compared with three (both tolerate one failure) but increases the quorum and the number of machines that must confirm each write. Two members are worse than one: the failure of either stops writes, and the chance that one of two machines fails is twice as high. Three members are the minimum for fault tolerance, five are for when you need to survive the failure of one machine during maintenance of another; more than five is rarely used, because each write waits for confirmation from a majority, and latency grows.

Losing quorum means fewer live members than a majority. Then there is no leader and none can be elected: writes are impossible, and since etcd makes reads linearizable by default (through confirmation with the leader), they fail too: etcdctl get ends with context deadline exceeded, and etcdctl endpoint status shows etcdserver: no leader in the errors column. Only a read without such confirmation answers, etcdctl get --consistency=s: it returns the local copy, possibly stale. For Kubernetes this means the API does not respond, no object can be created or changed, and the controllers and the scheduler stand still. Containers already running keep working, because the kubelet does not kill them over a lost connection to the API, but nothing is restarted on other nodes and nothing scales. If the failed members cannot be brought back, there is only one way out: restore the etcd cluster from a snapshot (or, as a last resort, start a surviving member with --force-new-cluster, which makes it forget the others and become a cluster of one).

Our cluster has one member, and it is its own quorum. A highly available control plane with three masters and etcd on each of them, a shared API address through kube-vip (a floating IP address held by one of the masters) and member maintenance are covered in the series on Kubernetes on BGP, so we will not repeat them here.

etcd in kubeadm

kubeadm runs etcd as a static pod: the kubelet on the master reads the manifest /etc/kubernetes/manifests/etcd.yaml and starts the container itself, without the API (otherwise there would be nothing to bring the API up). The main flags from the manifest:

user@k8s-master-1:~$ sudo grep -E -- '--(name|data-dir|listen|advertise|initial|cert|key|trusted|peer|client-cert|snapshot)' /etc/kubernetes/manifests/etcd.yaml
    - --advertise-client-urls=https://192.168.10.17:2379
    - --cert-file=/etc/kubernetes/pki/etcd/server.crt
    - --client-cert-auth=true
    - --data-dir=/var/lib/etcd
    - --initial-advertise-peer-urls=https://192.168.10.17:2380
    - --initial-cluster=k8s-master-1=https://192.168.10.17:2380
    - --key-file=/etc/kubernetes/pki/etcd/server.key
    - --listen-client-urls=https://127.0.0.1:2379,https://192.168.10.17:2379
    - --listen-metrics-urls=http://127.0.0.1:2381
    - --listen-peer-urls=https://192.168.10.17:2380
    - --name=k8s-master-1
    - --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt
    - --peer-client-cert-auth=true
    - --peer-key-file=/etc/kubernetes/pki/etcd/peer.key
    - --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
    - --snapshot-count=10000
    - --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt

Ports:

Port	What	Who connects
2379	client API, gRPC over TLS	kube-apiserver, `etcdctl`
2380	communication between members (Raft), TLS	other etcd members
2381	metrics and health checks, HTTP without TLS, only `127.0.0.1`	kubelet (`/livez` and `/readyz` checks), `curl` on the node

TLS here means an encrypted connection with certificate verification, where the client presents a certificate too. The data lives in /var/lib/etcd on the master's disk (the etcd-data volume of type hostPath). The certificates are in /etc/kubernetes/pki/etcd/, and etcd has its own CA, separate from the cluster CA:

File	Purpose
`ca.crt`, `ca.key`	the etcd CA; the key is the most valuable file on the master after the cluster `ca.key`
`server.crt`, `server.key`	serving certificate on 2379, names `localhost`, `k8s-master-1`, addresses `127.0.0.1`, `192.168.10.17`
`peer.crt`, `peer.key`	certificate for communication between members on 2380, both server and client
`healthcheck-client.crt`, `.key`	client certificate for health checks and for `etcdctl`

The client certificate of apiserver itself is /etc/kubernetes/pki/apiserver-etcd-client.crt, signed by the same etcd CA. An important detail: kubeadm does not enable etcd user and role authentication, so any client certificate signed by the etcd CA gives full access to all keys. To etcd there is no difference between healthcheck-client and apiserver-etcd-client; the keys of these certificates are readable only by root (0600), and it should stay that way.

etcdctl is the etcd network client, and etcdutl is a utility for working with data files directly, without a running etcd. Since version 3.6 their duties are strictly separated: etcdctl snapshot save takes a snapshot over the network, while snapshot status and snapshot restore have been removed from etcdctl and exist only in etcdutl. Both utilities are in the etcd image, so without installing anything you can do this:

$ kubectl -n kube-system exec etcd-k8s-master-1 -- etcdctl \
    --endpoints=https://127.0.0.1:2379 \
    --cacert=/etc/kubernetes/pki/etcd/ca.crt \
    --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
    --key=/etc/kubernetes/pki/etcd/healthcheck-client.key \
    endpoint health
https://127.0.0.1:2379 is healthy: successfully committed proposal: took = 96.565732ms

But during an etcd restore the API will not be working, so the utilities should be put on the master in advance, in the same version as etcd in the cluster (kubeadm 1.36 installs etcd 3.6.8):

user@k8s-master-1:~$ ETCD_VER=v3.6.8
user@k8s-master-1:~$ curl -fsSLO https://github.com/etcd-io/etcd/releases/download/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz
user@k8s-master-1:~$ curl -fsSLO https://github.com/etcd-io/etcd/releases/download/${ETCD_VER}/SHA256SUMS
user@k8s-master-1:~$ sha256sum --ignore-missing -c SHA256SUMS
etcd-v3.6.8-linux-amd64.tar.gz: OK
user@k8s-master-1:~$ tar xzf etcd-${ETCD_VER}-linux-amd64.tar.gz
user@k8s-master-1:~$ sudo install -m 0755 etcd-${ETCD_VER}-linux-amd64/etcdctl etcd-${ETCD_VER}-linux-amd64/etcdutl /usr/local/bin/

To avoid repeating four flags, etcdctl reads them from environment variables. Only root can read the keys, so we work in a root session:

user@k8s-master-1:~$ sudo -i
root@k8s-master-1:~# export ETCDCTL_ENDPOINTS=https://127.0.0.1:2379
root@k8s-master-1:~# export ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt
root@k8s-master-1:~# export ETCDCTL_CERT=/etc/kubernetes/pki/etcd/healthcheck-client.crt
root@k8s-master-1:~# export ETCDCTL_KEY=/etc/kubernetes/pki/etcd/healthcheck-client.key

The ETCDCTL_API=3 variable from old instructions is no longer needed: the third API version has been the default since etcd 3.4. A flag and a variable for the same parameter cannot be mixed: etcdctl --cert=… with ETCDCTL_CERT set ends with the error conflicting environment variable is shadowed by corresponding command-line flag.

Looking at it by hand

The state of a member:

root@k8s-master-1:~# etcdctl endpoint status -w table
+------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
|        ENDPOINT        |        ID        | VERSION | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA  | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED |
+------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| https://127.0.0.1:2379 | feb950520d2b15b7 |   3.6.8 |           3.6.0 |   35 MB | 3.4 MB |                   91% | 2.1 GB |      true |      false |         4 |    2948483 |            2948482 |        |                          |             false |
+------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+

DB SIZE is the size of the bbolt file, and IN USE is how much of it is occupied. The difference shows in the PERCENTAGE NOT IN USE column: 91% of the file is pages freed by compaction, which defrag would return. A cluster without applications quickly rewrites the same Leases, so the file here is much larger than the data. QUOTA is that same quota: 2.1 GB in decimal units is 2 GiB. RAFT TERM is the term number: on one member it grows only on restarts; in a cluster of three, frequent growth is a sign that the leader keeps getting lost, usually because of a slow disk or network. RAFT INDEX and RAFT APPLIED INDEX are the numbers of the last log entry and the last applied one; on a healthy member they are equal or almost equal.

The cluster members:

root@k8s-master-1:~# etcdctl member list -w table
+------------------+---------+--------------+----------------------------+----------------------------+------------+
|        ID        | STATUS  |     NAME     |         PEER ADDRS         |        CLIENT ADDRS        | IS LEARNER |
+------------------+---------+--------------+----------------------------+----------------------------+------------+
| feb950520d2b15b7 | started | k8s-master-1 | https://192.168.10.17:2380 | https://192.168.10.17:2379 |      false |
+------------------+---------+--------------+----------------------------+----------------------------+------------+

IS LEARNER marks a learner member: a new member that first catches up on the log without a vote and does not affect the quorum, and is then promoted to a full member. This is how members are added to a running cluster without risking quorum.

The keys of our application in the demo namespace, leaving out events, of which there are about ten more:

root@k8s-master-1:~# etcdctl get /registry/ --prefix --keys-only | grep '/demo/' | grep -v '/events/'
/registry/configmaps/demo/hello-config
/registry/configmaps/demo/kube-root-ca.crt
/registry/deployments/demo/hello
/registry/endpointslices/demo/hello-w68dr
/registry/networkpolicies/demo/default-deny
/registry/pods/demo/hello-599d6c557d-97s8z
/registry/pods/demo/hello-599d6c557d-kkvlg
/registry/replicasets/demo/hello-599d6c557d
/registry/secrets/demo/ghcr-pull
/registry/secrets/demo/hello-tls
/registry/serviceaccounts/demo/default
/registry/services/endpoints/demo/hello
/registry/services/specs/demo/hello

How many keys there are in total and which resources take the most:

root@k8s-master-1:~# etcdctl get /registry/ --prefix --keys-only | grep -v '^$' | cut -d/ -f3 | sort | uniq -c | sort -rn | head -5
     82 clusterroles
     69 clusterrolebindings
     53 serviceaccounts
     33 crd.projectcalico.org
     32 apiextensions.k8s.io

On an almost empty learning cluster, RBAC objects and CRD definitions come first, with fewer than five hundred keys in total. In a cluster with applications, events (events) usually take first place: every event from kubectl describe is a key in etcd that lives for an hour (the apiserver --event-ttl flag). To do this, apiserver attaches them to an etcd lease, an object with a time to live, after which etcd itself deletes all the keys attached to it:

root@k8s-master-1:~# EV=/registry/events/calico-system/calico-typha-55d594d885-dsh7d.18e50a011236234d
root@k8s-master-1:~# etcdctl lease timetolive $(printf '%x' $(etcdctl get $EV -w json | jq '.kvs[0].lease'))
lease 15b7a1a2e932ca15 granted with TTL(3660s), remaining(3604s)

The time to live is a little over an hour: apiserver does not create a lease for each event but uses the same one for all events during a minute, so it adds that minute to the time to live. Do not confuse this with Kubernetes Lease objects from kube-node-lease: those are ordinary keys, and their lease field is empty.

The value is protobuf with the k8s\x00 prefix:

root@k8s-master-1:~# etcdctl get /registry/deployments/demo/hello --print-value-only | head -c 32 | hexdump -C
00000000  6b 38 73 00 0a 15 0a 07  61 70 70 73 2f 76 31 12  |k8s.....apps/v1.|
00000010  0a 44 65 70 6c 6f 79 6d  65 6e 74 12 ae 15 0a ce  |.Deployment.....|
00000020

apiVersion and kind are visible, followed by binary fields. The auger utility from the etcd project can decode the whole value, but in practice it is easier to ask the API: kubectl get returns exactly this object as YAML. A Secret with encryption enabled:

root@k8s-master-1:~# etcdctl get /registry/secrets/demo/hello-tls --print-value-only | head -c 48 | hexdump -C | head -3
00000000  6b 38 73 3a 65 6e 63 3a  73 65 63 72 65 74 62 6f  |k8s:enc:secretbo|
00000010  78 3a 76 31 3a 6b 65 79  31 3a 93 9e 86 16 fc cf  |x:v1:key1:......|
00000020  94 dd b8 04 4f 14 6a 7d  0f 8d 8d 81 0c ae ae 56  |....O.j}.......V|

The prefix tells which provider and which key from EncryptionConfiguration encrypted the entry, and the rest is ciphertext.

Revisions and resourceVersion. Let's take the application's ConfigMap and change it:

root@k8s-master-1:~# etcdctl get /registry/configmaps/demo/hello-config -w json | jq '.kvs[0] | {create_revision, mod_revision, version}'
{
  "create_revision": 2206993,
  "mod_revision": 2206993,
  "version": 1
}

$ kubectl -n demo patch configmap hello-config --type merge -p '{"data":{"GREETING":"hi"}}'
configmap/hello-config patched
$ kubectl -n demo get configmap hello-config -o jsonpath='{.metadata.resourceVersion}{"\n"}'
2207071

root@k8s-master-1:~# etcdctl get /registry/configmaps/demo/hello-config -w json | jq '.kvs[0] | {create_revision, mod_revision, version}'
{
  "create_revision": 2206993,
  "mod_revision": 2207071,
  "version": 2
}
root@k8s-master-1:~# etcdctl get /registry/configmaps/demo/hello-config --rev=2206993 --print-value-only | strings | tail -2
GREETING
hello

resourceVersion matches mod_revision, and the old value can still be read at the old revision. Within ten minutes apiserver will run compaction (it compacts up to the revision that was current on the previous pass; on our cluster the old value lived for a little over seven minutes), and the same request will answer etcdserver: mvcc: required revision has been compacted.

Health metrics are on port 2381 of the node itself:

user@k8s-master-1:~$ curl -s http://127.0.0.1:2381/metrics | grep -E '^etcd_(server_has_leader|server_leader_changes_seen_total|mvcc_db_total_size_in(_use_in)?_bytes|server_quota_backend_bytes) '
etcd_mvcc_db_total_size_in_bytes 3.5418112e+07
etcd_mvcc_db_total_size_in_use_in_bytes 3.436544e+06
etcd_server_has_leader 1
etcd_server_leader_changes_seen_total 1
etcd_server_quota_backend_bytes 2.147483648e+09

The quota is 2 GiB and the file takes 35 MB, so there is a huge margin.

Performance: it all comes down to fsync

Every write in Kubernetes — a pod status change, a node Lease renewal every 10 seconds, an event — is a write to etcd, and every etcd write waits for fdatasync of the WAL. fdatasync is a system call that returns only when the data is physically saved on the medium rather than sitting in the kernel page cache. On an SSD or NVMe drive with power-loss protection this takes fractions of a millisecond, on a slow disk tens of milliseconds, and each of them is added to the latency of every API write.

Worse, a slow disk breaks Raft. While the leader waits for the disk, it may fail to send a heartbeat in time, the followers decide there is no leader and start an election, and for that time the cluster stops accepting writes. In the etcd log this shows up as the warnings slow fdatasync and apply request took too long (the expected time in them is 100 ms), and in the metrics as growth of etcd_server_leader_changes_seen_total.

Hence the requirements:

a local SSD or NVMe drive for /var/lib/etcd, not a network disk: NFS and SMB (network file systems) and iSCSI (a block disk over the network) add network latency to every fdatasync and share it with other workloads;
do not share the disk with heavy neighbours: container images, logs, emptyDir of pods on the master. A separate volume for /var/lib/etcd is good practice;
in a virtual machine, a disk with guaranteed IOPS and without the hypervisor's "lazy" caching modes, which acknowledge a write before it is saved: they give fast numbers and lose data when the host fails.

The measure is the 99th percentile of WAL fdatasync duration, the etcd_disk_wal_fsync_duration_seconds metric: it should be below 10 ms. The second metric is etcd_disk_backend_commit_duration_seconds, the time to commit a batch of changes to bbolt: the 99th percentile should be below 25 ms. In Prometheus from part 32:

histogram_quantile(0.99, sum by (le, instance) (rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])))

On the node you can see it without Prometheus, as a raw histogram:

user@k8s-master-1:~$ curl -s http://127.0.0.1:2381/metrics | grep -E '^etcd_disk_wal_fsync_duration_seconds_(bucket\{le="0.00[48]"\}|bucket\{le="0.0(16|32|64)"\}|bucket\{le="0.(128|256)"\}|count)'
etcd_disk_wal_fsync_duration_seconds_bucket{le="0.004"} 0
etcd_disk_wal_fsync_duration_seconds_bucket{le="0.008"} 0
etcd_disk_wal_fsync_duration_seconds_bucket{le="0.016"} 1162
etcd_disk_wal_fsync_duration_seconds_bucket{le="0.032"} 237943
etcd_disk_wal_fsync_duration_seconds_bucket{le="0.064"} 310419
etcd_disk_wal_fsync_duration_seconds_bucket{le="0.128"} 361250
etcd_disk_wal_fsync_duration_seconds_bucket{le="0.256"} 366913
etcd_disk_wal_fsync_duration_seconds_count 368315

The buckets are cumulative: not one of the 368,315 calls finished within 8 ms and only 0.3% within 16 ms, while the 99th percentile lies between 128 and 256 ms. This disk does not meet the 10 ms requirement: the learning cluster runs on virtual machines with a dynamic disk on shared storage. One etcd member on an almost empty cluster tolerates this, because it has nobody to send heartbeats to, although the log is full of apply request took too long; in a cluster of three members on such disks the leader would be lost regularly. Such a disk is not suitable for a production cluster.

For Prometheus in the cluster to collect the metrics, port 2381 must also listen on the node address: in the kubeadm configuration this is etcd.local.extraArgs with listen-metrics-urls: http://127.0.0.1:2381,http://192.168.10.17:2381 (127.0.0.1 stays first: the kubelet uses it to check the health of the pod), plus a firewall rule that lets only cluster nodes in. The metrics go without TLS, but they contain no secrets.

The disk for a future master is checked before installation with the fio utility (an I/O load generator), in a mode similar to how etcd writes to the WAL: sequential writes in blocks of about 2 KB with fdatasync after each one:

user@k8s-master-1:~$ sudo apt-get install -y fio
user@k8s-master-1:~$ sudo mkdir /var/lib/etcd-fio && sudo fio --rw=write --ioengine=sync --fdatasync=1 \
    --directory=/var/lib/etcd-fio --size=22m --bs=2300 --name=etcd-disk-test
...
  fsync/fdatasync/sync_file_range:
    sync (msec): min=17, max=799, avg=84.26, stdev=57.01
    sync percentiles (msec):
     |  1.00th=[   23],  5.00th=[   26], 10.00th=[   29], 20.00th=[   35],
     | 30.00th=[   50], 40.00th=[   66], 50.00th=[   70], 60.00th=[   84],
     | 70.00th=[  100], 80.00th=[  121], 90.00th=[  157], 95.00th=[  190],
     | 99.00th=[  271], 99.50th=[  300], 99.90th=[  485], 99.95th=[  642],
user@k8s-master-1:~$ sudo rm -rf /var/lib/etcd-fio

We look at the 99.00th line: 271 ms against a threshold of 10 ms, and even the fastest call took 17 ms. The test confirmed what the metrics showed, and on this disk it ran for 14 minutes: 22 MiB in blocks of 2300 bytes is almost 10,000 fdatasync calls. On a suitable SSD the same calls take about a millisecond, and the test takes a matter of seconds.

Snapshots

A snapshot is a consistent copy of the whole database at one revision, a file in bbolt format. It is taken over the network from a running member without stopping the cluster:

root@k8s-master-1:~# mkdir -p /var/backups/etcd
root@k8s-master-1:~# etcdctl snapshot save /var/backups/etcd/snapshot.db
{"level":"info","ts":"2026-11-05T07:59:13.528512Z","caller":"snapshot/v3_snapshot.go:83","msg":"created temporary db file","path":"/var/backups/etcd/snapshot.db.part"}
{"level":"info","ts":"2026-11-05T07:59:13.531968Z","logger":"client","caller":"v3@v3.6.8/maintenance.go:236","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2026-11-05T07:59:13.661844Z","caller":"snapshot/v3_snapshot.go:96","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
{"level":"info","ts":"2026-11-05T07:59:13.905603Z","logger":"client","caller":"v3@v3.6.8/maintenance.go:302","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2026-11-05T07:59:14.259633Z","caller":"snapshot/v3_snapshot.go:111","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"35 MB","took":"731.068883ms","etcd-version":"3.6.0"}
{"level":"info","ts":"2026-11-05T07:59:14.275147Z","caller":"snapshot/v3_snapshot.go:121","msg":"saved","path":"/var/backups/etcd/snapshot.db"}
Snapshot saved at /var/backups/etcd/snapshot.db
Server version 3.6.0
root@k8s-master-1:~# etcdutl snapshot status /var/backups/etcd/snapshot.db -w table
+----------+----------+------------+------------+---------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE | VERSION |
+----------+----------+------------+------------+---------+
| caf81aa3 |  2207072 |        497 |      35 MB |   3.6.0 |
+----------+----------+------------+------------+---------+

snapshot save takes the database from one member, so in a cluster of three --endpoints points to exactly one. The file permissions are 0600. etcd appends a checksum to the end of the file, and restore verifies it. A member/snap/db file copied by hand from the data directory has no such checksum: restore answers snapshot missing hash but --skip-hash-check=false, and it can be restored only with --skip-hash-check, and if it was copied while etcd was running, it may also turn out to be inconsistent. There is no need to copy the data directory instead of taking a snapshot.

A snapshot contains all the Secrets of the cluster. Database passwords, TLS keys, registry access tokens, cloud provider keys: everything in Secrets is also in the snapshot file. Without the encryption from part 12 it is plain text that an ordinary strings pulls out of the file. A snapshot must be handled as the most sensitive file in the infrastructure:

encrypt it before it leaves the master;
keep it outside the cluster: a snapshot in a PVC of the same cluster dies with it and is accessible to anyone who gets rights to that volume;
do not put it on a shared NFS export that pods can reach. In part 22 we saw that restricting an export to node addresses does not protect it from pods: their traffic goes out through NAT with the node address, and to the NFS server it is indistinguishable from the traffic of the node itself. A snapshot on such an export means all the Secrets of the cluster are available to any pod that manages to mount the share or whose volume with sec=sys and a suitable UID lives on it;
restrict write and delete access: whoever breaks into the cluster must not be able to erase the backups.

If Secret encryption in etcd is enabled, the snapshot holds ciphertext, and restoring requires the key from the EncryptionConfiguration file on the master. This file is copied separately from the snapshots and stored separately: the point of encryption is that a leaked snapshot without the key gives nothing.

A snapshot is not the whole control plane backup. To bring up a master on a new machine you also need:

/etc/kubernetes/pki/ — the cluster CA, the etcd CA, the front-proxy CA with their keys, and the sa.key/sa.pub pair that signs service account tokens. Without the old CAs, all kubeconfigs, kubelets on workers and tokens become invalid;
the kubeadm configuration (kubectl -n kube-system get configmap kubeadm-config -o yaml is also in the snapshot itself, but it is more convenient to have it alongside) and the EncryptionConfiguration file.

The pki directory holds the same kind of secrets as a snapshot and requires the same handling.

Regular snapshots: a CronJob

A scheduled snapshot is a job for a CronJob from part 9. The scheme:

 k8s-master-1, pod with hostNetwork
 ┌──────────────────────────────────────────────┐
 │ init container etcd:3.6.8-0                  │
 │   etcdctl snapshot save /work/etcd.db        │
 │       │ 127.0.0.1:2379, healthcheck-client   │
 │       ▼                                      │
 │ emptyDir /work                               │
 │       │                                      │
 │       ▼                                      │
 │ container etcd-backup                        │
 │   age -r <public key>      (encryption)      │
 │   rclone copyto            (upload)          │
 └───────┬──────────────────────────────────────┘
         │ HTTPS, write only
         ▼
 ┌──────────────────────────────────────────────┐
 │ S3 storage outside the cluster               │
 │ backup.example.com                           │
 │ versions, no deletion, retention periods     │
 └──────────────────────────────────────────────┘

The decisions behind it:

A pod on the master with hostNetwork: true. etcd listens on 127.0.0.1:2379 and the master address; in the node's network namespace the pod can reach the local address, and the etcd certificate does not have to be copied anywhere except the master itself. The pod gets to the master through nodeSelector and a toleration for the node-role.kubernetes.io/control-plane taint (part 13).
Only three files are mounted: ca.crt and the healthcheck-client pair. The etcd CA key does not get into the pod.
Encryption with a public key. age is a simple file encryption utility with short keys: the public key age1… encrypts, and only the private key decrypts. The cluster holds only the public key; the private key is kept outside the infrastructure (a password manager, a paper copy in a safe). Breaking into the cluster does not make it possible to read old snapshots.
Upload to S3-compatible storage outside the cluster. S3 is Amazon's object storage protocol, implemented by many systems, including self-hosted installations on your own servers. rclone is a utility for copying files to cloud and object storage. The upload account has the right only to write new objects (PutObject), with no reading or deleting; the bucket has versioning and object lock (a ban on deletion until a period expires) enabled. The retention period is set by the bucket's lifecycle rules, not by a script in the cluster: this way a compromised cluster can neither read nor erase the copies.

The image for the second container is built as in part 5:

FROM alpine:3.22
RUN apk add --no-cache age rclone ca-certificates

The manifest. The namespace is a separate one, with the Pod Security level privileged (part 30): pods with hostPath and hostNetwork are not allowed at the baseline and restricted levels. The right to create pods in this namespace equals cluster administrator rights, so it should not be given through RBAC (part 29) to anyone but administrators.

apiVersion: v1
kind: Namespace
metadata:
  name: etcd-backup
  labels:
    pod-security.kubernetes.io/enforce: privileged
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: etcd-backup
  namespace: etcd-backup
data:
  AGE_RECIPIENT: age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p   # public key
  RCLONE_CONFIG_BACKUP_TYPE: s3
  RCLONE_CONFIG_BACKUP_PROVIDER: Other
  RCLONE_CONFIG_BACKUP_ENDPOINT: https://backup.example.com
  RCLONE_CONFIG_BACKUP_REGION: us-east-1                                                  # the region the storage expects
  BUCKET: k8s-etcd-snapshots
---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: etcd-snapshot
  namespace: etcd-backup
spec:
  schedule: "17 * * * *"          # every hour at minute 17
  timeZone: Etc/UTC
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 2
      activeDeadlineSeconds: 900
      template:
        spec:
          restartPolicy: Never
          hostNetwork: true
          automountServiceAccountToken: false
          imagePullSecrets:
          - name: ghcr-pull
          nodeSelector:
            node-role.kubernetes.io/control-plane: ""
          tolerations:
          - key: node-role.kubernetes.io/control-plane
            operator: Exists
            effect: NoSchedule
          initContainers:
          - name: snapshot
            image: registry.k8s.io/etcd:3.6.8-0
            command:
            - etcdctl
            - --endpoints=https://127.0.0.1:2379
            - --cacert=/pki/ca.crt
            - --cert=/pki/client.crt
            - --key=/pki/client.key
            - snapshot
            - save
            - /work/etcd.db
            securityContext:
              runAsUser: 0
              allowPrivilegeEscalation: false
              readOnlyRootFilesystem: true
              capabilities:
                drop: ["ALL"]
            volumeMounts:
            - { name: etcd-ca,     mountPath: /pki/ca.crt,     readOnly: true }
            - { name: etcd-client, mountPath: /pki/client.crt, readOnly: true }
            - { name: etcd-key,    mountPath: /pki/client.key, readOnly: true }
            - { name: work,        mountPath: /work }
          containers:
          - name: upload
            image: ghcr.io/example/etcd-backup:1.0
            command: ["/bin/sh", "-ec"]
            args:
            - |
              name="etcd-$(date -u +%Y%m%dT%H%M%SZ).db.age"
              age -r "$AGE_RECIPIENT" -o "/work/$name" /work/etcd.db
              rm -f /work/etcd.db
              rclone copyto --no-check-dest --s3-no-check-bucket --s3-no-head \
                "/work/$name" "backup:$BUCKET/$name"
              echo "uploaded $name"
            envFrom:
            - configMapRef: { name: etcd-backup }
            - secretRef:    { name: etcd-backup-s3 }
            securityContext:
              runAsUser: 0
              allowPrivilegeEscalation: false
              readOnlyRootFilesystem: true
              capabilities:
                drop: ["ALL"]
            resources:
              requests: { cpu: 50m, memory: 64Mi }
              limits:   { memory: 256Mi }
            volumeMounts:
            - { name: work, mountPath: /work }
          volumes:
          - name: etcd-ca
            hostPath: { path: /etc/kubernetes/pki/etcd/ca.crt, type: File }
          - name: etcd-client
            hostPath: { path: /etc/kubernetes/pki/etcd/healthcheck-client.crt, type: File }
          - name: etcd-key
            hostPath: { path: /etc/kubernetes/pki/etcd/healthcheck-client.key, type: File }
          - name: work
            emptyDir: { sizeLimit: 4Gi }

Notes on the details:

Two containers. The etcd image is distroless, that is, it has no distribution inside: it contains etcd, etcdctl and etcdutl but no shell, so an init container takes the snapshot with a single command, and the main container does the encryption and upload. Init containers run before the main ones, so by the time upload starts the file is already in place.
Root without privileges. The healthcheck-client.key key on the node belongs to root with permissions 0600, and etcdctl writes the snapshot with the same permissions. Both containers run with UID 0, but without a single capability, without privilege escalation and with a read-only file system: root inside the pod remains the owner of its own files, and nothing more.
concurrencyPolicy: Forbid — if the previous run has hung, a new one does not start in parallel. activeDeadlineSeconds limits the time of a Job so that a hung run does not hang forever.
emptyDir with sizeLimit — on the node's disk, not in memory: a snapshot of hundreds of megabytes in medium: Memory would take up the pod's memory. The size limit protects the master's disk: if it is exceeded, the kubelet evicts the pod.
imagePullSecrets — the etcd-backup image is in a private registry, and a Secret works only in its own namespace, so etcd-backup needs its own copy of ghcr-pull (part 5).
S3 credentials — in a Secret created with a command rather than a manifest in Git:

$ read -rsp 'S3 secret key: ' S3_SECRET; echo
$ kubectl -n etcd-backup create secret generic etcd-backup-s3 \
    --from-literal=RCLONE_CONFIG_BACKUP_ACCESS_KEY_ID=k8s-etcd-writer \
    --from-literal=RCLONE_CONFIG_BACKUP_SECRET_ACCESS_KEY="$S3_SECRET"
secret/etcd-backup-s3 created
$ unset S3_SECRET

rclone takes the settings of the backup remote from the RCLONE_CONFIG_BACKUP_* variables, so no configuration file is needed. The three flags are required because of the write-only account, which is denied all reads. --no-check-dest turns off the check for whether such an object already exists in the storage: without it rclone first sends HeadObject and exits with api error Forbidden. --s3-no-check-bucket turns off the bucket check, and --s3-no-head turns off reading the object after the write. The restriction works the other way too: this key can neither list objects, nor download them, nor delete them.

The retention policy is set in the bucket: for example, hourly snapshots are kept for 7 days, and once a day a separate CronJob with a different prefix stores a copy that lives for 90 days. The frequency depends on how many changes in the cluster you are prepared to lose: for a cluster where everything is described in Git (more on this in part 41), losing an hour is usually fixed by re-applying the manifests and is no disaster.

Monitoring. A backup whose failure nobody knows about is worse than no backup: people rely on it. kube-state-metrics (an exporter of metrics about the state of API objects) from the Prometheus stack publishes kube_cronjob_status_last_successful_time, and an alert rule "the last successful run of etcd-snapshot is older than two hours" catches both failures and the case where the CronJob has stopped running altogether. And once a quarter a snapshot should be restored on a separate virtual machine: that is the only way to verify that the age key is in place, the files decrypt and the procedure below works.

A manual check:

$ kubectl -n etcd-backup create job --from=cronjob/etcd-snapshot etcd-snapshot-manual
job.batch/etcd-snapshot-manual created
$ kubectl -n etcd-backup wait --for=condition=complete --timeout=120s job/etcd-snapshot-manual
job.batch/etcd-snapshot-manual condition met
$ kubectl -n etcd-backup logs job/etcd-snapshot-manual -c upload
2026/11/05 08:06:53 NOTICE: Config file "/root/.rclone.conf" not found - using defaults
uploaded etcd-20261105T080653Z.db.age
$ kubectl -n etcd-backup get cronjob etcd-snapshot -o jsonpath='{.status}{"\n"}'
{"lastSuccessfulTime":"2026-11-05T08:06:57Z"}

The message about the configuration file is harmless: rclone takes its settings from the variables. A manual Job updates lastSuccessfulTime but not the LAST SCHEDULE column in kubectl get cronjob: that column shows only scheduled runs. The file in the storage starts with the line age-encryption.org/v1, and strings will not find a single object name in it.

Restoring a single-node control plane

The scenario: on k8s-master-1 someone has deleted a namespace with data, or an operator has corrupted hundreds of objects, or the database is damaged. The machine and the /etc/kubernetes/pki directory are intact; the API state must be returned to the moment of the snapshot.

The main rule from the Kubernetes documentation: etcd is not restored while even one kube-apiserver is running. First all apiservers are stopped, then etcd is restored, then the apiservers are started again. Otherwise an apiserver with an in-memory cache will keep writing over the restored database.

Step 0. Decrypt the snapshot on a workstation that has the private key, and copy it to the master:

$ age -d -i ~/secure/etcd-backup.agekey -o etcd.db etcd-20261105T080653Z.db.age
$ scp etcd.db user@k8s-master-1.example.com:/tmp/etcd.db
$ rm etcd.db

user@k8s-master-1:~$ sudo chmod 600 /tmp/etcd.db
user@k8s-master-1:~$ sudo etcdutl snapshot status /tmp/etcd.db -w table
+----------+----------+------------+------------+---------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE | VERSION |
+----------+----------+------------+------------+---------+
| 15f337e0 |  2207961 |        577 |      35 MB |   3.6.0 |
+----------+----------+------------+------------+---------+

Step 1. Stop the control plane static pods. The kubelet watches the manifest directory and stops a pod whose file has disappeared (it checks every 20 seconds). We move all four — etcd, apiserver, controller-manager, scheduler — to a neighbouring directory:

user@k8s-master-1:~$ sudo mkdir -p /etc/kubernetes/manifests-stopped
user@k8s-master-1:~$ sudo mv /etc/kubernetes/manifests/*.yaml /etc/kubernetes/manifests-stopped/
user@k8s-master-1:~$ sudo crictl ps --name 'etcd|kube-apiserver|kube-controller-manager|kube-scheduler'
CONTAINER           IMAGE               CREATED             STATE               NAME                ATTEMPT             POD ID              POD                 NAMESPACE

An empty list means all are stopped; on our cluster this took half a minute. The kubelet does not read a copy of the manifests in another directory: it looks only in /etc/kubernetes/manifests.

Step 2. Restore into the data directory. We do not delete the old directory but rename it: if something goes wrong, we can go back to it. We restore to the original place, /var/lib/etcd. The documentation also offers another path, a new directory with an edited hostPath in the manifest, but in a kubeadm cluster kubeadm upgrade (part 35) regenerates the manifest with the directory from the kubeadm configuration, and etcd suddenly starts on the old data.

user@k8s-master-1:~$ sudo mv /var/lib/etcd /var/lib/etcd.old
user@k8s-master-1:~$ sudo etcdutl snapshot restore /tmp/etcd.db \
    --name k8s-master-1 \
    --initial-cluster k8s-master-1=https://192.168.10.17:2380 \
    --initial-advertise-peer-urls https://192.168.10.17:2380 \
    --data-dir /var/lib/etcd \
    --bump-revision 1000000000 --mark-compacted
...
2026-11-05T08:08:49Z	info	snapshot/v3_snapshot.go:396	bumping latest revision	{"latest-revision": 2207961, "bump-amount": 1000000000, "new-latest-revision": 1002207961}
2026-11-05T08:08:49Z	info	snapshot/v3_snapshot.go:413	marking revision compacted	{"revision": 1002207961}
...
2026-11-05T08:08:50Z	info	snapshot/v3_snapshot.go:333	restored snapshot	{"path": "/tmp/etcd.db", "wal-dir": "/var/lib/etcd/member/wal", "data-dir": "/var/lib/etcd", "snap-dir": "/var/lib/etcd/member/snap"}

--name, --initial-cluster and --initial-advertise-peer-urls must match the flags from the manifest: restore creates a new cluster of one member with a new identifier, and etcd checks them against its own at startup. The last two flags matter specifically for Kubernetes. After a restore, revisions go backwards, while the kubelet, the controllers and other clients remember a resourceVersion from "the future", greater than the current database revision. --bump-revision raises the revision by a billion so that each new write is certainly newer than anything clients have seen. --mark-compacted marks all the old history as compacted: every watch gets 410 Gone, and clients read the object lists again instead of waiting for events that will never come.

Step 3. Start etcd, make sure it is alive, then start the others:

user@k8s-master-1:~$ sudo mv /etc/kubernetes/manifests-stopped/etcd.yaml /etc/kubernetes/manifests/
user@k8s-master-1:~$ sudo etcdctl --endpoints=https://127.0.0.1:2379 \
    --cacert=/etc/kubernetes/pki/etcd/ca.crt \
    --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
    --key=/etc/kubernetes/pki/etcd/healthcheck-client.key endpoint health
https://127.0.0.1:2379 is healthy: successfully committed proposal: took = 3.530884ms
user@k8s-master-1:~$ sudo mv /etc/kubernetes/manifests-stopped/*.yaml /etc/kubernetes/manifests/
user@k8s-master-1:~$ sudo rmdir /etc/kubernetes/manifests-stopped

Step 4. Restart the components that held state. The Kubernetes documentation recommends restarting kube-scheduler, kube-controller-manager and the kubelet so that none of them relies on cached data. The static pods have just been started again, so we restart the kubelet on all nodes in turn:

user@k8s-master-1:~$ sudo systemctl restart kubelet
user@k8s-node-1:~$ sudo systemctl restart kubelet
user@k8s-node-2:~$ sudo systemctl restart kubelet

$ kubectl get nodes
NAME           STATUS   ROLES           AGE   VERSION
k8s-master-1   Ready    control-plane   11d   v1.36.5
k8s-node-1     Ready    <none>          11d   v1.36.5
k8s-node-2     Ready    <none>          11d   v1.36.5
$ kubectl get pods -A | grep -v Running
NAMESPACE     NAME                         READY   STATUS      RESTARTS   AGE
etcd-backup   etcd-snapshot-manual-84l7s   0/1     Completed   0          3m12s

The completed pod of the manual snapshot is the only thing that is not Running. The /var/lib/etcd.old directory is deleted once it is clear that the cluster works, and /tmp/etcd.db right away.

Revisions now start from a billion, and a watch from the past gets 410:

$ kubectl -n demo patch configmap hello-config --type merge -p '{"data":{"GREETING":"again"}}'
configmap/hello-config patched
$ kubectl -n demo get configmap hello-config -o jsonpath='{.metadata.resourceVersion}{"\n"}'
1002208190
$ kubectl get --raw "/api/v1/namespaces/demo/configmaps?watch=1&resourceVersion=2207071&timeoutSeconds=2"
{"type":"ERROR","object":{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"too old resource version: 2207071 (1002207961)","reason":"Expired","code":410}}

What happens after a restore

The API again shows the world as it was at the moment of the snapshot, while the nodes live in the present. The controllers and the kubelet reconcile one with the other, as in any reconciliation loop, and this should be understood in advance:

Objects created after the snapshot are gone. The kubelet treats their pods on the nodes as extra and stops them: a pod that is not in the API does not live on a node. A Deployment created five minutes before the failure has to be applied again, and here a cluster with all its manifests in Git has the advantage.
Objects deleted after the snapshot are back. On our cluster the hello Deployment was deleted after the snapshot, and by the time of the restore its pods had already terminated. After the restore they appeared again under the same names, and the kubelet started their containers again: RESTARTS became 1 and the pod address changed. A deleted Secret is available again too. If the deletion was a fix (access revoked, a compromised key deleted), it has to be repeated.
Volumes. PVs dynamically created after the snapshot have disappeared from the API, but the directories on the NFS server and the data in them remain: these are orphans to be found and sorted out by hand. Conversely, a PV deleted after the snapshot together with its data returns to the API, but the data behind it is gone.
Nodes. A node joined after the snapshot re-registers itself: the kubelet creates a Node object if there is none. Bootstrap tokens issued after the snapshot are invalid.
cert-manager certificates issued after the snapshot are missing from the Secrets, and cert-manager issues them again.
Calico IPAM. Calico with its datastore in the Kubernetes API keeps pod address allocation in its own objects, that is, also in etcd. After a rollback they may diverge from reality; calicoctl ipam check helps check consistency (calicoctl is the Calico management utility, installed separately; it needs the variable DATASTORE_TYPE=kubernetes). On our cluster after the rollback it answered Check complete; found 0 problems, but almost nothing had changed there in that time.

For a cluster of several members the procedure is the same, but restore runs on each master from the same snapshot file, with its own --name and --initial-advertise-peer-urls and a shared --initial-cluster listing all members.

If everything in this part has been done, the cluster has an hourly encrypted snapshot outside the cluster, a copy of pki and a tested recovery procedure. With that you can calmly take on the riskiest planned operation: changing the version.

Sources

Operating etcd clusters for Kubernetes — Kubernetes documentation: snapshots, restore, defragmentation.
kube-apiserver — the --etcd-compaction-interval and --event-ttl flags.
Maintenance and Disaster recovery — etcd documentation: compaction, defrag, quota, etcdutl snapshot restore, --bump-revision, --mark-compacted.
Hardware recommendations, Tuning and Metrics — disk requirements, timeouts, fsync metrics.
Failure modes and Data model — quorum, revisions, bbolt.
The Raft Consensus Algorithm — description of the protocol.
etcd-defrag — the defragmentation utility.
age and rclone: Amazon S3, rclone global flags — encrypting and uploading snapshots.
CronJob metrics — kube-state-metrics metrics.

Next — part 35: upgrading the cluster. Version policy and the allowed skew between components, kubeadm upgrade on the master and workers, kubeadm certificates and their renewal, upgrading the node OS and replacing a node.