Vitaliy Leschenko’s blog
ru
Виталий Лещенко

Kubernetes for beginners. Part 2: Cluster architecture

All parts of the series “Kubernetes for beginners”

In the first part the cluster looked like a set of controllers that read the desired state through kube-apiserver and bring the world in line with it. Now let's take it apart piece by piece: which processes run on the master and on the nodes, what each of them does, who connects to whom and on which ports, how the control plane starts itself, and what happens in the few seconds it takes kubectl run to turn into a running container. We'll install the cluster in part 4; here, in the "Hands-on" section, we'll see how all of this looks on a lab cluster that is already up: the master k8s-master-1 (192.168.10.17) and the worker nodes k8s-node-1 (192.168.10.20) and k8s-node-2 (192.168.10.21).

The big picture

 ┌─ k8s-master-1: control plane ────────────────────────────────────────┐
 │                                                                      │
 │ ┌──────────┐        ┌────────────────┐         ┌───────────────────┐ │
 │ │ etcd     │◄─2379──┤ kube-apiserver │◄─6443───┤ kube-scheduler    │ │
 │ └──────────┘        │                │◄───┐    └───────────────────┘ │
 │                     └────┬─────┬─────┘    │                          │
 │                          ▲     │          │    ┌───────────────────┐ │
 │                          │     │          └────┤ kube-controller-  │ │
 │                          │     │               │ manager           │ │
 │                          │     │               └───────────────────┘ │
 └──────────────────────────┼─────┼─────────────────────────────────────┘
          6443: watch,      │     │ 10250: logs, exec,
          statuses, Lease   │     │ port-forward
                            │     ▼
 ┌─ k8s-node-1, k8s-node-2 ─┴───────────────────────────────────────────┐
 │ kubelet ──CRI──► containerd ──► containerd-shim-runc-v2 ──► runc     │
 │                   │                                                  │
 │                   └─► Calico CNI plugin: veth, address, route        │
 │                                                                      │
 │ kube-proxy (DaemonSet)  ──► nftables rules for services              │
 │ calico-node (DaemonSet) ──► routes between nodes, policies           │
 └──────────────────────────────────────────────────────────────────────┘

The main rule of the architecture: only kube-apiserver talks to etcd. All the other components are API clients: they connect to kube-apiserver over HTTPS on port 6443, receive changes through watch and write the results of their work back through the API. The components don't contact each other directly. The only notable exception is that kube-apiserver itself connects to kubelet on port 10250 when it needs to return a container's logs, run kubectl exec or forward a port.

kubelet, containerd, kube-proxy and calico-node run on the master too: in a kubeadm cluster the control plane itself consists of pods, and something has to run them. Ordinary applications don't land on the master because of the taint node-role.kubernetes.io/control-plane:NoSchedule, which kubeadm puts on it during installation.

Component Where and how it runs Ports Who connects
kube-apiserver master, static pod 6443/TCP all components, kubectl, add-ons
etcd master, static pod 2379/TCP (clients), 2380/TCP (other etcd members), 2381 (metrics, 127.0.0.1 only) only kube-apiserver; on 2380, the etcd cluster members among themselves
kube-scheduler master, static pod 10259/TCP, HTTPS: health checks and metrics, on 127.0.0.1 monitoring systems
kube-controller-manager master, static pod 10257/TCP, the same, on 127.0.0.1 monitoring systems
kubelet every node, systemd service 10250/TCP — the kubelet API; 10248 — health check on 127.0.0.1 kube-apiserver, metrics collectors
kube-proxy every node, DaemonSet pod 10256/TCP — health check; 10249 — metrics on 127.0.0.1 external load balancers, monitoring
NodePort services every node 30000–32767/TCP and UDP clients outside the cluster

A static pod is a pod that kubelet runs from a file on disk, without the API being involved; we'll see below why this is needed. A DaemonSet is an object that keeps one copy of a pod on every node; more on it in part 9.

kubeadm binds ports 10257 and 10259 to 127.0.0.1: there's no reason to reach the scheduler and the controller manager from outside, and every open port is extra attack surface. If their metrics need to be collected into Prometheus (a system for collecting metrics) from another machine, the address is changed deliberately and together with the firewall rules.

kube-apiserver

kube-apiserver is an HTTP server that implements the Kubernetes API. Every request goes through the same chain:

  1. TLS and authentication — who has come. A client certificate signed by the cluster CA (this is how the components and the administrator with admin.conf log in), a service account token (this is how pods log in), a bootstrap token (this is how new nodes log in), an external OIDC provider (single sign-on through a corporate identity server). More in part 30.
  2. Authorization — whether this subject is allowed this action. A kubeadm cluster runs two modules: Node (kubelet can read only what relates to the pods on its node) and RBAC (roles and role bindings).
  3. Admission — modules that can modify an object (mutating) or reject it (validating) after authorization: fill in default values, check quotas, forbid privileged pods. More in part 3.
  4. Schema validation — the fields and types match the API version.
  5. Write to etcd and the response to the client.

kube-apiserver has no state of its own: everything is in etcd. That's why there can be several instances behind a load balancer — this is how a highly available control plane is built, as covered in the BGP series, where the API address is held by kube-vip (a pod on the masters that brings up a shared virtual address on one of them and announces it over BGP). Internally, kube-apiserver keeps a watch cache: the latest version of every object and a short history of changes. Most reads and all subscriptions are served from this cache rather than from etcd — otherwise thousands of controllers would bring the database down.

Two more duties. kube-apiserver can proxy part of the API to external servers (the aggregation layer): this is how, for example, metrics-server, which collects current CPU and memory usage, serves its data under the path /apis/metrics.k8s.io/. And it shares its capacity among clients (API Priority and Fairness): requests are divided into priority levels and queues so that a single controller stuck in a loop can't take over the whole server and keep kubelet from reporting the state of the nodes.

etcd

etcd is a distributed key-value store whose consistency is ensured by the Raft algorithm: the members of the etcd cluster elect a leader, all writes go through it and are considered committed once a majority of members (a quorum) has saved them. That's why the number of members is odd: three survive the loss of one, five the loss of two. The lab cluster has a single etcd, and losing it means losing the cluster; how to back it up and restore it will be covered in part 35.

Objects are stored under keys of the form /registry/<resource>/<namespace>/<name>: the pod test from the default namespace is /registry/pods/default/test, a Deployment is /registry/deployments/default/web. The values are serialized in Protobuf (Google's compact binary format), not in JSON. Every record has an etcd revision number; it is essentially the object's resourceVersion in the API, which part 3 is about.

An important security detail: by default the values in etcd are not encrypted, Secrets included. Access to etcd or to its files in /var/lib/etcd is access to all the cluster's secrets. That's why etcd listens to clients only over TLS with client certificate verification (it accepts only certificates signed by a separate CA, etcd-ca), and encryption of secrets at the kube-apiserver level is enabled separately — that's the topic of part 13.

kube-scheduler

The scheduler solves one problem: for every pod whose spec.nodeName field is empty, choose a node. It subscribes to such pods through watch and puts each one through two stages:

  • filtering: discard the nodes the pod doesn't fit on — not enough resources according to requests, a taint the pod doesn't tolerate, labels from nodeSelector or affinity rules don't match, a required host port is taken, a volume can't be attached in this zone;
  • scoring: the remaining nodes get points — for even load, for spreading the replicas of one application across nodes, for an image already pulled — and the best one is chosen.

The scheduler writes the result to the API through the binding subresource: a short request saying "such-and-such pod goes to such-and-such node", after which the pod gets spec.nodeName. It doesn't start anything itself and doesn't contact the nodes. The placement rules are covered in detail in part 14.

kube-controller-manager

A single process inside which dozens of controllers run — each with its own reconciliation loop from part 1. The main ones:

Controller What it does
deployment creates and updates ReplicaSets from the Deployment template
replicaset keeps the required number of pods
statefulset, daemonset, job, cronjob the same for their own types of workloads
node-lifecycle watches the renewal of node Leases, sets NotReady and taints, evicts pods
endpointslice for every Service, collects the list of addresses of ready pods (EndpointSlice)
namespace deletes all the contents of a namespace being deleted
garbage-collector deletes objects whose owner has disappeared (cascading deletion)
serviceaccount, serviceaccount-token creates the default service account in every namespace
csrapproving, csrsigning approves and signs certificate signing requests (this is how nodes get certificates when they join)
bootstrapsigner, tokencleaner sign the data used to join nodes and delete expired bootstrap tokens
persistentvolume-binder, attachdetach bind volume claims to volumes and attach volumes to nodes

kubeadm runs it with the flag --use-service-account-credentials=true: each controller accesses the API under its own service account (system:serviceaccount:kube-system:replicaset-controller and so on) with minimal permissions, rather than under one shared account.

If there are several instances of kube-scheduler and kube-controller-manager (as in a highly available control plane), only one of each pair works — the leader. Leadership is a Lease object in the kube-system namespace: the leader renews it every few seconds, and the other instances wait until the lease expires without being renewed (15 seconds by default). This way two schedulers never assign the same pod at the same time.

For clouds there's also the cloud-controller-manager — controllers that know the API of a particular cloud: they create cloud load balancers, find out a node's zone, and delete the nodes of deleted virtual machines. On your own hardware it isn't there.

kubelet

kubelet is the agent on every node, the only Kubernetes component that creates containers itself (more precisely, asks the runtime to do so). It doesn't run in a container but as an ordinary systemd service: it needs full access to the host, and it must keep working even when the API is unavailable. Its duties:

  • registers the node: on first start it creates a Node object with its resources, addresses and versions;
  • reports liveness: every 10 seconds it renews its Lease in the kube-node-lease namespace and periodically updates the node's status (the conditions Ready, MemoryPressure, DiskPressure, PIDPressure);
  • runs pods: it subscribes through watch to pods whose spec.nodeName equals the name of its node and brings the containers on the node in line with their description;
  • attaches volumes, mounts Secrets and ConfigMaps into directories under /var/lib/kubelet/pods/<pod uid>/volumes/;
  • checks the health of containers (probes) and restarts the ones that crashed;
  • serves an API on port 10250: logs, exec, attach, port-forward, resource metrics. Access is only with a client certificate or a token; kubeadm disables anonymous requests;
  • evicts pods when the node runs short of memory or disk.

kubelet's configuration is the file /var/lib/kubelet/config.yaml (a KubeletConfiguration object), and its credentials for the API are /etc/kubernetes/kubelet.conf with the client certificate CN=system:node:k8s-node-1, O=system:nodes. The Node authorization module and the NodeRestriction admission plugin make sure that a kubelet with such a certificate can change only its own Node object and the pods of its own node: taking over one node doesn't give control over its neighbors.

Static pods: how the control plane starts itself

kube-apiserver, etcd, the scheduler and the controller manager in a kubeadm cluster are pods. But pods are created through the API, and there is no API yet. The chicken-and-egg problem is solved by the static pod mechanism: kubelet reads pod manifests straight from the directory given in staticPodPath (/etc/kubernetes/manifests with kubeadm) and runs them itself, without asking anyone.

The order at master startup:

  1. systemd starts containerd and kubelet.
  2. kubelet finds four files in /etc/kubernetes/manifests — etcd.yaml, kube-apiserver.yaml, kube-controller-manager.yaml, kube-scheduler.yaml — and starts the pods.
  3. etcd comes up, then kube-apiserver; now kubelet has someone to talk to.
  4. kubelet creates mirror pods in the API — objects named kube-apiserver-k8s-master-1 and so on, so that static pods are visible in kubectl get pods.

A mirror pod is only a reflection. kubectl delete pod kube-apiserver-k8s-master-1 will delete the object, but kubelet will immediately create it again, and the container won't even restart. A static pod can be changed only by editing its file: kubelet watches the directory and recreates the pod when the manifest changes. This is how kube-apiserver's flags are changed — and it's just as easy to bring it down with a typo, so before editing, the file is copied outside the manifests directory: a copy kube-apiserver.yaml.bak left next to it will also be read by kubelet as a manifest.

Everything else in kube-system — kube-proxy, CoreDNS (the cluster's DNS server, which answers for service names; more in part 17) — and the Calico network plugin are ordinary pods created through the API by DaemonSet and Deployment objects.

The runtime: CRI, containerd, shim and runc

kubelet can't run containers itself. It talks to the runtime over CRI (Container Runtime Interface) — a gRPC protocol (gRPC is a remote procedure call mechanism on top of HTTP/2) over a Unix socket; for containerd it's /run/containerd/containerd.sock. CRI has two services:

  • RuntimeService: RunPodSandbox (create the pod's "sandbox" — shared namespaces and networking), CreateContainer, StartContainer, StopContainer, RemovePodSandbox, ExecSync, ContainerStatus and so on;
  • ImageService: PullImage, ListImages, RemoveImage, ImageFsInfo.

Before version 1.24 kubelet could also talk to Docker through the built-in dockershim layer; it was removed, and now any runtime connects only over CRI. Images built with Docker still work as before: they share the same format, OCI.

containerd serves CRI with a built-in plugin. It keeps Kubernetes objects in its own namespace k8s.io (this is containerd's internal grouping, not kernel namespaces). It stores images in /var/lib/containerd and unpacks layers with the overlayfs snapshotter: each layer is a directory, and the container's root is their union through the overlay file system, with the container's writable layer on top.

For every pod, containerd starts a separate containerd-shim-runc-v2 process. The shim becomes the parent of the pod's containers: it holds their stdin/stdout, collects exit codes and writes logs. Thanks to it, containerd can be restarted or upgraded without touching running containers. The container itself is created by runc: the shim calls it with an OCI description (config.json), and runc creates the namespaces and cgroup, starts the process and exits.

The first container started in every pod is pause (the registry.k8s.io/pause image): a tiny process that does nothing but hold the pod's network namespace. The pod's other containers join its namespaces, which is why a pod has a single IP address and restarting an application container doesn't lose the network. More in part 7.

kubelet gives containers cgroups inside the kubepods.slice hierarchy: with the systemd cgroup driver (and it's the only correct one when systemd manages the system and cgroup v2 is used), each pod's cgroup is a systemd unit. How kubelet divides it by QoS class is in part 11.

containerd writes the containers' stdout and stderr logs to files /var/log/pods/<namespace>_<pod>_<uid>/<container>/0.log, and kubelet serves them on a kubectl logs request. For debugging on a node there's crictl — a CRI client that sees pods and containers the same way kubelet does.

The network plugin (CNI)

When containerd creates a pod sandbox, it needs to connect it to the network. It doesn't know how exactly, so it calls a CNI plugin: an executable from /opt/cni/bin with its configuration from /etc/cni/net.d. The CNI specification is simple: the plugin is run with the command ADD, DEL or CHECK in an environment variable, receives the path to the network namespace and the configuration as JSON on stdin, and prints the assigned address in response.

In the lab cluster the plugin is Calico. On ADD it creates a veth pair (a virtual "cable" of two interfaces: one end inside the pod as eth0, the other on the node as cali…), takes an address from the block allocated to the node from the 172.20.0.0/16 pool, and adds a route to the pod through this interface on the node. The rest is done by the Calico pods in the calico-system namespace:

  • calico-node (DaemonSet): it runs Felix — an agent that configures pod interfaces and writes network policy rules into nftables — and BIRD — a BGP daemon (BGP being the protocol routers use to tell each other which networks are reachable through them), through which the nodes exchange routes to each other's pod address blocks;
  • calico-typha: an intermediary between the API and Felix that holds a single subscription to changes and distributes them to all nodes, so that hundreds of Felix instances don't each load kube-apiserver with their own;
  • calico-kube-controllers: Calico's controllers, for example cleaning up the addresses of deleted nodes.

All of this is installed and upgraded by the Tigera operator (tigera-operator) from an Installation object; how, in part 4, and how packets travel between pods, in part 15.

kube-proxy

kube-proxy, despite its name, doesn't proxy anything. It runs as a DaemonSet pod on every node, subscribes to Service and EndpointSlice objects, and turns them into kernel rules: a packet to a service's virtual address from the 10.96.0.0/12 network must go to the address of one of the pods. In the lab cluster it runs in nftables mode — the rules live in the nftables tables ip kube-proxy and ip6 kube-proxy. The iptables, nftables and IPVS modes, how they work, and how Calico can replace kube-proxy with its eBPF dataplane are the topic of part 16.

Who talks to whom

From To Port Why How it authenticates
kube-apiserver etcd 2379 reading and writing objects the apiserver-etcd-client certificate
kube-scheduler, kube-controller-manager kube-apiserver 6443 watch, writing decisions certificates from scheduler.conf, controller-manager.conf
kubelet kube-apiserver 6443 watch of its pods, statuses, Lease the system:node:<name> certificate
kube-proxy, calico-node, CoreDNS kube-apiserver 6443 watch of services, endpoints, policies service account tokens
kubectl kube-apiserver 6443 everything the certificate from kubeconfig
kube-apiserver kubelet 10250 logs, exec, attach, port-forward the apiserver-kubelet-client certificate
kube-apiserver admission webhooks, aggregated APIs the service's 443 object validation, metrics-server the service's TLS
kubelet containerd Unix socket CRI permissions on the socket (root only)
containerd CNI plugin running the file connecting the pod to the network —
calico-node (BIRD) calico-node on other nodes 179 exchanging routes to pod blocks over BGP —

All API clients work the same way: at startup they do a list of the objects they need, put them in a local cache and open a watch from the point where the list ended. From then on the cache is updated by events, and the controller reads objects from the cache and puts the keys of changed objects into a work queue. In the client-go library this combination is called an informer. Hence an important property: controllers hardly ever read the API with GET requests; the whole load is one watch per object type.

The path of kubectl run

Let's trace what happens after the command:

$ kubectl run test --image=nginx:1.29
pod/test created

 kubectl         kube-apiserver      kube-scheduler        kubelet         containerd
    │                   │                   │                 │                 │
    ├── 1. POST pods ──►│                   │                 │                 │
    │                   │ 2. authn, authz,  │                 │                 │
    │                   │ admission, etcd   │                 │                 │
    │◄─ 201 Created ────┤                   │                 │                 │
    │                   ├── 3. watch ──────►│                 │                 │
    │                   │                   │ 4. filter,      │                 │
    │                   │                   │ score nodes     │                 │
    │                   │◄─ POST binding ───┤                 │                 │
    │                   ├── 5. watch ───────┼────────────────►│                 │
    │                   │                   │                 ├── 6. Sandbox ──►│
    │                   │                   │                 │ pause, netns,   │
    │                   │                   │                 │ CNI ADD         │
    │                   │                   │                 ├── 7. Pull, Run ►│
    │                   │◄─ 8. PATCH status ┼─────────────────┤                 │
    │                   │                   │                 │                 │
  1. kubectl reads the kubeconfig (~/.kube/config): the API address, the cluster CA, the client certificate. It builds a Pod object with one container and sends POST /api/v1/namespaces/default/pods.
  2. kube-apiserver verifies the client certificate, and RBAC allows creating pods. Admission plugins fill in the object: ServiceAccount sets the default account and a volume with its token, DefaultTolerationSeconds adds tolerations of 300 seconds for the not-ready and unreachable taints from part 1. The object is validated against the schema and written to etcd under the key /registry/pods/default/test. The client gets 201 Created — the pod exists, but for now it's only a record: spec.nodeName is empty, and the status is Pending.
  3. kube-scheduler receives an ADDED event through watch about a pod without a node and puts it in its queue.
  4. The scheduler filters and scores the nodes, chooses k8s-node-2 and sends POST .../pods/test/binding. kube-apiserver writes in spec.nodeName: k8s-node-2 and adds the condition PodScheduled=True.
  5. kubelet on k8s-node-2 is subscribed to the pods of its node and receives the event. It checks that the pod can be admitted (enough resources, no conflicts), creates the pod's cgroup, and prepares the volumes — including the file with the service account token.
  6. kubelet calls RunPodSandbox over CRI. containerd starts the shim, which through runc starts the pause container with a new network namespace; containerd calls the Calico CNI plugin, which creates the veth, assigns the address 172.20.140.5 and adds the route.
  7. kubelet calls PullImage for nginx:1.29 (if the image isn't on the node yet), then CreateContainer and StartContainer: the shim, through runc, starts nginx in the pause container's namespaces.
  8. kubelet writes the address, the container state and the conditions Initialized, ContainersReady, Ready into the pod's status — with a request to the pods/test/status subresource. Now kubectl get pod shows Running.

After that, if the pod matches the selector of some Service, the endpointslice controller will add its address to an EndpointSlice, and kube-proxy on all nodes will add it to the nftables rules. Note that no component called another one directly. Each saw a change of an object in the API and took its own step.

Hands-on

The nodes and the control plane pods:

$ kubectl get nodes -o wide
NAME           STATUS   ROLES           AGE   VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE           KERNEL-VERSION     CONTAINER-RUNTIME
k8s-master-1   Ready    control-plane   12d   v1.36.5   192.168.10.17   <none>        Ubuntu 26.04 LTS   7.0.0-15-generic   containerd://2.2.2
k8s-node-1     Ready    <none>          12d   v1.36.5   192.168.10.20   <none>        Ubuntu 26.04 LTS   7.0.0-15-generic   containerd://2.2.2
k8s-node-2     Ready    <none>          12d   v1.36.5   192.168.10.21   <none>        Ubuntu 26.04 LTS   7.0.0-15-generic   containerd://2.2.2

$ kubectl get pods -n kube-system -o wide
NAME                                   READY   STATUS    RESTARTS   AGE   IP              NODE           NOMINATED NODE   READINESS GATES
coredns-5d7c9f8b6d-7hq2x               1/1     Running   0          12d   172.20.62.130   k8s-master-1   <none>           <none>
coredns-5d7c9f8b6d-wb4kz               1/1     Running   0          12d   172.20.62.129   k8s-master-1   <none>           <none>
etcd-k8s-master-1                      1/1     Running   0          12d   192.168.10.17   k8s-master-1   <none>           <none>
kube-apiserver-k8s-master-1            1/1     Running   0          12d   192.168.10.17   k8s-master-1   <none>           <none>
kube-controller-manager-k8s-master-1   1/1     Running   0          12d   192.168.10.17   k8s-master-1   <none>           <none>
kube-proxy-6xk2d                       1/1     Running   0          12d   192.168.10.20   k8s-node-1     <none>           <none>
kube-proxy-p8vqn                       1/1     Running   0          12d   192.168.10.17   k8s-master-1   <none>           <none>
kube-proxy-zr5tw                       1/1     Running   0          12d   192.168.10.21   k8s-node-2     <none>           <none>
kube-scheduler-k8s-master-1            1/1     Running   0          12d   192.168.10.17   k8s-master-1   <none>           <none>

The control plane pods and kube-proxy have the node's address rather than one from the pod network: they run with hostNetwork: true, in the network namespace of the node itself. The -k8s-master-1 suffix on static pods is the name of the node that started them. CoreDNS runs on the master because there were no worker nodes yet when it started, and it does have a toleration for the control plane taint.

The static pod manifests and the ports on the master:

user@k8s-master-1:~$ ls -l /etc/kubernetes/manifests/
total 16
-rw------- 1 root root 2569 Oct  9 11:02 etcd.yaml
-rw------- 1 root root 3994 Oct  9 11:02 kube-apiserver.yaml
-rw------- 1 root root 3478 Oct  9 11:02 kube-controller-manager.yaml
-rw------- 1 root root 1463 Oct  9 11:02 kube-scheduler.yaml

user@k8s-master-1:~$ sudo ss -tlnp | grep -E 'kube|etcd'
LISTEN 0      4096       127.0.0.1:10248      0.0.0.0:*    users:(("kubelet",pid=1102,fd=19))
LISTEN 0      4096       127.0.0.1:10249      0.0.0.0:*    users:(("kube-proxy",pid=2231,fd=13))
LISTEN 0      4096       127.0.0.1:10257      0.0.0.0:*    users:(("kube-controller",pid=1688,fd=3))
LISTEN 0      4096       127.0.0.1:10259      0.0.0.0:*    users:(("kube-scheduler",pid=1702,fd=3))
LISTEN 0      4096       127.0.0.1:2379       0.0.0.0:*    users:(("etcd",pid=1655,fd=7))
LISTEN 0      4096       127.0.0.1:2381       0.0.0.0:*    users:(("etcd",pid=1655,fd=14))
LISTEN 0      4096   192.168.10.17:2379       0.0.0.0:*    users:(("etcd",pid=1655,fd=8))
LISTEN 0      4096   192.168.10.17:2380       0.0.0.0:*    users:(("etcd",pid=1655,fd=6))
LISTEN 0      4096               *:10250            *:*    users:(("kubelet",pid=1102,fd=22))
LISTEN 0      4096               *:10256            *:*    users:(("kube-proxy",pid=2231,fd=12))
LISTEN 0      4096               *:6443             *:*    users:(("kube-apiserver",pid=1671,fd=3))

Everything is as in the table: only 6443, 2379–2380 (etcd listens on the node's address so that other etcd cluster members can connect to it in the future), 10250 and 10256 are open to the outside. The name kube-controller is truncated: the kernel stores a process name of up to 15 characters.

A mirror pod remembers where it came from:

$ kubectl get pod -n kube-system kube-apiserver-k8s-master-1 \
    -o jsonpath='{.metadata.annotations}{"\n"}{.metadata.ownerReferences[0].kind}{"\n"}'
{"kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint":"192.168.10.17:6443","kubernetes.io/config.hash":"4b8e1c0a6f2d9e7b3c5a1f0e8d6b2a9c","kubernetes.io/config.mirror":"4b8e1c0a6f2d9e7b3c5a1f0e8d6b2a9c","kubernetes.io/config.seen":"2026-10-09T11:02:41.118203547+03:00","kubernetes.io/config.source":"file"}
Node

config.source: file means the pod comes from a file, config.mirror marks a mirror pod, and the owner is the Node object: if the node is deleted, the reflection is deleted too.

The kubelet configuration and the systemd service:

user@k8s-node-2:~$ sudo grep -E '^(cgroupDriver|clusterDNS|clusterDomain|rotateCertificates|staticPodPath)' -A1 /var/lib/kubelet/config.yaml
cgroupDriver: systemd
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
rotateCertificates: true
staticPodPath: /etc/kubernetes/manifests

user@k8s-node-2:~$ systemctl status kubelet --no-pager | head -5
● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; preset: enabled)
    Drop-In: /usr/lib/systemd/system/kubelet.service.d
             └─10-kubeadm.conf
     Active: active (running) since Fri 2026-10-09 11:14:20 MSK; 1 week 5 days ago

The static pod directory on a worker node is empty, but kubelet watches it just as it does on the master.

The node and leader leases:

$ kubectl get lease -n kube-node-lease
NAME           HOLDER         AGE
k8s-master-1   k8s-master-1   12d
k8s-node-1     k8s-node-1     12d
k8s-node-2     k8s-node-2     12d

$ kubectl get lease -n kube-system
NAME                                   HOLDER                                                                      AGE
apiserver-q7bx3mfl2kcnq4zt6r5h2dvw4e   apiserver-q7bx3mfl2kcnq4zt6r5h2dvw4e_1c9d4a2e-5b7f-4c3a-9e8d-2f6a1b0c7e45   12d
kube-controller-manager                k8s-master-1_8f2e6c1a-4d3b-4a7e-b9c0-5e1f2a3d6b78                           12d
kube-scheduler                         k8s-master-1_3a7d9e0b-2c4f-4b1a-8e6d-7c5b9a0f1e23                           12d

$ kubectl get lease -n kube-node-lease k8s-node-2 -o jsonpath='{.spec.renewTime}{"  "}{.spec.leaseDurationSeconds}{"\n"}'
2026-10-21T08:15:42.906114Z  40

renewTime is updated every 10 seconds — a quarter of leaseDurationSeconds. The apiserver-… Lease is the mark of each kube-apiserver instance: this is how the instances learn about each other.

What's in etcd. The etcdctl utility is in the etcd image, and the certificates are in the /etc/kubernetes/pki/etcd directory, mounted into the pod:

$ kubectl exec -n kube-system etcd-k8s-master-1 -- etcdctl \
    --endpoints=https://127.0.0.1:2379 \
    --cacert=/etc/kubernetes/pki/etcd/ca.crt \
    --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
    --key=/etc/kubernetes/pki/etcd/healthcheck-client.key \
    get /registry/pods/default --prefix --keys-only
/registry/pods/default/test

Without a certificate etcd won't answer at all; this is exactly the boundary that no one except kube-apiserver crosses.

The test pod on the node as containerd sees it:

user@k8s-node-2:~$ sudo crictl pods --name test
POD ID              CREATED             STATE               NAME                NAMESPACE           ATTEMPT             RUNTIME
3f1b0c9d2e8a4       3 minutes ago       Ready               test                default             0                   (default)

user@k8s-node-2:~$ sudo crictl ps --pod 3f1b0c9d2e8a4
CONTAINER           IMAGE               CREATED             STATE               NAME                ATTEMPT             POD ID              POD                 NAMESPACE
9c4e7a1b5d2f6       6a2f8c1d9e0b3       3 minutes ago       Running             test                0                   3f1b0c9d2e8a4       test                default

user@k8s-node-2:~$ ps -eo pid,ppid,cmd --forest | grep -A3 '[c]ontainerd-shim-runc-v2 -namespace k8s.io -id 3f1b'
   4711       1 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id 3f1b0c9d2e8a4c7f0e5b9d1a6c3e8f2b4d7a0c9e1f5b3d8a6c2e4f7b9d0a1c3e -address /run/containerd/containerd.sock
   4735    4711  \_ /pause
   4802    4711  \_ nginx: master process nginx -g daemon off;
   4840    4802      \_ nginx: worker process

The shim's parent is PID 1 (systemd), not containerd: the shim is detached from it and will survive its restart. Under the shim there are two containers — pause and nginx, while crictl ps shows only the application one: the pod sandbox is a separate CRI entity.

The pod's events put together the whole path from the section above:

$ kubectl get events --field-selector involvedObject.name=test
LAST SEEN   TYPE     REASON      OBJECT     MESSAGE
3m12s       Normal   Scheduled   pod/test   Successfully assigned default/test to k8s-node-2
3m11s       Normal   Pulling     pod/test   Pulling image "nginx:1.29"
3m7s        Normal   Pulled      pod/test   Successfully pulled image "nginx:1.29" in 4.212s (4.212s including waiting). Image size: 72178381 bytes.
3m7s        Normal   Created     pod/test   Created container: test
3m7s        Normal   Started     pod/test   Started container test

The first event was written by the scheduler, the rest by kubelet. And finally, kube-apiserver itself can report on its health piece by piece:

$ kubectl get --raw='/readyz?verbose' | head -8
[+]ping ok
[+]log ok
[+]etcd ok
[+]etcd-readiness ok
[+]informer-sync ok
[+]poststarthook/start-apiserver-admission-initializer ok
[+]poststarthook/generic-apiserver-start-informers ok
[+]poststarthook/priority-and-fairness-config-consumer ok

Let's remove the test pod: kubectl delete pod test.

Next — the API from the inside: what an object consists of, how API groups and versions are organized, what resourceVersion and watch are, how server-side apply works, and how to peek at every request kubectl sends to the server.

Only registered users can leave a comment.

Sign in or register to leave a comment.

© 2006-2026 Vitaliy Leschenko