Kubernetes for beginners. Part 6: What is a Pod?

In the previous part we built the image ghcr.io/example/hello:1.0, put it into a private registry and started it with one short kind: Pod manifest without giving it much thought. Now let's look at what this object is. A Pod is the smallest unit that Kubernetes schedules and runs: not a container but a group of containers with a shared network and shared volumes. Everything that comes later — Deployment, StatefulSet, Job — ultimately creates pods, so how a pod is built, how it starts, how it is checked and how it stops has to be understood thoroughly.

How the API server stores objects and how the kubelet learns about its pods through a watch was covered in parts 2 and 3. Here we look at what happens on the node k8s-node-2 once the kubelet has received a pod.

---

Why a pod and not a container

A container is a process in its own Linux namespaces (isolated views of the network, the file system, the processes, the hostname) and its own cgroup (resource limits). An application often needs a companion that needs the same things it does: a proxy that accepts traffic and passes it to the application over localhost; an agent that ships logs from a shared directory; a process that renews certificates in a file the application reads. Such companions have to run on the same node, together, with a shared network and shared files — and be scheduled, restarted and deleted as one unit.

A pod is exactly that unit:

• shared network: all containers of a pod have one network namespace — one IP address, one lo, shared ports. Containers of the same pod reach each other over 127.0.0.1 and cannot bind the same port;
• shared IPC: shared memory and System V and POSIX message queues;
• shared hostname (UTS namespace): hostname in any container of the pod returns the pod name;
• shared volumes: a volume declared in the pod can be mounted into any of its containers;
• shared fate: the whole pod is assigned to one node, is deleted as a whole and does not move — on another node it will be a different pod with a different IP.

What each container has of its own is the file system (a mount namespace with the root from its own image) and, by default, the process space (PID namespace): in the hello container the process /hello is PID 1, and it does not see the nginx processes of the neighbouring container. The shareProcessNamespace: true field in the pod spec makes the PID namespace shared — this is needed, for example, so that a debugging container can see the application's processes or so that one container can send a signal to another.

The most common case is a single-container pod. A multi-container pod is for containers that truly cannot be separated. A web application and its database do not go into one pod: they need to be scaled, updated and restarted independently.

---

The pause container

If the network namespace is shared, someone has to hold it. The containers of a pod are restarted individually: when hello crashes, the kubelet creates it again, and if the namespace belonged to hello, the pod's IP address would disappear with it. That is why the first thing created in a pod is the service container pause (in CRI terms, the sandbox of the pod). Its image registry.k8s.io/pause (kubeadm 1.36 expects version 3.10.2) is about 300 KB: a C program that calls pause() and sleeps until a signal arrives. Its only job is to exist: the pod's namespaces are created together with it, and the other containers join its namespaces.

The second role of pause shows up with shareProcessNamespace: true: pause is then PID 1 of the shared process space and reaps orphaned zombie processes, the way init does on an ordinary system.

 Pod hello (sandbox)
 ┌──────────────────────────────────────────────────────────────┐
 │ shared namespaces: net (eth0 172.20.172.5, lo), ipc, uts     │
 │                                                              │
 │ ┌──────────────┐ ┌─────────────────┐ ┌─────────────────────┐ │
 │ │ pause        │ │ proxy (nginx)   │ │ hello               │ │
 │ │ holds the    │ │ listens :8081   │ │ listens :8080       │ │
 │ │ namespaces   │ │ own mnt and PID │ │ own mnt and PID     │ │
 │ └──────────────┘ └────────┬────────┘ └──────────▲──────────┘ │
 │                           └──── 127.0.0.1:8080 ─┘            │
 │                                                              │
 │ emptyDir volume nginx-conf: mounted in init and in proxy     │
 └──────────────────────────────────────────────────────────────┘

containerd 2.x takes the pause image from its own configuration (pinned_images.sandbox in the io.containerd.cri.v1.images section) and protects it from garbage collection. kubeadm pulls its own version but does not warn about a mismatch with containerd, and the version that actually runs is the one configured in containerd — which is why in part 4 we set the same 3.10.2 in config.toml that kubeadm expects.

---

How a pod starts

 kubectl apply ──► kube-apiserver ──► etcd: /registry/pods/default/hello   (nodeName empty)
                         │
 kube-scheduler: watch ──┤ picks a node ──► binding: nodeName = k8s-node-2
                         │
 kubelet k8s-node-2: watch on pods with its own nodeName
   1. pod cgroup    /sys/fs/cgroup/kubepods.slice/…/kubepods-besteffort-pod<uid>.slice
   2. volumes       /var/lib/kubelet/pods/<uid>/volumes/…
   3. CRI RunPodSandbox ──► containerd: pause, new netns, ipc, uts
                              └──► CNI ADD (Calico): veth, address 172.20.172.5
   4. images        CRI PullImage for each container (imagePullPolicy)
   5. init containers, one at a time, each until it completes successfully
   6. sidecar containers (init with restartPolicy: Always) — in order,
      the next one waits until the previous one has started
   7. main containers — all at once
   8. probes ──► pod status in the API: conditions, Ready

A pod is created in the API with an empty spec.nodeName. The scheduler (part 13) picks a node and records it through the binding subresource. The kubelet on that node sees the pod in its watch and gets to work. First comes the pod cgroup: with the systemd driver the kubelet creates a slice unit inside kubepods.slice, and which subdirectory it goes into (besteffort, burstable, or the root for Guaranteed) depends on the pod's QoS class, covered in part 10. Then it prepares the volumes. Then it asks the runtime over CRI to create the sandbox: containerd starts pause with new namespaces and calls the CNI plugin, which for Calico creates a veth pair — one end eth0 inside the pod, the other cali… on the node — and assigns an address from the 172.20.0.0/16 pool; details are in part 14. After that come the containers, in turn.

Most fields of a pod cannot be changed after creation: you cannot add a container or change the command or the environment variables. Only a little is allowed — the container image, activeDeadlineSeconds, new tolerations and, through a separate subresource, resources. Changing a pod means deleting it and creating a new one. That is the job of controllers such as Deployment, the subject of the next part.

---

Init containers

Init containers (spec.initContainers) run before the main ones, strictly one at a time and in the order they are declared; each must exit with code 0, otherwise the kubelet restarts it according to the pod's restartPolicy, and the main containers are never reached. Typical tasks: prepare a configuration or a directory on a volume, wait for a dependency (a database, a DNS name), run a schema migration, download data. Their convenience lies in having a separate image: tools needed only for preparation (a shell, curl, a database client) do not end up in the application image.

While init containers are running, the pod status shows Init:0/2, Init:1/2; a failed init container shows Init:Error or Init:CrashLoopBackOff.

Sidecar containers

For a long time a "sidecar" was just a convention: a second container in containers. Such a convention has three problems. The start order is not guaranteed: the application may start before the proxy it uses to reach the network. Neither is the stop order: the proxy may die before the application, and the last requests go nowhere. And in a Job pod (part 9), where the main container finishes, an endless sidecar never lets the pod finish.

Native sidecar containers solve all three problems. A native sidecar is an init container with restartPolicy: Always. The feature has been stable (GA) since Kubernetes 1.33 and works without any flags in our 1.36. The behaviour:

• a sidecar starts in the order of init containers, but the kubelet does not wait for it to finish: once it has started (and, if the sidecar has a startupProbe, once that probe has succeeded), the kubelet moves on to the next init container and then to the main ones;
• a sidecar runs for the whole life of the pod and is restarted if it crashes, regardless of the pod's restartPolicy;
• its readinessProbe counts towards the pod's readiness;
• when the pod stops, the kubelet sends the stop signal to sidecar containers only after all main containers have exited, and stops them in reverse order of declaration;
• in a Job pod, the main containers finishing finishes the pod; sidecars do not get in the way.

Examples of sidecars: a proxy accepting traffic — this is how service meshes such as Istio and Linkerd work, putting a proxy into every pod and using it to encrypt and account for traffic between services; an agent shipping logs from a file on a shared volume; a process renewing a certificate or a token.

---

A pod with hello and a sidecar

Let's build a pod from three parts: an init container prepares the nginx configuration on a shared emptyDir volume (a temporary directory on the node's disk that lives as long as the pod; details are in part 21), an nginx sidecar accepts requests on port 8081 and proxies them to the application at 127.0.0.1:8080, and the main container is hello.

First we delete the test pod from the previous part: a pod with the same name cannot be created, and changing it into what we need is even less possible.

$ kubectl delete pod hello
pod "hello" deleted from default namespace

hello-pod.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: hello
  labels:
    app.kubernetes.io/name: hello
  annotations:
    kubectl.kubernetes.io/default-container: hello
spec:
  terminationGracePeriodSeconds: 30
  initContainers:
  - name: render-config
    image: busybox:1.37
    command:
    - sh
    - -c
    - |
      cat > /etc/nginx/conf.d/default.conf <<'EOF'
      server {
        listen 8081;
        location = /nginx-health { return 200 "ok\n"; }
        location / {
          proxy_pass http://127.0.0.1:8080;
          proxy_set_header Host $host;
          proxy_set_header X-Forwarded-For $remote_addr;
        }
      }
      EOF
    volumeMounts:
    - name: nginx-conf
      mountPath: /etc/nginx/conf.d
  - name: proxy
    image: nginx:1.29
    restartPolicy: Always          # this is what makes the init container a sidecar
    ports:
    - name: proxy
      containerPort: 8081
    volumeMounts:
    - name: nginx-conf
      mountPath: /etc/nginx/conf.d
      readOnly: true
    startupProbe:
      httpGet: {path: /nginx-health, port: proxy}
      periodSeconds: 1
      failureThreshold: 10
    readinessProbe:
      httpGet: {path: /nginx-health, port: proxy}
      periodSeconds: 10
  containers:
  - name: hello
    image: ghcr.io/example/hello:1.0
    ports:
    - name: http
      containerPort: 8080
    startupProbe:
      httpGet: {path: /healthz, port: http}
      periodSeconds: 2
      failureThreshold: 15         # up to 30 seconds to start
    livenessProbe:
      httpGet: {path: /healthz, port: http}
      periodSeconds: 10
      timeoutSeconds: 2
      failureThreshold: 3
    readinessProbe:
      httpGet: {path: /readyz, port: http}
      periodSeconds: 5
      timeoutSeconds: 2
      failureThreshold: 2
    lifecycle:
      preStop:
        sleep:
          seconds: 5
  volumes:
  - name: nginx-conf
    emptyDir: {}

Notes on what is not obvious:

• the kubectl.kubernetes.io/default-container annotation tells kubectl logs and kubectl exec which container to use when -c is not given; without it, in a multi-container pod kubectl takes the first one from containers and warns: Defaulted container "hello" out of: hello, render-config (init), proxy (init);
• imagePullSecrets is not specified: in the previous part we attached the registry secret to the default ServiceAccount, and the admission plugin fills it in at creation;
• the nginx-conf volume is mounted over /etc/nginx/conf.d of the nginx image: the stock default.conf with a listener on port 80 is hidden, and only ours is active;
• $host and $remote_addr are nginx variables, not shell ones: the quoted <<'EOF' stops the shell from substituting variables inside.

$ kubectl apply -f hello-pod.yaml && kubectl get pod hello -w
pod/hello created
NAME    READY   STATUS    RESTARTS   AGE
hello   0/2     Pending   0          0s
hello   0/2     Init:0/2   0          1s
hello   0/2     Init:0/2   0          2s
hello   0/2     Init:0/2   0          3s
hello   0/2     Init:0/2   0          4s
hello   0/2     Init:1/2   0          5s
hello   0/2     Init:1/2   0          5s
hello   0/2     PodInitializing   0          6s
hello   0/2     PodInitializing   0          6s
hello   1/2     PodInitializing   0          7s
hello   1/2     Running           0          7s
hello   1/2     Running           0          9s
hello   2/2     Running           0          9s

kubectl get -w prints a line for every change of the object and does not realign the columns, so they drift, and neighbouring lines can be identical: fields that are not in the table changed. Init:1/2 means render-config has finished and the kubelet is waiting for the sidecar's startupProbe; 1/2 PodInitializing means the sidecar is ready and the main container is starting. READY 2/2 means both the sidecar and the main container are ready: both are counted. The init container render-config has done its job and exited, so it is not counted.

$ kubectl port-forward pod/hello 8081:8081 &
Forwarding from 127.0.0.1:8081 -> 8081
Forwarding from [::1]:8081 -> 8081
$ curl -s http://127.0.0.1:8081/
Handling connection for 8081
hello from hello
$ kubectl logs hello --tail=2
2026/10/25 23:14:44 hello 1.0 listening on [::]:8080
2026/10/25 23:15:13 127.0.0.1:43052 GET /

The request went laptop → API server → kubelet → port 8081 of the pod (nginx) → 127.0.0.1:8080 (hello). In the hello log the client address is 127.0.0.1: nginx reached it over the shared lo. The response is hello from hello: the container's hostname equals the pod name.

---

Probes: startup, liveness, readiness

Probes are executed by the kubelet on the pod's node. There are four mechanisms: httpGet (a response code of 200–399 is success), tcpSocket (a TCP connection could be opened), grpc (the standard health checking protocol of gRPC, a remote procedure call framework on top of HTTP/2) and exec (a command inside the container exited with code 0). HTTP and TCP probes are made by the kubelet itself, from the node's network namespace to the pod's IP — in the nginx log such requests come from the node's address with User-Agent: kube-probe/1.36; exec starts a process inside the container through the runtime — this is more expensive and requires the image to have the command (distroless does not).

There are three probes, and they answer different questions:

Probe	Question	What the kubelet does on failure
startupProbe	has the application started yet?	until it succeeds, liveness and readiness are not run; failureThreshold failures in a row kill the container, then restartPolicy applies
livenessProbe	is the process alive and not stuck?	kills and restarts the container
readinessProbe	is it ready to accept requests right now?	marks the container and the pod not ready; the pod's address is removed from the services' EndpointSlices; the container is left alone

EndpointSlice is an API object with the list of addresses of the pods behind a service; it is maintained by a controller in the controller-manager, and kube-proxy writes load-balancing rules from it. Details are in part 15. For this part it is enough to know that a pod with Ready=False gets no traffic through a Service.

All three have the same parameters:

Field	Default	Meaning
initialDelaySeconds	0	pause after the container starts before the first probe
periodSeconds	10	interval between probes
timeoutSeconds	1	how long to wait for a response
failureThreshold	3	how many failures in a row count as a failure
successThreshold	1	how many successes in a row count as recovery (only 1 for liveness and startup)

The time a container gets to start is failureThreshold × periodSeconds of the startupProbe: for hello that is 15 × 2 = 30 seconds, and probing begins immediately, so a fast start does not wait for nothing. This is better than initialDelaySeconds on liveness: that delay is the same for every start and is always chosen with a margin for the worst case, while the startupProbe releases the application as soon as it is ready.

Common mistakes

• Liveness checks dependencies. A /healthz that queries the database: the database slows down, liveness fails on all replicas at once, the kubelet restarts them, the restart does not cure the database, and the load on it from reconnections grows. Liveness should answer only the question "is this process alive" — for hello it returns ok as long as the HTTP server responds at all. Dependencies are a matter for readiness, and even then with care: if the dependency is shared by all replicas, readiness removes all of them from load balancing, and the service becomes entirely unavailable instead of failing some of the requests.
• The same check for liveness and readiness. Then an overloaded pod, which should rightly be taken out of load balancing for a while, also gets restarted.
• Timeouts that are too tight. The default timeoutSeconds: 1 is too little for an application with a garbage collector that has pauses. Under load the probes start failing, a restart adds load to the others, and they start failing too.
• No startupProbe on a slowly starting application. Liveness starts probing while the application is still coming up and kills it before it manages to start. An endless restart loop out of nowhere.
• No readinessProbe. The pod gets into load balancing the moment the container starts, before the application has opened its port, and the first requests get connection refused.
• An exec probe with a heavy command (psql -c 'select 1' every 5 seconds in a hundred pods) — a noticeable load on both the node and the database.

---

restartPolicy, phases and states

The pod's spec.restartPolicy determines what the kubelet does with main containers that have exited:

Value	Behaviour	Where it is used
Always (default)	restart on any exit, including code 0	services, Deployment; the only value allowed there
OnFailure	restart only on a non-zero code	Job
Never	do not restart	Job, one-off pods

A restart is a new container in the same pod, on the same node, with the same IP; the RESTARTS counter grows. Since 1.35 the container field restartPolicyRules (beta, enabled by default) allows finer control: for example, restarting only on certain exit codes and treating the others as final.

The pod phase (status.phase) is a coarse overall state:

Phase	Meaning
Pending	the pod has been accepted, but not all containers have been created yet: it is waiting for scheduling, an image download, init containers
Running	the pod is on a node, all containers have been created, at least one is running or restarting
Succeeded	all containers have exited successfully and will not be restarted
Failed	all containers have exited, at least one with an error
Unknown	the state is unknown, usually because contact with the node is lost

The Running phase does not mean "working fine": a pod in CrashLoopBackOff is Running too. The details are in the container states (status.containerStatuses[].state): Waiting with a reason (ContainerCreating, ErrImagePull, ImagePullBackOff, CrashLoopBackOff, CreateContainerConfigError), Running with the start time, Terminated with the exit code and a reason (Completed, Error, OOMKilled). The lastState field holds how the previous attempt ended — the first place to look when there are restarts. The STATUS column in kubectl get pods is not the phase but a summary that kubectl assembles from the container states.

And the conditions (status.conditions): PodScheduled — the pod is assigned to a node; PodReadyToStartContainers — the sandbox has been created and the network configured (beta in 1.36, enabled by default); Initialized — the init containers have finished; ContainersReady — all containers are ready; Ready — the pod is ready to accept traffic. Ready can also depend on external readiness gates — extra conditions set by a third-party controller (for example, a cloud load balancer once it has registered the pod).

CrashLoopBackOff

If a container crashes over and over, the kubelet does not restart it instantly but waits, doubling the pause: 10 s, 20 s, 40 s, 80 s, 160 s and then 300 s (five minutes) each time — that is the cap. While the kubelet waits, the container's state is Waiting with the reason CrashLoopBackOff. If after a restart the container runs for 10 minutes without crashing, the delay is reset.

CrashLoopBackOff is not an error in itself but a message: "the container keeps crashing, I'm waiting before the next attempt". The cause has to be found in the logs of the previous run and in lastState. Since 1.35 the 300-second cap can be configured per node with the crashLoopBackOff.maxContainerRestartPeriod field in the kubelet configuration (beta, from 1 to 300 seconds) — for example, for nodes where a quick recovery after brief dependency outages matters.

---

The termination lifecycle

Deleting a pod is not an instant event but a process, and most of the requests lost during rollouts are lost right here.

 t=0   kubectl delete pod hello
       API: deletionTimestamp, deletionGracePeriodSeconds=30; pod is Terminating
         │
         ├──► EndpointSlice controller: pod address → ready=false, terminating=true
         │      └──► kube-proxy on every node removes the pod from load balancing
         │           (in parallel, within a fraction of a second or a few seconds)
         │
         └──► kubelet k8s-node-2:
                1. preStop of the main container hello (sleep 5 s)
                2. SIGTERM to PID 1 of the hello container
                   hello: ready=false, srv.Shutdown(), wait for requests
                3. hello has exited ──► stop signal to the sidecar proxy (nginx)
                4. everything has exited ──► remove the sandbox, release the IP
 t=30  if anything is still alive: SIGKILL (preStop gets a one-off +2 s)
       kubelet reports to the API: the pod is finally deleted

The key subtlety is that the two branches run in parallel. The kubelet starts stopping the container, while kube-proxy on other nodes keeps sending new connections to the pod for a while: EndpointSlice changes do not spread across the cluster instantly. If the application closes its port right away on SIGTERM, those connections are refused.

The cure is a pause before SIGTERM: a preStop hook that simply waits a few seconds until the pod disappears from load balancing. Hooks come in three kinds: exec (a command in the container), httpGet (a request to the container) and sleep (a pause done by the kubelet, stable since 1.34). People used to write exec: {command: ["sleep", "5"]}, but the distroless hello image has no sleep command, and such a hook would simply fail; the sleep action is carried out by the kubelet itself and requires nothing from the image.

The rules that follow from this:

• the application handles SIGTERM: it finishes the requests in progress and exits. This is especially important when the application process is PID 1 in the container: the kernel does not apply the default signal action to PID 1, and a process without a SIGTERM handler simply will not notice it — it will be killed with SIGKILL after 30 seconds;
• terminationGracePeriodSeconds (30 by default) is the shared budget for preStop and the graceful shutdown of all the pod's containers, counted from the start of deletion. A preStop sleep 5 and 20 seconds for srv.Shutdown in hello fit into 30 with room to spare. Sidecars get whatever is left of the budget after the main containers; if the budget runs out earlier, the kubelet stops all remaining containers at once with a short grace period. If the application needs more, increase it rather than hope;
• the stop signal can be changed with the STOPSIGNAL instruction in the image; the official nginx image does exactly that — STOPSIGNAL SIGQUIT, on which nginx finishes the current requests, whereas SIGTERM means a fast shutdown for it. The lifecycle.stopSignal field in the container spec is still alpha;
• kubectl delete pod --grace-period=0 --force removes the object from the API without waiting for the node. The containers on the node may still be running at that point. For StatefulSet pods (part 8) this is a direct path to two instances with the same name — use it only when the node is definitely dead.

The postStart hook is the counterpart of preStop: it runs right after the container is created, in parallel with its start and with no guarantee that it fires before the entrypoint. It is rarely useful.

---

kubectl logs, exec, port-forward

All three commands work through the API server: kubectl calls the pod's subresources (pods/log, pods/exec, pods/portforward), the API server proxies the request to the kubelet of the right node on port 10250, and the kubelet to the CRI streaming server in containerd. The laptop needs no direct network access to the node or the pod, and permissions on these subresources are granted through RBAC separately from the permission to read pods (part 29). The streams run over WebSocket; in recent versions this is the default, while earlier SPDY was used — an experimental predecessor of HTTP/2 long abandoned by browsers.

kubectl logs reads the files into which containerd writes the containers' stdout and stderr: /var/log/pods/<namespace>_<pod>_<uid>/<container>/<restart number>.log. Useful flags: -c — the container, -f — follow, --previous — the log of the previous, crashed instance (indispensable with CrashLoopBackOff), --since=10m, --timestamps, --all-containers. The kubelet keeps the log of the current and one previous instance of a container and rotates the files itself (containerLogMaxSize, 10 MiB by default, and containerLogMaxFiles, 5). This is not a log storage system: the pod is deleted — the logs are deleted; storage needs a collector (part 33).

kubectl exec starts a new process in the namespaces and cgroup of a running container: kubectl exec hello -c proxy -- nginx -T. You cannot get into the hello container this way — distroless has no shell:

$ kubectl exec -it hello -- sh
error: Internal error occurred: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "d1ce3a58…": OCI runtime exec failed: exec failed: unable to start container process: exec: "sh": executable file not found in $PATH

For debugging such containers there is kubectl debug with an ephemeral container that joins the pod's namespaces and brings its own tools — part 31.

kubectl port-forward forwards a local port into the pod: containerd on the node enters the pod's network namespace and connects to localhost:<port>. That is why port-forward reaches even a port that listens only on the pod's localhost, and goes through neither a Service nor network policies. It is a debugging tool, not a way for users to get access.

---

Hands-on

Pod status and events (the output is shortened: without container IDs, the init container's command, mounts and volumes):

$ kubectl describe pod hello
Name:             hello
Namespace:        default
Priority:         0
Service Account:  default
Node:             k8s-node-2/192.168.10.21
Labels:           app.kubernetes.io/name=hello
Status:           Running
IP:               172.20.172.5
Init Containers:
  render-config:
    Image:          busybox:1.37
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
    Ready:          True
    Restart Count:  0
  proxy:
    Image:          nginx:1.29
    Port:           8081/TCP (proxy)
    State:          Running
    Ready:          True
    Restart Count:  0
    Readiness:      http-get http://:proxy/nginx-health delay=0s timeout=1s period=10s #success=1 #failure=3
    Startup:        http-get http://:proxy/nginx-health delay=0s timeout=1s period=1s #success=1 #failure=10
Containers:
  hello:
    Image:          ghcr.io/example/hello:1.0
    Image ID:       ghcr.io/example/hello@sha256:a5066cd40ef37b9b889ddfbe9c1860b3afe231b96b975000e0edca3fdd6f4a4a
    Port:           8080/TCP (http)
    State:          Running
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:http/healthz delay=0s timeout=2s period=10s #success=1 #failure=3
    Readiness:      http-get http://:http/readyz delay=0s timeout=2s period=5s #success=1 #failure=2
    Startup:        http-get http://:http/healthz delay=0s timeout=1s period=2s #success=1 #failure=15
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       True
  ContainersReady             True
  PodScheduled                True
QoS Class:                   BestEffort
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  52s   default-scheduler  Successfully assigned default/hello to k8s-node-2
  Normal  Pulled     50s   kubelet            spec.initContainers{render-config}: Container image "busybox:1.37" already present on machine and can be accessed by the pod
  Normal  Created    49s   kubelet            spec.initContainers{render-config}: Container created
  Normal  Started    49s   kubelet            spec.initContainers{render-config}: Container started
  Normal  Pulled     48s   kubelet            spec.initContainers{proxy}: Container image "nginx:1.29" already present on machine and can be accessed by the pod
  Normal  Created    47s   kubelet            spec.initContainers{proxy}: Container created
  Normal  Started    47s   kubelet            spec.initContainers{proxy}: Container started
  Normal  Pulled     46s   kubelet            spec.containers{hello}: Container image "ghcr.io/example/hello:1.0" already present on machine and can be accessed by the pod
  Normal  Created    45s   kubelet            spec.containers{hello}: Container created
  Normal  Started    45s   kubelet            spec.containers{hello}: Container started

The order of events is exactly the one in the startup diagram: init, then the sidecar, then the main container; the kubelet prefixes each event with the container it refers to (spec.initContainers{proxy}). The words "and can be accessed by the pod" in the Pulled event are the check from the previous part: the kubelet has made sure the pod is entitled to the image already on the node. QoS Class: BestEffort means the containers have no resources set; how to fix that and what it risks is covered in part 10.

The sandbox and the containers on the node. crictl talks to containerd over the same CRI as the kubelet:

user@k8s-node-2:~$ sudo crictl pods --name hello
POD ID              CREATED             STATE               NAME                NAMESPACE           ATTEMPT             RUNTIME
51ed2323f4fa5       58 seconds ago      Ready               hello               default             0                   (default)

user@k8s-node-2:~$ sudo crictl ps -a --pod 51ed2323f4fa5
CONTAINER           IMAGE               CREATED             STATE               NAME                ATTEMPT             POD ID              POD                 NAMESPACE
79b420d295565       f768da2949737       53 seconds ago      Running             hello               0                   51ed2323f4fa5       hello               default
8fe828832e979       5dfe511714e1f       55 seconds ago      Running             proxy               0                   51ed2323f4fa5       hello               default
17203c058fb4c       30ecbe1509090       57 seconds ago      Exited              render-config       0                   51ed2323f4fa5       hello               default

crictl ps does not show the pause container itself: to CRI it is a sandbox, not a container. It is visible in crictl inspectp — together with the path to the network namespace that all containers of the pod share:

user@k8s-node-2:~$ sudo crictl inspectp 51ed2323f4fa5 | jq -r '.info.runtimeSpec.linux.namespaces[] | "\(.type) \(.path // "-")"'
pid -
ipc -
uts -
mount -
network /var/run/netns/cni-6436b1e4-179c-4a6a-df3b-f86c74a11268

user@k8s-node-2:~$ for c in $(sudo crictl ps --pod 51ed2323f4fa5 -q); do
>   pid=$(sudo crictl inspect $c | jq .info.pid)
>   echo "${c:0:13} net=$(sudo readlink /proc/$pid/ns/net) mnt=$(sudo readlink /proc/$pid/ns/mnt) pid=$(sudo readlink /proc/$pid/ns/pid)"
> done
79b420d295565 net=net:[4026532287] mnt=mnt:[4026532358] pid=pid:[4026532359]
8fe828832e979 net=net:[4026532287] mnt=mnt:[4026532355] pid=pid:[4026532356]

The network namespace of hello and proxy is one and the same, 4026532287; the file system and the process space are their own. Let's look into this shared network from the node; nsenter enters the namespace of a process:

user@k8s-node-2:~$ pid=$(sudo crictl inspect 79b420d295565 | jq .info.pid)
user@k8s-node-2:~$ sudo nsenter -t $pid -n ss -ltn
State  Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0      511          0.0.0.0:8081      0.0.0.0:*
LISTEN 0      4096               *:8080            *:*
user@k8s-node-2:~$ sudo nsenter -t $pid -n ip -br addr
lo               UNKNOWN        127.0.0.1/8 ::1/128
eth0@if10        UP             172.20.172.5/32 fe80::18c2:82ff:fed2:9b0f/64

Both ports — nginx and hello — are in the same network namespace. The /32 pod address and the eth0 interface paired with if10 on the node are Calico's work, covered in part 14.

The pod directory of the kubelet (tree comes from the tree package):

$ kubectl get pod hello -o jsonpath='{.metadata.uid}{"\n"}'
84f07d9e-e9e9-4e13-912f-c062a13a46e0

user@k8s-node-2:~$ uid=84f07d9e-e9e9-4e13-912f-c062a13a46e0
user@k8s-node-2:~$ sudo tree -L 3 /var/lib/kubelet/pods/$uid
/var/lib/kubelet/pods/84f07d9e-e9e9-4e13-912f-c062a13a46e0
├── containers
│   ├── hello
│   │   └── 86f8f74a
│   ├── proxy
│   │   └── 1526bf9a
│   └── render-config
│       └── 6765c03e
├── etc-hosts
├── plugins
│   └── kubernetes.io~empty-dir
│       ├── nginx-conf
│       └── wrapped_kube-api-access-4rv9d
└── volumes
    ├── kubernetes.io~empty-dir
    │   └── nginx-conf
    └── kubernetes.io~projected
        └── kube-api-access-4rv9d

14 directories, 4 files

• volumes/kubernetes.io~empty-dir/nginx-conf/ is that very emptyDir: an ordinary directory on the node's disk, bind-mounted into the containers. Inside is default.conf, written by the init container;
• volumes/kubernetes.io~projected/kube-api-access-… is the volume with the ServiceAccount token, the cluster CA certificate and the namespace name, which Kubernetes mounts into every pod by default at /var/run/secrets/kubernetes.io/serviceaccount (part 29); plugins/…/wrapped_kube-api-access-… is the kubelet's bookkeeping entry for the emptyDir this volume is built on;
• etc-hosts is the pod's /etc/hosts file, written by the kubelet: the pod name with its IP;
• containers/<name>/<id> are the terminationMessagePath files: what a container wrote to /dev/termination-log before dying, the kubelet shows in lastState.terminated.message.

user@k8s-node-2:~$ sudo cat /var/lib/kubelet/pods/$uid/etc-hosts
# Kubernetes-managed hosts file.
127.0.0.1	localhost
::1	localhost ip6-localhost ip6-loopback
fe00::0	ip6-localnet
fe00::0	ip6-mcastprefix
fe00::1	ip6-allnodes
fe00::2	ip6-allrouters
172.20.172.5	hello

The pod cgroup is a slice with the pod UID, containing a scope for each container and for the sandbox (container IDs shortened):

user@k8s-node-2:~$ systemd-cgls --no-pager -u kubepods-besteffort-pod84f07d9e_e9e9_4e13_912f_c062a13a46e0.slice
Unit kubepods-besteffort-pod84f07d9e_e9e9_4e13_912f_c062a13a46e0.slice (/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84f07d9e_e9e9_4e13_912f_c062a13a46e0.slice):
├─cri-containerd-79b420d29556….scope
│ └─204908 /hello
├─cri-containerd-8fe828832e97….scope
│ ├─204864 nginx: master process nginx -g daemon off;
│ ├─204892 nginx: worker process
│ └─204893 nginx: worker process
└─cri-containerd-51ed2323f4fa….scope
  └─204782 /pause

Logs on disk:

user@k8s-node-2:~$ sudo ls /var/log/pods/default_hello_$uid/
hello  proxy  render-config
user@k8s-node-2:~$ sudo tail -1 /var/log/pods/default_hello_$uid/hello/0.log
2026-10-26T02:15:13.083108848+03:00 stderr F 2026/10/25 23:15:13 127.0.0.1:43052 GET /

The line format is the time, the stream (stdout or stderr), a flag for a full (F) or split (P) line, and the text itself. This is the CRI format, and every log collector understands it. containerd writes the time in the node's time zone (+03:00 here), while hello itself writes UTC: the distroless image has no time zone set, and Go treats it as UTC.

CrashLoopBackOff with your own eyes. A pod that was given a wrong port:

apiVersion: v1
kind: Pod
metadata:
  name: crash
spec:
  containers:
  - name: hello
    image: ghcr.io/example/hello:1.0
    env:
    - name: PORT
      value: "abc"

$ kubectl apply -f crash.yaml && kubectl get pod crash -w
pod/crash created
NAME    READY   STATUS    RESTARTS   AGE
crash   0/1     Pending   0          0s
crash   0/1     ContainerCreating   0          0s
crash   0/1     ContainerCreating   0          1s
crash   0/1     ContainerCreating   0          2s
crash   1/1     Running             0          3s
crash   0/1     Error               0          4s
crash   1/1     Running             1 (2s ago)   5s
crash   0/1     Error               1 (3s ago)   6s
crash   0/1     CrashLoopBackOff    1 (13s ago)   18s
crash   1/1     Running             2 (15s ago)   20s
crash   0/1     Error               2 (15s ago)   20s
crash   0/1     CrashLoopBackOff    2 (22s ago)   41s
crash   1/1     Running             3 (24s ago)   43s
crash   0/1     Error               3 (25s ago)   44s

Each attempt briefly shows 1/1 Running: the container starts and crashes right away. The first restart (at second 5) comes almost immediately, the next ones at seconds 20 and 43: pauses of 10 and 20 seconds plus a few seconds until the kubelet gets back to the pod. CrashLoopBackOff appears in the column not right after the crash but when the kubelet updates the pod status. The kubelet writes the current pause into state.waiting.message: watch longer, and after the fifth restart it will say back-off 2m40s restarting failed container=hello pod=crash_default(…) — the very 160 seconds. The cause:

$ kubectl logs crash --previous
2026/10/25 23:19:43 listen: listen tcp: lookup tcp/abc: unknown port
$ kubectl get pod crash -o jsonpath='{.status.containerStatuses[0].lastState}' | jq
{
  "terminated": {
    "containerID": "containerd://46c00258…",
    "exitCode": 1,
    "finishedAt": "2026-10-25T23:19:43Z",
    "reason": "Error",
    "startedAt": "2026-10-25T23:19:43Z"
  }
}
$ kubectl delete pod crash
pod "crash" deleted from default namespace

Shutdown step by step. Let's delete the hello pod while following the logs of both containers with timestamps; in a neighbouring terminal curl requests go through port-forward, two per second:

$ for i in $(seq 1 40); do curl -s -m 1 -o /dev/null -w '%{http_code} ' http://127.0.0.1:8081/; sleep 0.5; done

$ kubectl logs hello -f --timestamps > hello.log &
$ kubectl logs hello -c proxy -f --timestamps > proxy.log &
$ date +%T.%N; kubectl delete pod hello; date +%T.%N
02:23:04.604314715
pod "hello" deleted from default namespace
02:23:12.618510482
$ tail -4 hello.log
2026-10-26T02:23:08.732183328+03:00 2026/10/25 23:23:08 127.0.0.1:35846 GET /
2026-10-26T02:23:09.241063991+03:00 2026/10/25 23:23:09 127.0.0.1:35862 GET /
2026-10-26T02:23:09.733872142+03:00 2026/10/25 23:23:09 signal received, shutting down
2026-10-26T02:23:09.735109251+03:00 2026/10/25 23:23:09 bye
$ grep -E 'SIGQUIT|1#1: exit' proxy.log
2026-10-26T02:23:10.192158970+03:00 2026/10/25 23:23:10 [notice] 1#1: signal 3 (SIGQUIT) received, shutting down
2026-10-26T02:23:10.201916590+03:00 2026/10/25 23:23:10 [notice] 1#1: exit

And the curl loop in the neighbouring terminal printed:

200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 502 000 000 000 000 000 000 000

delete was sent at 02:23:04.6, SIGTERM arrived at 02:23:09.7 — five seconds of preStop later. All requests in those five seconds were served: the pod is already Terminating, but the application keeps working. The shutdown itself took about a millisecond, because there were no open requests. Only after hello exited did the kubelet send the stop signal to the sidecar: nginx received SIGQUIT (that very STOPSIGNAL from the image) half a second later. In that half second one request got a 502 — nginx is still alive, but the application behind it is gone; then 000: nginx has stopped, and port-forward lost its connection. Through a Service such a request would not have arrived here — by that time the pod's address has already been removed from the EndpointSlice — but port-forward goes straight into the pod. kubectl delete waits for the object to be finally deleted by default and returned after eight seconds. At the default verbosity the kubelet journal has no lines about stopping containers: the order is visible in the logs of the containers themselves.

---

A pod can start in the right order, report its readiness and stop gracefully. But it lives only until the first problem: deleted — gone; the node died — gone with it, and nobody will create a new one. That is what controllers are for.

Sources

• Kubernetes: pods, pod lifecycle (phases, conditions, restarts and CrashLoopBackOff, termination), init containers, sidecar containers, sharing the process namespace.
• Probes: liveness, readiness and startup and configuring them; container lifecycle hooks (postStart, preStop, the sleep action).
• Logging in Kubernetes (files on the node and rotation), port-forward, crictl.
• Kubelet configuration (crashLoopBackOff, containerLogMaxSize, containerLogMaxFiles) and feature gates — which version made what stable.
• The pause container source and containerd CRI configuration (pinned_images).

---

Next — Deployment and ReplicaSet: how controllers keep the required number of pods, what pod-template-hash is, how a rolling update with maxSurge and maxUnavailable proceeds, how the readiness probe drives a rollout, and how to roll back to the previous version.