3 November 2026, 09:49Виталий Лещенко

Kubernetes for beginners. Part 31: How to troubleshoot a cluster

All parts of the series “Kubernetes for beginners”

Over thirty parts the cluster has grown a lot of components: pods and controllers from part 6 and part 7, the scheduler from part 13, networking from Calico, kube-proxy and CoreDNS (parts 14, 15, 16), volumes on NFS and SMB, security policies from the previous part. Each of them has its own way of breaking, and each reports it in its own place. This part opens the observability block and is about manual troubleshooting: in what order to look, with which commands, where events, logs and container state live, how to get into a pod without a shell and onto a node without SSH, and what the statuses you see most often in kubectl get pods mean. Metrics, log collection and traces come in the next two parts; here everything is done with plain kubectl and, as a last resort, crictl on the node.

The demo namespace stays as the previous part left it: under the restricted level of Pod Security Admission, with hello running as UID 10001 without capabilities. That affects debugging, and we will see how.

Top down

Kubernetes is a chain of controllers, where each next one reacts to the object created by the previous one (part 3). When something does not work, one link is broken, while the symptom shows up at the end of the chain: "the site does not open". So the cause is searched for in reverse order — from the object a person works with down to the node where the process lives:

 level          what to look at                              with what
 ───────────    ─────────────────────────────────────────    ─────────────────────────────────────
 object         Deployment, Service, Ingress, PVC:           kubectl get / describe
                status, conditions, how many are ready
    │
    ▼
 events         what controllers, the scheduler and          kubectl events --for, describe
                kubelet said about the object in the
                last hour
    │
    ▼
 pod            phase, conditions, node, IP,                 kubectl get pod -o wide / -o yaml
                state of every container
    │
    ▼
 container      logs of the current and previous run,        kubectl logs, exec, debug
                exit code, what is inside
    │
    ▼
 node           kubelet, containerd, CNI, disk, memory,      kubectl describe node, debug node,
                certificates                                 crictl, journalctl

At every level there is one question: did things get as far as the next link. Did the Deployment create a ReplicaSet? Did the ReplicaSet create pods? Are the pods assigned to a node? Did kubelet start the container? Did the container pass readiness? The first link that answers "no" is where the breakage is, and its events usually name the cause in plain words. Jumping to application logs before it is clear whether the application is running at all is a common waste of time.

Events

An Event is an ordinary API object in the events.k8s.io/v1 group (and in the old core v1, which kubectl describe still reads): a short message attached to another object. They are written by the components that do something with objects: the scheduler (Scheduled, FailedScheduling), kubelet (Pulling, Pulled, Created, Started, Killing, BackOff, Unhealthy, FailedMount), controllers (ScalingReplicaSet, SuccessfulCreate), CSI drivers through their sidecar containers (ProvisioningSucceeded), cert-manager and any operators.

$ kubectl get events -n demo --field-selector reason=BackOff -o yaml
apiVersion: v1
items:
- apiVersion: v1
  count: 8
  eventTime: null
  firstTimestamp: "2026-11-02T13:57:25Z"
  involvedObject:
    apiVersion: v1
    fieldPath: spec.containers{hello}
    kind: Pod
    name: hello-798bd968d9-vkz9z
    namespace: demo
    resourceVersion: "1699629"
    uid: cab28704-1a6a-40d9-bd16-172760839f9d
  kind: Event
  lastTimestamp: "2026-11-02T13:58:06Z"
  message: Back-off restarting failed container hello in pod hello-798bd968d9-vkz9z_demo(cab28704-1a6a-40d9-bd16-172760839f9d)
  metadata:
    creationTimestamp: "2026-11-02T13:57:25Z"
    name: hello-798bd968d9-vkz9z.18e43f05b2daee83
    namespace: demo
    resourceVersion: "1699788"
    uid: e12dcf66-8527-4226-9af6-5775de85c8f8
  reason: BackOff
  reportingComponent: kubelet
  reportingInstance: k8s-node-2
  source:
    component: kubelet
    host: k8s-node-2
  type: Warning
kind: List
metadata:
  resourceVersion: ""

What matters in how events work:

Type is one of two: Normal and Warning. The --types=Warning filter cuts the noise of "image pulled, container started".
Aggregation. A repeating event is not created again; it increments count (in the new API, series.count) and moves lastTimestamp. "8 times in 41 seconds" is one event.
Lifetime. Events are stored in etcd as separate objects and are removed after kube-apiserver's --event-ttl, 1 hour by default. An hour after an incident not a single event about it is left in the cluster. That is why they are shipped to a log system — how, we will see in part 33 — and in manual troubleshooting you look at them right away.
Events are not guaranteed. Components write them "best effort" and rate-limited; the absence of an event does not prove nothing happened.

The handiest way to view them is kubectl events: unlike kubectl get events, it sorts by time by default and can filter by object:

$ kubectl events -n demo --for pod/hello-798bd968d9-vkz9z
LAST SEEN          TYPE      REASON      OBJECT                       MESSAGE
55s                Normal    Scheduled   Pod/hello-798bd968d9-vkz9z   Successfully assigned demo/hello-798bd968d9-vkz9z to k8s-node-2
9s (x4 over 51s)   Normal    Pulled      Pod/hello-798bd968d9-vkz9z   Container image "ghcr.io/example/hello:1.1" already present on machine and can be accessed by the pod
8s (x4 over 49s)   Normal    Created     Pod/hello-798bd968d9-vkz9z   Container created
8s (x4 over 49s)   Normal    Started     Pod/hello-798bd968d9-vkz9z   Container started
8s (x6 over 49s)   Warning   Unhealthy   Pod/hello-798bd968d9-vkz9z   Readiness probe failed: Get "http://172.20.172.6:8080/readyz": dial tcp 172.20.172.6:8080: connect: connection refused
4s (x8 over 45s)   Warning   BackOff     Pod/hello-798bd968d9-vkz9z   Back-off restarting failed container hello in pod hello-798bd968d9-vkz9z_demo(cab28704-1a6a-40d9-bd16-172760839f9d)

$ kubectl events -A --types=Warning --watch

--for takes kind/name, -A looks across all namespaces, --watch keeps the stream open. Node events live in the default namespace and are attached to the Node object: kubectl events --for node/k8s-node-2.

Object and pod: describe, get, conditions

kubectl describe is a summary that the client assembles from several requests: the object itself in human-readable form plus the related events. For a pod that is the node, IP, containers with their state, volumes, tolerations and the events at the end. For a Deployment, the strategy, replicas and ReplicaSets; for a node, conditions, resources allocated to pods, and taints.

kubectl get with output formats answers questions more precisely:

$ kubectl get pods -n demo -o wide
NAME                     READY   STATUS             RESTARTS     AGE   IP              NODE         NOMINATED NODE   READINESS GATES
hello-798bd968d9-vkz9z   0/1     CrashLoopBackOff   3 (8s ago)   55s   172.20.172.6    k8s-node-2   <none>           <none>
hello-865b67cc76-682fc   1/1     Running            0            67s   172.20.211.15   k8s-node-1   <none>           <none>
hello-865b67cc76-6vcck   1/1     Running            0            67s   172.20.172.8    k8s-node-2   <none>           <none>
hello-865b67cc76-ntcj4   1/1     Running            0            67s   172.20.211.16   k8s-node-1   <none>           <none>

-o wide adds the node and IP: if all the failing pods are on one node, the problem is the node, not the application. The STATUS column is not an API field but a string kubectl builds from the pod phase and the waiting or termination reason of its containers. The real state is in -o yaml; sed cuts out the status branch (the lastProbeTime and observedGeneration lines are removed below):

$ kubectl get pod hello-798bd968d9-vkz9z -n demo -o yaml | sed -n '/^status:/,$p'
status:
  conditions:
  - lastTransitionTime: "2026-11-02T13:57:19Z"
    status: "True"
    type: PodReadyToStartContainers
  - lastTransitionTime: "2026-11-02T13:57:15Z"
    status: "True"
    type: Initialized
  - lastTransitionTime: "2026-11-02T13:57:15Z"
    message: 'containers with unready status: [hello]'
    reason: ContainersNotReady
    status: "False"
    type: Ready
  - lastTransitionTime: "2026-11-02T13:57:15Z"
    message: 'containers with unready status: [hello]'
    reason: ContainersNotReady
    status: "False"
    type: ContainersReady
  - lastTransitionTime: "2026-11-02T13:57:15Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://13884b1b4b6703e8cf8001a66cf0831dda91aafeb4fad1c0d5dc1133271b1ae4
    image: ghcr.io/example/hello:1.1
    imageID: ghcr.io/example/hello@sha256:2d8774f801594f2b1b115963e54ec54b1b49f95c2dece1d6b0fd1e59ab447c7b
    lastState:
      terminated:
        containerID: containerd://13884b1b4b6703e8cf8001a66cf0831dda91aafeb4fad1c0d5dc1133271b1ae4
        exitCode: 1
        finishedAt: "2026-11-02T13:58:02Z"
        reason: Error
        startedAt: "2026-11-02T13:58:02Z"
    name: hello
    ready: false
    resources: {}
    restartCount: 3
    started: false
    state:
      waiting:
        message: back-off 40s restarting failed container=hello pod=hello-798bd968d9-vkz9z_demo(cab28704-1a6a-40d9-bd16-172760839f9d)
        reason: CrashLoopBackOff
    user:
      linux:
        gid: 10001
        supplementalGroups:
        - 10001
        uid: 10001
    volumeMounts:
    - mountPath: /tmp
      name: tmp
  hostIP: 192.168.10.21
  hostIPs:
  - ip: 192.168.10.21
  phase: Running
  podIP: 172.20.172.6
  podIPs:
  - ip: 172.20.172.6
  qosClass: BestEffort
  resources: {}
  startTime: "2026-11-02T13:57:15Z"

Pod conditions are five flags in the order they become True:

Condition	Meaning	If `False`, look at
`PodScheduled`	the scheduler picked a node	the `FailedScheduling` event
`PodReadyToStartContainers`	kubelet created the pod sandbox: network namespace, address from CNI, pause container	`FailedCreatePodSandBox` events, CNI on the node
`Initialized`	all init containers finished successfully	init container logs, `-c <name>`
`ContainersReady`	all containers passed readiness	`Unhealthy` events, logs
`Ready`	the pod is ready for traffic and is in the service endpoints	the same plus readiness gates

Each container has three possible states: waiting with a reason (ContainerCreating, ErrImagePull, CrashLoopBackOff, CreateContainerConfigError), running with a start time, and terminated with exitCode, reason and signal. lastState keeps the previous run — that is where you see why the container died last time while the current one waits for a restart. The user.linux field shows which UID and groups the process really runs with, after all securityContext settings and the image's own.

For scripts and quick questions there are jsonpath and custom-columns:

$ kubectl get pods -n demo -o custom-columns='POD:.metadata.name,NODE:.spec.nodeName,REASON:.status.containerStatuses[0].lastState.terminated.reason,EXIT:.status.containerStatuses[0].lastState.terminated.exitCode'
POD                      NODE         REASON   EXIT
hello-798bd968d9-vkz9z   k8s-node-2   Error    1
hello-865b67cc76-682fc   k8s-node-1   <none>   <none>
hello-865b67cc76-6vcck   k8s-node-2   <none>   <none>
hello-865b67cc76-ntcj4   k8s-node-1   <none>   <none>

The exit code follows the shell convention: up to 128 it is the code the application returned, above 128 the process was killed by the signal number code − 128:

Code	What happened
0	the process exited on its own, successfully; for a Deployment it is still a restart: `restartPolicy: Always`
1, 2	application error: config not read, port not opened, a Go panic gives 2
126, 127	command not executable or not found: a typo in `command`, no shell in a distroless image
137	128 + 9, SIGKILL: the OOM killer, or kubelet that did not wait for the exit after SIGTERM
139	128 + 11, SIGSEGV: a memory error in the process itself
143	128 + 15, SIGTERM: the process was stopped cleanly by the signal

Logs: kubectl logs

kubectl logs reads neither from the application nor from containerd directly: the request goes to kube-apiserver, which proxies it to kubelet on the right node (port 10250, the pods/log subresource), and kubelet reads the file containerd writes the container's stdout and stderr to. How these files are laid out and rotated is covered in part 33; what matters now is that kubectl logs sees only what the process wrote to stdout and stderr, and only while the files are on the node.

$ kubectl logs -n demo hello-798bd968d9-vkz9z --previous
2026/11/02 13:58:02 listen: listen tcp: lookup tcp/8080	: unknown port

The main flags:

Flag	What it does
`--previous`, `-p`	logs of the previous run of the container. The key flag for CrashLoopBackOff: the current container has just started or is not running yet, the reason for the crash is in the previous one
`-c <container>`	a container in a multi-container pod; without the flag, the `kubectl.kubernetes.io/default-container` annotation or the first container
`--all-containers`	all containers of the pod, including init
`--since=10m`, `--since-time=...`	only recent lines
`--tail=100`	the last N lines; with `-l` the default is 10 per pod
`-f`	follow the stream
`-l app.kubernetes.io/name=hello`	by selector: logs of all pods of a Deployment at once; with `--prefix`, with the pod name at the start of each line
`--timestamps`	the time containerd recorded, not the application
`deploy/hello`	instead of a pod name: kubectl picks one pod of the Deployment itself

--previous shows exactly one previous run: kubelet keeps only the pod's last exited container and its log file on the node and removes older ones. While the container waits for a restart there is no new run, and kubectl logs without the flag shows the same failed run as --previous. If the pod was deleted or recreated by the controller on another node, its logs are gone with it — one more reason to collect them centrally.

Inside the container: exec and ephemeral containers

kubectl exec starts a new process in the namespaces of a running container — through the same kubelet and the CRI Exec call. In an image with a shell it is the usual way to look inside: kubectl exec -it <pod> -- sh. In hello even this does not work:

$ kubectl exec -n demo deploy/hello -- cat /etc/resolv.conf
error: Internal error occurred: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "d817a624cf51c53c50ac96b2a8d0818a31674ae132f8a9cc324a6a31d1900f23": OCI runtime exec failed: exec failed: unable to start container process: exec: "cat": executable file not found in $PATH

This is the normal situation with an image built properly in part 5: a distroless or scratch image has no shell, cat, curl or ps, and that is good for security. There is no need to put debugging tools back into the image — that is what ephemeral containers are for.

An ephemeral container is a container added to an already running pod through the pods/ephemeralcontainers subresource. The pod's regular containers list is immutable, but ephemeralContainers can be appended to. An ephemeral container has no ports, probes or resources, is not restarted and is not removed while the pod lives. It runs next to the pod's other containers — in the same network namespace, with the same IP and localhost. kubectl debug adds it:

$ kubectl debug -it -n demo hello-865b67cc76-682fc --image=nicolaka/netshoot:v0.15 --target=hello
Warning: Non-root user is configured for the entire target Pod, and some capabilities granted by debug profile may not work. Please consider using "--custom" with a custom profile that specifies "securityContext.runAsUser: 0".
Error from server (Forbidden): pods "hello-865b67cc76-682fc" is forbidden: violates PodSecurity "restricted:v1.36": allowPrivilegeEscalation != false (container "debugger-x2gvb" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "debugger-x2gvb" must set securityContext.capabilities.drop=["ALL"]; container "debugger-x2gvb" must not include "SYS_PTRACE" in securityContext.capabilities.add)

The refusal comes from Pod Security Admission of part 30: an ephemeral container is part of the pod spec, and demo under the restricted level checks it just like a regular one. The debug container's security settings are set by a kubectl debug profile. In kubectl 1.36 the default profile is general: it adds SYS_PTRACE so that strace and /proc/<pid>/root of another process work, and restricts nothing. The other options are baseline and restricted (compatible with the Pod Security Standards levels of the same names), netadmin (the NET_ADMIN and NET_RAW capabilities for tcpdump and ip) and sysadmin (a privileged container). In a restricted namespace only --profile=restricted gets through:

$ kubectl debug -it -n demo hello-865b67cc76-682fc --image=nicolaka/netshoot:v0.15 --target=hello --profile=restricted
Targeting container "hello". If you don't see processes from this container it may be because the container runtime doesn't support this feature.
Defaulting debug container name to debugger-lxwkm.
All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt.
If you don't see a command prompt, try pressing enter.
 hello-865b67cc76-682fc  ~  id
uid=10001 gid=10001 groups=10001
 hello-865b67cc76-682fc  ~  ps
PID   USER     TIME  COMMAND
    1 10001     0:00 /hello
   14 10001     0:00 zsh
   22 10001     0:00 ps
 hello-865b67cc76-682fc  ~  ss -ltnp
State  Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
LISTEN 0      4096               *:8080            *:*    users:(("hello",pid=1,fd=4))
 hello-865b67cc76-682fc  ~  curl -s localhost:8080/healthz
ok
 hello-865b67cc76-682fc  ~  ls /proc/1/root/
bin    dev    hello  lib    proc   run    sys    usr
boot   etc    home   lib64  root   sbin   tmp    var
 hello-865b67cc76-682fc  ~  exit
Session ended, the ephemeral container will not be restarted but may be reattached using 'kubectl attach hello-865b67cc76-682fc -c debugger-lxwkm -n demo -i -t' if it is still running

netshoot is an image with a set of network tools for debugging: curl, dig, nslookup, tcpdump, ss, ip, iperf3, nmap, mtr and others. The tag is given explicitly: the "no latest" rule from part 5 applies to debugging images too. --target=hello puts the ephemeral container into the PID namespace of the hello container: its process is visible as number 1, and its file system through /proc/1/root/. Without --target the ephemeral container sees only its own processes, unless the pod has shareProcessNamespace enabled.

Why all of this worked without SYS_PTRACE: runAsUser: 10001 is set at the pod level and applies to all its containers, ephemeral ones included — kubectl debug warned about exactly that on the first attempt. The debugger runs with the same UID as the application, and the kernel lets you look into /proc of your own user's processes without capabilities. If the application ran as root, it would take SYS_PTRACE, and with it a namespace without restricted and baseline, or debugging from the node. The line All commands and output ... will be recorded in container logs is a kubectl 1.36 reminder: the input and output of an interactive session end up in the ephemeral container's log, and a password typed in it will be seen by anyone with access to pods/log.

What kubectl debug did in the API — appended an element to the pod spec:

$ kubectl get pod hello-865b67cc76-682fc -n demo -o jsonpath='{.spec.ephemeralContainers}' | jq
[
  {
    "image": "nicolaka/netshoot:v0.15",
    "imagePullPolicy": "IfNotPresent",
    "name": "debugger-lxwkm",
    "resources": {},
    "securityContext": {
      "allowPrivilegeEscalation": false,
      "capabilities": {
        "drop": [
          "ALL"
        ]
      },
      "runAsNonRoot": true,
      "seccompProfile": {
        "type": "RuntimeDefault"
      }
    },
    "stdin": true,
    "targetContainerName": "hello",
    "terminationMessagePath": "/dev/termination-log",
    "terminationMessagePolicy": "File",
    "tty": true
  }
]

An ephemeral container cannot be removed: after you exit the shell it stays in terminated state until the pod is recreated. So debugging on production pods is done deliberately, and when it is over the pods can be restarted with kubectl rollout restart.

Permissions. All of the above are separate subresources in RBAC (part 29): pods/log (get) for logs, pods/exec (create) for exec, pods/ephemeralcontainers (patch) for debug, pods/portforward for port-forward. The right to exec into a pod is the right to everything the process inside can do, including its ServiceAccount token and mounted secrets; it is given to those you would give the secrets themselves.

A copy of the pod

An ephemeral container does not help if the main container crashes right after start: there is nothing to look into. For that kubectl debug can create a copy of the pod with changes:

$ kubectl debug -n demo hello-798bd968d9-vkz9z -it --copy-to=hello-debug \
    --container=hello --profile=restricted -- sh

Here the copy runs sh in the hello container instead of the application — if the image has one. For distroless the image is replaced:

$ kubectl debug -n demo hello-798bd968d9-vkz9z --copy-to=hello-debug \
    --set-image=hello=busybox:1.37 --container=hello --profile=restricted -it -- sh

--set-image=<container>=<image> (or *=<image> for all) changes the image, --container says in which container to replace the command; without it kubectl debug would add one more container to the copy. The profile is needed here too: general would add SYS_PTRACE to the copy's hello container itself, and PSA would reject the pod. restricted replaces the container's securityContext with its own set, so readOnlyRootFilesystem is gone in the copy. A shared PID namespace (--share-processes) is on by default in the copy, so the processes of the other containers are visible from the debug container. The copy is a separate pod without an owner: the ReplicaSet does not know about it, labels are not copied by default (--keep-labels copies them), which means it will not get into the service endpoints and will not receive user traffic. Probes are also dropped by default (--keep-liveness, --keep-readiness keep them) so that kubelet does not kill the copy. After debugging it is deleted by hand: kubectl delete pod -n demo hello-debug.

On the node: debug node, crictl, journalctl

When the problem is below the pod — the container is not created, the node is NotReady, volumes do not mount — you need access to the node itself. SSH is not always available, but kubectl debug can start a pod right on the node:

$ kubectl debug node/k8s-node-2 -it --profile=sysadmin --image=ubuntu:26.04
Creating debugging pod node-debugger-k8s-node-2-8lwdc with container debugger on node k8s-node-2.
All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt.
If you don't see a command prompt, try pressing enter.
root@k8s-node-2:/# chroot /host
# crictl ps -a --name hello --state exited | head -3
CONTAINER           IMAGE               CREATED             STATE               NAME                ATTEMPT             POD ID              POD                      NAMESPACE
40bef406c15b6       c16806bc0dd63       58 seconds ago      Exited              hello               4                   712c804be67d4       hello-798bd968d9-vkz9z   demo
# exit
root@k8s-node-2:/# exit
Session ended, resume using 'kubectl attach node-debugger-k8s-node-2-8lwdc -c debugger -n default -i -t' command

The node-debugger-... pod is created in the current namespace (here default, with no PSA labels) with nodeName set directly (the scheduler is not involved), in the node's network, PID and IPC namespaces, and the node's root file system is mounted at /host via hostPath. Without --profile=sysadmin the pod is not privileged, and chroot /host or reading parts of /proc will not work. After chroot /host you are effectively in a shell on the node: crictl, journalctl, systemctl and kubelet's files are available. Such a pod is root on the node, so the right to create pods with hostPath and privileged is given only to administrators, and Pod Security Admission in namespaces like demo will not let it through anyway. The pod is deleted by hand: after you exit it stays in Completed.

crictl is the command-line client for CRI, the gRPC interface kubelet uses to drive containerd (part 2). It sees the same things kubelet sees and works even when the API server is unavailable. The socket address comes from /etc/crictl.yaml (runtime-endpoint: unix:///run/containerd/containerd.sock):

user@k8s-node-2:~$ sudo crictl pods --namespace demo
POD ID              CREATED             STATE               NAME                     NAMESPACE           ATTEMPT             RUNTIME
712c804be67d4       2 minutes ago       Ready               hello-798bd968d9-vkz9z   demo                0                   (default)
f61cf58e184f0       2 minutes ago       Ready               hello-865b67cc76-6vcck   demo                0                   (default)

user@k8s-node-2:~$ sudo crictl ps -a --pod 712c804be67d4
CONTAINER           IMAGE               CREATED              STATE               NAME                ATTEMPT             POD ID              POD                      NAMESPACE
40bef406c15b6       c16806bc0dd63       About a minute ago   Exited              hello               4                   712c804be67d4       hello-798bd968d9-vkz9z   demo

user@k8s-node-2:~$ sudo crictl logs 40bef406c15b6
2026/11/02 13:58:45 listen: listen tcp: lookup tcp/8080	: unknown port

user@k8s-node-2:~$ sudo crictl inspect 40bef406c15b6 | jq '.status | {state, exitCode, reason, logPath}'
{
  "state": "CONTAINER_EXITED",
  "exitCode": 1,
  "reason": "Error",
  "logPath": "/var/log/pods/demo_hello-798bd968d9-vkz9z_cab28704-1a6a-40d9-bd16-172760839f9d/hello/4.log"
}

crictl inspect also shows what the API does not have: the process PID on the node for a running container (.info.pid), the cgroup path, the OCI spec runc started the container with, the mount points. crictl images lists images on the node, crictl pull checks pulling an image from the node with the same containerd, crictl stats shows usage per container.

The services' own logs are in journald: kubelet and containerd run as systemd units on kubeadm nodes.

user@k8s-node-2:~$ sudo journalctl -u kubelet --since "10 min ago" --no-pager | grep hello-798bd968d9 | tail -1
Nov 02 13:58:56 k8s-node-2 kubelet[763948]: E1102 13:58:56.206825  763948 pod_workers.go:1338] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"hello\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=hello pod=hello-798bd968d9-vkz9z_demo(cab28704-1a6a-40d9-bd16-172760839f9d)\"" pod="demo/hello-798bd968d9-vkz9z" podUID="cab28704-1a6a-40d9-bd16-172760839f9d"

user@k8s-node-2:~$ sudo journalctl -u containerd -p warning --since today --no-pager
-- No entries --

The letter at the start of a kubelet line is the level: I info, W warning, E error. The verbosity is set by logging.verbosity in /var/lib/kubelet/config.yaml (kubeadm sets 0) or the -v flag; 4 and above are for hunting a specific error, and only briefly.

Typical states

Pending

The pod is created but not assigned to a node, or assigned but has not yet reached sandbox creation. If PodScheduled=False, the cause is named by the scheduler's event — one message summarising all nodes:

$ kubectl events -n demo --for pod/big
LAST SEEN           TYPE      REASON             OBJECT    MESSAGE
60s (x2 over 88s)   Warning   FailedScheduling   Pod/big   0/3 nodes are available: 1 node(s) had untolerated taint(s), 2 Insufficient memory. no new claims to deallocate, preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.

It reads like this: the big pod requests 5 GiB of memory; of the three nodes the master dropped out because of a taint, and the two workers do not have enough unallocated memory — the scheduler counts the sum of requests of the pods already placed, not actual usage (part 10). Preemption will not help: the taint keeps the pod off the master, and the workers would lack memory even if all lower-priority pods were removed. The phrase no new claims to deallocate comes from dynamic resource allocation (DRA) scheduling; without resourceClaims requests you can ignore it. Typical phrases and what they mean:

Phrase in the event	Cause	What to do
`Insufficient cpu` / `Insufficient memory`	the sum of requests on the nodes is used up	`kubectl describe node` → "Allocated resources"; lower the requests or add a node
`had untolerated taint(s)`	the node has a taint, the pod has no toleration	`kubectl describe node \| grep Taints`; the node may have got `node.kubernetes.io/disk-pressure` by itself
`didn't match Pod's node affinity/selector`	`nodeSelector` or `nodeAffinity` finds no nodes with such labels	`kubectl get nodes --show-labels`
`didn't match pod anti-affinity rules`	more replicas than nodes with required anti-affinity	relax to `preferred…` or add nodes
`didn't match pod topology spread constraints`	`whenUnsatisfiable: DoNotSchedule` and a skew across zones or nodes	check `maxSkew` and the topology labels
`pod has unbound immediate PersistentVolumeClaims`	the PVC is `Pending`: no StorageClass, the CSI driver did not create a volume	`kubectl describe pvc`, logs of the CSI driver's controller
`node(s) had volume node affinity conflict`	a local volume is bound to another node	the volume and the pod must meet on the same node
`node(s) were unschedulable`	the node is cordoned after `drain`	`kubectl uncordon`

ContainerCreating

The pod is assigned, kubelet is creating the sandbox and mounting volumes, but the container is not running yet. More than a minute is a reason to look at the events:

Warning   FailedCreatePodSandBox   Pod/sandbox   Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b099e20b0b04a67fe6cc7f85455d66eb87b745ce177e151e6d09bc89601b9c81": plugin type="calico" failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/

This is CNI: containerd called the Calico plugin, and it did not find the file the calico-node agent writes on startup. So calico-node on this node is not running or not ready — look at kubectl get pods -n calico-system -o wide and its logs. Other variants of the same event: the IP pool is exhausted, the plugin cannot reach the API server.

Warning   FailedMount   Pod/no-volume-cm   MountVolume.SetUp failed for volume "config" : configmap "hello-settings" not found
Warning   FailedMount   Pod/data-reader    MountVolume.MountDevice failed for volume "pvc-8f1c..." : rpc error: code = DeadlineExceeded desc = context deadline exceeded

The first is a volume from a ConfigMap or Secret that does not exist in the namespace (part 12). The second is a CSI driver that did not mount a network volume in time: for NFS that is an unreachable server, a firewall on 2049 or an export that does not allow the node's address (part 22); details are in the logs of the driver's node plugin on that node and in the node's dmesg.

ErrImagePull and ImagePullBackOff

ErrImagePull is a failed attempt to pull the image, ImagePullBackOff means kubelet is waiting before the next one, and the pause grows from 10 seconds to 5 minutes. The cause is in the text of the Failed event:

$ kubectl events -n demo --for pod/pull-missing --types=Warning
LAST SEEN           TYPE      REASON   OBJECT             MESSAGE
35s (x3 over 81s)   Warning   Failed   Pod/pull-missing   Failed to pull image "ghcr.io/example/hello:1.2": rpc error: code = NotFound desc = failed to pull and unpack image "ghcr.io/example/hello:1.2": failed to resolve image: ghcr.io/example/hello:1.2: not found
35s (x3 over 81s)   Warning   Failed   Pod/pull-missing   Error: ErrImagePull
8s (x4 over 80s)    Warning   Failed   Pod/pull-missing   Error: ImagePullBackOff

Error text	Cause
`failed to resolve image: ...hello:1.2: not found`	no such tag or a typo in the name; check the registry
`failed to fetch anonymous token: ... 401 Unauthorized`	a private registry, and containerd came without credentials: no `imagePullSecrets` on the pod or its ServiceAccount, or the secret is in another namespace
`x509: certificate signed by unknown authority`	ghcr.io has a public certificate; this happens with your own registry with an internal CA (the "Your own registry" section of part 5): containerd does not trust its CA, a file in `/etc/containerd/certs.d/registry.example.com/` is needed on every node
`dial tcp: lookup ghcr.io: no such host`	the node cannot resolve the registry name: the node's DNS, not CoreDNS
`toomanyrequests`	a public registry limit; a mirror or your own registry

The quickest check is to pull the image from the node with the same containerd: sudo crictl pull <image>. For a public image this immediately tells you whether the tag exists and whether the node reaches the registry. A private registry answers an anonymous client with 401 Unauthorized for any tag, even an existing one, so it needs crictl pull --creds <user>:<token>; then "pulls from the node" versus "does not pull into the pod" points to the pod's credentials.

CrashLoopBackOff

The container starts and exits, and kubelet restarts it with a growing pause: 10 s, 20, 40, 80 … up to 5 minutes; after 10 minutes of normal running the counter resets. This is not a cause but a waiting mode, and the cause is always in the previous run:

kubectl get pod ... -o yaml → lastState.terminated: the exit code and reason.
kubectl logs --previous — what the application wrote before it died.
If there are no logs and the code is 127 or 126 — the problem is in command/args or the image.
If the code is 137 and reason: OOMKilled — the next section. If 137 without OOM — kubelet killed the container: look at Killing events and the liveness probe, which may have been killing a container that had not finished starting.
If the code is 0 — the process simply finished its work: it is not a long-running program and belongs in a Job (part 9).

When the logs say too little, a copy of the pod with sh instead of the application helps: run the binary by hand and see what it does not like.

OOMKilled

$ kubectl get pod oom -n demo -o jsonpath='{.status.containerStatuses[0].lastState}' | jq
{
  "terminated": {
    "containerID": "containerd://4f06596a129437c311776738cf4f195302872e962c70d645c2ac68950af12fe3",
    "exitCode": 137,
    "finishedAt": "2026-11-02T13:46:55Z",
    "reason": "OOMKilled",
    "startedAt": "2026-11-02T13:46:55Z"
  }
}

The kernel killed the process because the container's cgroup reached memory.max, that is, the memory limit (part 10). The confirmation on the node is in the kernel log:

user@k8s-node-2:~$ sudo journalctl -k --since "10 min ago" --no-pager | grep -i "out of memory\|oom.group"
Nov 02 13:47:38 k8s-node-2 kernel: Memory cgroup out of memory: Killed process 763094 (tail) total-vm:69260kB, anon-rss:64948kB, file-rss:2060kB, shmem-rss:0kB, UID:0 pgtables:184kB oom_score_adj:982
Nov 02 13:47:38 k8s-node-2 kernel: Tasks in /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod637609c0_506e_417f_b645_73a37ad52b03.slice/cri-containerd-feeeb009a4322979cfd70bb0fd0fca347a47184f9505a94a034f3f3a92f5f11d.scope are going to be killed due to memory.oom.group set

The second line says kubelet on cgroup v2 enables memory.oom.group: when memory runs out, the kernel kills not just the single greediest process but all of the container's processes at once, so the container is not left half-alive. The fix is not "raise the limit until it stops" but to find out how much memory the application needs (metrics are in the next part) and set requests and limits from that number. For runtimes with their own heap (JVM, .NET, Node.js) it matters that the runtime knows the limit and does not plan a heap bigger than it. A separate case is a pod evicted by kubelet because the whole node ran out of memory: that is Evicted, not OOMKilled.

Running but not Ready

The process runs, but the readiness probe fails, and the pod is excluded from the service endpoints:

Warning   Unhealthy   Pod/ready-path   Readiness probe failed: HTTP probe failed with statuscode: 404
Warning   Unhealthy   Pod/ready-port   Readiness probe failed: Get "http://172.20.172.48:8081/readyz": dial tcp 172.20.172.48:8081: connect: connection refused

The first: the application answers, but not with what is expected — the probe path does not match the application's path (404), or the application considers itself not ready (503, often because it cannot reach its database). The second: the probe port does not match the application's port, or the application listens on 127.0.0.1 instead of all addresses — kubelet goes to the pod IP, not localhost. Check from inside the pod with kubectl debug, netshoot and curl to the same address.

Terminating forever

kubectl delete set deletionTimestamp, but the object does not go away. There are two independent causes.

Finalizers. An object with a non-empty metadata.finalizers is not deleted until every controller in the list removes its entry. A PVC with kubernetes.io/pvc-protection waits until pods stop using it; a namespace waits until all objects inside are deleted; a CRD object waits for its operator — and if the operator was removed first, it will wait forever:

$ kubectl get pvc data -n demo -o jsonpath='{.metadata.finalizers}'
["kubernetes.io/pvc-protection"]

Removing a finalizer by hand (kubectl patch ... --type=merge -p '{"metadata":{"finalizers":null}}') is fine only once you understand what it protects: it means giving up the cleanup the controller was supposed to do — for example, deleting the volume in the storage.

A pod kubelet cannot stop. Finalizers are rare on pods; usually kubelet cannot terminate the process or unmount a volume. The classic case is an NFS volume with the hard option (part 23) whose server is unreachable: the process hangs in the D state (uninterruptible I/O wait), SIGKILL does not reach it, umount hangs too. That is not a fault of the option: hard guarantees a write is not silently lost, and everything continues when the server comes back. The cure is bringing the server back. On the node it looks like this:

user@k8s-node-1:~$ ps -eo pid,stat,wchan:32,cmd | awk '$2 ~ /D/'
  48211 D    rpc_wait_bit_killable            /hello
user@k8s-node-1:~$ sudo dmesg | tail -2
[512334.118] nfs: server nfs.example.com not responding, still trying

kubectl delete pod --grace-period=0 --force deletes only the API object and does nothing on the node: the process and the mount stay. For a Deployment that is tolerable, for a StatefulSet it is dangerous — the controller will create a pod with the same name and the same volume on another node while the old one is still writing. If the node is dead, the right way is to make sure it is powered off and put the node.kubernetes.io/out-of-service=nodeshutdown:NoExecute taint on it: then Kubernetes detaches the volumes and recreates the pods properly.

Evicted

Kubelet evicts pods on its own in two cases. The first: the node is running out of memory, disk space (nodefs is /var/lib/kubelet, imagefs is containerd's images) or PIDs. The default thresholds are memory.available<100Mi, nodefs.available<10%, imagefs.available<15%, nodefs.inodesFree<5% and imagefs.inodesFree<5%. Pods that exceed their requests go first, in QoS class order, and the message says The node was low on resource: .... The second: the pod exceeded its own ephemeral-storage limit or the sizeLimit of an emptyDir volume. This is a pod that wrote 64 MiB to a temporary directory with a 32 MiB limit:

$ kubectl get pods -n demo disk-eater
NAME         READY   STATUS                   RESTARTS   AGE
disk-eater   0/1     ContainerStatusUnknown   1          105s
$ kubectl get pod disk-eater -n demo -o jsonpath='{.status.phase} {.status.reason}{"\n"}{.status.message}{"\n"}'
Failed Evicted
Pod ephemeral local storage usage exceeds the total limit of containers 32Mi.
$ kubectl events -n demo --for pod/disk-eater
LAST SEEN   TYPE      REASON    OBJECT           MESSAGE
65s         Warning   Evicted   Pod/disk-eater   Pod ephemeral local storage usage exceeds the total limit of containers 32Mi.

The STATUS column is misleading here: it shows the container's termination reason (ContainerStatusUnknown; kubelet killed it together with the pod), and the fact that the pod was evicted is visible only in .status.reason and the event. An evicted pod stays in the Failed phase, the controller creates a replacement, and the objects themselves are removed by the garbage collector only when the cluster has more terminated pods than kube-controller-manager's --terminated-pod-gc-threshold (12,500). To remove them by hand: kubectl delete pods -n demo --field-selector=status.phase=Failed. The cause is fixed case by case: who writes logs to files inside the container, who needs an emptyDir with sizeLimit, whether there is enough disk for images.

CreateContainerConfigError

kubelet cannot assemble the container configuration from what the spec refers to:

Warning   Failed   Pod/no-env-cm    Error: configmap "hello-settings" not found
Warning   Failed   Pod/no-key       Error: couldn't find key DB_PASSWORD in Secret demo/hello-db
Warning   Failed   Pod/root-image   Error: container has runAsNonRoot and image will run as root (pod: "root-image_demo(01ba9ce7-a9a3-45ed-b522-127686918b02)", container: busybox)

A missing ConfigMap or Secret, a missing key in it (for env.valueFrom, unless the reference is marked optional: true), or a securityContext that contradicts the image (part 30). As soon as the missing object appears, kubelet continues the start by itself.

Node NotReady

Every 10 seconds the node renews its Lease object in the kube-node-lease namespace and updates its status when it changes, but at least once every 5 minutes. If there are no renewals for longer than --node-monitor-grace-period (50 seconds in 1.36), the node lifecycle controller sets the conditions to Unknown and the node.kubernetes.io/unreachable taint; pods on that node will be evicted after 300 seconds (the default tolerationSeconds). This is a node one minute after systemctl stop kubelet:

$ kubectl describe node k8s-node-2 | sed -n '/Conditions/,/Addresses/p'
Conditions:
  Type                 Status    LastHeartbeatTime                 LastTransitionTime                Reason              Message
  ----                 ------    -----------------                 ------------------                ------              -------
  NetworkUnavailable   False     Sun, 01 Nov 2026 20:32:25 +0000   Sun, 01 Nov 2026 20:32:25 +0000   CalicoIsUp          Calico is running on this node
  MemoryPressure       Unknown   Mon, 02 Nov 2026 13:44:13 +0000   Mon, 02 Nov 2026 13:49:43 +0000   NodeStatusUnknown   Kubelet stopped posting node status.
  DiskPressure         Unknown   Mon, 02 Nov 2026 13:44:13 +0000   Mon, 02 Nov 2026 13:49:43 +0000   NodeStatusUnknown   Kubelet stopped posting node status.
  PIDPressure          Unknown   Mon, 02 Nov 2026 13:44:13 +0000   Mon, 02 Nov 2026 13:49:43 +0000   NodeStatusUnknown   Kubelet stopped posting node status.
  Ready                Unknown   Mon, 02 Nov 2026 13:44:13 +0000   Mon, 02 Nov 2026 13:49:43 +0000   NodeStatusUnknown   Kubelet stopped posting node status.
Addresses:
$ kubectl get node k8s-node-2 -o jsonpath='{.spec.taints}' | jq -c '.[]'
{"effect":"NoSchedule","key":"node.kubernetes.io/unreachable","timeAdded":"2026-11-02T13:49:43Z"}
{"effect":"NoExecute","key":"node.kubernetes.io/unreachable","timeAdded":"2026-11-02T13:49:43Z"}

kubelet was stopped at 13:48:53, and Unknown was set at 13:49:43 — exactly 50 seconds later. LastHeartbeatTime is five minutes older: kubelet sends the full status only when something changes, and the controller sees the node's liveness through the Lease. NetworkUnavailable is set by Calico, not kubelet, which is why it stayed False.

Ready=False with a reason means kubelet is alive and reports the problem itself. For example, when the node's CNI configuration disappears, the Ready line says container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized. Typical causes, from common to rare:

kubelet is not running: systemctl status kubelet, journalctl -u kubelet. A common case after manual edits is swap turned on or an error in /var/lib/kubelet/config.yaml.
containerd is not running or not responding: systemctl status containerd, crictl info.
CNI is not ready — no configuration in /etc/cni/net.d/, because calico-node did not start on the node. containerd's network state is shown by crictl info | jq '.status.conditions'. If the configuration file was put back by a rename (mv from another directory), containerd may not notice it: it skips the event of a rename into the directory and rereads the directory only on the next write to it.
kubelet's client certificate expired: x509: certificate has expired in the log. Kubelet renews /var/lib/kubelet/pki/kubelet-client-current.pem by itself if the node was powered on; after a long downtime the renewal does not happen in time, and the node has to be joined again. kubeadm certificates in detail are in part 35.
PLEG (Pod Lifecycle Event Generator) is the kubelet loop that polls containerd for the state of all containers. If one pass takes more than three minutes, kubelet considers itself unhealthy: PLEG is not healthy: pleg was last seen active 3m5s ago; threshold is 3m0s. Causes: an overloaded containerd, thousands of exited containers, a process stuck on NFS, a slow disk.
Resource pressure — MemoryPressure, DiskPressure, PIDPressure set to True; the node is ready but gets a taint and stops accepting new pods.
Clock skew: TLS to the API server breaks with a large time offset; timedatectl on the node.

A service without endpoints

The pod runs, the service exists, but requests to it get connection refused or hang:

$ kubectl get endpointslices -n demo -l kubernetes.io/service-name=hello-typo
NAME               ADDRESSTYPE   PORTS     ENDPOINTS   AGE
hello-typo-5mzdd   IPv4          <unset>   <unset>     89s
$ kubectl get svc hello-typo -n demo -o jsonpath='{.spec.selector}'; echo
{"app.kubernetes.io/name":"helo"}

An empty EndpointSlice (part 15) means either the service selector does not match the pod labels (as here: a typo), or matching pods exist but are not Ready, and then they show up in the EndpointSlice with ready: false. If there are endpoints but traffic does not flow, targetPort does not match the container port, or a NetworkPolicy cuts the traffic (part 20).

DNS errors

Inside a pod a name does not resolve, or resolves slowly. Check from the same namespace:

$ kubectl debug -it -n demo hello-865b67cc76-682fc --image=nicolaka/netshoot:v0.15 --profile=restricted -- \
    dig +search +short hello
10.100.210.84
$ kubectl debug -it -n demo hello-865b67cc76-682fc --image=nicolaka/netshoot:v0.15 --profile=restricted -- \
    dig @10.96.0.10 kubernetes.default.svc.cluster.local +short
10.96.0.1

If the second query fails, CoreDNS itself is unreachable: its pods (kubectl get pods -n kube-system -l k8s-app=kube-dns), their logs, a NetworkPolicy that forgot to allow egress on UDP and TCP 53 to kube-system. If internal names work but external ones are slow, it is ndots:5 (part 16): the name api.example.org has fewer than five dots, so the resolver first tries the three suffixes from search, getting NXDOMAIN, and only then asks for the name as is — four queries instead of one, each for A and AAAA. The fix is a trailing dot in the name in the application config (api.example.org.) or dnsConfig.options with ndots: 2 on the pod.

Summary table

Symptom	Common cause	Where to start
Deployment `0/3` ready, no pods	the ReplicaSet cannot create pods: quota, admission	`kubectl describe rs`, `kubectl events --for deploy/...`
`Pending`	no resources, taints, affinity, PVC not bound	`kubectl events --for pod/...` → `FailedScheduling`
`ContainerCreating` for more than a minute	CNI, a volume does not mount, no ConfigMap/Secret for a volume	`FailedCreatePodSandBox`, `FailedMount` events
`ErrImagePull`, `ImagePullBackOff`	tag, name, `imagePullSecrets`, registry CA	text of the `Failed` event, `crictl pull` on the node
`CrashLoopBackOff`	an application or configuration error	`lastState.terminated`, `kubectl logs --previous`
`OOMKilled`, code 137	memory limit too low or a leak	`lastState`, `journalctl -k` on the node, memory metrics
`Running`, `0/1`	readiness fails	`Unhealthy` events, `curl` from an ephemeral container
`Terminating` does not go away	a finalizer; a process in `D` on unreachable NFS	`metadata.finalizers`; `ps` and `dmesg` on the node
`Evicted` (or `ContainerStatusUnknown`)	memory, disk or PID pressure on the node; the `ephemeral-storage` limit	`.status.reason` and `.status.message`, `kubectl describe node`
`CreateContainerConfigError`	no ConfigMap, Secret or key	text of the `Failed` event
Node `NotReady`	kubelet, containerd, CNI, certificate, PLEG	`describe node`, `journalctl -u kubelet`
Service does not answer	selector, unready pods, `targetPort`, policy	`kubectl get endpointslices -l kubernetes.io/service-name=...`
Names do not resolve or are slow	CoreDNS, egress on 53, `ndots:5`	`dig` from netshoot, CoreDNS logs

Hands-on

Let us go through one case from start to finish. Version hello:1.1 is released; together with it, the PORT port was typed by hand into the hello-config ConfigMap that hello takes its environment variables from. Ten minutes after kubectl set image the rollout has not finished.

Object. The Deployment reports that the new version is not ready:

$ kubectl rollout status deploy/hello -n demo --timeout=10s
error: deployment "hello" exceeded its progress deadline

$ kubectl get deploy hello -n demo -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'
Available=True MinimumReplicasAvailable
Progressing=False ProgressDeadlineExceeded
$ kubectl get deploy hello -n demo
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
hello   3/3     1            3           11m

Progressing=False with ProgressDeadlineExceeded: within progressDeadlineSeconds (600 seconds by default) the rollout did not progress. The old pods are running (Available=True): with three replicas the default maxUnavailable (25%) rounds down to zero, and the rolling update from part 7 does not remove an old pod until a new one is ready. What counts is not the time since the rollout started but the time since the last progress: if the number of ready replicas went up even once in those ten minutes — say, a failed node with old pods came back — the countdown starts over.

Events and pod. The new ReplicaSet created one pod, and it is in CrashLoopBackOff — the output above: exit code 1, restartCount growing.

Container. The logs of the previous run name the cause:

$ kubectl logs -n demo hello-798bd968d9-vkz9z --previous
2026/11/02 13:58:02 listen: listen tcp: lookup tcp/8080	: unknown port

The application did not recognise port 8080, and in the message there is a gap between 8080 and the colon that should not be there. Let us see what the application gets in the variable. The image has no shell, and an ephemeral container will not help: the container crashes milliseconds after start, and --target will not catch it. We make a copy with busybox instead of the application:

$ kubectl debug -n demo hello-798bd968d9-vkz9z --copy-to=hello-debug \
    --set-image=hello=busybox:1.37 --container=hello --profile=restricted -it -- sh
/ $ id
uid=10001 gid=10001 groups=10001
/ $ printenv PORT | cat -A
8080^I$
/ $ exit
$ kubectl delete pod -n demo hello-debug
pod "hello-debug" deleted from demo namespace

cat -A shows invisible characters: ^I after 8080 is a tab, $ is the end of the line. The value was copied from a table together with a tab, and kubectl and YAML faithfully kept it.

Node, for completeness. The cause is found, but while the failing pod still exists, let us see what it looks like on the node — with the same kubectl debug node:

$ kubectl debug node/k8s-node-2 -it --profile=sysadmin --image=ubuntu:26.04
root@k8s-node-2:/# chroot /host
# ls /var/log/pods/demo_hello-798bd968d9-vkz9z_*/hello/
4.log
# exit
root@k8s-node-2:/# exit
$ kubectl get pods -o name | grep node-debugger | xargs kubectl delete
pod "node-debugger-k8s-node-2-8lwdc" deleted from default namespace

One file — from the last run (number 4), the one kubectl logs and --previous showed the error from. Everything older is already deleted.

Fix. A rollback makes the service uniform again, then the corrected ConfigMap and a new rollout:

$ kubectl rollout undo deploy/hello -n demo
Warning: resource deployments/hello was previously managed with 'kubectl apply'. Rolling back will not update the kubectl.kubernetes.io/last-applied-configuration annotation, which may cause unexpected behavior on future 'kubectl apply' operations. Consider using 'kubectl apply' with your previous configuration file instead.
deployment.apps/hello rolled back
$ kubectl apply -f hello-config.yaml
configmap/hello-config configured
$ kubectl set image deploy/hello -n demo hello=ghcr.io/example/hello:1.1
deployment.apps/hello image updated
$ kubectl rollout status deploy/hello -n demo
Waiting for deployment spec update to be observed...
Waiting for deployment "hello" rollout to finish: 0 out of 3 new replicas have been updated...
Waiting for deployment "hello" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "hello" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "hello" rollout to finish: 1 old replicas are pending termination...
deployment "hello" successfully rolled out

The warning on rollback says that rollout undo changes the Deployment bypassing kubectl apply, and the last-applied-configuration annotation still describes version 1.1. The right habit is to fix the manifest in Git after a rollback and apply it. The pod hello-798bd968d9-vkz9z is deleted together with its log files on the node, and in an hour the events about it will be gone too. If this failure had happened at night and resolved itself, there would be not a single trace of it in the cluster by morning. That is exactly why metrics and centralised logs come next.

Cleanup: kubectl delete pod -n demo <debug pods> for everything created in the "Typical states" section; the ephemeral containers in the hello pods will disappear on the Deployment's next restart.

Manual troubleshooting answers the question "what is broken now". To answer "what happened at night", "how much memory is really needed" and "when will we run out of resources", you need data collected in advance.

Sources

Debug Pods and Determine the Reason for Pod Failure — the order of pod troubleshooting, lastState and the termination message.
Debug Running Pods — kubectl exec, ephemeral containers, pod copies, kubectl debug profiles.
Ephemeral Containers — what an ephemeral container can and cannot have.
kubectl debug and kubectl events — flag reference.
Debugging Kubernetes Nodes With Kubectl and Debugging Kubernetes nodes with crictl — kubectl debug node and crictl.
Pod Lifecycle — phases, conditions, container states, the restart back-off.
Deployments: Failed Deployment — progressDeadlineSeconds and ProgressDeadlineExceeded.
Resource Management for Pods and Containers and Node-pressure Eviction — memory and ephemeral-storage limits, eviction thresholds.
Node Status and Node Shutdowns — node conditions and Lease, the out-of-service taint.
Finalizers — why an object gets stuck in deletion.
Debug Services and Debugging DNS Resolution — a service without endpoints, CoreDNS checks.
nicolaka/netshoot — what the debugging image contains.

Next — part 32: metrics and autoscaling. metrics-server and kubectl top, the HorizontalPodAutoscaler and its algorithm, Prometheus with kube-prometheus-stack, PromQL queries and alerting rules.