28 October 2026, 16:33Виталий Лещенко

Kubernetes for beginners. Part 10: How do requests, limits and QoS work?

All parts of the series “Kubernetes for beginners”

The node-exporter in part 8 and the database dump in the previous part had a resources block in their containers with two sets of numbers — requests and limits. It is the most underrated part of a manifest. It decides which node a pod lands on, how much CPU time it gets, when the kernel kills its process and when the kubelet evicts the whole pod from the node, and in what order that happens to its neighbours. Let's look at what Kubernetes does with these numbers: how the scheduler reads them, what the kubelet turns them into in cgroup v2 on the node, where CPU throttling and OOMKilled come from, how QoS classes and eviction under memory pressure work, and how to restrict all of this at the namespace level.

Requests and limits

spec:
  containers:
  - name: hello
    image: ghcr.io/example/hello:1.0
    resources:
      requests:
        cpu: 250m
        memory: 128Mi
      limits:
        cpu: 500m
        memory: 128Mi

requests — how much of a resource the container is guaranteed. The scheduler places a pod only on a node where the sum of the requests of all pods plus the new one fits into what is available; on the node the CPU request becomes a weight when CPU is shared.
limits — the ceiling. The container gets no CPU time above its CPU limit even if the node is idle; above the memory limit the kernel kills its process.

Units. CPU is measured in cores: 1 is one core (physical or virtual, whatever the node sees), 250m is a quarter of a core, "250 millicores". Nothing finer than 1m can be set. Memory is in bytes, with the suffixes Ki, Mi, Gi (powers of two) or k, M, G (powers of ten): 128Mi is 134,217,728 bytes, 128M is 128,000,000. The classic typo is memory: 400m: a lowercase m means "milli", and that is a request for 0.4 bytes.

Values are set per container; a pod's request and limit are the sum over its containers. Init containers run one at a time and before the main ones, so scheduling uses the larger of two: the "heaviest" init container and the sum of the main containers; the sidecar containers from part 6 run all the time and are added to the sum. Since 1.34 (beta, enabled by default) resources can also be set for the pod as a whole — spec.resources — letting the containers share a common budget; that is handy for pods with a dozen small containers, but the per-container way is still the main one.

Default rules: if only a limit is set, the request is taken equal to it. If only a request is set, there is no limit — the container can take everything the node has.

How the scheduler uses requests

For every node the kubelet publishes two figures per resource in the status of the Node object:

capacity — the total: the machine's cores and memory;
allocatable — how much of it can be given to pods: capacity minus the reserve for the system (systemReserved), minus the reserve for the kubelet and the runtime (kubeReserved), minus the eviction threshold (evictionHard, see below).

 capacity: 2 CPU, 3.32 GiB
 ┌──────────────────────────────────────────────────────────────┐
 │ systemReserved │ kubeReserved │ evictionHard │  allocatable  │
 │ sshd, journald │ kubelet,     │ margin at    │  what the     │
 │ kernel,        │ containerd   │ which pods   │  pods share   │
 │ systemd        │              │ are evicted  │               │
 └──────────────────────────────────────────────────────────────┘

The scheduler kube-scheduler (part 13) adds up the requests of all pods already assigned to each candidate node, adds the requests of the new one and compares the sum with allocatable. It does not look at actual usage: a node loaded at 5 % but with requests at 100 % is full for it. This is a deliberate decision — the schedule is built on promises, not on a momentary snapshot, otherwise pods placed at night would not fit at rush hour. The flip side is that a pod without requests weighs zero for the scheduler, and a hundred of them can be crammed onto one node.

If a pod fits no node, it stays Pending with the event:

Warning  FailedScheduling  default-scheduler  0/3 nodes are available: 1 node(s) had untolerated taint(s), 2 Insufficient memory. no new claims to deallocate, preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.

The system reserve. By default kubeadm reserves nothing except the 100 MiB memory eviction threshold: allocatable is almost equal to capacity, and pods can take all the memory the kubelet, containerd and sshd need. On a busy node this ends with the kubelet unable to answer the API server and the node turning NotReady. The best practice is to set a reserve in the kubelet configuration:

# KubeletConfiguration fragment
systemReserved:
  cpu: 200m
  memory: 512Mi
kubeReserved:
  cpu: 200m
  memory: 512Mi
evictionHard:
  memory.available: 200Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
  imagefs.available: 15%
  imagefs.inodesFree: 5%

With kubeadm the kubelet configuration lives in the kubelet-config ConfigMap in kube-system and is copied to the nodes as /var/lib/kubelet/config.yaml; you edit the ConfigMap, then on every node run kubeadm upgrade node phase kubelet-config and systemctl restart kubelet — otherwise the next cluster upgrade (part 35) brings the old one back. Note evictionHard: set one threshold and the others are zeroed unless you list them explicitly or enable mergeDefaultEvictionSettings. That is why the example lists them all.

By default the kubelet enforces only the pods entry of enforceNodeAllocatable: the whole cgroup hierarchy of pods (kubepods.slice) is limited to capacity minus systemReserved and kubeReserved, and pods together cannot eat the reserve even if they have no limits. The eviction threshold is not part of that limit: it is there so that the kubelet starts evicting before the kernel hits the limit of the whole pod group. Without reserves, as on our lab, the memory.max of kubepods.slice holds all of the node's memory. The reserves for the system and the kubelet are only subtracted from allocatable and do not become cgroup limits of their own — that is safer, and changing it only makes sense if you understand that a system.slice with a memory limit can bring sshd down.

What the numbers turn into: cgroup v2

Requests and limits are data in the API. The kubelet applies them: it creates a cgroup hierarchy for pods and passes the container parameters to containerd, which passes them to runc, which writes the files in /sys/fs/cgroup. With SystemdCgroup = true from part 4 the groups are created through systemd and named like its units — slices and scopes:

/sys/fs/cgroup/
 └─ kubepods.slice                              capacity minus reserves
     ├─ kubepods-pod<uid>.slice                 Guaranteed pods — right here
     │   └─ cri-containerd-<id>.scope           container
     ├─ kubepods-burstable.slice                Burstable
     │   └─ kubepods-burstable-pod<uid>.slice
     │       ├─ cri-containerd-<id>.scope       pause container
     │       └─ cri-containerd-<id>.scope       hello container
     └─ kubepods-besteffort.slice               BestEffort, cpu.weight = 1
         └─ kubepods-besteffort-pod<uid>.slice

Three levels: one common to all pods, one per QoS class (classes are below), one per pod and one per container. In the slice name the dashes of the pod UID are replaced with underscores: for systemd a dash separates hierarchy levels.

What goes into the files:

Manifest field	cgroup v2 file	Value for `250m`/`500m`/`128Mi`
`requests.cpu`	`cpu.weight`	35 (new formula) or 10 (old)
`limits.cpu`	`cpu.max`	`50000 100000`
no `limits.cpu`	`cpu.max`	`max 100000`
`limits.memory`	`memory.max`	`134217728`
no `limits.memory`	`memory.max`	`max`
`requests.memory`	—	written nowhere (without Memory QoS)

cpu.weight is a relative weight for sharing CPU, from 1 to 10000, 100 by default for any group. While there is enough CPU the weight changes nothing; when several groups want more than there is, time is split in proportion to the weights. Historically Kubernetes converts the request into cpu.shares from cgroup v1 (1000m → 1024), and the runtime converts shares into a cgroup v2 weight. The old linear formula gave one core a weight of 39 — less than any system process by default — so runc 1.3.2 and crun 1.23 replaced it with a non-linear one where one core gives about 100 and 250m gives 35. On the lab with containerd 2.2 and runc 1.4 from the Ubuntu packages you see the new values; with an old runc it would be 10 for 250m. But only in container groups: the pod and QoS class groups are configured by the kubelet itself through systemd, still with the old formula — a pod with 250m has weight 10, and kubepods.slice on a two-core node has 79. What matters is not the absolute number but the ratio between neighbours on the same level.

cpu.max is a quota and a period in microseconds: "no more than 50,000 µs of CPU time for every 100,000 µs". That is the 500m limit. How it works and why it is dangerous is in the next section.

memory.max is the hard memory limit of the group, including the page cache of files the container reads and writes (the kernel reclaims cache first before declaring a shortage). When it is exceeded — OOM inside the group.

requests.memory does not reach the kernel: it is a promise only for the scheduler and for the eviction order. The alpha Memory QoS feature can turn it into memory.min/memory.low (protection from page reclaim) and set memory.high (soft throttling below the limit), but in 1.36 it is still alpha and disabled by default.

And one more file the kubelet sets itself: memory.oom.group = 1 in the group of every container (on cgroup v2 since 1.28; since 1.32 it can be turned off with the kubelet parameter singleProcessOOMKill: true). On OOM the kernel kills all processes of the container at once rather than one — so as not to leave a half-dead container where the worker process died while the supervisor keeps pretending everything is fine.

CPU: the CFS quota and throttling

CPU is a compressible resource: if there is not enough of it, a process runs slower but does not die. The CPU limit is implemented by the kernel mechanism CFS bandwidth control (bandwidth control of CFS, the main Linux process scheduler): in every period (100 ms) the group may spend no more than its quota; once it has spent it, all of its threads are frozen until the end of the period. That is what throttling is.

The trap is that the quota counts CPU time across all cores at once. A service with a limit of 1 and eight threads handling a request in parallel spends 100 ms of quota in 12.5 ms of real time and stands still for the next 87.5 ms:

 limit 1 CPU = 100 ms of CPU for every 100 ms of time
 8 threads work in parallel:

 time, ms    0    12.5                                         100   112.5
             ├─────┼────────────────────────────────────────────┼─────┼────
 threads     ████████ quota used up ░░░░░░░ frozen ░░░░░░░░░░░░░ ██████ ...
 response     request arrives ───────────────────── response delayed ~90 ms

Averaged over a minute such a service may use 30 % of its limit and still regularly answer a hundred milliseconds late. The throttled metric in cpu.stat and container_cpu_cfs_throttled_periods_total in the Prometheus monitoring system (part 32) show this directly, an average utilisation graph does not.

The second problem is programs that size their thread count by the number of cores. The node has 32 cores, the limit is 2, but the runtime sees 32 and starts 32 garbage collector threads that eat the quota in milliseconds. Modern runtimes read cpu.max: Java has done it for a long time (its processor count can be set with -XX:ActiveProcessorCount), Go since version 1.25 (the default GOMAXPROCS takes the cgroup limit into account). For the rest the number of worker threads has to be set explicitly.

Kubernetes cannot kill for CPU: neither the kubelet nor the kernel terminates a container for exceeding cpu.max. They only slow it down.

Memory: OOMKilled and the node OOM killer

Memory is incompressible: a page once given cannot be taken back without killing the process (there is no swap on Kubernetes nodes by default; since 1.34 it can be enabled in LimitedSwap mode for Burstable pods only, but that is a separate story). So there are two different punishments for memory, and it is important to tell them apart.

OOM in the container cgroup. The container hits memory.max, the kernel cannot free cache — the OOM killer fires inside this group and kills its processes (with memory.oom.group, all of them). The kubelet sees that the container exited on SIGKILL (exit code 137 = 128 + 9) with the reason OOMKilled, and restarts it according to the pod's restartPolicy. The pod stays on the node, RESTARTS grows, repeated failures give CrashLoopBackOff. The neighbours are not affected.

Node OOM. If the whole node runs out of memory (pods have no limits or their sum exceeds physical memory), the kernel's global OOM killer fires. It picks a victim by oom_score — a score that grows with memory usage and is shifted by oom_score_adj from −1000 to 1000. The kubelet sets this value for every container in advance according to its QoS class:

Class	`oom_score_adj`
Guaranteed	−997
Burstable	from 2 to 999: `1000 − 1000 × request / node memory`
BestEffort	1000
pause container of any pod	−998
kubelet, containerd	−999

The bigger a Burstable pod's memory request relative to the node, the smaller the chance it is the one killed. BestEffort die first, Guaranteed last, system daemons almost never.

It should not come to a global OOM: the kubelet steps in earlier with its eviction.

QoS classes

From requests and limits every pod is assigned a class at creation, recorded in status.qosClass:

Class	Condition
Guaranteed	every container has requests and limits for both CPU and memory, and they are equal
Burstable	not Guaranteed, but at least one container has a CPU or memory request or limit
BestEffort	no container has any CPU or memory requests or limits

The class is determined once and does not change for the life of the pod. It affects three things: which slice the pod goes into on the node, which oom_score_adj its processes get and — indirectly — the eviction order. There is a fourth: only Guaranteed pods with whole numbers of cores can get exclusive cores from the static policy of the kubelet's CPU manager, which latency-sensitive applications need.

Eviction under node resource pressure

Every 10 seconds the kubelet checks node resource signals and compares them with thresholds:

Signal	What it is	Default threshold
`memory.available`	memory capacity minus the working set (without inactive file cache)	`< 100Mi`
`nodefs.available`	free space on the kubelet filesystem (`/var/lib/kubelet`, logs)	`< 10%`
`nodefs.inodesFree`	free inodes there	`< 5%`
`imagefs.available`	free space where containerd keeps images and layers	`< 15%`
`imagefs.inodesFree`	free inodes there	`< 5%`
`pid.available`	free process IDs	not set

Thresholds are hard (evictionHard: crossed — evict immediately, with no grace period) and soft (evictionSoft plus evictionSoftGracePeriod: evict if the breach lasts longer than the given time, and give pods evictionMaxPodGracePeriod to stop).

When a threshold is crossed, the kubelet:

sets the node condition MemoryPressure, DiskPressure or PIDPressure, and the node controller adds a taint like node.kubernetes.io/memory-pressure:NoSchedule so new pods do not come here;
tries to free the resource itself: on disk shortage it deletes dead containers and unused images;
if that does not help, it evicts pods. The order: first those whose usage exceeds their request, among them by pod priority and by how large the excess is; then the rest, also by priority.

In practice this matches the QoS classes: BestEffort always exceed their zero request and go first, Burstable go if they have exceeded their request, Guaranteed and Burstable within their request go last. That is why an honest memory request is the best protection against eviction.

An evicted pod moves to phase Failed with the reason Evicted and stays in the API until it is removed; if a ReplicaSet or StatefulSet owns it, the controller creates a replacement — on another node. Eviction and OOM are different things: eviction is done by the kubelet and removes the whole pod, OOM is done by the kernel and kills the container's processes.

ephemeral-storage

The third resource pods share on a node is the local disk. A pod writes to it in three places: the writable layer of the container (everything the application writes to its filesystem outside volumes), emptyDir volumes on disk and its logs /var/log/pods/.... All of this is ephemeral storage, temporary storage that disappears with the pod:

resources:
  requests:
    ephemeral-storage: 1Gi
  limits:
    ephemeral-storage: 2Gi

The request is counted by the scheduler just like CPU and memory. The kubelet checks the limit by periodic accounting and, when it is exceeded, evicts the pod with a message like Pod ephemeral local storage usage exceeds the total limit of containers 2Gi. An emptyDir can have its own sizeLimit. And an emptyDir with medium: Memory is tmpfs, and it counts not as disk but as the container's memory and runs into memory.max: a 1 GiB file in such a volume with a 512 MiB limit ends in OOM. Volumes in detail — in part 21.

LimitRange: defaults and bounds in a namespace

Setting requests on every container by hand is discipline, and discipline breaks. LimitRange is a namespace object checked when a pod is created (it is handled by the LimitRanger admission controller in the API server — the stage every write request passes through before it is stored in etcd):

apiVersion: v1
kind: LimitRange
metadata:
  name: defaults
  namespace: quota-test
spec:
  limits:
  - type: Container
    defaultRequest:          # request when none is given
      cpu: 100m
      memory: 128Mi
    default:                 # limit when none is given
      memory: 128Mi
    min:
      memory: 32Mi
    max:
      cpu: "2"
      memory: 2Gi

A container without resources gets a request of 100m/128Mi, a memory limit of 128Mi — and a CPU limit of two cores: if default is not set for a resource but max is, max becomes the default limit. If you do not want CPU limits by default, do not set max.cpu either. A container asking for 4 GiB is rejected. You can also set maxLimitRequestRatio — how many times the limit may exceed the request. Constraints exist for Container, Pod (the sum) and PersistentVolumeClaim (the size of a volume claim).

Important: a LimitRange acts only at creation of a pod. Change the LimitRange and the running pods stay as they are. And it does not check whether the namespace as a whole has enough resources — ResourceQuota does that. Namespaces in detail — in the next part; here quota-test is created for the experiments.

ResourceQuota: the namespace budget

ResourceQuota limits the sum over the whole namespace:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: budget
  namespace: quota-test
spec:
  hard:
    requests.cpu: "2"
    requests.memory: 4Gi
    limits.memory: 4Gi
    requests.ephemeral-storage: 10Gi
    pods: "20"
    persistentvolumeclaims: "5"
    requests.storage: 50Gi
    count/jobs.batch: "10"

On every object creation the ResourceQuota admission controller adds its requests to what is already used and rejects the request if the sum would exceed hard; a separate controller in kube-controller-manager recalculates the usage in status.used. You can limit resources (CPU, memory, storage — including per StorageClass) and the number of objects of any type through count/<resource>.<group>.

One consequence worth remembering: if a quota limits requests.cpu or requests.memory, then every new pod in the namespace must specify these requests — otherwise there is nothing to count, and it is rejected. That is why a ResourceQuota is almost always paired with a LimitRange that fills in the defaults.

Quotas can apply to only some pods — through scopes: BestEffort and NotBestEffort, Terminating and NotTerminating (whether the pod has activeDeadlineSeconds), and by priority class (scopeSelector with PriorityClass). For example, to forbid BestEffort pods entirely — a pods: "0" quota with the BestEffort scope.

Changing resources on the fly

The resources of a pod used to be immutable: to give a container more memory, the pod was recreated. Since 1.35 in-place resize is stable: CPU and memory requests and limits of a running pod are changed through the resize subresource, and the kubelet rewrites cpu.max, cpu.weight and memory.max in the cgroup without touching the process.

$ kubectl patch pod res-demo --subresource resize --patch \
    '{"spec":{"containers":[{"name":"hello","resources":{"limits":{"cpu":"1"}}}]}}'
pod/res-demo patched

You need kubectl 1.32 or newer. What can be changed without a restart is decided by the container's resizePolicy: by default NotRequired for both resources, and for memory RestartContainer is sensible — most runtimes (JVM, Node.js) pick their heap size at startup and will not see the increase anyway. Progress is visible in the pod conditions PodResizePending (reasons Infeasible — never possible on this node, Deferred — no room yet) and PodResizeInProgress, and the values actually applied are in status.containerStatuses[].resources.

Restrictions: the pod's QoS class cannot change (for Guaranteed, requests and limits must stay equal); the kubelet lowers a memory limit carefully and defers it if current usage is above the new limit, but there is no guarantee against OOM; requests or limits cannot be removed entirely. And most importantly: this is an operation on a pod. Changing the Deployment template still rolls out new pods. Resources are changed in place by those who manage individual pods — for example, the Vertical Pod Autoscaler (VPA, a component that picks requests based on actual usage): its newer versions can apply recommendations without recreating the pod.

What to set: recommendations

Requests — on every container, always. Without requests the scheduler places pods blindly, the pod lands in BestEffort, goes first on eviction and dies first on a node OOM. The values come from observation: usage under real load over a week, the request around the normal level (for example, the 90th percentile), not the peak and not "4 GiB just in case". How to collect this data — in part 32.

Memory limit equal to the memory request. If the limit is above the request, the sum of limits on the node exceeds its memory, and the promise "you may go up to the limit" holds only while the neighbours behave modestly. When everyone heads for their limits at once, evictions and global OOM start, and the ones who suffer are not the ones to blame. With limit = request the pod gets exactly what is reserved for it: exceed it — OOM only for this pod, fair and predictable. A memory leak shows up immediately as growing RESTARTS, not as a night-time eviction of the neighbours.

A CPU limit is debatable. Arguments against:

requests already guarantee everyone their share through cpu.weight at the moment of contention; a limit adds nothing to that guarantee;
a limit forbids using idle CPU: at night the node is free, and a batch job crawls in its 500m;
throttling hits the latency of multithreaded services even at low average utilisation, and it is hard to diagnose.

Arguments for:

predictability: a service that ran on six cores of a free node in testing will get its one core on a busy node in production — better to find that out in testing;
isolation in a shared cluster: when different teams work in it, limits keep one service from monopolising a node for a while and making things worse for the neighbours than they planned;
Guaranteed requires a CPU limit, and the CPU manager's exclusive cores require Guaranteed;
some runtimes and applications size their thread pools by the limit — without it they see all of the node's cores.

The practical conclusion: in a single-team cluster, correct requests without CPU limits are usually better for latency-sensitive services; in a shared cluster, for batch jobs and where reproducibility matters, a limit is set, but noticeably above the request, and the throttling metric is watched. CPU requests, however, are needed in any case.

Node reserves — systemReserved and kubeReserved, as above. A LimitRange with defaults — in every namespace where people deploy, not only a pipeline, and a ResourceQuota where a namespace is used by a separate team.

Hands-on

A pod with the resources from the beginning of the part — it is Burstable, because its CPU requests and limits differ:

$ kubectl apply -f res-demo.yaml
pod/res-demo created

$ kubectl get pod res-demo -o jsonpath='{.status.qosClass}{" "}{.metadata.uid}{" "}{.spec.nodeName}{"\n"}'
Burstable 61fc9751-ad7b-4318-9db3-078bbf2e5eb5 k8s-node-2

On node k8s-node-2 let's find its group and read the files:

user@k8s-node-2:~$ cd /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61fc9751_ad7b_4318_9db3_078bbf2e5eb5.slice
user@k8s-node-2:~$ ls -d cri-containerd-*
cri-containerd-4507bad2a06b244791b8f45f3e1c4cb5be2edec0c219688831e39b7fe68b30db.scope
cri-containerd-70e2164251a90996f1ef42026416c808f3beab6db6ffc351e7d6b257846ad5c5.scope

user@k8s-node-2:~$ sudo crictl ps --pod $(sudo crictl pods --name res-demo -q) -o table
CONTAINER           IMAGE               CREATED             STATE               NAME                ATTEMPT             POD ID              POD                 NAMESPACE
70e2164251a90       f768da2949737       16 seconds ago      Running             hello               0                   4507bad2a06b2       res-demo            default

Two container groups: 4507… is the pod's pause container (its ID matches the ID of the pod "sandbox" in crictl — the command-line client of CRI, the interface between the kubelet and the runtime), 70e2… is our hello. Let's read:

user@k8s-node-2:~$ C=cri-containerd-70e2164251a90996f1ef42026416c808f3beab6db6ffc351e7d6b257846ad5c5.scope
user@k8s-node-2:~$ cat $C/cpu.weight $C/cpu.max $C/memory.max $C/memory.oom.group
35
50000 100000
134217728
1
user@k8s-node-2:~$ cat cpu.weight cpu.max memory.max memory.oom.group
10
50000 100000
134217728
0
user@k8s-node-2:~$ cat $C/memory.min $C/memory.low $C/memory.high
0
0
max

Weight 35 is the new formula (an old runc would give 10), a quota of 50 ms per 100 ms, a 128 MiB memory limit, oom.group enabled. The pod group repeats the quota and the memory limit — the sum over the containers — but its weight is 10: the kubelet computes it with the old formula. memory.oom.group is off for the pod group — the processes of one container are killed, not of the whole pod. The pause container has cpu.weight 1, and cpu.max and memory.max are max. memory.min, memory.low, memory.high are untouched — Memory QoS is off. The process's oom_score_adj:

user@k8s-node-2:~$ cat /proc/$(cat $C/cgroup.procs | head -1)/oom_score_adj
963

The lab node has 3,479,660 KiB of memory (3.32 GiB): 1000 − 1000 × 128 MiB / 3.32 GiB ≈ 962.3, the kubelet uses integer division and gets 963. On the same node a Guaranteed pod's process got −997, a BestEffort one 1000, the pause containers of all pods −998, the kubelet and containerd −999. Now throttling: let's run an endless loop in another pod with a 200m limit:

$ kubectl run burner --image=busybox:1.37 --restart=Never \
    --overrides='{"spec":{"containers":[{"name":"burner","image":"busybox:1.37","command":["sh","-c","while :; do :; done"],"resources":{"requests":{"cpu":"100m","memory":"16Mi"},"limits":{"cpu":"200m","memory":"16Mi"}}}]}}'
pod/burner created

$ kubectl get pod burner -o jsonpath='{.metadata.uid}{" "}{.spec.nodeName}{"\n"}'
342f2a06-f06b-4169-a1a0-2db441b449b4 k8s-node-1

user@k8s-node-1:~$ cat /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod342f2a06_f06b_4169_a1a0_2db441b449b4.slice/cpu.stat
usage_usec 12693723
user_usec 12679482
system_usec 14240
nice_usec 0
core_sched.force_idle_usec 0
nr_periods 637
nr_throttled 633
throttled_usec 18443637
nr_bursts 0
burst_usec 0

This is the pod group about a minute after start. In 64 seconds there were 637 periods, and in 633 of them the group hit its quota: 12.7 seconds of CPU were spent — exactly 200m, the rest of the time the thread stood still. throttled_usec here is only 18 seconds because the 20000 100000 quota is set on both the pod group and the container group, and the kernel records the stall at the level where the quota ran out first: the container's cpu.stat showed another 32 seconds at the same moment, about 51 seconds of stall out of 64 in total. From the process's point of view the node is "slow", although the second core is free.

OOM. A pod with a 64 MiB limit that eats memory without bound (tail reads /dev/zero, which has no newlines, and accumulates the "line" in memory). The default restartPolicy is Always, so the kubelet keeps restarting the container:

$ kubectl run eater --image=busybox:1.37 \
    --overrides='{"spec":{"containers":[{"name":"eater","image":"busybox:1.37","command":["tail","/dev/zero"],"resources":{"requests":{"cpu":"50m","memory":"64Mi"},"limits":{"memory":"64Mi"}}}]}}'
pod/eater created

$ kubectl get pod eater
NAME    READY   STATUS      RESTARTS      AGE
eater   0/1     OOMKilled   3 (25s ago)   46s

$ kubectl get pod eater -o jsonpath='{.status.containerStatuses[0].lastState.terminated}{"\n"}'
{"containerID":"containerd://7c2505a705bc3fcb923cf099e74da786d8990f455c02b6be026e9c6e0b3f1a73","exitCode":137,"finishedAt":"2026-10-28T11:11:19Z","reason":"OOMKilled","startedAt":"2026-10-28T11:11:19Z"}

The container dies a fraction of a second after starting, and between attempts the kubelet waits the backoff pause from part 6. The pod landed on k8s-node-2; on the node the kernel writes every OOM to the log and the cgroup to its counters. The container group disappears with its processes, but the pod group lives until the pod finishes (for a pod with restartPolicy: Never the kubelet removes it right after the container stops), and the memory.events counters in cgroup v2 are hierarchical — they count events in nested groups:

user@k8s-node-2:~$ sudo journalctl -k --since -3min | grep -E 'oom-kill:|Killed process|Tasks in' | grep -A3 7c2505a7
Oct 28 14:11:19 k8s-node-2 kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=cri-containerd-7c2505a705bc3fcb923cf099e74da786d8990f455c02b6be026e9c6e0b3f1a73.scope,mems_allowed=0,oom_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcd90933_a943_41de_84fc_fc9b63303d88.slice,task_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcd90933_a943_41de_84fc_fc9b63303d88.slice/cri-containerd-7c2505a705bc3fcb923cf099e74da786d8990f455c02b6be026e9c6e0b3f1a73.scope,task=tail,pid=595791,uid=0
Oct 28 14:11:19 k8s-node-2 kernel: Memory cgroup out of memory: Killed process 595791 (tail) total-vm:69252kB, anon-rss:64940kB, file-rss:1960kB, shmem-rss:0kB, UID:0 pgtables:184kB oom_score_adj:982
Oct 28 14:11:19 k8s-node-2 kernel: Tasks in /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcd90933_a943_41de_84fc_fc9b63303d88.slice/cri-containerd-7c2505a705bc3fcb923cf099e74da786d8990f455c02b6be026e9c6e0b3f1a73.scope are going to be killed due to memory.oom.group set
Oct 28 14:11:19 k8s-node-2 kernel: Memory cgroup out of memory: Killed process 595791 (tail) total-vm:69252kB, anon-rss:64940kB, file-rss:1960kB, shmem-rss:0kB, UID:0 pgtables:184kB oom_score_adj:982

user@k8s-node-2:~$ cat /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcd90933*.slice/memory.events
low 0
high 0
max 145
oom 4
oom_kill 8
oom_group_kill 4
sock_throttled 0

CONSTRAINT_MEMCG means OOM inside a group, not on the whole node. oom_memcg is the pod group: its limit is the sum of the container limits, the same 64 MiB, and the kernel hit it at the pod level, but it kills the processes by the container group's memory.oom.group (Tasks in … are going to be killed). tail itself appears in the log twice — as the chosen victim and as a member of the group — so oom_kill is twice oom and oom_group_kill. oom_score_adj:982 is the same formula for a 64 MiB request. The node and the neighbours were not affected.

What the scheduler sees on the node:

$ kubectl describe node k8s-node-2 | sed -n '/^Capacity:/,/^System Info:/p;/^Allocated resources:/,/^Events:/p'
Capacity:
  cpu:                2
  ephemeral-storage:  31761232Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             3479660Ki
  pods:               110
Allocatable:
  cpu:                2
  ephemeral-storage:  29271151363
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             3377260Ki
  pods:               110
System Info:
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests    Limits
  --------           --------    ------
  cpu                250m (12%)  500m (25%)
  memory             128Mi (3%)  128Mi (3%)
  ephemeral-storage  0 (0%)      0 (0%)
  hugepages-1Gi      0 (0%)      0 (0%)
  hugepages-2Mi      0 (0%)      0 (0%)
Events:

Allocatable is smaller than Capacity by exactly 100 MiB of memory (the kubeadm default eviction threshold) and 10 % of the disk; CPU is not reserved at all. With the reserve from the scheduler section it would become cpu: 1600m, and memory would shrink by about another 1.1 GiB — a noticeable share on a 3.3 GiB node. Requests is the sum of the requests of all pods on the node; calico-node, csi-node-driver and kube-proxy have zero requests on our lab, so only res-demo is here. The full list with each pod's requests is in the Non-terminated Pods block of the same describe.

In-place resize — let's raise the CPU limit of res-demo to one core:

$ kubectl patch pod res-demo --subresource resize --patch \
    '{"spec":{"containers":[{"name":"hello","resources":{"limits":{"cpu":"1"}}}]}}'
pod/res-demo patched

$ kubectl get pod res-demo -o jsonpath='{.status.containerStatuses[0].resources.limits.cpu}{" restarts="}{.status.containerStatuses[0].restartCount}{"\n"}'
1 restarts=0

user@k8s-node-2:~$ cat $C/cpu.max cpu.max
100000 100000
100000 100000

The quota grew to 100 ms per period for both the container and the pod group, and the process was not restarted. A request that can never fit on the node is rejected by the 1.36 API server right away, without getting to Infeasible:

$ kubectl patch pod res-demo --subresource resize --patch \
    '{"spec":{"containers":[{"name":"hello","resources":{"requests":{"cpu":"3"},"limits":{"cpu":"4"}}}]}}'
Error from server (Forbidden): pods "res-demo" is forbidden: node didn't have enough allocatable resources: cpu, requested: 3000, allocatable: 2000

Let's try to make the pod Guaranteed by making the CPU request equal to the limit:

$ kubectl patch pod res-demo --subresource resize --patch \
    '{"spec":{"containers":[{"name":"hello","resources":{"requests":{"cpu":"1"}}}]}}'
The Pod "res-demo" is invalid: spec: Invalid value: "Burstable": Pod QOS Class may not change as a result of resizing

The QoS class is immutable — the API refused.

Finally, quotas. In the quota-test namespace with the LimitRange and ResourceQuota from the examples above:

$ kubectl create namespace quota-test
namespace/quota-test created
$ kubectl apply -f limitrange.yaml -f quota.yaml
limitrange/defaults created
resourcequota/budget created

$ kubectl -n quota-test describe limitrange defaults
Name:       defaults
Namespace:  quota-test
Type        Resource  Min   Max  Default Request  Default Limit  Max Limit/Request Ratio
----        --------  ---   ---  ---------------  -------------  -----------------------
Container   cpu       -     2    100m             2              -
Container   memory    32Mi  2Gi  128Mi            128Mi          -

$ kubectl -n quota-test run plain --image=nginx:1.29
pod/plain created
$ kubectl -n quota-test get pod plain -o jsonpath='{.spec.containers[0].resources}{"\n"}'
{"limits":{"cpu":"2","memory":"128Mi"},"requests":{"cpu":"100m","memory":"128Mi"}}

$ kubectl -n quota-test run huge --image=nginx:1.29 \
    --overrides='{"spec":{"containers":[{"name":"huge","image":"nginx:1.29","resources":{"requests":{"memory":"4Gi"},"limits":{"memory":"4Gi"}}}]}}'
Error from server (Forbidden): pods "huge" is forbidden: maximum memory usage per Container is 2Gi, but limit is 4Gi
$ kubectl -n quota-test run tiny --image=nginx:1.29 \
    --overrides='{"spec":{"containers":[{"name":"tiny","image":"nginx:1.29","resources":{"requests":{"memory":"16Mi"},"limits":{"memory":"16Mi"}}}]}}'
Error from server (Forbidden): pods "tiny" is forbidden: minimum memory usage per Container is 32Mi, but request is 16Mi

$ kubectl -n quota-test create deployment big --image=nginx:1.29 --replicas=40
deployment.apps/big created
$ kubectl -n quota-test get rs -l app=big
NAME             DESIRED   CURRENT   READY   AGE
big-7b45b7cd5b   40        19        2       21s
$ kubectl -n quota-test describe rs -l app=big | grep -m1 'Warning  FailedCreate'
  Warning  FailedCreate      16s                replicaset-controller  Error creating: pods "big-7b45b7cd5b-l55kz" is forbidden: exceeded quota: budget, requested: pods=1,requests.cpu=100m, used: pods=20,requests.cpu=2, limited: pods=20,requests.cpu=2

$ kubectl -n quota-test describe resourcequota budget
Name:                       budget
Namespace:                  quota-test
Resource                    Used    Hard
--------                    ----    ----
count/jobs.batch            0       10
limits.memory               2560Mi  4Gi
persistentvolumeclaims      0       5
pods                        20      20
requests.cpu                2       2
requests.ephemeral-storage  0       10Gi
requests.memory             2560Mi  4Gi
requests.storage            0       50Gi

The pod without resources got the LimitRange defaults, including the CPU limit from max; the too-large and too-small containers were rejected right away. The 40-replica Deployment hit the quota at the 20th pod — and at the same time requests.cpu: 2 (20 × 100m); the message lists both resources. The refusal goes not to the user but to the ReplicaSet controller: the Deployment is created successfully, and the missing pods are visible only in the ReplicaSet events.

One subtlety: the admission controller learns about quotas from its cache, and in the first seconds after a quota is created it may not see it yet. On the lab a pod created by the same command right after kubectl apply of the quota slipped past it, and Used showed pods 21 with Hard 20. So a quota is created in advance, not in the same script as the pods; the output above was taken with a pause of a few seconds between them.

Let's clean up: kubectl delete namespace quota-test, kubectl delete pod res-demo burner eater.

Sources

Resource Management for Pods and Containers — requests and limits, units, ephemeral-storage, pod-level resources.
Pod Quality of Service Classes — QoS class conditions and Memory QoS.
Node-pressure Eviction — signals, thresholds, eviction order and oom_score_adj.
Reserve Compute Resources for System Daemons — capacity, allocatable, systemReserved, kubeReserved, enforceNodeAllocatable.
Limit Ranges and Resource Quotas — namespace constraints.
Resize CPU and Memory Resources assigned to Containers — in-place resize and resizePolicy.
Control Group v2 — cpu.weight, cpu.max, memory.max, memory.oom.group, memory.events in the Linux kernel documentation.
CFS Bandwidth Control — the CFS quota and period, throttling and cpu.stat.

This is the second time we create a namespace in passing and select pods by labels without stopping to look at what they are.

Next — Namespace, labels and annotations: what a namespace isolates and what it does not, which objects belong to no namespace and what happens when a namespace is deleted, how label and field selectors work, which labels the project recommends and how annotations differ from labels. From it on the training namespace is demo.