Kubernetes for beginners. Part 13: How does the scheduler work?

A pod created by a Deployment is not bound anywhere in the first milliseconds of its life: its spec.nodeName is empty, and no kubelet runs it. Someone has to decide which node it will live on — that is the job of kube-scheduler, one of the control plane components from part 2. Today we look at how it makes that decision: the queue, filtering, scoring, binding; which pod and node fields influence it (nodeSelector, affinity, taints and tolerations, topology spread); what the scheduler does when there is no room (priorities and preemption), and how to take a node out for maintenance without bringing the application down (cordon, drain and PodDisruptionBudget). The resource requests the scheduler uses to count free room come from part 10, labels and selectors from part 11. At the end we spread the replicas of hello (the Deployment from part 7, living in the demo namespace since part 11) across the two workers so that losing one node does not take the whole application with it.

---

What the scheduler does and what it does not

The scheduler is an ordinary API client. On a kubeadm cluster it runs as a static pod from /etc/kubernetes/manifests/kube-scheduler.yaml on k8s-master-1, talks to kube-apiserver with the kubeconfig /etc/kubernetes/scheduler.conf and watches two sets of objects: pods whose spec.nodeName is empty and whose spec.schedulerName equals its name (default-scheduler by default), and everything that affects the decision — nodes, pods on nodes, PVCs, PVs and StorageClasses.

The scheduler's decision is a single write to the API: the pods/binding subresource, a Binding object with the node name. In response kube-apiserver writes spec.nodeName into the pod. The scheduler does nothing more: it does not start containers, does not talk to nodes and does not re-check its decision later. From there the kubelet on the chosen node takes over: it sees in its watch a pod with its own name in spec.nodeName, checks again whether there are enough resources (this is admission on the node: if there really is no room, the pod gets the Failed phase with the reason OutOfcpu or OutOfmemory), and starts it through the CRI (the interface the kubelet uses to drive containerd).

Two consequences worth remembering right away:

• the decision is made once. If the node later becomes overloaded or a better one appears, the scheduler will not move the pod. Moving is deleting a pod and creating a new one; a controller does it (a Deployment, preemption, drain), and the new pod goes through scheduling again;
• if spec.nodeName already has a value, the scheduler does not touch the pod. DaemonSets in old versions relied on this, and so does anyone who wants to bypass the scheduler (more on that below).

Several schedulers can run in a cluster at once, and a pod picks its own with the schedulerName field. Usually one scheduler with several profiles is enough — they come at the end of the next section.

In a kubeadm cluster the scheduler is a single process, but the control plane can be highly available (the series on Kubernetes on BGP), and then there are three instances. So that two of them do not schedule the same pod, leader election is used: the instances compete for the Lease object kube-system/kube-scheduler, and only the one whose name is written in holderIdentity works; the others wait for the leader to stop renewing the lease (the lease lasts 15 seconds).

---

The scheduling cycle

              ┌──────────────────────── queue ───────────────────────────┐
  new pod   → │ activeQ       ready for an attempt, by priority          │
              │ backoffQ      failed attempt, wait 1s, 2s … up to 10s    │
              │ unschedulable wait for an event that may help            │
              │ gated         wait for schedulingGates to be removed     │
              └──────────────────────────────┬───────────────────────────┘
                                             ▼
   scheduling cycle (one pod at a time)
     PreFilter → Filter → PostFilter      which nodes fit at all;
                          (preemption)    if none, try to free one
     PreScore  → Score  → NormalizeScore  which of the fitting ones is best
     Reserve   → Permit                   reserve resources in its cache
                                             │
                                             ▼
   binding cycle (in parallel for different pods)
     PreBind  → Bind → PostBind           wait for volumes, write Binding

The queue. A pod lands in activeQ and is picked from there by priority (priorities come below), with equal priorities by creation time. If an attempt fails, the pod does not spin in the loop: it goes to unschedulable and waits for an event that may change the result — a node appeared, a pod was deleted, a taint was removed. Which events are useful to a particular pod is decided by the plugins themselves through QueueingHint (stable since 1.34): a plugin that refused because of a CPU shortage will say that deleting a pod on a node is a reason to try again, and a change of node labels is not. After such an event the pod goes to backoffQ with a growing delay and then back to activeQ. Every five minutes all stuck pods get another attempt in any case.

A pod can be put into the queue and forbidden from being scheduled with the schedulingGates field (stable since 1.30): while the list of gates is not empty, the pod sits in Pending (with status SchedulingGated in kubectl get pods), lies in a separate gated queue and does not enter the cycle. Gates are set only when the pod is created and can only be removed later; while they are in place, the pod's placement may only be narrowed — by adding a nodeSelector or node affinity terms. External controllers that need to prepare something before placement use this, for example to allocate quota.

Filtering (Filter). For each node the plugins answer "fits" or "no, because…". A node fits when all of them approve. The main filters:

Plugin	What it checks
NodeUnschedulable	the node is not marked spec.unschedulable (kubectl cordon)
NodeName	if the pod sets nodeName, this is that node
NodeResourcesFit	the sum of requests of the pods on the node plus the new pod does not exceed the node's allocatable
NodePorts	the hostPort is free, if the pod asks for one
NodeAffinity	nodeSelector and required node affinity
TaintToleration	the pod tolerates all node taints with the NoSchedule and NoExecute effects
InterPodAffinity	required pod affinity and anti-affinity
PodTopologySpread	spread constraints with DoNotSchedule
VolumeBinding, VolumeZone, NodeVolumeLimits	the pod's volumes can be attached on this node (in detail in part 21)

Important: NodeResourcesFit counts requests, not real consumption. A node where pods requested 1.9 CPU out of 2 is full for the scheduler even if the CPU load is 5%. And the other way round: a node with pods without requests looks empty to the filter, although it may be choking. That is why requests on every container is the first condition for predictable scheduling (part 10).

Checking every node is expensive on a large cluster, so the scheduler stops once it has found enough feasible ones: the share percentageOfNodesToScore is computed from the cluster size by default — 50% for a hundred nodes, 10% for five thousand and never less than 5% — but at least 100 nodes are checked. On clusters of up to a hundred nodes, let alone the three nodes of the lab, all of them are checked.

PostFilter. If no node fits, PostFilter is called — in the default scheduler this is the DefaultPreemption plugin, preemption. More on it below.

Scoring (Score). Each plugin scores the feasible nodes from 0 to 100, the scores are multiplied by the plugin's weight and added up. The node with the highest sum wins; on a tie, a random one of the best. Examples of criteria:

Plugin	Prefers
NodeResourcesFit	by default the LeastAllocated strategy — the node with the largest share of free requested resources; MostAllocated — the opposite, tight packing
NodeResourcesBalancedAllocation	the node where CPU and memory will be used in equal shares after placement
ImageLocality	the node where the image is already pulled
NodeAffinity, InterPodAffinity	nodes that match the soft (preferred) rules, according to their weights
PodTopologySpread	nodes where placement breaks evenness less (ScheduleAnyway)
TaintToleration	nodes with fewer PreferNoSchedule taints the pod does not tolerate

The default weights in 1.36: TaintToleration — 3; NodeAffinity, PodTopologySpread, InterPodAffinity and DynamicResources — 2; NodeResourcesFit, NodeResourcesBalancedAllocation and ImageLocality — 1. DynamicResources handles pods that ask for devices through a ResourceClaim (DRA, dynamic allocation of devices such as GPUs); it gives every node zero for other pods. And one subtlety of scoring: unlike the filter, when NodeResourcesFit scores, a container without requests counts as if it requested 100m CPU and 200 MiB of memory — otherwise a node packed with pods without requests would look like the freest one.

Reserve and Permit. Having chosen a node, the scheduler immediately counts the pod as placed in its cache (assume) — so that the next pod from the queue already sees the resources as taken, without waiting for the write to reach the API and come back through the watch. Permit can hold a pod back (used for gang scheduling, when several pods must start together or not at all).

Binding. The binding cycle runs asynchronously: while VolumeBinding waits for a PV to bind to a PVC for one pod, the scheduler is already processing the next one. The DefaultBinder plugin creates the Binding. If binding fails, the reservation in the cache is rolled back and the pod returns to the queue.

Plugins and profiles

Everything listed above are scheduling framework plugins: the extension points (PreFilter, Filter, Score and so on) are fixed, and each one is implemented by a set of plugins compiled into kube-scheduler or into your own build. The set, the weights and the plugin parameters are set by configuration:

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
clientConnection:
  kubeconfig: /etc/kubernetes/scheduler.conf
profiles:
  - schedulerName: default-scheduler
  - schedulerName: bin-packing
    pluginConfig:
      - name: NodeResourcesFit
        args:
          scoringStrategy:
            type: MostAllocated
            resources:
              - name: cpu
                weight: 1
              - name: memory
                weight: 1
      - name: PodTopologySpread
        args:
          defaultingType: List

These are two profiles in one process: pods with schedulerName: bin-packing are scheduled with the tight packing strategy (handy for batch jobs, to free whole nodes), the rest as usual. The second entry in pluginConfig is needed for the packing to actually work: without it the profile keeps the default soft spread constraints (more on them in the topology spread section), PodTopologySpread has weight 2 against 1 for NodeResourcesFit, and in the lab three bin-packing pods ended up two on one node and one on the other. defaultingType: List with no list of constraints switches these defaults off for the profile, and then all three pods land on one node (the output is in "Hands-on").

The file is passed to kube-scheduler with the --config flag; on kubeadm that means editing the static manifest and mounting the file into the pod. So that kubeadm upgrade does not lose such an edit, it also has to go into ClusterConfiguration: in kubeadm.k8s.io/v1beta4 this is scheduler.extraArgs — a list of name/value pairs, for example name: config, value: /etc/kubernetes/scheduler-config.yaml — and scheduler.extraVolumes with the file itself. On an existing cluster ClusterConfiguration is stored in the kubeadm-config ConfigMap of the kube-system namespace; the lab has scheduler: {} there.

Changing the default profile is rarely needed. Fields in the pod itself are almost always enough, and we move on to them.

---

nodeName and nodeSelector

nodeName names the node directly:

spec:
  nodeName: k8s-node-2

The scheduler does not see a pod with nodeName set at all: the kubelet on k8s-node-2 picks it up right away. Neither taints nor affinity are checked, and resources are checked only by the kubelet at startup; if they are short, the pod immediately gets the Failed phase with the reason OutOfcpu instead of waiting for room in Pending (the output is in "Hands-on"). For applications this is a bad tool: the node name is hard-coded in the manifest, and when the node is replaced the manifest breaks. Its place is debugging and situations when the scheduler itself is unavailable.

nodeSelector is a map of labels the node must have, all at once:

spec:
  nodeSelector:
    kubernetes.io/os: linux
    disktype: ssd

Node labels are set by the kubelet (kubernetes.io/hostname, kubernetes.io/os, kubernetes.io/arch, in the cloud topology.kubernetes.io/zone and region) and by the administrator:

$ kubectl label node k8s-node-1 disktype=ssd
node/k8s-node-1 labeled

The kubelet cannot set labels with the node-role.kubernetes.io/ prefix on itself. This is a restriction of the admission plugin (a kube-apiserver module that checks or amends every object before it is written to etcd) NodeRestriction, which keeps a compromised node from declaring itself control plane and attracting sensitive pods. But other labels outside the kubernetes.io/ and k8s.io/ prefixes — both disktype and example.com/gpu — the kubelet can change on its own node. So labels that bring special or sensitive workloads to a node are better created with the node-restriction.kubernetes.io/ prefix, which NodeRestriction leaves to the administrator only, for example example.com.node-restriction.kubernetes.io/gpu=true.

---

Node affinity

nodeSelector can only say "all labels are equal". Node affinity is the same, but with operators and soft rules:

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: kubernetes.io/arch
                operator: In
                values: ["amd64", "arm64"]
              - key: node-role.kubernetes.io/control-plane
                operator: DoesNotExist
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 80
          preference:
            matchExpressions:
              - key: disktype
                operator: In
                values: ["ssd"]

• required… is a filter: a node without matching labels is excluded. Several nodeSelectorTerms are combined with OR, expressions within one term with AND.
• preferred… is scoring: nodes that match the rule get the weight (1–100) added to their sum. The pod may still end up on a node without an SSD if the SSD nodes have no room.
• Operators: In, NotIn, Exists, DoesNotExist, Gt, Lt (the last two compare the label value as an integer). NotIn and DoesNotExist give anti-affinity to nodes.
• IgnoredDuringExecution in the name is an honest warning: if the label is removed from the node, pods already running there stay. The rule only applies at scheduling time.

If a pod sets both nodeSelector and nodeAffinity, both must be satisfied.

---

Pod affinity and anti-affinity

Sometimes what matters is not which node, but who already lives on it: a cache should be next to the application, two database replicas further apart. For this, rules are expressed relative to other pods:

spec:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              app.kubernetes.io/name: hello
          topologyKey: kubernetes.io/hostname
    podAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 50
          podAffinityTerm:
            labelSelector:
              matchLabels:
                app: redis
            topologyKey: kubernetes.io/hostname

The key field is topologyKey: a node label that divides nodes into topology domains. Nodes with the same label value are one domain. With kubernetes.io/hostname each node is its own domain, and the rule reads "not on the same node where a pod app.kubernetes.io/name=hello already is". With topology.kubernetes.io/zone the domain is a zone, and the same rule means "not in the same zone". The lab has no zones, but they can be modelled with your own label, racks for example: kubectl label node k8s-node-1 topology.kubernetes.io/zone=rack-a.

How it works when checking a candidate node: the scheduler takes the candidate's topologyKey value, finds all nodes with the same value and looks for pods on them matching the labelSelector (by default in the same namespace, otherwise namespaces or namespaceSelector). Affinity needs such pods, anti-affinity forbids them.

Three warnings:

• Required anti-affinity by hostname limits the number of replicas to the number of nodes. A third replica with the rule above on two workers stays in Pending forever (exactly what the anti Deployment shows in the section on Pending). For web applications a soft rule or topology spread (next section) is usually the right choice.
• It is expensive. To check one node the scheduler has to look at the pods of all nodes in the same domain, and recompute the rules when any pod changes. The documentation does not recommend pod affinity and anti-affinity in clusters larger than several hundred nodes.
• The selector must tell versions apart if that matters. During a rolling update the old and new pods of a Deployment share the label app.kubernetes.io/name: hello. The field matchLabelKeys: ["pod-template-hash"] (stable since 1.33) adds the value of the pod's own label to the selector, and the rule applies only to pods of the same revision.

---

Taints and tolerations

Affinity is a pod drawn to nodes. A taint is the opposite, a node that repels pods. A taint is an entry in the node's spec.taints made of a key, a value and an effect; a toleration in a pod says "this taint does not bother me".

$ kubectl taint node k8s-node-2 dedicated=batch:NoSchedule
node/k8s-node-2 tainted

spec:
  tolerations:
    - key: dedicated
      operator: Equal
      value: batch
      effect: NoSchedule

Effect	New pods without a toleration	Running pods without a toleration
NoSchedule	are not placed	stay
PreferNoSchedule	are placed only if there is nowhere else	stay
NoExecute	are not placed	are evicted (immediately or after tolerationSeconds)

The Equal operator requires the value to match, Exists only the key. A toleration with an empty key and operator: Exists tolerates everything — system DaemonSets (CNI, log collection) that must run on every node do this. For applications it is bad practice: such a pod ends up on the control plane and on a node that is being taken out of service. (1.35 added the numeric operators Gt and Lt for tolerations, but it is an alpha feature, disabled by default in 1.36 too.)

The main thing to understand: a toleration permits, it does not attract. A pod tolerating dedicated=batch can end up on any node: in the lab, with this taint on k8s-node-2, four pods without the toleration went to k8s-node-1, while four pods with it spread over both nodes. To dedicate nodes to a workload you need both mechanisms: the taint keeps strangers out, node affinity (or nodeSelector) brings your own in.

The control plane taint

kubeadm puts a taint on the master during init:

$ kubectl describe node k8s-master-1 | grep -A1 Taints
Taints:             node-role.kubernetes.io/control-plane:NoSchedule
Unschedulable:      false

That is why application pods do not get onto k8s-master-1, while CoreDNS, kube-proxy and calico-node do: they have the matching tolerations. Removing this taint (kubectl taint node k8s-master-1 node-role.kubernetes.io/control-plane:NoSchedule-, with a minus at the end) only makes sense in a single-node training cluster. Otherwise the control plane should stay free of applications: a pod that ate the memory can push kube-apiserver or etcd into OOM, and the whole cluster suffers.

Taints from the node lifecycle controller

Some taints are set by Kubernetes itself. The node lifecycle controller in kube-controller-manager checks every 5 seconds (--node-monitor-period) when a node last reported. The kubelet reports in two ways: it renews its Lease in the kube-node-lease namespace (every 10 seconds) and updates the node's status.conditions. If there are no reports for longer than --node-monitor-grace-period (50 seconds by default), the controller sets the Ready condition to Unknown and puts taints on the node:

Taint	When
node.kubernetes.io/not-ready	the kubelet reports Ready=False (for example, the CNI network plugin or containerd is not working)
node.kubernetes.io/unreachable	no word from the kubelet, Ready=Unknown
node.kubernetes.io/unschedulable	the node is cordoned
node.kubernetes.io/memory-pressure, disk-pressure, pid-pressure	the kubelet reports a resource shortage
node.kubernetes.io/network-unavailable	the node's network is not configured

The unschedulable taint is also set by the controller, not by kubectl: right after kubectl cordon the spec.taints field is still empty, the taint with a timeAdded mark appears a moment later and likewise disappears with a delay after uncordon.

not-ready and unreachable are set with two effects: NoSchedule (new pods do not come) and NoExecute (old ones are evicted). And so that a short disappearance of a node — a reboot, a flapping switch — does not trigger a mass move, the DefaultTolerationSeconds admission plugin adds two tolerations to every pod that has none of its own:

$ kubectl -n demo get pod hello-6d5c6db6b4-247xb -o jsonpath='{.spec.tolerations}' | jq
[
  {
    "effect": "NoExecute",
    "key": "node.kubernetes.io/not-ready",
    "operator": "Exists",
    "tolerationSeconds": 300
  },
  {
    "effect": "NoExecute",
    "key": "node.kubernetes.io/unreachable",
    "operator": "Exists",
    "tolerationSeconds": 300
  }
]

So from a node failure to the deletion of its pods about 50 + 300 seconds pass — almost six minutes. Only then does the ReplicaSet see that there are fewer pods than needed and create replacements on live nodes. This default (the kube-apiserver flags --default-not-ready-toleration-seconds and --default-unreachable-toleration-seconds) is a compromise; for a stateless application with several replicas it can be reduced in the pod template (for example to 30–60 seconds), for a pod with its own volume on the node it can be increased. A DaemonSet gets these tolerations without tolerationSeconds: its pods are never evicted from a node, because they have nowhere to go.

A separate note on StatefulSets: the controller does not automatically recreate a pod on an unreachable node until the old one is deleted for good, otherwise you could get two instances with one volume (part 8).

---

Topology spread constraints

Anti-affinity says "no more than one per domain". More often something else is needed: "roughly equally across domains". That is what topologySpreadConstraints are for:

spec:
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: kubernetes.io/hostname
      whenUnsatisfiable: DoNotSchedule
      labelSelector:
        matchLabels:
          app.kubernetes.io/name: hello
      matchLabelKeys:
        - pod-template-hash

• maxSkew is the allowed skew: the difference between the number of matching pods in the fullest and the emptiest domain after placement. With 1 on two nodes four replicas lie 2 + 2, five lie 3 + 2.
• topologyKey is, as in affinity, the label that defines the domains.
• whenUnsatisfiable is DoNotSchedule (a filter: if the placement breaks the skew, the node does not fit) or ScheduleAnyway (scoring: prefer nodes where the skew is smaller).
• labelSelector says which pods to count. Usually the pods of the same application.
• matchLabelKeys — as in affinity, label keys whose values are taken from the pod itself (here it is beta, enabled by default). With pod-template-hash the old and new revisions are counted separately, and a rolling update does not hit the skew created by the departing pods.
• minDomains is the minimum number of domains (stable since 1.30, only with DoNotSchedule): if there are fewer domains with fitting nodes, the missing ones count as empty, and pods do not pile into the existing ones.
• nodeAffinityPolicy and nodeTaintsPolicy (stable since 1.33) decide whether to count, when computing domains, the nodes the pod cannot go to anyway because of affinity (Honor by default) or taints (Ignore by default). Honor for taints is useful: otherwise the control plane with its taint counts as a domain with zero pods, and the skew computed from it will not let replicas be placed on the workers — the lab shows this below.

If a pod has no constraints of its own and belongs to a Service, ReplicaSet, StatefulSet or ReplicationController, the scheduler applies default constraints to it: a soft spread by kubernetes.io/hostname with maxSkew: 3 and by topology.kubernetes.io/zone with maxSkew: 5, with the selector taken from that service or controller. This explains why Deployment replicas usually spread across nodes anyway — but only "usually": the constraint is soft, with maxSkew: 3 a 4 + 0 layout is allowed, and a node with the image already pulled and free resources can outweigh it.

---

Priorities and preemption

When the cluster is full, someone has to be left without room. A PriorityClass says who matters more:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: business-critical
value: 100000
globalDefault: false
preemptionPolicy: PreemptLowerPriority
description: "User-facing services that may preempt background jobs"
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: batch-low
value: 1000
preemptionPolicy: Never
description: "Background jobs: wait for room, never preempt anyone"

A pod refers to a class with the priorityClassName field, and at creation the Priority admission plugin writes the number into spec.priority (a pod with a non-existent class is rejected). User classes go up to one billion; above that lie two built-in ones: system-cluster-critical (2 000 000 000, used by CoreDNS and calico-typha) and system-node-critical (2 000 001 000, used by kube-proxy, calico-node and the static control plane pods). A class with globalDefault: true applies to pods without priorityClassName; otherwise they get 0.

Priority works in two places:

1. The queue. A pod with a higher priority is picked from activeQ earlier.
2. Preemption. If no node was found for a pod, the PostFilter plugin DefaultPreemption looks for a node where removing lower-priority pods frees enough room. Among the candidates it chooses the one with fewer victims and lower victim priority, and where, if possible, no PodDisruptionBudget is violated. The victims are deleted gracefully (with terminationGracePeriodSeconds and a Preempted event), and the preempting pod gets status.nominatedNodeName — the node is reserved for it, and the scheduler takes that into account when placing other pods. Once room frees up, the pod goes through the normal cycle and most likely lands there; nominatedNodeName is then cleared.

preemptionPolicy: Never is for a class that needs priority only for queue order: a background job goes ahead of other background jobs but does not touch running ones.

Preemption respects PDBs when it can, but does not guarantee it: if there is no other way, the budget will be violated. And one more trap: if you hand out high priority to all applications "just in case", preemption stops meaning anything. There should be few classes, two or three, each with a clear description of who is entitled to it; the use of high classes can be limited by a quota — the ResourceQuota from part 10 with a scopeSelector on PriorityClass. How this looks in the lab is in "Hands-on".

---

Node maintenance: cordon, drain, PodDisruptionBudget

Nodes have to be taken out of service: update the kernel, replace a disk, upgrade Kubernetes (part 35). Three commands:

• kubectl cordon sets spec.unschedulable: true on the node. New pods are not scheduled there (the NodeUnschedulable filter and the taint node.kubernetes.io/unschedulable:NoSchedule), running ones keep running.
• kubectl drain is cordon plus eviction of all pods from the node.
• kubectl uncordon removes unschedulable. Pods do not come back by themselves: the scheduler moves nothing, new pods simply become able to land on the node again.

drain does not delete pods directly; for each one it calls the Eviction API — creates the pods/eviction subresource. The difference is fundamental: before deleting the pod, kube-apiserver checks all the PodDisruptionBudgets that select it. If the eviction would violate a budget, the API answers 429 Too Many Requests, and drain retries every 5 seconds until the controller brings up a replacement somewhere else.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: hello
  namespace: demo
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: hello
  unhealthyPodEvictionPolicy: AlwaysAllow

• minAvailable or maxUnavailable (one of the two, a number or a percentage; percentages are rounded up) — how many pods must stay ready, or how many may be missing.
• selector says which pods the budget protects. It should match the pods of one controller; if several budgets select one pod, the Eviction API answers with error 500 and the pod cannot be evicted.
• unhealthyPodEvictionPolicy (stable since 1.31): IfHealthyBudget (the default) allows evicting unready pods only if the budget is satisfied anyway; AlwaysAllow always allows evicting unready pods. The second is almost always better: a pod in CrashLoopBackOff serves no traffic, and with the default it can block drain forever.

The disruption controller in kube-controller-manager keeps recomputing the budget status:

$ kubectl -n demo get pdb hello
NAME    MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
hello   N/A             1                 1                     3s

ALLOWED DISRUPTIONS is how many pods can be evicted right now (status.disruptionsAllowed). Zero means drain will wait. For a Deployment maxUnavailable is more convenient than minAvailable: the budget does not need recomputing when the number of replicas changes, and when scaled down to one replica it does not block maintenance forever.

A PDB only protects against voluntary disruptions — the ones that go through the Eviction API: drain, node autoscaling, preemption. It does not protect against a node failure, OOM, kubectl delete pod or an update of the Deployment itself (that has its own maxUnavailable).

Typical mistakes:

• minAvailable: 100% or maxUnavailable: 0 — drain never finishes. For a single instance this is done deliberately when a manual procedure is needed before shutdown, but then it is part of the runbook, not a "just in case" value.
• A PDB on a one-replica Deployment with minAvailable: 1 — the same. Either two replicas or no PDB.
• A budget that holds only with all replicas on one node. minAvailable: 2 with three replicas on one node — drain evicts one, the second waits until the first comes up on another node. This is correct, but slow; spreading across nodes (next section) makes the drain fast.

---

Spreading hello across the two workers

By now three hello replicas from part 11 run in demo. Let's scale them to four and see where they landed:

$ kubectl -n demo scale deployment hello --replicas=4
deployment.apps/hello scaled
$ kubectl -n demo get pods -l app.kubernetes.io/name=hello -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP              NODE         NOMINATED NODE   READINESS GATES
hello-6d5c6db6b4-247xb   1/1     Running   0          8s    172.20.211.12   k8s-node-1   <none>           <none>
hello-6d5c6db6b4-2czsx   1/1     Running   0          25s   172.20.172.34   k8s-node-2   <none>           <none>
hello-6d5c6db6b4-ntb9b   1/1     Running   0          25s   172.20.211.7    k8s-node-1   <none>           <none>
hello-6d5c6db6b4-ssg42   1/1     Running   0          25s   172.20.172.29   k8s-node-2   <none>           <none>

Two and two: on empty workers LeastAllocated and the default soft spread split the pods evenly. But these are soft rules, and the decision is made once. Let's see what is left after ordinary maintenance: while the pods are restarted, k8s-node-1 is closed for scheduling.

$ kubectl cordon k8s-node-1
node/k8s-node-1 cordoned
$ kubectl -n demo rollout restart deployment hello
deployment.apps/hello restarted
$ kubectl -n demo rollout status deployment hello
...
deployment "hello" successfully rolled out
$ kubectl uncordon k8s-node-1
node/k8s-node-1 uncordoned
$ kubectl -n demo get pods -l app.kubernetes.io/name=hello -o wide
NAME                     READY   STATUS        RESTARTS   AGE   IP              NODE         NOMINATED NODE   READINESS GATES
hello-6d5c6db6b4-ntb9b   1/1     Terminating   0          67s   172.20.211.7    k8s-node-1   <none>           <none>
hello-6d95689dbb-gfld8   1/1     Running       0          13s   172.20.172.38   k8s-node-2   <none>           <none>
hello-6d95689dbb-ng8hq   1/1     Running       0          22s   172.20.172.33   k8s-node-2   <none>           <none>
hello-6d95689dbb-p86qx   1/1     Running       0          23s   172.20.172.35   k8s-node-2   <none>           <none>
hello-6d95689dbb-z9jv5   1/1     Running       0          9s    172.20.172.36   k8s-node-2   <none>           <none>

All four new pods are on k8s-node-2, the last old one is finishing, and nothing changes after uncordon: the default maxSkew: 3 allows this layout. Losing k8s-node-2 would now take the whole application down. Let's add a hard spread across nodes to the pod template, a soft anti-affinity as an extra argument and a shorter wait for an unreachable node (the labels are the same as in part 11):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello
  namespace: demo
  labels: &labels
    app.kubernetes.io/name: hello
    app.kubernetes.io/instance: hello
    app.kubernetes.io/version: "1.0"
    app.kubernetes.io/component: web
    app.kubernetes.io/part-of: k8s-for-beginners
    app.kubernetes.io/managed-by: kubectl
  annotations:
    example.com/source: https://git.example.com/demo/hello
spec:
  replicas: 4
  selector:
    matchLabels:
      app.kubernetes.io/name: hello
      app.kubernetes.io/instance: hello
  template:
    metadata:
      labels: *labels
    spec:
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: DoNotSchedule
        nodeTaintsPolicy: Honor
        labelSelector:
          matchLabels:
            app.kubernetes.io/name: hello
        matchLabelKeys:
        - pod-template-hash
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              topologyKey: kubernetes.io/hostname
              labelSelector:
                matchLabels:
                  app.kubernetes.io/name: hello
      tolerations:
      - key: node.kubernetes.io/unreachable
        operator: Exists
        effect: NoExecute
        tolerationSeconds: 60
      - key: node.kubernetes.io/not-ready
        operator: Exists
        effect: NoExecute
        tolerationSeconds: 60
      containers:
      - name: hello
        image: ghcr.io/example/hello:1.0
        ports:
        - name: http
          containerPort: 8080
        resources:
          requests:
            cpu: 100m
            memory: 64Mi
          limits:
            memory: 64Mi
        readinessProbe:
          httpGet:
            path: /readyz
            port: http

nodeTaintsPolicy: Honor is not decoration here. First let's apply the same manifest without this line (the file hello-ignore.yaml):

$ kubectl apply -f hello-ignore.yaml
deployment.apps/hello configured
$ kubectl -n demo get pods -l app.kubernetes.io/name=hello -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP              NODE         NOMINATED NODE   READINESS GATES
hello-6d95689dbb-p86qx   1/1     Running   0          68s   172.20.172.35   k8s-node-2   <none>           <none>
hello-7c4f947b68-5xngf   0/1     Pending   0          14s   <none>          <none>       <none>           <none>
hello-7c4f947b68-j5dq7   1/1     Running   0          24s   172.20.172.45   k8s-node-2   <none>           <none>
hello-7c4f947b68-t2hlc   1/1     Running   0          25s   172.20.211.13   k8s-node-1   <none>           <none>
hello-7c4f947b68-xslgl   0/1     Pending   0          17s   <none>          <none>       <none>           <none>
$ kubectl -n demo describe pod hello-7c4f947b68-5xngf
...
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  14s   default-scheduler  0/3 nodes are available: 1 node(s) had untolerated taint(s), 2 node(s) didn't match pod topology spread constraints. no new claims to deallocate, preemption: 0/3 nodes are available: 1 Preemption is not helpful for scheduling, 2 No preemption victims found for incoming pod.

The rollout is stuck: with the default Ignore, k8s-master-1 counts as a third domain with zero hello pods, and with maxSkew: 1 each worker can take only one pod of the new revision. Two new-revision pods wait in Pending, and the last old one stays because the Deployment's maxUnavailable does not let it drop below three ready replicas. Now apply the manifest as a whole:

$ kubectl apply -f hello-deployment.yaml
deployment.apps/hello configured
$ kubectl -n demo rollout status deployment hello
...
deployment "hello" successfully rolled out
$ kubectl -n demo get pods -l app.kubernetes.io/name=hello -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP              NODE         NOMINATED NODE   READINESS GATES
hello-67479df8df-9dbfx   1/1     Running   0          36s   172.20.211.14   k8s-node-1   <none>           <none>
hello-67479df8df-9qg9b   1/1     Running   0          36s   172.20.172.46   k8s-node-2   <none>           <none>
hello-67479df8df-c2l78   1/1     Running   0          28s   172.20.211.3    k8s-node-1   <none>           <none>
hello-67479df8df-q4c8v   1/1     Running   0          25s   172.20.172.39   k8s-node-2   <none>           <none>

Two and two, and now it is a rule rather than chance. Let's add the PDB from the previous section and take k8s-node-1 out for maintenance:

$ kubectl apply -f hello-pdb.yaml
poddisruptionbudget.policy/hello created
$ kubectl drain k8s-node-1 --ignore-daemonsets --delete-emptydir-data
node/k8s-node-1 cordoned
Warning: ignoring DaemonSet-managed Pods: calico-system/calico-node-6d2qb, calico-system/csi-node-driver-q6t25, kube-system/kube-proxy-xmjz2
evicting pod demo/hello-67479df8df-c2l78
evicting pod calico-system/calico-typha-55d594d885-hxzs8
evicting pod demo/hello-67479df8df-9dbfx
error when evicting pods/"hello-67479df8df-9dbfx" -n "demo" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
pod/calico-typha-55d594d885-hxzs8 evicted
evicting pod demo/hello-67479df8df-9dbfx
error when evicting pods/"hello-67479df8df-9dbfx" -n "demo" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
pod/hello-67479df8df-c2l78 evicted
evicting pod demo/hello-67479df8df-9dbfx
pod/hello-67479df8df-9dbfx evicted
node/k8s-node-1 drained

What happened: drain cordoned the node and started evicting all pods at once. The first hello eviction went through (the budget allowed one), the second at the same moment did not. After 5 seconds, again no: the replacement of the first pod had not passed its readiness probe yet, and without the second only two of four pods would have been ready. The line pod/… evicted is printed when the pod is really gone, that is after its graceful termination. The third attempt went through, and the application served requests with at least three replicas the whole time. A calico-typha replica left the node too — it has its own PDB, and the Tigera operator also reduces the number of its replicas while a node is cordoned (part 8).

The flags: --ignore-daemonsets — do not evict DaemonSet pods (the controller would immediately recreate them on the same node, because they tolerate unschedulable); --delete-emptydir-data — consent to losing the contents of emptyDir volumes. Pods not managed by any controller drain refuses to evict without --force: nobody would recreate them.

But what about the DoNotSchedule spread with maxSkew: 1, if nothing may go to k8s-node-1? Here nodeTaintsPolicy: Honor helps again: the cordoned node carries the taint node.kubernetes.io/unschedulable, which hello pods do not tolerate, so it is excluded from the domain count, one domain remains, and all four pods legitimately end up on k8s-node-2. With Ignore the replacements would have been stuck in Pending, as in the experiment above, and drain would have waited forever. Let's bring the node back:

$ kubectl uncordon k8s-node-1
node/k8s-node-1 uncordoned
$ kubectl -n demo get pods -l app.kubernetes.io/name=hello -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP              NODE         NOMINATED NODE   READINESS GATES
hello-67479df8df-9qg9b   1/1     Running   0          65s   172.20.172.46   k8s-node-2   <none>           <none>
hello-67479df8df-c7n4b   1/1     Running   0          13s   172.20.172.37   k8s-node-2   <none>           <none>
hello-67479df8df-q4c8v   1/1     Running   0          54s   172.20.172.39   k8s-node-2   <none>           <none>
hello-67479df8df-z5r8f   1/1     Running   0          24s   172.20.172.42   k8s-node-2   <none>           <none>

The pods did not come back by themselves. To spread them again, restarting the Deployment is enough: the new pods follow the rule.

$ kubectl -n demo rollout restart deployment hello
deployment.apps/hello restarted
$ kubectl -n demo get pods -l app.kubernetes.io/name=hello -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP              NODE         NOMINATED NODE   READINESS GATES
hello-6df567dbbb-7vqns   1/1     Running   0          20s   172.20.211.16   k8s-node-1   <none>           <none>
hello-6df567dbbb-jvgh5   1/1     Running   0          17s   172.20.172.44   k8s-node-2   <none>           <none>
hello-6df567dbbb-p4rc5   1/1     Running   0          29s   172.20.211.15   k8s-node-1   <none>           <none>
hello-6df567dbbb-s2mf9   1/1     Running   0          28s   172.20.172.40   k8s-node-2   <none>           <none>

For automatic rebalancing there is a separate project, descheduler — a controller that, following configured policies, evicts pods that break the spread or affinity so that the scheduler places them again; it is not part of a standard cluster.

---

Pending and events

When a pod cannot be placed, the scheduler does not stay silent. It writes the reason into a pod condition and an event with a per-node breakdown. Let's create a separate Deployment anti of three nginx:1.29 replicas with required anti-affinity by kubernetes.io/hostname to its own pods (the label app: anti): two replicas go to the workers, there is no room for the third.

$ kubectl -n demo get pods -l app=anti -o wide
NAME                    READY   STATUS    RESTARTS   AGE   IP              NODE         NOMINATED NODE   READINESS GATES
anti-79d5d64c98-9rmgd   0/1     Pending   0          20s   <none>          <none>       <none>           <none>
anti-79d5d64c98-jszc8   1/1     Running   0          20s   172.20.172.47   k8s-node-2   <none>           <none>
anti-79d5d64c98-zlmnb   1/1     Running   0          20s   172.20.211.18   k8s-node-1   <none>           <none>
$ kubectl -n demo describe pod anti-79d5d64c98-9rmgd
...
Conditions:
  Type           Status
  PodScheduled   False 
...
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  20s   default-scheduler  0/3 nodes are available: 1 node(s) had untolerated taint(s), 2 node(s) didn't match pod anti-affinity rules. no new claims to deallocate, preemption: 0/3 nodes are available: 1 Preemption is not helpful for scheduling, 2 No preemption victims found for incoming pod.

The message reads as a sum of filter refusals over all nodes: the master refused because of the taint (in 1.36 the taint name is not printed in the message), both workers because of anti-affinity. The phrase no new claims to deallocate belongs to DRA: there is nothing to release. The second part is the result of the preemption attempt: on the master it is useless (the taint will not go away), on the workers there are no victims whose removal would help (a pod of the same priority cannot be preempted). The same text is in the pod's PodScheduled condition with the reason Unschedulable. Typical phrases:

Message fragment	Cause
Insufficient cpu, Insufficient memory	requests do not fit into allocatable minus the requests of other pods
had untolerated taint(s)	no toleration for a node taint
didn't match Pod's node affinity/selector	nodeSelector or required node affinity
didn't match pod anti-affinity rules, didn't match pod affinity rules	pod affinity rules
didn't match pod topology spread constraints	DoNotSchedule with maxSkew violated
node(s) were unschedulable	the node is cordoned
didn't have free ports for the requested pod ports	the hostPort is taken
pod has unbound immediate PersistentVolumeClaims	the PVC is not bound yet (part 21)

A node can refuse for several reasons at once, but in the summary it is counted once — by the first filter that refused. That is why the master always lands in untolerated taint(s), even if the pod would not match it by selector either.

How much of a node's resources is already requested is visible in describe node:

$ kubectl describe node k8s-node-2
...
Allocatable:
  cpu:                2
  ephemeral-storage:  29271151363
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             3377252Ki
  pods:               110
...
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests    Limits
  --------           --------    ------
  cpu                300m (15%)  0 (0%)
  memory             256Mi (7%)  256Mi (7%)
  ephemeral-storage  0 (0%)      0 (0%)
  hugepages-1Gi      0 (0%)      0 (0%)
  hugepages-2Mi      0 (0%)      0 (0%)

Two hello pods with 100m and 64 Mi each and one anti with 100m and 128 Mi from the LimitRange of the demo namespace (part 11); the Calico and kube-proxy pods request no resources.

---

Hands-on

Who is the scheduler leader right now and how often it renews the lease:

$ kubectl -n kube-system get lease kube-scheduler -o yaml | grep -E 'holderIdentity|renewTime|leaseDurationSeconds'
  holderIdentity: k8s-master-1_f994e4e0-ad31-4daa-abc4-db19947249d2
  leaseDurationSeconds: 15
  renewTime: "2026-10-28T15:43:11.754706Z"

Binding from the inside: if you create a pod with a schedulerName that does not exist in the cluster, it hangs in Pending without events — nobody schedules it. Let's bind it by hand with the same request the scheduler makes:

$ kubectl -n demo run manual --image=nginx:1.29 --overrides='{"spec":{"schedulerName":"nobody"}}'
pod/manual created
$ kubectl -n demo get pod manual
NAME     READY   STATUS    RESTARTS   AGE
manual   0/1     Pending   0          6s
$ kubectl -n demo describe pod manual
...
Events:                      <none>
$ kubectl create --raw /api/v1/namespaces/demo/pods/manual/binding -f - <<'EOF'
{"apiVersion":"v1","kind":"Binding","metadata":{"name":"manual"},
 "target":{"apiVersion":"v1","kind":"Node","name":"k8s-node-2"}}
EOF
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Success","code":201}
$ kubectl -n demo get pod manual -o wide
NAME     READY   STATUS    RESTARTS   AGE   IP              NODE         NOMINATED NODE   READINESS GATES
manual   1/1     Running   0          14s   172.20.172.60   k8s-node-2   <none>           <none>

That is the scheduler's entire "output": a Binding. Everything else is computation inside it. Such a pod gets no Scheduled event — the scheduler writes it, and it took no part here. A pod cannot be bound a second time:

$ kubectl create --raw /api/v1/namespaces/demo/pods/manual/binding -f - <<'EOF'
{"apiVersion":"v1","kind":"Binding","metadata":{"name":"manual"},
 "target":{"apiVersion":"v1","kind":"Node","name":"k8s-node-1"}}
EOF
Error from server (Conflict): Operation cannot be fulfilled on pods/binding "manual": pod manual is already assigned to node "k8s-node-2"
$ kubectl -n demo delete pod manual
pod "manual" deleted from demo namespace

Selectors and affinity. Four pods on the pause image (it is on every node — it is the pod's pause container): ssd with the nodeSelector from the section above, nvme with nodeSelector: {disktype: nvme}, which no node has, aff with the node affinity from its section, and near-redis with a required pod affinity to app: redis pods, of which there are none in the cluster. The label disktype=ssd is already on k8s-node-1:

$ kubectl apply -f sel.yaml
pod/ssd created
pod/nvme created
pod/aff created
pod/near-redis created
$ kubectl -n demo get pods ssd nvme aff near-redis -o wide
NAME         READY   STATUS    RESTARTS   AGE   IP              NODE         NOMINATED NODE   READINESS GATES
ssd          1/1     Running   0          9s    172.20.211.17   k8s-node-1   <none>           <none>
nvme         0/1     Pending   0          9s    <none>          <none>       <none>           <none>
aff          1/1     Running   0          9s    172.20.211.20   k8s-node-1   <none>           <none>
near-redis   0/1     Pending   0          9s    <none>          <none>       <none>           <none>
$ kubectl -n demo get events --field-selector involvedObject.name=nvme -o custom-columns=R:.reason,M:.message
R                  M
FailedScheduling   0/3 nodes are available: 1 node(s) had untolerated taint(s), 2 node(s) didn't match Pod's node affinity/selector. no new claims to deallocate, preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.
$ kubectl -n demo get events --field-selector involvedObject.name=near-redis -o custom-columns=R:.reason,M:.message
R                  M
FailedScheduling   0/3 nodes are available: 1 node(s) had untolerated taint(s), 2 node(s) didn't match pod affinity rules. no new claims to deallocate, preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.

aff landed on k8s-node-1 by the soft rule: after normalization the SSD node gets the top score from NodeAffinity, the plugin's weight of 2 doubles it, and that outweighs the rest. For nvme and near-redis preemption is useless on all three nodes: removing other pods will not add a label to a node or create a redis pod.

Taints on live nodes. Two Deployments of four pods each: plain without tolerations and batch with a toleration for dedicated=batch:NoSchedule:

$ kubectl taint node k8s-node-2 dedicated=batch:NoSchedule
node/k8s-node-2 tainted
$ kubectl apply -f taint.yaml
deployment.apps/plain created
deployment.apps/batch created
$ kubectl -n demo get pods -l 'app in (plain,batch)' -o wide
NAME                     READY   STATUS              RESTARTS   AGE   IP              NODE         NOMINATED NODE   READINESS GATES
batch-5cb5d7476d-cxjpx   0/1     ContainerCreating   0          10s   <none>          k8s-node-1   <none>           <none>
batch-5cb5d7476d-fnnzh   1/1     Running             0          10s   172.20.172.43   k8s-node-2   <none>           <none>
batch-5cb5d7476d-rr2gm   0/1     ContainerCreating   0          10s   <none>          k8s-node-1   <none>           <none>
batch-5cb5d7476d-slht7   0/1     ContainerCreating   0          11s   <none>          k8s-node-2   <none>           <none>
plain-7b575454-76mnm     0/1     ContainerCreating   0          11s   <none>          k8s-node-1   <none>           <none>
plain-7b575454-gkqxw     0/1     ContainerCreating   0          11s   <none>          k8s-node-1   <none>           <none>
plain-7b575454-rk8m6     0/1     ContainerCreating   0          11s   <none>          k8s-node-1   <none>           <none>
plain-7b575454-xn4zp     0/1     ContainerCreating   0          11s   <none>          k8s-node-1   <none>           <none>
$ kubectl delete -f taint.yaml
deployment.apps "plain" deleted from demo namespace
deployment.apps "batch" deleted from demo namespace
$ kubectl taint node k8s-node-2 dedicated=batch:NoSchedule-
node/k8s-node-2 untainted

All plain pods are on k8s-node-1, batch on both: a toleration permits, it does not attract. Now a soft taint on k8s-node-1 and four pods without tolerations:

$ kubectl taint node k8s-node-1 maintenance=soon:PreferNoSchedule
node/k8s-node-1 tainted
$ kubectl -n demo create deployment soft --image=registry.k8s.io/pause:3.10.2 --replicas=4
deployment.apps/soft created
$ kubectl -n demo get pods -l app=soft -o wide
NAME                    READY   STATUS              RESTARTS   AGE   IP       NODE         NOMINATED NODE   READINESS GATES
soft-7ddd9f8d46-57c9h   0/1     ContainerCreating   0          8s    <none>   k8s-node-2   <none>           <none>
soft-7ddd9f8d46-9prdb   0/1     ContainerCreating   0          8s    <none>   k8s-node-2   <none>           <none>
soft-7ddd9f8d46-lchnp   0/1     ContainerCreating   0          8s    <none>   k8s-node-2   <none>           <none>
soft-7ddd9f8d46-r64bt   0/1     ContainerCreating   0          10s   <none>   k8s-node-2   <none>           <none>
$ kubectl -n demo delete deployment soft
deployment.apps "soft" deleted from demo namespace
$ kubectl taint node k8s-node-1 maintenance=soon:PreferNoSchedule-
node/k8s-node-1 untainted

All four on k8s-node-2: k8s-node-1 had room, but TaintToleration with weight 3 outweighed evenness.

Scheduling gates. A pod with schedulingGates and attempts to change it:

apiVersion: v1
kind: Pod
metadata:
  name: gated
  namespace: demo
spec:
  schedulingGates:
  - name: example.com/quota-check
  containers:
  - name: pause
    image: registry.k8s.io/pause:3.10.2

$ kubectl apply -f gated.yaml
pod/gated created
$ kubectl -n demo get pod gated
NAME    READY   STATUS            RESTARTS   AGE
gated   0/1     SchedulingGated   0          3s
$ kubectl -n demo get pod gated -o jsonpath='{.status.conditions}{"\n"}'
[{"lastProbeTime":null,"lastTransitionTime":"2026-10-28T15:43:40Z","message":"Scheduling is blocked due to non-empty scheduling gates","reason":"SchedulingGated","status":"False","type":"PodScheduled"}]
$ kubectl -n demo patch pod gated --type=merge -p '{"spec":{"nodeSelector":{"disktype":"ssd"}}}'
pod/gated patched
$ kubectl -n demo patch pod gated --type=json -p '[{"op":"add","path":"/spec/schedulingGates/-","value":{"name":"x"}}]'
The Pod "gated" is invalid: spec.schedulingGates[1].name: Forbidden: only deletion is allowed, but found new scheduling gate 'x'
$ kubectl -n demo patch pod gated --type=json -p '[{"op":"remove","path":"/spec/schedulingGates"}]'
pod/gated patched
$ kubectl -n demo get pod gated -o wide
NAME    READY   STATUS    RESTARTS   AGE   IP              NODE         NOMINATED NODE   READINESS GATES
gated   1/1     Running   0          7s    172.20.211.31   k8s-node-1   <none>           <none>
$ kubectl -n demo delete pod gated
pod "gated" deleted from demo namespace

nodeSelector cannot be changed in a running pod, but it can in a gated one: the pod is not placed yet. New gates cannot be added, only removed. Once the gate was removed, the pod went straight to k8s-node-1 — where the added selector now sends it.

Priorities and preemption — on requests, with no real load. We create the business-critical and batch-low classes from the section above, a Deployment filler of two pause pods with priorityClassName: batch-low and a request of cpu: 1500m — one per worker, after which each has less than 300m free — and a pod important with priorityClassName: business-critical and a request of cpu: "1":

$ kubectl apply -f priority.yaml
priorityclass.scheduling.k8s.io/business-critical created
priorityclass.scheduling.k8s.io/batch-low created
$ kubectl get priorityclass
NAME                      VALUE        GLOBAL-DEFAULT   AGE    PREEMPTIONPOLICY
batch-low                 1000         false            0s     Never
business-critical         100000       false            0s     PreemptLowerPriority
system-cluster-critical   2000000000   false            4d5h   PreemptLowerPriority
system-node-critical      2000001000   false            4d5h   PreemptLowerPriority
$ kubectl apply -f filler.yaml
deployment.apps/filler created
$ kubectl -n demo get pods -l app=filler -o wide
NAME                      READY   STATUS    RESTARTS   AGE   IP              NODE         NOMINATED NODE   READINESS GATES
filler-5775b46cdb-jdl6m   1/1     Running   0          6s    172.20.172.57   k8s-node-2   <none>           <none>
filler-5775b46cdb-kpt2p   1/1     Running   0          6s    172.20.211.33   k8s-node-1   <none>           <none>
$ kubectl apply -f important.yaml
pod/important created
$ kubectl -n demo get pods -l app=filler -o wide
NAME                      READY   STATUS    RESTARTS   AGE   IP              NODE         NOMINATED NODE   READINESS GATES
filler-5775b46cdb-c78g8   0/1     Pending   0          9s    <none>          <none>       <none>           <none>
filler-5775b46cdb-kpt2p   1/1     Running   0          16s   172.20.211.33   k8s-node-1   <none>           <none>
$ kubectl -n demo get pod important -o wide
NAME        READY   STATUS    RESTARTS   AGE   IP              NODE         NOMINATED NODE   READINESS GATES
important   1/1     Running   0          10s   172.20.172.55   k8s-node-2   <none>           <none>
$ kubectl -n demo get events --sort-by=.lastTimestamp | grep -E 'important|filler|Preempt'
...
10s         Warning   FailedScheduling       pod/important                  0/3 nodes are available: 1 node(s) had untolerated taint(s), 2 Insufficient cpu.
9s          Normal    Preempted              pod/filler-5775b46cdb-jdl6m    Preempted by pod 28de96fc-8832-40ee-85db-ab7e77867e42 on node k8s-node-2
9s          Normal    Killing                pod/filler-5775b46cdb-jdl6m    Stopping container pause
7s          Warning   FailedScheduling       pod/filler-5775b46cdb-c78g8    0/3 nodes are available: 1 node(s) had untolerated taint(s), 2 Insufficient cpu. no new claims to deallocate, preemption: not eligible due to preemptionPolicy=Never.
7s          Normal    Scheduled              pod/important                  Successfully assigned demo/important to k8s-node-2
...

Everything as described: important did not fit (its FailedScheduling has no preemption: part — preemption succeeded), the scheduler picked a victim on k8s-node-2, the Preempted event names the preempting pod by UID, the victim terminated, and two seconds later important took the freed room. The ReplicaSet's replacement filler waits in Pending and preempts nobody itself — its class has preemptionPolicy: Never. The reserved node is visible in the NOMINATED NODE column, but only briefly, while the victim terminates. Repeating the experiment (deleted important, filler got its second replica back, created important again), we catch it:

$ kubectl -n demo get pod important -o wide
NAME        READY   STATUS    RESTARTS   AGE   IP       NODE     NOMINATED NODE   READINESS GATES
important   0/1     Pending   0          2s    <none>   <none>   k8s-node-2       <none>

How the scheduler reasoned. At verbosity level 10 it logs the plugin scores for each node; in production you cannot leave it like that (far too much output), but in the lab you can temporarily add - --v=10 to command in /etc/kubernetes/manifests/kube-scheduler.yaml, having first saved a copy of the manifest outside the manifests directory. Let's check profiles at the same time: into the same manifest we add - --config=/etc/kubernetes/scheduler-config.yaml with the file from the profiles section, and the file itself to volumeMounts and volumes. The kubelet recreates the static pod as soon as the manifest file changes, and the logs of the previous container go with it. Let's restart hello and look at how one of the new pods was scored:

$ kubectl -n kube-system logs kube-scheduler-k8s-master-1 | grep -E 'Plugin scored node|Calculated node' | grep 99dfn | grep -v 'score=0$'
I1028 15:46:48.335751       1 schedule_one.go:982] "Plugin scored node for pod" pod="demo/hello-55ddc6bb64-99dfn" plugin="TaintToleration" node="k8s-node-1" score=300
I1028 15:46:48.335755       1 schedule_one.go:982] "Plugin scored node for pod" pod="demo/hello-55ddc6bb64-99dfn" plugin="NodeResourcesFit" node="k8s-node-1" score=54
I1028 15:46:48.335760       1 schedule_one.go:982] "Plugin scored node for pod" pod="demo/hello-55ddc6bb64-99dfn" plugin="NodeResourcesBalancedAllocation" node="k8s-node-1" score=74
I1028 15:46:48.335763       1 schedule_one.go:982] "Plugin scored node for pod" pod="demo/hello-55ddc6bb64-99dfn" plugin="TaintToleration" node="k8s-node-2" score=300
I1028 15:46:48.335764       1 schedule_one.go:982] "Plugin scored node for pod" pod="demo/hello-55ddc6bb64-99dfn" plugin="NodeResourcesFit" node="k8s-node-2" score=63
I1028 15:46:48.335765       1 schedule_one.go:982] "Plugin scored node for pod" pod="demo/hello-55ddc6bb64-99dfn" plugin="InterPodAffinity" node="k8s-node-2" score=200
I1028 15:46:48.335768       1 schedule_one.go:982] "Plugin scored node for pod" pod="demo/hello-55ddc6bb64-99dfn" plugin="NodeResourcesBalancedAllocation" node="k8s-node-2" score=74
I1028 15:46:48.335772       1 schedule_one.go:1050] "Calculated node's final score for pod" pod="demo/hello-55ddc6bb64-99dfn" node="k8s-node-1" score=428
I1028 15:46:48.335774       1 schedule_one.go:1050] "Calculated node's final score for pod" pod="demo/hello-55ddc6bb64-99dfn" node="k8s-node-2" score=637

The scores are already multiplied by the plugin weight. TaintToleration gave both workers 300 each (100 × 3): they have no PreferNoSchedule taints. NodeResourcesFit preferred the less loaded k8s-node-2 (63 against 54), and the soft anti-affinity settled it: k8s-node-2 had fewer hello pods at that moment, and InterPodAffinity gave it 200 (100 × 2). PodTopologySpread takes no part in scoring: the pod has only a DoNotSchedule constraint, which works as a filter, and the defaults do not apply to a pod with constraints of its own. Lines with score=0 (ImageLocality, DynamicResources) are filtered out. Nearby in the log you can see the numbers NodeResourcesFit worked with: for k8s-node-2 requestedResource=[800,1174405120] — 800m CPU and 1120 MiB, although the pods there, together with the new one, requested 400m and 320 MiB (that number is in the NodeResourcesBalancedAllocation line): the four containers of calico-node, csi-node-driver and kube-proxy without requests are counted as 100m and 200 MiB each.

The bin-packing profile with the configuration from the profiles section — three pause pods with 300m CPU each and schedulerName: bin-packing:

$ kubectl -n demo create deployment packed --image=registry.k8s.io/pause:3.10.2 --replicas=3 --dry-run=client -o json | jq '.spec.template.spec.schedulerName="bin-packing" | .spec.template.spec.containers[0].resources={requests:{cpu:"300m"}}' | kubectl apply -f -
deployment.apps/packed created
$ kubectl -n demo get pods -l app=packed -o wide
NAME                     READY   STATUS              RESTARTS   AGE   IP              NODE         NOMINATED NODE   READINESS GATES
packed-f474646d7-q249j   1/1     Running             0          8s    172.20.211.37   k8s-node-1   <none>           <none>
packed-f474646d7-rttx8   1/1     Running             0          8s    172.20.211.36   k8s-node-1   <none>           <none>
packed-f474646d7-sqnr4   0/1     ContainerCreating   0          8s    <none>          k8s-node-1   <none>           <none>

All three on one node; in the Scheduled events of these pods the From column shows bin-packing — the profile name. After the experiment we restore the manifest from the copy (checking sha256sum) and delete the packed Deployment.

Taints and eviction by NoExecute. Besides the hello and anti pods, k8s-node-2 has a pod patient that tolerates the maintenance taint for 30 seconds:

apiVersion: v1
kind: Pod
metadata:
  name: patient
  namespace: demo
spec:
  nodeSelector:
    kubernetes.io/hostname: k8s-node-2
  tolerations:
  - key: maintenance
    operator: Exists
    effect: NoExecute
    tolerationSeconds: 30
  containers:
  - name: pause
    image: registry.k8s.io/pause:3.10.2

$ kubectl taint node k8s-node-2 maintenance=now:NoExecute
node/k8s-node-2 tainted
$ kubectl get events -n demo --field-selector reason=TaintManagerEviction
LAST SEEN   TYPE     REASON                 OBJECT                       MESSAGE
...
38s         Normal   TaintManagerEviction   pod/anti-79d5d64c98-z6d9h    Marking for deletion Pod demo/anti-79d5d64c98-z6d9h
38s         Normal   TaintManagerEviction   pod/hello-55ddc6bb64-99dfn   Marking for deletion Pod demo/hello-55ddc6bb64-99dfn
38s         Normal   TaintManagerEviction   pod/hello-55ddc6bb64-w56nb   Marking for deletion Pod demo/hello-55ddc6bb64-w56nb
8s          Normal   TaintManagerEviction   pod/patient                  Marking for deletion Pod demo/patient
$ kubectl get pods -A -o wide --field-selector spec.nodeName=k8s-node-2
NAMESPACE       NAME                    READY   STATUS    RESTARTS       AGE    IP              NODE         NOMINATED NODE   READINESS GATES
calico-system   calico-node-7mx4w       1/1     Running   2 (138m ago)   4d4h   192.168.10.21   k8s-node-2   <none>           <none>
calico-system   csi-node-driver-6nzck   2/2     Running   4 (138m ago)   4d4h   172.20.172.16   k8s-node-2   <none>           <none>
kube-system     kube-proxy-xjhjp        1/1     Running   2 (138m ago)   4d4h   192.168.10.21   k8s-node-2   <none>           <none>
$ kubectl taint node k8s-node-2 maintenance=now:NoExecute-
node/k8s-node-2 untainted

The hello and anti pods were marked for deletion in the same second, patient exactly 30 seconds later. They are deleted by the taint-eviction-controller in kube-controller-manager (that is what the source field of the events says), and deleted bypassing PDBs: eviction by taint is a deletion, not the Eviction API. That is why planned maintenance uses drain, and a manual NoExecute is only for when a node really has to be freed immediately. The hello replacements went to k8s-node-1 (with nodeTaintsPolicy: Honor a node with a foreign taint is excluded from the count), while the calico-node, csi-node-driver and kube-proxy pods stayed on the node: DaemonSets tolerate everything.

A pod bypassing the scheduler and admission on the node: a pod with nodeName: k8s-node-2 that asks for more CPU than the node has:

$ kubectl -n demo run direct --image=registry.k8s.io/pause:3.10.2 --overrides='{"spec":{"nodeName":"k8s-node-2","containers":[{"name":"direct","image":"registry.k8s.io/pause:3.10.2","resources":{"requests":{"cpu":"3"}}}]}}'
pod/direct created
$ kubectl -n demo get pod direct
NAME     READY   STATUS     RESTARTS   AGE
direct   0/1     OutOfcpu   0          5s
$ kubectl -n demo get pod direct -o jsonpath='{.status.phase} {.status.reason}: {.status.message}{"\n"}'
Failed OutOfcpu: Pod was rejected: Node didn't have enough resource: cpu, requested: 3000, used: 300, capacity: 2000
$ kubectl -n demo delete pod direct
pod "direct" deleted from demo namespace

The pod does not wait for room but fails right away: the kubelet rejected it at admission, and nobody will reschedule it. capacity in the message is the node's allocatable.

Scheduler metrics. kube-scheduler serves them at https://127.0.0.1:10259/metrics only to authorized clients — an anonymous request gets forbidden: User "system:anonymous" cannot get path "/metrics". On the master the simplest way is to take the administrator certificate from admin.conf without saving it to files:

user@k8s-master-1:~$ sudo -i
root@k8s-master-1:~# curl -sk --cert <(grep client-certificate-data /etc/kubernetes/admin.conf | awk '{print $2}' | base64 -d) \
     --key <(grep client-key-data /etc/kubernetes/admin.conf | awk '{print $2}' | base64 -d) \
     https://127.0.0.1:10259/metrics | grep -E '^scheduler_(pending_pods|schedule_attempts_total)'
scheduler_pending_pods{queue="active"} 0
scheduler_pending_pods{queue="backoff"} 0
scheduler_pending_pods{queue="gated"} 0
scheduler_pending_pods{queue="unschedulable"} 1
scheduler_schedule_attempts_total{profile="default-scheduler",result="scheduled"} 8
scheduler_schedule_attempts_total{profile="default-scheduler",result="unschedulable"} 6

These are the queues from the diagram at the start of the article: the third anti replica lies in unschedulable. The attempt counters count from the last kube-scheduler restart and are split by profile.

Finally, the full list of what hinders or helps placement on a node, in one command:

$ kubectl get node k8s-node-1 -o json | jq '{unschedulable: .spec.unschedulable, taints: .spec.taints, labels: .metadata.labels}'
{
  "unschedulable": null,
  "taints": null,
  "labels": {
    "beta.kubernetes.io/arch": "amd64",
    "beta.kubernetes.io/os": "linux",
    "disktype": "ssd",
    "kubernetes.io/arch": "amd64",
    "kubernetes.io/hostname": "k8s-node-1",
    "kubernetes.io/os": "linux"
  }
}

Cleaning up: kubectl -n demo delete deployment anti, kubectl -n demo delete pod ssd nvme aff near-redis, kubectl -n demo delete deployment filler, kubectl -n demo delete pod important, kubectl delete priorityclass business-critical batch-low and kubectl label node k8s-node-1 disktype-. The hello PDB and the spread in the hello Deployment can stay.

---

Sources

• Kubernetes Scheduler — filtering and scoring of nodes.
• Scheduling Framework — extension points, QueueingHint, the scheduling cycle and the binding cycle.
• Scheduler Configuration — KubeSchedulerConfiguration, default plugins, profiles.
• Scheduler Performance Tuning — percentageOfNodesToScore and its default.
• Resource Bin Packing — the MostAllocated and RequestedToCapacityRatio strategies.
• Assigning Pods to Nodes — nodeName, nodeSelector, node affinity, pod affinity and anti-affinity, matchLabelKeys.
• Taints and Tolerations — effects, tolerationSeconds, taints by node condition, the Gt and Lt operators.
• Pod Topology Spread Constraints — constraint fields, nodeTaintsPolicy, default constraints.
• Pod Priority and Preemption — PriorityClass, built-in classes, nominatedNodeName, PDBs during preemption.
• Pod Scheduling Readiness — schedulingGates and which fields may change behind the gates.
• Nodes — node heartbeats (Lease and status), the node controller.
• Admission Control in Kubernetes — NodeRestriction, DefaultTolerationSeconds, Priority.
• Disruptions — voluntary and involuntary disruptions, what a PDB protects.
• Specifying a Disruption Budget for your Application — minAvailable, maxUnavailable, rounding of percentages, unhealthyPodEvictionPolicy.
• API-initiated Eviction — the Eviction API and the 200, 429 and 500 responses.
• Safely Drain a Node — kubectl drain and PDBs.
• Customizing components with the kubeadm API — scheduler.extraArgs and extraVolumes in ClusterConfiguration.
• Descheduler — rebalancing pods that are already placed.

---

Next — the Kubernetes network model: why every pod has its own address and a pod reaches another pod without NAT, what CNI is and how containerd calls Calico, where the cali… veth and the /32 route on the node come from, and how a packet travels between pods on the same node and on different ones.