Kubernetes for beginners. Part 8: How do StatefulSet and DaemonSet work?

In the previous part a Deployment kept four copies of hello, replaced them during an update and rolled them back. To it all pods are the same: the name is random (hello-85854d9fd8-4mjwb), the address is new after every re-creation, and if you delete any of them, the ReplicaSet creates a replacement that is just as good. That is how stateless applications are built, and they are the majority in a cluster. But there are two classes of workloads that such interchangeability gets in the way of. Databases, message brokers and any clusters with leader election need permanent names and their own disks: replica number two must stay number two after a restart and find its data. And log and metrics collection agents, network and storage drivers need exactly one copy on every node, however many nodes there are. For the first, Kubernetes has StatefulSet; for the second, DaemonSet. Both are controllers in kube-controller-manager just like the ReplicaSet from part 7, with the same reconciliation loop from part 1; what differs are the rules by which they create and delete pods.

---

What is wrong with a Deployment for a stateful application

Imagine that hello is not a web server but a node of a three-node database. What breaks if you run it with a Deployment:

• Names. Database nodes know each other by names or addresses: "replica db-1 follows the leader db-0". The name of a Deployment pod contains a random tail and changes with every re-creation, and so does the address. You cannot write the member list into the configuration.
• Disks. A Deployment template has one set of volumes for all replicas. If there is a PersistentVolumeClaim there (a claim for a persistent volume, covered in detail in part 21), all three pods refer to the same volume — but the database needs three different ones, one per node, and after a pod is recreated it must get its own volume back.
• Order. A ReplicaSet creates and deletes pods in bulk, in any order. Many clustered systems need the cluster initiator to come up first and the others to join it one by one.
• Updates. A Deployment rolling update first creates a new pod (maxSurge) and then kills the old one. For a database this means that for a minute there are two pods claiming the same role.

StatefulSet solves all four points and nothing beyond them.

---

StatefulSet: names in order

The manifest hardly differs from a Deployment. Let us take our image from part 5: it answers hello from <pod name>, which makes it easy to see who is who. Access to the private package in ghcr.io was set up there too; the training namespace until part 11 is default.

apiVersion: v1
kind: Service
metadata:
  name: web
  labels:
    app: web
spec:
  clusterIP: None            # headless: no virtual address
  selector:
    app: web
  ports:
  - name: http
    port: 8080
    targetPort: http
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
spec:
  serviceName: web           # the Service that gives the pods DNS names
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      terminationGracePeriodSeconds: 30
      containers:
      - name: hello
        image: ghcr.io/example/hello:1.0
        ports:
        - name: http
          containerPort: 8080
        readinessProbe:
          httpGet:
            path: /healthz
            port: http

$ kubectl apply -f web.yaml
service/web created
statefulset.apps/web created

$ kubectl get pods -l app=web -w
NAME    READY   STATUS    RESTARTS   AGE
web-0   0/1     Pending   0          0s
web-0   0/1     Pending   0          0s
web-0   0/1     ContainerCreating   0          1s
web-0   0/1     ContainerCreating   0          2s
web-0   0/1     ContainerCreating   0          3s
web-0   0/1     Running             0          4s
web-0   1/1     Running             0          4s
web-1   0/1     Pending             0          0s
web-1   0/1     Pending             0          0s
web-1   0/1     ContainerCreating   0          0s
web-1   0/1     ContainerCreating   0          1s
web-1   0/1     ContainerCreating   0          2s
web-1   0/1     Running             0          3s
web-1   1/1     Running             0          3s
web-2   0/1     Pending             0          1s
web-2   0/1     Pending             0          1s
web-2   0/1     ContainerCreating   0          1s
web-2   0/1     ContainerCreating   0          2s
web-2   0/1     ContainerCreating   0          3s
web-2   0/1     Running             0          4s
web-2   1/1     Running             0          4s

kubectl get -w prints a line for every change of the object, so identical lines follow one another, and the columns are aligned only to the first one. Two things are visible at once. The names are web-0, web-1, web-2: the StatefulSet name and an ordinal, with no random tails. And the pods appear strictly one at a time: web-1 is not created until web-0 has become Ready, that is, until its readiness probe from part 6 has passed.

If you delete web-1, the controller creates a new pod with the same name web-1, not web-3. Its address will be different — the CNI still assigns a pod IP on every creation — but the name, the DNS record and the disk stay the same. This is what is called a stable identity: the number, the network name and the volumes are tied not to a particular pod instance but to a position in the set.

The controller puts labels on every pod by which it can be selected individually:

Label	Value	Why
statefulset.kubernetes.io/pod-name	web-1	a Service for one particular pod (for example, only for the leader)
apps.kubernetes.io/pod-index	1	a filter in logs and metrics, routing by number
controller-revision-hash	web-867b65b898	the template revision the pod was created from; more on it below

By default numbering starts from zero. The .spec.ordinals.start field (stable since 1.31) shifts the start: with start: 1 the pods will be web-1, web-2, web-3. It is needed mostly when moving a set between clusters piece by piece.

The headless Service and DNS names

A regular Service from part 15 gets a virtual address from 10.96.0.0/12, and kube-proxy spreads connections to this address over the pods. A headless Service (clusterIP: None) gets no address: kube-proxy does not handle it at all, and the cluster DNS server CoreDNS answers a query for the Service name with the list of the pods' own addresses. For a StatefulSet something else matters more: the controller writes hostname: web-1 and subdomain: web (the value of serviceName) into the spec of every pod, and from this pair CoreDNS creates a separate name for every pod:

web-0.web.default.svc.cluster.local  →  172.20.211.19
web-1.web.default.svc.cluster.local  →  172.20.172.23
web-2.web.default.svc.cluster.local  →  172.20.211.20
  │    │     │
  │    │     └─ namespace
  │    └─ serviceName (headless Service)
  └─ pod name = hostname

Inside the namespace the short web-1.web is enough: the search list in the pod's /etc/resolv.conf adds the rest. When web-1 is recreated, the record points to the new address. This is exactly the name the cluster members write into each other's configuration. You have to create the Service yourself — the StatefulSet does not create it, it only refers to it by name.

Besides A records with addresses, CoreDNS publishes SRV records for a headless Service. SRV is a DNS record type that answers the question "which hosts provide such-and-such service and on which port": the answer contains a priority, a weight, a port and a host name. A query for the Service name returns one SRV record per ready pod — with its port 8080 and its personal name web-N.web.default.svc.cluster.local, and for a named port there is also a name of the form _http._tcp.web.default.svc.cluster.local (_<port name>._<protocol>.<service>). This way a client learns both the member list and the port without knowing either in advance.

Two subtleties. By default DNS returns only pods that have passed the readiness probe; clustered systems that need to find their neighbours before they are ready (to form a cluster first and only then become Ready) set publishNotReadyAddresses: true in the Service. And about caching: CoreDNS itself in kubeadm does not cache answers about cluster.local names — neither positive nor negative ones (the Corefile has disable success cluster.local and disable denial cluster.local), so the name web-2.web starts resolving as soon as the pod is ready. But a "no such name" received before that may be remembered by something on the client side — the application's resolver or a caching agent on the node — so programs that look for neighbours through DNS must repeat the query rather than give up after the first failure. How CoreDNS, the search list and ndots work is covered in part 16; the DNS protocol itself — in part 12 of the networking series.

---

Its own disk for every pod: volumeClaimTemplates

The main reason StatefulSet exists is the volumeClaimTemplates field. It is a template not of a volume but of a volume claim (PersistentVolumeClaim, PVC): the controller creates one claim per number and puts it into the pod with the same number.

spec:
  # ... everything as above
  template:
    spec:
      containers:
      - name: hello
        # ...
        volumeMounts:
        - name: data
          mountPath: /data
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 1Gi

The claim name is made of the template name, the set name and the number: data-web-0, data-web-1, data-web-2. When the pod web-1 is recreated — after a failure, an update, a move to another node — the controller does not create a new claim but finds the existing data-web-1 and mounts it. The data returns to the same number.

 StatefulSet web
   ├─ web-0 ── PVC data-web-0 ── PV (disk 1)    web-0.web.default.svc
   ├─ web-1 ── PVC data-web-1 ── PV (disk 2)    web-1.web.default.svc
   └─ web-2 ── PVC data-web-2 ── PV (disk 3)    web-2.web.default.svc

 pod recreated → new pod name? no, the same: web-1
               → new PVC?      no, the same: data-web-1
               → new IP?       yes, but the same DNS name

On our stand such a manifest gets stuck for now: the cluster has no StorageClass yet — a description of where and how to provision volumes — and the claims stay Pending:

$ kubectl get pvc
NAME         STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
data-web-0   Pending                                                     <unset>                 40s

$ kubectl get pods -l app=web
NAME    READY   STATUS    RESTARTS   AGE
web-0   0/1     Pending   0          40s

$ kubectl describe pvc data-web-0 | tail -3
  Type    Reason         Age               From                         Message
  ----    ------         ----              ----                         -------
  Normal  FailedBinding  9s (x4 over 40s)  persistentvolume-controller  no persistent volumes available for this claim and no storage class is set

Note: there is one claim, data-web-0, and one pod. web-1 is not created because web-0 is not ready — the ordering works here too. The pod waits with the scheduler event pod has unbound immediate PersistentVolumeClaims. Volumes, PVs, StorageClass and CSI drivers are the subject of parts 21–23; there this example will work on NFS.

To try the example right now, you can create volumes by hand. A PersistentVolume (PV) is an object that describes a particular volume; in a cluster without a default StorageClass, a claim without storageClassName binds to any free PV without a class of a suitable size and access mode. The simplest option for an experiment is a local directory on a node: a volume of type local with node affinity, so that the scheduler places the pod where its data lives. The directory /srv/pv/a has to be created on k8s-node-1 in advance, and you need one such PV per replica:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-node-1-a
spec:
  capacity:
    storage: 1Gi
  accessModes: ["ReadWriteOnce"]
  persistentVolumeReclaimPolicy: Retain
  storageClassName: ""
  local:
    path: /srv/pv/a
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values: ["k8s-node-1"]

As soon as free PVs appear, the volume controller binds the claims to them, and the set comes up further:

$ kubectl get pvc
NAME         STATUS   VOLUME        CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
data-web-0   Bound    pv-node-1-b   1Gi        RWO                           <unset>                 87s
data-web-1   Bound    pv-node-1-a   1Gi        RWO                           <unset>                 22s
data-web-2   Bound    pv-node-2-a   1Gi        RWO                           <unset>                 18s

A pod with such a volume is tied to the directory's node for good: if the node dies, the data stays on it. That is fine for an experiment and not for real data; after the experiment the PVs and directories are deleted by hand.

volumeClaimTemplates cannot be changed after the set is created: the API allows editing only replicas, ordinals, template, updateStrategy, revisionHistoryLimit, persistentVolumeClaimRetentionPolicy and minReadySeconds in a StatefulSet, and answers an attempt to change anything else with Forbidden: updates to statefulset spec for fields other than ... are forbidden. You can increase the disk size by expanding the PVCs themselves (if the StorageClass allows it), but the template stays old — new replicas get the old size, which is worth remembering when scaling.

What happens to the disks on deletion and scale-down

By default, nothing. You deleted the StatefulSet or reduced replicas from three to one — the claims data-web-1 and data-web-2 and the volumes behind them stay. This is deliberate: losing data because of a typo in replicas is worse than a forgotten disk. You brought back three replicas — web-1 and web-2 got their old data.

The behaviour is set by persistentVolumeClaimRetentionPolicy (stable since 1.32):

spec:
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain   # the StatefulSet was deleted
    whenScaled: Delete    # replicas was reduced

Both fields take Retain (the default) and Delete. The mechanism is the same owner references (ownerReferences) and garbage collector that delete pods together with a ReplicaSet. With whenDeleted: Delete the controller sets the StatefulSet itself as the owner of every claim, and after it is deleted the garbage collector deletes the claims. With whenScaled: Delete the controller, before deleting a surplus pod, makes that pod the owner of the claim, and the claim disappears after it — once the pod has stopped and unmounted the volume. What happens to the disk itself is decided by the volume's reclaimPolicy (part 21): Delete deletes the data, Retain leaves it to the administrator.

You can see this on the claims themselves. With both set to Delete, every claim is owned by the StatefulSet, and when replicas is reduced from three to two, the owner of the claim data-web-2 changes to the pod:

$ kubectl get pvc -w -o custom-columns=NAME:.metadata.name,STATUS:.status.phase,OWNER-KIND:.metadata.ownerReferences[*].kind,OWNER:.metadata.ownerReferences[*].name
NAME         STATUS   OWNER-KIND    OWNER
data-web-0   Bound    StatefulSet   web
data-web-1   Bound    StatefulSet   web
data-web-2   Bound    StatefulSet   web
data-web-2   Bound    Pod           web-2

A few seconds later, when the pod has stopped, the claim data-web-2 is gone, and its PV with Retain has moved to the Released status: the data is in place, but such a volume will not bind to a new claim by itself — the administrator decides what to do with it.

The policies apply only when the set is deleted and when the number of replicas is reduced. If a pod is recreated because of a node failure, the claim is always kept. For databases a sensible combination is Retain for both, and for caches and temporary replicas that resynchronise on every start anyway, whenScaled: Delete.

---

Startup order and podManagementPolicy

The default rules (the OrderedReady policy):

• on creation and scale-up, pods are created in ascending order of numbers, and each next one only when all previous ones are Running and Ready (and if minReadySeconds is set, ready for at least that long);
• on scale-down, pods are deleted in descending order, and each next one only when the previous one has stopped completely;
• if an earlier pod fails during scaling, the process stops until it comes back up.

 scale 0 → 3:  web-0 ──Ready──► web-1 ──Ready──► web-2
 scale 3 → 1:  web-2 ──deleted──► web-1 ──deleted──►  (web-0 stays)

For a system with an initiator this is exactly what is needed: web-0 brings up the cluster, the others join it one by one and leave one by one. But for many stateful applications the order does not matter — each node finds its neighbours by itself — and waiting for readiness one by one is slow: a hundred replicas at ten seconds each is almost 17 minutes. For them there is podManagementPolicy: Parallel: pods are created and deleted all at once, without waiting. Names, DNS and disks stay just as stable; only the queue is removed. The field is set when the set is created and does not change afterwards.

Never set terminationGracePeriodSeconds: 0 in a StatefulSet. The set guarantees that there is no more than one pod with a given name, and everything else relies on this guarantee: two web-1 pods at once would write to the same disk. A zero grace period and forced deletion (kubectl delete pod --force --grace-period=0) remove the pod from the API without waiting for the kubelet to confirm that the processes have stopped.

The same explains the behaviour when a node fails. If k8s-node-2 has powered off, its pods go to Unknown or hang in Terminating: the API server cannot get confirmation that they have stopped, since the kubelet is unreachable. A Deployment in this situation simply creates a replacement on another node. A StatefulSet does not — from its point of view web-1 may still be running and writing to the disk, and a second web-1 is not allowed. The pod stays in this state until the node comes back or the administrator confirms that the node is really dead: deletes the Node object or puts the taint node.kubernetes.io/out-of-service on it (taints are covered in part 13). Only after that does the controller create web-1 on another node, and the volumes are detached from the dead one. This is not a shortcoming but the price of the guarantee.

---

Updates: RollingUpdate, partition and OnDelete

The strategy is set by .spec.updateStrategy.type, and in 1.36 there are two options (1.37 adds a third, Recreate, an alpha behind the StatefulSetRecreateStrategy feature gate).

RollingUpdate (the default). After the template changes, the controller deletes and recreates pods one at a time, from the highest number to the lowest: web-2, then web-1, then web-0, each time waiting for the new pod to become ready. Unlike a Deployment, the new pod is not created next to the old one: first the old one is deleted, then a new one appears under the same name. There are no extra instances, but one replica is unavailable while it is being recreated. The highest number goes first because in clustered systems web-0 is most often the initiator or the leader, and it makes sense to update it last.

A staged update with partition. The .spec.updateStrategy.rollingUpdate.partition field sets the number from which pods are updated. Pods with a number lower than partition stay on the old version — and even if you delete them, they are recreated from the old template. This is a built-in canary release mechanism:

spec:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 2       # only web-2 is updated

$ kubectl set image statefulset/web hello=ghcr.io/example/hello:1.1
statefulset.apps/web image updated

$ kubectl rollout status statefulset/web
Waiting for partitioned roll out to finish: 0 out of 1 new pods have been updated...
Waiting for 1 pods to be ready...
Waiting for 1 pods to be ready...
Waiting for 1 pods to be ready...
partitioned roll out complete: 1 new pods have been updated...

$ kubectl get pods -l app=web -L controller-revision-hash
NAME    READY   STATUS    RESTARTS   AGE   CONTROLLER-REVISION-HASH
web-0   1/1     Running   0          24s   web-867b65b898
web-1   1/1     Running   0          20s   web-867b65b898
web-2   1/1     Running   0          5s    web-846985d455

We looked at web-2 and made sure the new version works with the data and the neighbours — now we lower partition to 0 (kubectl patch statefulset web -p '{"spec":{"updateStrategy":{"rollingUpdate":{"partition":0}}}}'), and the rest are updated: first web-1, then web-0, each going through Terminating and Completed and being created again. If partition is greater than the number of replicas, template changes reach no pod at all. Until partition is lowered, a web-0 deleted by hand comes back with the old image hello:1.0. kubectl rollout status always says "partitioned", even without partition in the manifest: the API fills in partition: 0 in the strategy by default.

maxUnavailable — how many pods of the set to update at once (.spec.updateStrategy.rollingUpdate.maxUnavailable, a number or a percentage, 1 by default). In 1.36 this field is beta but disabled by default: it was enabled in the first 1.35 releases and disabled again in 1.35.4. To use it, you have to enable the feature gate (a switch that turns a feature on) MaxUnavailableStatefulSet on the API server and in kube-controller-manager. Without it the API server silently drops the maxUnavailable field from the strategy — there is no error, but the field is not in the stored object either — and the update goes strictly one at a time: slow for a set of a hundred replicas, but predictable. Whether the gate is on can be seen in the API server's metrics: kubectl get --raw /metrics | grep 'kubernetes_feature_enabled{name="MaxUnavailableStatefulSet"' on our stand shows stage="BETA"} 0. With podManagementPolicy: Parallel and maxUnavailable greater than one, the controller recreates pods in batches.

OnDelete. The controller does not touch pods at all when the template changes. A new pod with the new version appears only when someone else deletes the old one — the administrator or an operator (part 28). kubectl rollout status does not work for such a set and answers error: rollout status is only available for RollingUpdate strategy type; which pods are already on the new version can be seen from the controller-revision-hash label. This is how systems are updated where the order and moment of each replica's restart are decided by a person or by external logic: first the replicas, then the leader switchover, then the former leader.

Revisions and rollback. The controller saves every template change in a ControllerRevision object — a numbered snapshot of the template. The revision hash is the controller-revision-hash label on the pods; by it the controller knows which pod is of which version. Up to revisionHistoryLimit revisions are kept (10 by default), and kubectl rollout history statefulset/web and kubectl rollout undo statefulset/web work the same way as with a Deployment.

There is one known trap with OrderedReady. If the new version never becomes Ready (an error in the image or configuration), the update stops at the very first pod and waits. Rolling back the template does not help in this situation: the controller keeps waiting for the broken pod to become ready and only then would it recreate it from the fixed template. The documentation describes this as a known issue: after rolling back the template you have to delete by hand the pods that managed to start with the broken version — then the controller recreates them from the fixed template. On the stand this is reproduced with a tag that does not exist in the registry:

$ kubectl set image statefulset/web hello=ghcr.io/example/hello:1.2
statefulset.apps/web image updated

$ kubectl get pods -l app=web -L controller-revision-hash
NAME    READY   STATUS         RESTARTS   AGE   CONTROLLER-REVISION-HASH
web-0   1/1     Running        0          95s   web-846985d455
web-1   1/1     Running        0          48s   web-846985d455
web-2   0/1     ErrImagePull   0          36s   web-7697f865bd

$ kubectl rollout undo statefulset/web
Warning: resource statefulsets/web was previously managed with 'kubectl apply'. Rolling back will not update the kubectl.kubernetes.io/last-applied-configuration annotation, which may cause unexpected behavior on future 'kubectl apply' operations. Consider using 'kubectl apply' with your previous configuration file instead.
statefulset.apps/web rolled back

$ kubectl get pods -l app=web -L controller-revision-hash
NAME    READY   STATUS             RESTARTS   AGE     CONTROLLER-REVISION-HASH
web-0   1/1     Running            0          2m20s   web-846985d455
web-1   1/1     Running            0          93s     web-846985d455
web-2   0/1     ImagePullBackOff   0          81s     web-7697f865bd

$ kubectl delete pod web-2
pod "web-2" deleted from default namespace

$ kubectl get pods -l app=web -L controller-revision-hash
NAME    READY   STATUS    RESTARTS   AGE     CONTROLLER-REVISION-HASH
web-0   1/1     Running   0          2m33s   web-846985d455
web-1   1/1     Running   0          106s    web-846985d455
web-2   1/1     Running   0          4s      web-846985d455

The template rolled back at once, but web-2 stayed on the broken revision for a minute and a half until it was deleted by hand. And one more detail of the rollback: it does not create a new revision but moves the existing ControllerRevision with the same template to the next number, so after it kubectl rollout history shows revisions 1, 3 and 4 — the second has become the fourth.

---

A StatefulSet does not make an application clustered

This is the most common misunderstanding. A StatefulSet gives three things: a permanent name, a permanent disk and a startup order. It does not know that there is a database inside the pod and does nothing of what makes a database fault-tolerant:

• it does not set up replication — three PostgreSQL pods in a StatefulSet with replicas: 3 are three independent databases with three empty disks, not a cluster;
• it does not elect a leader or switch it over on failure — if the pod with the primary database fails, the StatefulSet recreates it when it can, but does not promote another replica to primary;
• it does not know which pod is currently accepting writes and does not route traffic there — a regular Service spreads clients over all three, including read-only replicas;
• it does not make backups and does not check that the data on the disk is intact after a move.

All of this must be done by the application itself, or it is done by an operator — a separate controller that knows how to manage a particular system: it creates the StatefulSet, sets up replication, switches the leader, makes backups. An example of the first is etcd, the store Kubernetes itself rests on: its nodes elect a leader and confirm every write by a majority using the Raft consensus protocol, and it needs a StatefulSet only for names and disks (in detail in part 34). An example of the second is the operators for the PostgreSQL database and the RabbitMQ message broker, which are covered in part 28. In practice databases in a cluster are almost always installed with an operator, and the StatefulSet turns out to be a detail inside it.

---

DaemonSet: a pod on every node

A DaemonSet solves the opposite task: not "N pods anywhere" but "one pod on every suitable node". A node is added — a pod appears on it; a node is removed — the pod disappears with it. A DaemonSet has no replicas field: the number of pods equals the number of nodes.

Typical DaemonSet residents:

• the network plugin. Calico runs calico-node on every node: it programs routes and rules for the pods of that node and, in BGP mode, exchanges routes with its neighbours (BGP is the protocol routers use to tell each other which networks are reachable through them, see part 22 of the networking series); on our stand it was installed by the Tigera operator — a controller that installs and updates Calico — in part 4. The same goes for kube-proxy, which turns Services into nftables rules on every node (part 15);
• the node part of a CSI driver. CSI (Container Storage Interface) is the standard interface through which Kubernetes connects external storage; every driver has a node plugin that mounts volumes on a particular node, and it is always a DaemonSet (part 21);
• log agents. Containers write to stdout, containerd stores it in /var/log/pods/... files on the node, and an agent such as Fluent Bit (a lightweight log collector and forwarder) or Grafana Alloy (Grafana's agent for logs, metrics and traces) reads these files and sends them to storage, for example to Loki — the log storage and search system from Grafana Labs (part 33);
• metrics agents. node-exporter is a Prometheus project program that serves over HTTP the metrics of the node itself: CPU load, memory, disks, network. Prometheus is a monitoring system that periodically collects such metrics and stores them as time series (part 32).

Example: node-exporter

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
  labels:
    app: node-exporter
spec:
  selector:
    matchLabels:
      app: node-exporter
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  template:
    metadata:
      labels:
        app: node-exporter
    spec:
      hostNetwork: true          # listen on port 9100 at the node's address
      hostPID: true              # see the node's processes
      tolerations:
      - key: node-role.kubernetes.io/control-plane
        operator: Exists
        effect: NoSchedule       # allow it on the master too
      containers:
      - name: node-exporter
        image: quay.io/prometheus/node-exporter:v1.12.1
        args:
        - --path.rootfs=/host
        - --path.udev.data=/host/run/udev/data
        ports:
        - name: metrics
          containerPort: 9100
        resources:
          requests:
            cpu: 20m
            memory: 32Mi
          limits:
            memory: 64Mi
        securityContext:
          runAsNonRoot: true
          runAsUser: 65534
          readOnlyRootFilesystem: true
          allowPrivilegeEscalation: false
        volumeMounts:
        - name: root
          mountPath: /host
          readOnly: true
          mountPropagation: HostToContainer
      volumes:
      - name: root
        hostPath:
          path: /

Node agents almost always ask for something from the node: the node's network (hostNetwork), its processes (hostPID), its files (hostPath). This is direct access that bypasses container isolation, and it should be granted minimally: here the node's root is mounted read-only, the process does not run as root (UID 65534 is nobody), the container's file system is immutable, privilege escalation is forbidden. mountPropagation: HostToContainer is needed so that disks mounted on the node later are visible inside the container too. --path.rootfs=/host tells node-exporter where to find the node's root when counting space on file systems, but this flag does not change the path to the udev database (disk information) — without --path.udev.data the pod's log shows Failed to open directory, disabling udev device properties. Recipes from the internet with privileged: true are not needed for node-exporter: it only has to read /proc and /sys. securityContext is covered in detail in part 30, requests and limits — in the next part.

$ kubectl apply -f node-exporter.yaml
daemonset.apps/node-exporter created

$ kubectl get daemonset node-exporter
NAME            DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
node-exporter   3         3         3       3            3           <none>          52s

$ kubectl get pods -l app=node-exporter -o wide
NAME                  READY   STATUS    RESTARTS   AGE   IP              NODE           NOMINATED NODE   READINESS GATES
node-exporter-ns8mn   1/1     Running   0          52s   192.168.10.21   k8s-node-2     <none>           <none>
node-exporter-nslxs   1/1     Running   0          52s   192.168.10.20   k8s-node-1     <none>           <none>
node-exporter-snhlc   1/1     Running   0          52s   192.168.10.17   k8s-master-1   <none>           <none>

For the first half-minute READY is 0: the kubelet on every node pulls the image (Successfully pulled image … in 20.42s in the pod's events).

The pods have the nodes' addresses, not addresses from 172.20.0.0/16: with hostNetwork: true the pod lives in the node's network namespace, and the CNI does not assign it an address.

How a DaemonSet places pods

For a long time the DaemonSet controller chose the node itself and wrote it into spec.nodeName, bypassing the scheduler. Now it works differently: for every suitable node it creates a pod with a hard binding to that node through node affinity — a placement rule based on node properties — and from there the pod is placed by the regular scheduler kube-scheduler, like any other:

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchFields:
          - key: metadata.name
            operator: In
            values:
            - k8s-node-1

This way DaemonSet pods go through the same resource and priority checks as the others. If a node does not have enough room, the scheduler may preempt lower-priority pods; that is why critical agents (CNI, CSI) get priorityClassName: system-node-critical — a built-in class with the highest priority for node components.

Which nodes are "suitable" is set by nodeSelector and affinity in the template — for example, a GPU agent only on nodes labelled gpu=true. Without them — all nodes, except those where taints get in the way. A taint is a mark on a node saying "not here", and a pod gets onto such a node only if it has a matching toleration. On the master kubeadm puts the taint node-role.kubernetes.io/control-plane:NoSchedule, so in the example above a toleration for it is written explicitly: without it node-exporter would not appear on the master, and DESIRED would be 2.

Some tolerations the controller adds to all DaemonSet pods by itself:

Taint	Effect	What it gives
node.kubernetes.io/not-ready	NoExecute	the pod is placed on a node that is not ready and is not evicted from it
node.kubernetes.io/unreachable	NoExecute	it is not evicted from a node that has lost contact with the control plane
node.kubernetes.io/disk-pressure	NoSchedule	it is placed on a node running out of disk
node.kubernetes.io/memory-pressure	NoSchedule	it is placed on a node running out of memory
node.kubernetes.io/pid-pressure	NoSchedule	it is placed on a node running out of process IDs
node.kubernetes.io/unschedulable	NoSchedule	it is placed on a node closed to new pods with kubectl cordon
node.kubernetes.io/network-unavailable	NoSchedule	only for pods with hostNetwork: true: it is placed on a node without a network

The point is in the first row and the last. A new node does not become Ready until the network plugin is running on it, and the network plugin is a DaemonSet pod. Without the automatic toleration for not-ready there would be a deadlock. For the same reason kubectl drain — the command that closes a node to new pods and evicts the existing ones before maintenance — refuses by default to evict DaemonSet pods and requires the --ignore-daemonsets flag: evicting them is pointless, the controller would immediately put them back on the same node. Taints, tolerations, affinity and drain are covered in detail in part 13.

Updating a DaemonSet

A DaemonSet has the same two strategies, RollingUpdate (the default) and OnDelete, but its RollingUpdate parameters are its own:

• maxUnavailable (1 by default) — on how many nodes at once the old pod may be stopped before the new one appears. A number or a percentage of the number of nodes. For a cluster of a hundred nodes, 1 means a hundred sequential steps, and metrics agents usually get 10–20%.
• maxSurge (0 by default, stable since 1.25) — on how many nodes the new pod may first be started next to the old one, removing the old one only when the new one is ready. This way a node is never left without an agent even for a second, which matters for network and storage plugins.

In a DaemonSet exactly one of the two fields must be non-zero. The API rejects both being zero (maxUnavailable: Required value: cannot be 0 when maxSurge is 0), but also both being non-zero: spec.updateStrategy.rollingUpdate.maxSurge: Invalid value: 1: may not be set when maxUnavailable is non-zero. So when you turn on maxSurge, you have to write maxUnavailable: 0 explicitly — otherwise the default of 1 is filled in and the manifest is not accepted.

maxSurge does not always work. If the pod occupies a port on the node — through hostNetwork or hostPort — a second instance on the same node will not start: port 9100 is already taken by the old one. With hostNetwork: true the API itself writes a hostPort with the same number into every container port, and the scheduler finds no node for the new pod:

$ kubectl get pods -l app=node-exporter -o wide
NAME                  READY   STATUS    RESTARTS   AGE   IP              NODE           NOMINATED NODE   READINESS GATES
node-exporter-dhvjz   0/1     Pending   0          20s   <none>          <none>         <none>           <none>
node-exporter-ns8mn   1/1     Running   0          45s   192.168.10.21   k8s-node-2     <none>           <none>
...

$ kubectl describe pod node-exporter-dhvjz | tail -1
  Warning  FailedScheduling  20s   default-scheduler  0/3 nodes are available: 1 node(s) didn't have free ports for the requested pod ports, 2 node(s) didn't satisfy plugin(s) [NodeAffinity]. no new claims to deallocate, preemption: 0/3 nodes are available: 1 No preemption victims found for incoming pod, 2 Preemption is not helpful for scheduling.

One node is taken by the port, the other two do not match the node binding described above — and the rollout stands still until the strategy is switched back to maxUnavailable. So for our node-exporter with hostNetwork: true only maxUnavailable fits: the old pod on the node stops (node-exporter exits on SIGTERM with a non-zero code, so Error flashes by in get -w), and only then is the new one placed on that node. The same applies to agents that hold an exclusive file lock on the node.

minReadySeconds (0 by default) adds a pause: a new pod is considered available only after being Ready for the given time. This is cheap protection against a version that starts, passes the probe and crashes ten seconds later.

OnDelete in a DaemonSet is used for the most sensitive components — the network plugin and CSI: the administrator takes the node out of service (kubectl drain), deletes the old pod, checks the new one and moves to the next node. DaemonSet revisions are also stored in ControllerRevision, and kubectl rollout history daemonset/node-exporter and kubectl rollout undo work as usual.

---

Hands-on

Let us start with the StatefulSet web from the beginning of the part. What the controller wrote into the pod spec:

$ kubectl get pod web-1 -o jsonpath='{.spec.hostname}{" "}{.spec.subdomain}{"\n"}'
web-1 web

$ kubectl get pod web-1 -o jsonpath='{.metadata.ownerReferences[0].kind}/{.metadata.ownerReferences[0].name}{"\n"}'
StatefulSet/web

$ kubectl get pods -l app=web -o wide --show-labels
NAME    READY   STATUS    RESTARTS   AGE   IP              NODE         NOMINATED NODE   READINESS GATES   LABELS
web-0   1/1     Running   0          34s   172.20.211.19   k8s-node-1   <none>           <none>            app=web,apps.kubernetes.io/pod-index=0,controller-revision-hash=web-846985d455,statefulset.kubernetes.io/pod-name=web-0
web-1   1/1     Running   0          41s   172.20.172.23   k8s-node-2   <none>           <none>            app=web,apps.kubernetes.io/pod-index=1,controller-revision-hash=web-846985d455,statefulset.kubernetes.io/pod-name=web-1
web-2   1/1     Running   0          49s   172.20.211.20   k8s-node-1   <none>           <none>            app=web,apps.kubernetes.io/pod-index=2,controller-revision-hash=web-846985d455,statefulset.kubernetes.io/pod-name=web-2

The owner of the pod is the StatefulSet itself, with no intermediate ReplicaSet as with a Deployment. All three pods are already on the second revision after the update to hello:1.1, and their ages show its order: web-2 was recreated first, web-0 last. Now DNS. Let us start a temporary pod with busybox and ask for names from it:

$ kubectl run dns-test --restart=Never --image=busybox:1.37 -- sleep 3600
pod/dns-test created

$ kubectl exec dns-test -- nslookup web.default.svc.cluster.local
Server:		10.96.0.10
Address:	10.96.0.10:53


Name:	web.default.svc.cluster.local
Address: 172.20.211.19
Name:	web.default.svc.cluster.local
Address: 172.20.172.23
Name:	web.default.svc.cluster.local
Address: 172.20.211.20

$ kubectl exec dns-test -- nslookup web-1.web.default.svc.cluster.local
Server:		10.96.0.10
Address:	10.96.0.10:53


Name:	web-1.web.default.svc.cluster.local
Address: 172.20.172.23

$ kubectl exec dns-test -- nslookup -type=SRV web.default.svc.cluster.local
Server:		10.96.0.10
Address:	10.96.0.10:53

web.default.svc.cluster.local	service = 0 33 8080 web-2.web.default.svc.cluster.local
web.default.svc.cluster.local	service = 0 33 8080 web-1.web.default.svc.cluster.local
web.default.svc.cluster.local	service = 0 33 8080 web-0.web.default.svc.cluster.local

$ kubectl exec dns-test -- wget -qO- http://web-1.web:8080/
hello from web-1

The Service name returns all three addresses (headless: there is no virtual address), the name web-1.web… exactly one. The SRV answer has priority 0, weight 33 (CoreDNS divides 100 equally between the records), port 8080 and the pod's personal name. The empty line before Name: is the empty answer to the second query, for an IPv6 address (AAAA), which busybox sends together with the A query.

The full names in nslookup are no accident. busybox nslookup appends the search domains only to names without a dot: nslookup web goes through the whole list and prints ** server can't find web.cluster.local: NXDOMAIN for every failed variant mixed in with the answer, and nslookup web-1.web answers NXDOMAIN at once, although that name works in the pod — wget goes through the regular resolver, which honours search and ndots:5, and finds the short web-1.web. Now let us delete web-1 and immediately ask for its name again:

$ kubectl delete pod web-1
pod "web-1" deleted from default namespace

$ kubectl exec dns-test -- nslookup web-1.web.default.svc.cluster.local
Server:		10.96.0.10
Address:	10.96.0.10:53

** server can't find web-1.web.default.svc.cluster.local: NXDOMAIN

** server can't find web-1.web.default.svc.cluster.local: NXDOMAIN

$ kubectl get pod web-1 -o wide
NAME    READY   STATUS    RESTARTS   AGE   IP              NODE         NOMINATED NODE   READINESS GATES
web-1   1/1     Running   0          8s    172.20.172.25   k8s-node-2   <none>           <none>

$ kubectl exec dns-test -- nslookup web-1.web.default.svc.cluster.local
Server:		10.96.0.10
Address:	10.96.0.10:53


Name:	web-1.web.default.svc.cluster.local
Address: 172.20.172.25

$ kubectl exec dns-test -- wget -qO- http://web-1.web:8080/
hello from web-1

Until the new web-1 has passed the readiness probe, the name does not exist at all — six or seven seconds on the stand; then it appears with the new address. The address changed, the name stayed. After the experiments we delete the dns-test pod (kubectl delete pod dns-test); the sleep in it runs as PID 1 and does not handle SIGTERM, so the deletion waits the full 30-second grace period. The template revisions:

$ kubectl get controllerrevisions -l app=web
NAME             CONTROLLER             REVISION   AGE
web-846985d455   statefulset.apps/web   2          79s
web-867b65b898   statefulset.apps/web   1          94s

$ kubectl rollout history statefulset/web
statefulset.apps/web 
REVISION  CHANGE-CAUSE
1         <none>
2         <none>

The name of a ControllerRevision is the set name plus the template hash, the same as in the pods' controller-revision-hash label; the same template gives the same hash, so after the set is recreated from the same manifest the hashes do not change.

Now the deletion order. Let us scale the set down to one and watch the pods go:

$ kubectl scale statefulset web --replicas=1
statefulset.apps/web scaled

$ kubectl get pods -l app=web -w
NAME    READY   STATUS    RESTARTS   AGE
web-0   1/1     Running   0          12s
web-1   1/1     Running   0          26s
web-2   1/1     Running   0          59s
web-2   1/1     Terminating   0          60s
web-2   1/1     Terminating   0          61s
web-2   1/1     Terminating   0          61s
web-2   0/1     Completed     0          62s
web-2   0/1     Completed     0          62s
web-2   0/1     Completed     0          62s
web-1   1/1     Terminating   0          30s
web-1   1/1     Terminating   0          30s
web-1   1/1     Terminating   0          31s
web-1   0/1     Completed     0          32s
web-1   0/1     Completed     0          32s
web-1   0/1     Completed     0          32s

web-1 started stopping only after web-2 had finished completely and disappeared from the API. Completed is the phase of a pod whose container exited with code 0: hello exits cleanly on SIGTERM.

On to DaemonSet. Which DaemonSets are already running in the cluster:

$ kubectl get daemonsets -A
NAMESPACE       NAME              DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
calico-system   calico-node       3         3         3       3            3           kubernetes.io/os=linux   2d11h
calico-system   csi-node-driver   3         3         3       3            3           kubernetes.io/os=linux   2d11h
default         node-exporter     3         3         3       3            3           <none>                   11m
kube-system     kube-proxy        3         3         3       3            3           kubernetes.io/os=linux   2d11h

calico-node and kube-proxy are already familiar; csi-node-driver is the node CSI plugin that the Tigera operator installs together with Calico for its own service volumes. All of them run on three nodes, including the master: their templates have a toleration for any taint (operator: Exists without a key). Let us see what the controller added to the node-exporter pod:

$ kubectl get pod node-exporter-nslxs -o yaml | grep -A 24 '^  tolerations:'
  tolerations:
  - effect: NoSchedule
    key: node-role.kubernetes.io/control-plane
    operator: Exists
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
  - effect: NoSchedule
    key: node.kubernetes.io/disk-pressure
    operator: Exists
  - effect: NoSchedule
    key: node.kubernetes.io/memory-pressure
    operator: Exists
  - effect: NoSchedule
    key: node.kubernetes.io/pid-pressure
    operator: Exists
  - effect: NoSchedule
    key: node.kubernetes.io/unschedulable
    operator: Exists
  - effect: NoSchedule
    key: node.kubernetes.io/network-unavailable
    operator: Exists

The first one is ours, the other seven were added by the controller, including network-unavailable, because the pod uses the node's network. The node's metrics are right at its address:

$ curl -s http://192.168.10.20:9100/metrics | grep -E '^node_(load1|memory_MemAvailable_bytes|filesystem_avail_bytes\{.*mountpoint="/"\}) '
node_filesystem_avail_bytes{device="/dev/sda2",device_error="",fstype="ext4",mountpoint="/"} 2.5520345088e+10
node_load1 0.14
node_memory_MemAvailable_bytes 2.789015552e+09

The space at the end of the expression cuts off node_load15, which would otherwise match node_load1 too.

And what happens when a node is taken out of service:

$ kubectl drain k8s-node-1 --ignore-daemonsets --delete-emptydir-data
node/k8s-node-1 cordoned
Warning: ignoring DaemonSet-managed Pods: calico-system/calico-node-6d2qb, calico-system/csi-node-driver-q6t25, default/node-exporter-nslxs, kube-system/kube-proxy-xmjz2
evicting pod default/web-0
pod/web-0 evicted
node/k8s-node-1 drained

$ kubectl get pods -o wide | grep -E 'NAME|k8s-node-1|web-0'
NAME                  READY   STATUS    RESTARTS   AGE   IP              NODE         NOMINATED NODE   READINESS GATES
node-exporter-nslxs   1/1     Running   0          12m   192.168.10.20   k8s-node-1   <none>           <none>
web-0                 1/1     Running   0          10s   172.20.172.30   k8s-node-2   <none>           <none>

web-0 was evicted and recreated with the same name on k8s-node-2, the DaemonSet pods stayed in place: the node is marked with the taint node.kubernetes.io/unschedulable:NoSchedule, but they have a toleration for it. Without --ignore-daemonsets, drain refuses to work and lists them: cannot delete DaemonSet-managed Pods (use --ignore-daemonsets to ignore): calico-system/calico-node-6d2qb, …. If the node has a pod without a controller, like our dns-test, drain stops too — cannot delete Pods that declare no controller (use --force to override): nobody would recreate such a pod after eviction. Let us bring the node back: kubectl uncordon k8s-node-1.

Which pods get evicted depends on what runs on the node: if a calico-typha replica happens to be there, drain evicts it too. Typha is Calico's intermediary between the API server and calico-node: it holds one connection to the API and distributes changes to the agents on the nodes, so that the API server does not serve a separate watch for every node; it is a Deployment, and the Tigera operator keeps its replicas on some of the nodes. The operator also sizes Typha by the number of nodes open for scheduling, and while one of the three nodes is cordoned it reduces its replicas from two to one, restoring them after uncordon.

---

StatefulSet and DaemonSet, like Deployment, keep pods that must run all the time. But there are tasks that must run and finish: a database schema migration, a nightly dump, recalculating a report.

Sources

• Kubernetes: StatefulSet (ordinal indexes and .spec.ordinals, pod management policies, partition, maxUnavailable, forced rollback, ControllerRevision, persistentVolumeClaimRetentionPolicy), StatefulSet basics step by step, force deleting StatefulSet pods, DaemonSet (placement through node affinity, automatic tolerations), updating a DaemonSet (maxSurge, maxUnavailable, OnDelete).
• API reference: StatefulSet, DaemonSet, ControllerRevision; well-known labels, annotations and taints (statefulset.kubernetes.io/pod-name, apps.kubernetes.io/pod-index, node.kubernetes.io/out-of-service), feature gates (MaxUnavailableStatefulSet).
• Network and DNS: Service (headless Services), DNS for Services and Pods (hostname, subdomain, SRV records), the CoreDNS plugins kubernetes and cache, RFC 2782 (SRV).
• Storage: PersistentVolume and PersistentVolumeClaim, volumes (local, hostPath).
• Nodes and scheduling: taints and tolerations, pod priority and preemption, API-initiated eviction, node shutdowns (the out-of-service taint), safely draining a node, kubectl drain.
• node-exporter (running in a container, --path.rootfs, --path.udev.data), Typha in Calico.

---

Next — Job and CronJob: how Kubernetes runs a task until it completes successfully, how many times it retries on failure, how it runs it in parallel in pieces and on a schedule, and why scheduled jobs must be idempotent.