Kubernetes for beginners. Part 7: How do Deployment and ReplicaSet work?

In the previous part we took the pod apart: how it starts, how the kubelet checks its health and how it stops gracefully. But a pod on its own is defenceless. Delete it — it is gone. The node k8s-node-1 powers off — its pods disappear, and nobody creates them again on k8s-node-2. You need four identical pods — write four manifests with different names. You need to update the version — delete and create them one by one, making sure the application stays up. All of this is the job of controllers, and for stateless applications the main one is Deployment, which manages pods not directly but through a ReplicaSet.

The reconciliation loop — a controller compares the desired state from spec with the actual one and removes the difference — was described in the first part, and how controllers follow objects through a watch, in the third. Here we look at two specific controllers: how they are built, how together they roll out a new version of hello and how they roll it back.

---

The chain of controllers

 Deployment hello                       spec: 4 replicas, template T2, strategy
   │
   │  deployment-controller (kube-controller-manager)
   │
   ├──► ReplicaSet hello-8c9694f75      template T1 (hello:1.0), replicas: 0   revision 1
   └──► ReplicaSet hello-85854d9fd8     template T2 (hello:1.1), replicas: 4   revision 2
          │
          │  replicaset-controller (kube-controller-manager)
          │
          ├──► Pod hello-85854d9fd8-4mjwb
          ├──► Pod hello-85854d9fd8-cmcqg
          ├──► Pod hello-85854d9fd8-ptchf
          └──► Pod hello-85854d9fd8-z4h9z

The work is split like this:

• ReplicaSet is responsible for one thing: that there are exactly replicas pods made from its template. Too few — it creates more, too many — it deletes some. What is in the template and why does not matter to it.
• Deployment is responsible for versions. Each variant of the pod template is a separate ReplicaSet. Updating the template means a new ReplicaSet and a gradual transfer of replicas from the old one to the new one. Old ReplicaSets with zero replicas remain as history you can return to.

Both controllers are loops inside the kube-controller-manager process on the master k8s-master-1. Neither of them goes to the nodes: they only create, change and delete objects in the API. The pods on the nodes are started by the kubelet, as in the previous part.

The links between objects are recorded in metadata.ownerReferences: a ReplicaSet's owner is the Deployment, a pod's owner is the ReplicaSet, with the flag controller: true. The garbage collector in the controller-manager works by these links: when a Deployment is deleted, it deletes its ReplicaSets, and after them the pods. kubectl delete deployment hello --cascade=orphan deletes only the Deployment itself: the garbage collector removes the owner reference from its ReplicaSets, and they and their pods are left as orphans — they keep running, but nobody manages them any more.

---

ReplicaSet

hello-rs.yaml:

apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: hello-rs
spec:
  replicas: 3
  selector:
    matchLabels:
      app.kubernetes.io/name: hello-rs
  template:
    metadata:
      labels:
        app.kubernetes.io/name: hello-rs
    spec:
      containers:
      - name: hello
        image: ghcr.io/example/hello:1.0

Three fields: how many (replicas), which pods count as its own (selector) and from which template to create the missing ones (template). The template's labels must match the selector, otherwise the API server rejects the object: the controller would create pods it cannot see itself and would keep creating them forever. If the template says app.kubernetes.io/name: other, the answer is:

$ kubectl apply -f hello-rs.yaml
The ReplicaSet "hello-rs" is invalid: spec.template.metadata.labels: Invalid value: {"app.kubernetes.io/name":"other"}: `selector` does not match template `labels`

Labels, selectors and the app.kubernetes.io/* naming conventions are covered in detail in part 11. ReplicaSet replaced ReplicationController — an object with the same purpose but with a selector by exact label equality only; ReplicationController is still in the API for compatibility, but it is not used for anything new.

How the replicaset-controller loop works:

1. Through a watch it receives changes of ReplicaSets and pods into its local cache (an informer) and puts a changed ReplicaSet into a queue.
2. For a ReplicaSet from the queue it finds the pods that match the selector: its own (with an ownerReference to this ReplicaSet) and ownerless ones. An ownerless pod with matching labels the ReplicaSet adopts — it adds an ownerReference to it. Its own pod that no longer matches the selector (someone changed its label) it releases.
3. It counts the active pods (not finished and not being deleted) and compares the number with replicas. Too few — it creates pods from the template named <ReplicaSet name>-<5 random characters>. Extra ones — it deletes.
4. It writes the number of ready and available pods into status.

To avoid creating extra pods while freshly created ones have not reached the cache yet, the controller keeps expectations — "I created three pods and expect to see three creation events". Until the expectation is fulfilled, the next pass for this ReplicaSet is skipped.

Which pods are deleted when replicas goes down is not random either: first those not yet assigned to a node and not started (Pending), then not ready ones, then pods with a lower "deletion cost" — the controller.kubernetes.io/pod-deletion-cost annotation, — then those on nodes with more replicas of this ReplicaSet, then younger ones (creation times are compared roughly, on a logarithmic scale). The annotation is the way to influence the choice: pods with a lower cost are deleted first.

Why not create a ReplicaSet directly

A ReplicaSet cannot update pods. Change its image to hello:1.1 — the running pods stay on 1.0: the template is used only when creating new ones, and the controller does not care how existing pods differ from the template. The new version appears only in pods created after the change — that is, after someone deletes the old ones. A ReplicaSet has no rollback, no history and no control over the replacement rate.

The second surprise is adoption. A ReplicaSet with the selector app.kubernetes.io/name: hello-rs takes over any ownerless pod with that label in the namespace, created by hand or by another tool, and when replicas is exceeded it may delete exactly that pod. Both effects take a minute to see:

$ kubectl apply -f hello-rs.yaml
replicaset.apps/hello-rs created
$ kubectl run stray --image=ghcr.io/example/hello:1.0 --labels=app.kubernetes.io/name=hello-rs
pod/stray created
$ kubectl get pods -l app.kubernetes.io/name=hello-rs
NAME             READY   STATUS              RESTARTS   AGE
hello-rs-8x855   1/1     Running             0          11s
hello-rs-sbbkz   0/1     ContainerCreating   0          11s
hello-rs-v9247   1/1     Running             0          11s
stray            0/1     Terminating         0          3s
$ kubectl get pod stray -o jsonpath='{.metadata.ownerReferences[0].kind}/{.metadata.ownerReferences[0].name}{"\n"}'
ReplicaSet/hello-rs
$ kubectl set image replicaset/hello-rs hello=ghcr.io/example/hello:1.1
replicaset.apps/hello-rs image updated
$ kubectl get pods -l app.kubernetes.io/name=hello-rs -o custom-columns=NAME:.metadata.name,IMAGE:.spec.containers[0].image
NAME             IMAGE
hello-rs-8x855   ghcr.io/example/hello:1.0
hello-rs-sbbkz   ghcr.io/example/hello:1.0
hello-rs-v9247   ghcr.io/example/hello:1.0
stray            ghcr.io/example/hello:1.0

The ReplicaSet adopted the pod stray (its ownerReference already points to hello-rs), counted four pods with replicas: 3 and deleted the extra one — the youngest, that is, the foreign one. And changing the image in the template did not touch a single running pod.

That is why ReplicaSets are created only through a Deployment. The selector of a ReplicaSet created by a Deployment has an extra label pod-template-hash, and foreign pods do not fall under it.

$ kubectl delete replicaset hello-rs
replicaset.apps "hello-rs" deleted from default namespace

---

Deployment

hello-deploy.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello
  labels:
    app.kubernetes.io/name: hello
  annotations:
    kubernetes.io/change-cause: "hello 1.0"
spec:
  replicas: 4
  revisionHistoryLimit: 5
  minReadySeconds: 5
  progressDeadlineSeconds: 120
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app.kubernetes.io/name: hello
  template:
    metadata:
      labels:
        app.kubernetes.io/name: hello
    spec:
      terminationGracePeriodSeconds: 30
      containers:
      - name: hello
        image: ghcr.io/example/hello:1.0
        ports:
        - name: http
          containerPort: 8080
        startupProbe:
          httpGet: {path: /healthz, port: http}
          periodSeconds: 2
          failureThreshold: 15
        livenessProbe:
          httpGet: {path: /healthz, port: http}
          periodSeconds: 10
          timeoutSeconds: 2
        readinessProbe:
          httpGet: {path: /readyz, port: http}
          periodSeconds: 5
          timeoutSeconds: 2
          failureThreshold: 2
        lifecycle:
          preStop:
            sleep:
              seconds: 5

The pod template is the same hello as in the previous part, without the nginx sidecar: that one was needed to show how a pod is built. The Deployment fields that concern rollouts:

Field	Default	Meaning
strategy.type	RollingUpdate	gradual replacement; the other option is Recreate
rollingUpdate.maxSurge	25%	by how many pods replicas may be exceeded during a rollout
rollingUpdate.maxUnavailable	25%	by how many pods the number of available ones may drop below replicas
minReadySeconds	0	how many seconds a pod must be ready before it counts as available
progressDeadlineSeconds	600	how many seconds a rollout may make no progress before it is declared stuck
revisionHistoryLimit	10	how many old ReplicaSets to keep for rollback

The kubernetes.io/change-cause annotation on the Deployment itself is the description of a revision that kubectl rollout history shows: when a new ReplicaSet is created, the controller copies it there. That is why it is changed together with the image, in the same edit. The old way — the --record flag of kubectl — is deprecated.

The selector of a Deployment in apps/v1 is immutable: once created, it cannot be changed by apply or by edit. Changing the application's labels means creating a new Deployment. That is why the selector is made narrow and stable from the start: only the application name, without the version or anything else that will change.

$ kubectl apply -f hello-deploy.yaml
deployment.apps/hello created
$ kubectl rollout status deployment/hello
Waiting for deployment spec update to be observed...
Waiting for deployment spec update to be observed...
Waiting for deployment "hello" rollout to finish: 0 out of 4 new replicas have been updated...
Waiting for deployment "hello" rollout to finish: 0 of 4 updated replicas are available...
Waiting for deployment "hello" rollout to finish: 0 of 4 updated replicas are available...
Waiting for deployment "hello" rollout to finish: 0 of 4 updated replicas are available...
Waiting for deployment "hello" rollout to finish: 0 of 4 updated replicas are available...
Waiting for deployment "hello" rollout to finish: 0 of 4 updated replicas are available...
Waiting for deployment "hello" rollout to finish: 1 of 4 updated replicas are available...
Waiting for deployment "hello" rollout to finish: 2 of 4 updated replicas are available...
Waiting for deployment "hello" rollout to finish: 3 of 4 updated replicas are available...
deployment "hello" successfully rolled out
$ kubectl get deployment,replicaset,pods -l app.kubernetes.io/name=hello
NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/hello   4/4     4            4           19s

NAME                              DESIRED   CURRENT   READY   AGE
replicaset.apps/hello-8c9694f75   4         4         4       19s

NAME                        READY   STATUS    RESTARTS   AGE
pod/hello-8c9694f75-4nf7n   1/1     Running   0          18s
pod/hello-8c9694f75-9ghql   1/1     Running   0          18s
pod/hello-8c9694f75-g2rqk   1/1     Running   0          18s
pod/hello-8c9694f75-j87gg   1/1     Running   0          19s

kubectl rollout status prints a line on every change of the Deployment's status, so identical lines repeat. The Deployment columns: READY — ready pods out of desired, UP-TO-DATE — pods with the current template, AVAILABLE — pods ready for at least minReadySeconds.

pod-template-hash

8c9694f75 in the ReplicaSet name is the pod-template-hash: a hash of the pod template computed by the deployment-controller (up to 10 characters, so the names of one Deployment's ReplicaSets can differ in length). It is also written as a label into the ReplicaSet's template and into its selector:

$ kubectl get replicaset hello-8c9694f75 -o jsonpath='{.spec.selector.matchLabels}{"\n"}'
{"app.kubernetes.io/name":"hello","pod-template-hash":"8c9694f75"}

This way the pods of different ReplicaSets of one Deployment do not overlap: they all have the label app.kubernetes.io/name=hello, but each ReplicaSet sees only pods with its own hash. The same template gives the same hash — that is why, when the template is returned to its previous form, the Deployment does not create a new ReplicaSet but finds the old one. The pod-template-hash label is never touched by hand.

---

Rolling update step by step

An update starts with any change of the pod template (spec.template): the image, variables, probes, template labels, template annotations. A change of replicas or of the strategy fields is not a rollout — it is just scaling or a new setting.

The deployment-controller moves replicas from the old ReplicaSet to the new one within two limits:

• total pods (old and new, ready or not) no more than replicas + maxSurge;
• available pods no fewer than replicas − maxUnavailable.

Terminating pods are not counted in either: a removed old pod spends a few more seconds running preStop and serving the last requests, while a new one already takes its place. So during a rollout there can be more pods, and more resources consumed, than replicas + maxSurge.

Percentages are converted into pods: maxSurge is rounded up, maxUnavailable down, and they cannot both be zero — then the rollout could not make a single step.

By default (25% and 25%) with replicas: 4 this gives maxSurge = 1, maxUnavailable = 1: at most 5 pods, at least 3 available. This is how the rollout went on the test stand (the exact order depends on when pods become ready):

Step	What the controller does	Old RS	New RS (available)	Total	Available
0	—	4	0	4	4
1	new +1 (up to 5 total); old −1 (down to 3 available)	3	1 (0)	4	3
2	new +1: the removed old pod is terminating and not counted, 5 again	3	2 (0)	5	3
3	two new ones became available: old −1, new +1	2	3 (2)	5	4
4	old −1, new +1	1	4 (2)	5	3
5	new ones available: old −1	0	4 (4)	4	4

Fast — 25 seconds on the stand — but during the process only three pods out of four are available: a quarter of the capacity goes into the rollout. For an application with four replicas, all of which are needed under load, this is noticeable.

maxSurge: 1, maxUnavailable: 0 — what our manifest has: not a single old pod is removed until an available new one has come to replace it.

Step	What the controller does	Old RS	New RS (available)	Total	Available
0	—	4	0	4	4
1	new +1	4	1 (0)	5	4
2	new one available: old −1, new +1	3	2 (1)	5	4
3	old −1, new +1	2	3 (2)	5	4
4	old −1, new +1	1	4 (3)	5	4
5	old −1	0	4 (4)	4	4

Slower — one pod per step, 40 seconds on the stand — but the capacity never drops below four replicas, and during the rollout the nodes need resources for only one extra pod (not counting terminating ones). For a small number of replicas this is the best choice. The opposite variant, maxSurge: 0, maxUnavailable: 1, is needed when there is nowhere to place an extra pod: the cluster is full or each pod holds a unique resource.

minReadySeconds and the readiness probe

"Available" is the key word. The controller moves to the next step only when the new pods have become available: ready by the readiness probe (the Ready=True condition) and staying ready for at least minReadySeconds. Hence the direct link between probes and rollout safety:

• Without a readinessProbe a pod counts as ready the moment its container starts. If the new version crashes a second after start or answers with errors, the controller will not notice and will replace all four pods with broken ones — the rollout is "successful".
• With a readinessProbe a broken version never becomes ready, and the rollout stops at the first new pod. The old ones keep working.
• minReadySeconds catches what readiness does not: an application that passes the first check and crashes a few seconds later (it did not wait for the database, it ran out of memory warming up a cache). The pod must stay ready for minReadySeconds before an old one is removed in its place. We have 5 seconds; for applications with a long warm-up — 30 and more.

progressDeadlineSeconds

If a rollout is stuck — a new pod does not become available — the controller waits progressDeadlineSeconds (120 here, 600 by default) since the last progress and sets the Progressing condition in status.conditions with status False and reason ProgressDeadlineExceeded. kubectl rollout status then exits with an error, and the deployment pipeline that called it learns about the failure.

What a Deployment does not do matters: it does not roll back by itself. The new ReplicaSet stays with its pods that are not ready, the old one with the working ones, and all of it waits for a decision by a human or an external tool. The states of the Progressing condition:

status	reason	Meaning
True	NewReplicaSetCreated, FoundNewReplicaSet, ReplicaSetUpdated	rollout in progress
True	NewReplicaSetAvailable	rollout complete
False	ProgressDeadlineExceeded	rollout stuck
Unknown	DeploymentPaused	the Deployment is paused (more on that below)

Rollover

If you change the template again while the previous rollout is not finished, the Deployment does not wait for it to complete: it creates one more ReplicaSet and starts moving replicas to it, and the half-rolled-out one becomes old and is scaled down together with the rest. The intermediate version never fully arrives — that is normal.

---

Rolling out hello 1.0 → 1.1

We build and push the new version — the same Dockerfile from part 5, a different version:

$ docker buildx build --platform linux/amd64,linux/arm64 \
    --build-arg VERSION=1.1 \
    -t ghcr.io/example/hello:1.1 --push .

We change the image and the revision description in the manifest and first look at what exactly will change in the cluster — kubectl diff compares the file with the live object on the server:

$ sed -i 's/hello:1.0/hello:1.1/; s/"hello 1.0"/"hello 1.1"/' hello-deploy.yaml
$ kubectl diff -f hello-deploy.yaml
diff -u -N /tmp/LIVE-4064112325/apps.v1.Deployment.default.hello /tmp/MERGED-16719779/apps.v1.Deployment.default.hello
--- /tmp/LIVE-4064112325/apps.v1.Deployment.default.hello	2026-10-26 22:09:53.564641082 +0300
+++ /tmp/MERGED-16719779/apps.v1.Deployment.default.hello	2026-10-26 22:09:53.565641119 +0300
@@ -5,9 +5,9 @@
     deployment.kubernetes.io/revision: "1"
     kubectl.kubernetes.io/last-applied-configuration: |
       {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{"kubernetes.io/change-cause":"hello 1.0"},…}
-    kubernetes.io/change-cause: hello 1.0
+    kubernetes.io/change-cause: hello 1.1
   creationTimestamp: "2026-10-26T19:09:12Z"
-  generation: 1
+  generation: 2
   labels:
     app.kubernetes.io/name: hello
   name: hello
@@ -33,7 +33,7 @@
         app.kubernetes.io/name: hello
     spec:
       containers:
-      - image: ghcr.io/example/hello:1.0
+      - image: ghcr.io/example/hello:1.1
         imagePullPolicy: IfNotPresent
         lifecycle:
           preStop:

The last-applied-configuration line is shortened here: it is an annotation into which kubectl apply writes the whole applied manifest as one line of JSON — we will come back to it. generation will grow by one: this is how the API server marks every change of spec. imagePullPolicy: IfNotPresent in the live object is that very default from part 5, filled in by the API server at creation. If there are differences, kubectl diff exits with code 1, if not — with 0, which is handy to check in scripts.

We apply and watch the ReplicaSets:

$ kubectl apply -f hello-deploy.yaml
deployment.apps/hello configured
$ kubectl get replicaset -l app.kubernetes.io/name=hello -w
NAME              DESIRED   CURRENT   READY   AGE
hello-8c9694f75   4         4         4       47s
hello-85854d9fd8   1         0         0       0s
hello-85854d9fd8   1         0         0       0s
hello-85854d9fd8   1         1         0       0s
hello-85854d9fd8   1         1         1       5s
hello-85854d9fd8   1         1         1       10s
hello-8c9694f75    3         4         4       58s
hello-85854d9fd8   2         1         1       10s
hello-8c9694f75    3         4         4       58s
hello-8c9694f75    3         3         3       58s
hello-85854d9fd8   2         1         1       10s
hello-85854d9fd8   2         2         1       11s
hello-85854d9fd8   2         2         2       16s
hello-8c9694f75    3         3         3       65s
hello-85854d9fd8   2         2         2       20s
hello-8c9694f75    2         3         3       68s
…
hello-85854d9fd8   4         4         4       36s
hello-8c9694f75    1         1         1       85s
hello-85854d9fd8   4         4         4       40s
hello-8c9694f75    0         1         1       88s
hello-8c9694f75    0         1         1       88s
hello-8c9694f75    0         0         0       88s
hello-8c9694f75    0         0         0       95s

-w prints a line on every change of an object, so identical lines follow each other, and the columns are aligned only to the first one. This is the maxSurge: 1, maxUnavailable: 0 table: the new pod is ready after 5 seconds, another 5 is minReadySeconds, and only after that the old ReplicaSet goes down by one and the new one gets the next replica. If the 1.1 image is not on the node yet, the time to pull it is added to the first step. The old ReplicaSet's CURRENT drops at once together with DESIRED: the pod being deleted is no longer counted there. It terminates for seven more seconds — preStop for 5 seconds and the exit on SIGTERM from the previous part — and the line with the same numbers 7 seconds later (65s after 58s) is the moment it is finally gone.

The same steps in the Deployment's events:

$ kubectl describe deployment hello | sed -n '/^Events:/,$p'
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  100s  deployment-controller  Scaled up replica set hello-8c9694f75 from 0 to 4
  Normal  ScalingReplicaSet  52s   deployment-controller  Scaled up replica set hello-85854d9fd8 from 0 to 1
  Normal  ScalingReplicaSet  42s   deployment-controller  Scaled down replica set hello-8c9694f75 from 4 to 3
  Normal  ScalingReplicaSet  42s   deployment-controller  Scaled up replica set hello-85854d9fd8 from 1 to 2
  Normal  ScalingReplicaSet  32s   deployment-controller  Scaled down replica set hello-8c9694f75 from 3 to 2
  Normal  ScalingReplicaSet  32s   deployment-controller  Scaled up replica set hello-85854d9fd8 from 2 to 3
  Normal  ScalingReplicaSet  22s   deployment-controller  Scaled down replica set hello-8c9694f75 from 2 to 1
  Normal  ScalingReplicaSet  22s   deployment-controller  Scaled up replica set hello-85854d9fd8 from 3 to 4
  Normal  ScalingReplicaSet  12s   deployment-controller  Scaled down replica set hello-8c9694f75 from 1 to 0

The deployment-controller does not touch pods — only the replicas field of the two ReplicaSets. Pods are created and deleted by the replicaset-controller.

We check the version with a port-forward to the Deployment (kubectl picks one of its pods):

$ kubectl port-forward deployment/hello 8080:8080 &
Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080
$ curl -si http://127.0.0.1:8080/ | grep -E 'X-App-Version|hello from'
Handling connection for 8080
X-App-Version: 1.1
hello from hello-85854d9fd8-lnnbv
$ kill %1

History and rollback

$ kubectl rollout history deployment/hello
deployment.apps/hello
REVISION  CHANGE-CAUSE
1         hello 1.0
2         hello 1.1

$ kubectl rollout history deployment/hello --revision=1 | grep -E 'Image|change-cause'
  Annotations:	kubernetes.io/change-cause: hello 1.0
    Image:	ghcr.io/example/hello:1.0

A revision is the deployment.kubernetes.io/revision annotation on a ReplicaSet; the history is simply the list of this Deployment's ReplicaSets, including those with zero replicas. revisionHistoryLimit: 5 limits how many such empty ReplicaSets are kept; older ones are deleted, and you can no longer roll back to them. revisionHistoryLimit: 0 disables rollback entirely.

Rolling back to the previous revision:

$ kubectl rollout undo deployment/hello
Warning: resource deployments/hello was previously managed with 'kubectl apply'. Rolling back will not update the kubectl.kubernetes.io/last-applied-configuration annotation, which may cause unexpected behavior on future 'kubectl apply' operations. Consider using 'kubectl apply' with your previous configuration file instead.
deployment.apps/hello rolled back
$ kubectl rollout status deployment/hello
…
deployment "hello" successfully rolled out
$ kubectl get replicaset -l app.kubernetes.io/name=hello
NAME               DESIRED   CURRENT   READY   AGE
hello-85854d9fd8   0         0         0       98s
hello-8c9694f75    4         4         4       2m10s
$ kubectl rollout history deployment/hello
deployment.apps/hello
REVISION  CHANGE-CAUSE
2         hello 1.1
3         hello 1.0

undo is not a time machine but an ordinary rollout: kubectl copies the template from the ReplicaSet of the required revision into the Deployment (together with the change-cause annotation), and everything follows the same maxSurge and maxUnavailable rules. The template matched the old one, the hash is the same — so the former hello-8c9694f75 came back to life instead of a new one being created. Its revision became 3: revision 1 disappeared from the history because the same ReplicaSet is now the latest. To return to a specific revision — kubectl rollout undo deployment/hello --to-revision=2.

One caveat about undo in real work, and kubectl itself warns about it. It changes the object in the cluster but not the file hello-deploy.yaml and not the last-applied-configuration annotation: the file still has 1.1, and the next kubectl apply will roll out 1.1 again. If the manifests live in Git, you roll back with a commit in Git and apply it — this way the cluster and the repository do not diverge. undo is an emergency tool, after which the manifest is brought back in line. We will come back to this in part 41, where the cluster is synchronised from Git automatically.

Let's bring 1.1 back — the file already says so:

$ kubectl apply -f hello-deploy.yaml
deployment.apps/hello configured

A rollout that failed

A typo in the tag — there is no version 1.2 in the registry:

$ kubectl set image deployment/hello hello=ghcr.io/example/hello:1.2
deployment.apps/hello image updated
$ kubectl rollout status deployment/hello --timeout=3m
Waiting for deployment spec update to be observed...
Waiting for deployment "hello" rollout to finish: 0 out of 4 new replicas have been updated...
Waiting for deployment "hello" rollout to finish: 1 out of 4 new replicas have been updated...
error: deployment "hello" exceeded its progress deadline
$ kubectl get pods -l app.kubernetes.io/name=hello
NAME                     READY   STATUS             RESTARTS   AGE
hello-85854d9fd8-5zw99   1/1     Running            0          2m33s
hello-85854d9fd8-h87n6   1/1     Running            0          2m55s
hello-85854d9fd8-pqkhz   1/1     Running            0          2m22s
hello-85854d9fd8-rfhcx   1/1     Running            0          2m44s
hello-8d8f8775c-hk7jb    0/1     ImagePullBackOff   0          2m1s
$ kubectl get deployment hello -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'
Available=True MinimumReplicasAvailable
Progressing=False ProgressDeadlineExceeded

kubectl set image is an imperative change of one field, convenient here for the demonstration; in normal work the image is changed in the manifest. The new pod failed to pull the image (its events say failed to resolve image: ghcr.io/example/hello:1.2: not found), went to ErrImagePull, then to ImagePullBackOff, and never became ready. With maxUnavailable: 0 none of the four old pods were touched — the application runs on 1.1 at full capacity. After 120 seconds without progress the Deployment declared the rollout stuck, and rollout status returned an error and exit code 1. The Available=True condition says that there are at least as many available pods as the strategy requires.

With the default settings (maxUnavailable: 25%) the picture would be worse: one old pod would be removed at once, and the application would run on three pods out of four until someone stepped in.

The same goes for a broken version that pulls fine but fails readiness: the rollout stops at the very first pod. That is exactly why the readiness probe and maxUnavailable: 0 are the best insurance against a bad release that a Deployment has without external tools.

Incidentally, this shows what the history is worth without the annotation:

$ kubectl rollout history deployment/hello
deployment.apps/hello
REVISION  CHANGE-CAUSE
3         hello 1.0
4         hello 1.1
5         hello 1.1

Revision 5 is 1.2, but set image did not touch the kubernetes.io/change-cause annotation, and the new ReplicaSet got the previous description. The history is exactly as honest as the care with which this annotation is changed.

Rolling back:

$ kubectl rollout undo deployment/hello
Warning: resource deployments/hello was previously managed with 'kubectl apply'. Rolling back will not update the kubectl.kubernetes.io/last-applied-configuration annotation, which may cause unexpected behavior on future 'kubectl apply' operations. Consider using 'kubectl apply' with your previous configuration file instead.
deployment.apps/hello rolled back
$ kubectl get replicaset -l app.kubernetes.io/name=hello
NAME               DESIRED   CURRENT   READY   AGE
hello-85854d9fd8   4         4         4       4m39s
hello-8c9694f75    0         0         0       5m11s
hello-8d8f8775c    0         0         0       2m7s

The broken pod is deleted, and hello-85854d9fd8 never changed. The empty hello-8d8f8775c does not need to be deleted by hand: in time revisionHistoryLimit will push it out. And the manifest hello-deploy.yaml still says 1.1 — set image did not touch it, so there is no divergence from the file.

---

pause, resume, restart

kubectl rollout pause freezes rollouts: template changes are accepted by the API server, but no new ReplicaSet is created. This is needed when there are several changes (image, variable, resources) and you want to roll them out as one rollout rather than three:

$ kubectl rollout pause deployment/hello
deployment.apps/hello paused
$ kubectl set env deployment/hello GOMAXPROCS=2
deployment.apps/hello env updated
$ kubectl set env deployment/hello GOMEMLIMIT=96MiB
deployment.apps/hello env updated
$ kubectl get replicaset -l app.kubernetes.io/name=hello | grep -c hello
3
$ kubectl rollout undo deployment/hello
error: you cannot rollback a paused deployment; resume it first with 'kubectl rollout resume' and try again
$ kubectl rollout resume deployment/hello
deployment.apps/hello resumed
$ kubectl get replicaset -l app.kubernetes.io/name=hello
NAME               DESIRED   CURRENT   READY   AGE
hello-77b6f6d957   4         4         4       41s
hello-85854d9fd8   0         0         0       5m23s
hello-8c9694f75    0         0         0       5m55s
hello-8d8f8775c    0         0         0       2m51s

GOMAXPROCS and GOMEMLIMIT are Go runtime variables: how many OS threads to use for running code, and a soft memory limit near which the garbage collector works harder. While the Deployment is paused there are still three ReplicaSets — no new ones appeared, the Progressing condition is Unknown with the reason DeploymentPaused, and rollout undo refuses to work. After resume both changes went out in one rollout — the new hello-77b6f6d957.

This was an experiment, and you would like to return the Deployment to the manifest with a simple kubectl apply -f hello-deploy.yaml. But that does not work:

$ kubectl apply -f hello-deploy.yaml
deployment.apps/hello unchanged
$ kubectl get deployment hello -o jsonpath='{.spec.template.spec.containers[0].env}{"\n"}'
[{"name":"GOMAXPROCS","value":"2"},{"name":"GOMEMLIMIT","value":"96MiB"}]

kubectl apply removes from an object only the fields it once wrote itself: it keeps their list in the last-applied-configuration annotation. The variables were never there — set env added them — and apply leaves them alone. A field that is in the file apply returns to the file's value (the image, replicas), while a field added around the file quietly stays in the cluster. So changes made with kubectl set are either also put into the manifest or removed with the same command:

$ kubectl set env deployment/hello GOMAXPROCS- GOMEMLIMIT-
deployment.apps/hello env updated
$ kubectl get replicaset -l app.kubernetes.io/name=hello
NAME               DESIRED   CURRENT   READY   AGE
hello-77b6f6d957   0         0         0       86s
hello-85854d9fd8   4         4         4       6m8s
hello-8c9694f75    0         0         0       6m40s
hello-8d8f8775c    0         0         0       3m36s

The template matched revision 1.1 again, and hello-85854d9fd8 came back to life — with new pods, while the ReplicaSet with the variables stayed in the history.

A pause also stops a rollout halfway if something went wrong and you need time to investigate: some pods are on the new version, some on the old one, and nothing changes. While the Deployment is paused, progressDeadlineSeconds is not counted.

kubectl rollout restart recreates all pods without changing the version: kubectl adds the kubectl.kubernetes.io/restartedAt annotation with the current time (2026-10-26T22:17:39+03:00) to the template; the template has changed — which means a new ReplicaSet and an ordinary rollout following the strategy. It is needed so that the application rereads something at start: a mounted Secret, a ConfigMap in environment variables (part 12), an updated root certificate.

---

The Recreate strategy

spec:
  strategy:
    type: Recreate

All old pods are deleted, the controller waits for them to terminate completely and only then creates new ones. Between these moments the application is unavailable. The guarantee covers only rollouts: if you delete a pod by hand, the ReplicaSet creates a replacement at once, without waiting for the old one to terminate.

Recreate is needed when two versions cannot run at the same time: the application holds a volume with ReadWriteOnce access that mounts on one node only (part 21); the new version changes the database schema so that the old one cannot work with it; the application is the only consumer of a queue where two instances are not allowed. If none of these reasons apply — RollingUpdate.

---

Scaling

$ kubectl scale deployment/hello --replicas=6
deployment.apps/hello scaled
$ kubectl get replicaset -l app.kubernetes.io/name=hello
NAME               DESIRED   CURRENT   READY   AGE
hello-77b6f6d957   0         0         0       98s
hello-85854d9fd8   6         6         6       6m20s
hello-8c9694f75    0         0         0       6m52s
hello-8d8f8775c    0         0         0       3m48s

Scaling changes only the Deployment's spec.replicas, and the Deployment carries the number over to the current ReplicaSet — no new revision appears. Under the hood kubectl scale writes to the scale subresource; the Horizontal Pod Autoscaler (HPA) works through it too (part 32).

If you scale in the middle of a rollout, the Deployment distributes the new replicas proportionally between the old and the new ReplicaSet, so as not to break the strategy's limits and not to skew the rollout to one side. On the stand a stuck rollout (6 old pods and 1 new one) became 9 and 2 after --replicas=10.

One trap. The manifest says replicas: 4, and someone ran kubectl scale --replicas=6 — the next kubectl apply brings it back to 4: this field is in the file. The same with an HPA: it set 8, and apply reset it to 4. When the number of replicas is managed by an HPA, the replicas field is removed from the manifest.

$ kubectl scale deployment/hello --replicas=4
deployment.apps/hello scaled

---

Hands-on

Ownership links. A pod points to a ReplicaSet, a ReplicaSet to a Deployment:

$ kubectl get pod hello-85854d9fd8-4mjwb -o jsonpath='{.metadata.ownerReferences}' | jq
[
  {
    "apiVersion": "apps/v1",
    "blockOwnerDeletion": true,
    "controller": true,
    "kind": "ReplicaSet",
    "name": "hello-85854d9fd8",
    "uid": "a439717d-1518-4c16-8ef4-935e092e9f5f"
  }
]
$ kubectl get replicaset hello-85854d9fd8 -o jsonpath='{.metadata.ownerReferences[0].kind}/{.metadata.ownerReferences[0].name}{"\n"}'
Deployment/hello

The Deployment's status — what kubectl rollout status and all delivery tools rely on:

$ kubectl get deployment hello -o jsonpath='{.metadata.generation}{"\n"}'
13
$ kubectl get deployment hello -o json | jq '.status | del(.conditions)'
{
  "availableReplicas": 4,
  "observedGeneration": 13,
  "readyReplicas": 4,
  "replicas": 4,
  "terminatingReplicas": 0,
  "updatedReplicas": 4
}

generation has grown to 13: every change of spec in this part — apply, undo, set, pause, resume, scale — added one. observedGeneration equals metadata.generation — the controller has processed the latest version of spec. If generation is greater, the controller has not seen the change yet, and any conclusions from status are premature. updatedReplicas — pods of the current template, availableReplicas — available ones, terminatingReplicas — pods in the middle of graceful termination that are no longer counted in replicas (the DeploymentReplicaSetTerminatingReplicas feature gate, beta since 1.35, enabled by default).

What is stored in etcd. On the master, with the etcdctl client from the etcd pod, using the certificates kubeadm put into /etc/kubernetes/pki/etcd/ (etcd in detail is in part 34):

$ kubectl -n kube-system exec etcd-k8s-master-1 -- etcdctl \
    --endpoints=https://127.0.0.1:2379 \
    --cacert=/etc/kubernetes/pki/etcd/ca.crt \
    --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
    --key=/etc/kubernetes/pki/etcd/healthcheck-client.key \
    get --prefix --keys-only /registry/ | grep -E 'deployments/default/hello|replicasets/default/hello|pods/default/hello'
/registry/deployments/default/hello
/registry/pods/default/hello-85854d9fd8-4mjwb
/registry/pods/default/hello-85854d9fd8-cmcqg
/registry/pods/default/hello-85854d9fd8-ptchf
/registry/pods/default/hello-85854d9fd8-z4h9z
/registry/replicasets/default/hello-77b6f6d957
/registry/replicasets/default/hello-85854d9fd8
/registry/replicasets/default/hello-8c9694f75
/registry/replicasets/default/hello-8d8f8775c

One key per object; the empty ReplicaSets of the history are objects just as complete as the working ones. The values are stored in protobuf — a binary serialisation format from Google, not JSON — so they are easier to read through the API.

Self-healing. Let's delete a pod and look at the list a second later:

$ kubectl delete pod hello-85854d9fd8-4mjwb --wait=false
pod "hello-85854d9fd8-4mjwb" deleted from default namespace
$ kubectl get pods -l app.kubernetes.io/name=hello
NAME                     READY   STATUS              RESTARTS   AGE
hello-85854d9fd8-2rzmr   0/1     ContainerCreating   0          1s
hello-85854d9fd8-4mjwb   1/1     Terminating         0          60s
hello-85854d9fd8-cmcqg   1/1     Running             0          50s
hello-85854d9fd8-ptchf   1/1     Running             0          39s
hello-85854d9fd8-z4h9z   1/1     Running             0          73s

The pod being deleted is still terminating, but the ReplicaSet has already created a replacement: a pod in Terminating does not count as active for it. That is exactly why terminatingReplicas appeared in the status — to show that while pods terminate there are actually more of them than replicas.

The controller-manager at work. The same episode in the controller-manager's journal — the logs of the static pod on the master. At the default verbosity these lines are absent; with --v=4 in the manifest /etc/kubernetes/manifests/kube-controller-manager.yaml (the kubelet restarts the pod itself after the file changes) you can see the replicaset-controller notice the deletion, create a replacement, and the Deployment sync right after (the command was run a few seconds after the pod was deleted):

$ kubectl -n kube-system logs kube-controller-manager-k8s-master-1 --since=20s | grep 'default/hello' \
    | grep -E '"(Too few replicas|Controller created pod|Updating status for|Started syncing deployment)' | head -5
I1026 19:41:10.935036       1 replica_set.go:668] "Too few replicas" replicaSet="default/hello-85854d9fd8" need=4 creating=1
I1026 19:41:11.106686       1 controller_utils.go:613] "Controller created pod" controller="hello-85854d9fd8" pod="default/hello-85854d9fd8-2rzmr"
I1026 19:41:11.106840       1 replica_set_utils.go:68] "Updating status for : default/hello-85854d9fd8, replicas 4->3 (need 4), fullyLabeledReplicas 4->3, readyReplicas 4->3, availableReplicas 4->3, terminatingReplicas 0->1, sequence No: 22->22"
I1026 19:41:11.168476       1 deployment_controller.go:598] "Started syncing deployment" deployment="default/hello" startTime="2026-10-26 19:41:11.168470276 +0000 UTC m=+41.308722235"
I1026 19:41:11.168857       1 replica_set_utils.go:68] "Updating status for : default/hello-85854d9fd8, replicas 3->4 (need 4), fullyLabeledReplicas 3->4, readyReplicas 3->3, availableReplicas 3->3, terminatingReplicas 1->1, sequence No: 22->22"

The controller-manager's journal is in UTC (19:41 is 22:41 in the stand's time zone). Raised verbosity is kept only for the time of an investigation and then turned back: at --v=4 the controller-manager writes hundreds of lines a minute even in a small cluster.

---

A Deployment covers almost everything a stateless application needs: the required number of pods, recovery, rollouts without downtime and rollback. But all its pods are identical and interchangeable: they have random names, a shared volume template and no order. That is not enough for a database or a message broker, and not the right thing for an agent that has to run on every node.

Sources

• Kubernetes: Deployment (strategies, proportional scaling, progressDeadlineSeconds, status conditions, terminating pods), ReplicaSet (pod deletion order, pod-deletion-cost), ReplicationController, controllers and the reconciliation loop.
• Owners and dependents and garbage collection (ownerReferences, cascading deletion, --cascade=orphan).
• Deployment API reference, the controller.kubernetes.io/pod-deletion-cost annotation, feature gates (DeploymentReplicaSetTerminatingReplicas).
• kubectl: rollout, diff, scale; declarative management with kubectl apply (how apply calculates differences and which fields it removes).
• kube-controller-manager options (--v), Go runtime environment variables (GOMAXPROCS, GOMEMLIMIT).

---

Next — StatefulSet and DaemonSet: stable names and volumes for stateful pods, ordered startup and updates, and one pod per node for agents.