Vitaliy Leschenko’s blog
ru
Виталий Лещенко

Kubernetes for beginners. Part 3: The API from the inside

All parts of the series “Kubernetes for beginners”

Everything that happens in the cluster goes through the API: in the first part the controllers read the desired state from it, and in the second we saw that kube-apiserver is the only one that talks to etcd, while all the other components are its clients. Today let's look into the API itself: how an object is built, what groups, versions, resources and subresources are, what kubectl's HTTP requests look like, how two clients avoid overwriting each other's changes, how subscribing to changes works, how objects are deleted, and how server-side apply differs from the ordinary kubectl apply. This is the foundation for all the other parts: every object from here on — Pod, Service, PersistentVolumeClaim (a volume claim), Application in Argo CD (the GitOps tool from the final part) — is built the same way.

The commands are run on the workstation against the lab cluster (its installation is in part 4), in the default namespace. The examples need the Deployment web from the first part:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  labels:
    app: web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: nginx
          image: nginx:1.29
          ports:
            - containerPort: 80

The object: apiVersion, kind, metadata, spec, status

Let's see what lies in the cluster after kubectl apply -f web.yaml:

$ kubectl get deployment web -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app":"web"},"name":"web","namespace":"default"},"spec":{"replicas":3,"selector":{"matchLabels":{"app":"web"}},"template":{"metadata":{"labels":{"app":"web"}},"spec":{"containers":[{"image":"nginx:1.29","name":"nginx","ports":[{"containerPort":80}]}]}}}}
  creationTimestamp: "2026-10-21T09:12:40Z"
  generation: 1
  labels:
    app: web
  name: web
  namespace: default
  resourceVersion: "1843217"
  uid: 7d1e9c2a-4b3f-4e8a-9c6d-2f0b1a5e8c34
spec:
  progressDeadlineSeconds: 600
  replicas: 3
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: web
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - image: nginx:1.29
        imagePullPolicy: IfNotPresent
        name: nginx
        ports:
        - containerPort: 80
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
status:
  availableReplicas: 3
  conditions:
  - lastTransitionTime: "2026-10-21T09:12:44Z"
    lastUpdateTime: "2026-10-21T09:12:44Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: "2026-10-21T09:12:40Z"
    lastUpdateTime: "2026-10-21T09:12:44Z"
    message: ReplicaSet "web-7c5ddbdf54" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  observedGeneration: 1
  readyReplicas: 3
  replicas: 3
  terminatingReplicas: 0
  updatedReplicas: 3

The object is twice as long as the manifest. Let's go through it piece by piece.

apiVersion and kind say what the object is: the Deployment type from the apps group, version v1. By this pair kube-apiserver finds the schema, and kubectl finds the path to send the request to.

metadata is the part shared by all objects:

Field Set by Meaning
name client a name unique within the namespace and the type
namespace client the namespace; cluster-scoped objects (Node, Namespace, PersistentVolume) don't have one
uid server a unique identifier for the whole lifetime of the cluster: an object deleted and created again with the same name is a different object with a different uid
resourceVersion server the version of this state of the object; changes with every write
generation server the version number of spec; grows only when the desired state changes
creationTimestamp server the creation time
labels client labels for selection: service and Deployment selectors, kubectl get -l (part 12)
annotations client and controllers arbitrary data that isn't searched by: service marks, add-on configuration
ownerReferences controllers who owns the object; cascading deletion rests on this
finalizers controllers what must happen before the object is deleted
deletionTimestamp server the object is being deleted and is waiting for finalizers
managedFields server which fields were set by which client (server-side apply); kubectl get hides them, --show-managed-fields shows them

The kubectl.kubernetes.io/last-applied-configuration annotation is a trace of client-side kubectl apply; we'll come back to it below.

spec is the desired state. The server has filled it in with default values: the RollingUpdate update strategy with 25% for maxSurge and maxUnavailable, imagePullPolicy: IfNotPresent (because the tag isn't latest), restartPolicy: Always, 30 seconds for graceful termination and so on. Each such value is a decision made for you, and they're worth knowing: in the parts on pods and Deployments we'll go through almost all of them.

status is the observed state, written by the controller. It usually contains conditions (conditions): a list of facts of the form "type — true/false — reason — message — when it changed". For a Deployment these are Available ("enough replicas") and Progressing ("an update is in progress or has finished"). Conditions are read by people, by kubectl wait --for=condition=Available and by other controllers.

Not every object has a spec/status pair. ConfigMap and Secret have data instead, Event has the fields of the event itself, RBAC roles have rules. But every object has apiVersion, kind and metadata.

Groups, versions and resources

The API is divided into groups — sets of related types that evolve independently. The types that appeared first (Pod, Service, ConfigMap, Secret, Node, Namespace) live in the core group, which has no name: in apiVersion you write just v1. The rest are in named groups: apps (Deployment, StatefulSet, DaemonSet, ReplicaSet), batch (Job, CronJob), networking.k8s.io (Ingress, NetworkPolicy), rbac.authorization.k8s.io, storage.k8s.io and so on. Add-ons add groups of their own: the Calico network plugin adds crd.projectcalico.org and projectcalico.org, cert-manager (which issues TLS certificates) adds cert-manager.io.

Every group has versions with a clear level of stability:

  • alpha (v1alpha1) — experimental, disabled by default, may disappear without warning;
  • beta (v1beta1) — tested, but the schema may still change; deprecated beta versions are removed a few releases after the announcement;
  • GA, stable (v1, v2) — not removed for as long as Kubernetes 1.x lives.

The same object can be read through different versions of one group if the server serves them: kube-apiserver stores the object in etcd in one version (the storage version) and converts it on read. When a version is deprecated, the server adds a Warning header to the response, and kubectl prints it: Warning: <group>/<version> <type> is deprecated in v1.X+, unavailable in v1.Y+. Such warnings in the output of kubectl apply are a reason to update the manifests before a cluster upgrade breaks them (part 36).

Kind and resource are different things. A kind is the type of the object in the request body (Deployment, capitalized, singular). A resource is the name of the collection in the URL (deployments, lowercase, plural). They usually correspond one to one. Subresources are a special case: they have no stored object of their own, they read and change part of the parent object, and their kind is either the parent's own (deployments/status returns a Deployment) or a service kind (deployments/scale returns a Scale).

 /apis/apps/v1/namespaces/default/deployments/web/scale
  │    │    │             │       │           │   │
  │    │    │             │       │           │   └─ subresource
  │    │    │             │       │           └─ object name
  │    │    │             │       └─ resource: the type in the plural
  │    │    │             └─ namespace
  │    │    └─ version
  │    └─ group
  └─ /apis — named groups, /api — the core group
Action Method and path
list pods in a namespace GET /api/v1/namespaces/default/pods
list pods in all namespaces GET /api/v1/pods
one pod GET /api/v1/namespaces/default/pods/test
create a pod POST /api/v1/namespaces/default/pods
replace an object entirely PUT /apis/apps/v1/namespaces/default/deployments/web
change part of an object PATCH /apis/apps/v1/namespaces/default/deployments/web
delete DELETE /apis/apps/v1/namespaces/default/deployments/web
subscribe to changes GET /api/v1/namespaces/default/pods?watch=1
a cluster-scoped object GET /api/v1/nodes/k8s-node-1

In authorization terms, HTTP methods turn into verbs: get, list, watch, create, update (PUT), patch, delete, deletecollection. RBAC rules are written in exactly these (part 30).

Subresources are additional paths on an object with their own permissions and their own semantics:

Subresource Why
/status writing status separately from spec: a controller changes only the status and can't accidentally overwrite the desired state, and a user can't forge the status through the main path
/scale a common "how many replicas" interface for Deployment, StatefulSet, ReplicaSet and custom types; used by kubectl scale and autoscaling
/log a pod container's logs (kubectl logs); kube-apiserver proxies the request to kubelet
/exec, /attach, /portforward interactive streams to a container through kube-apiserver and kubelet
/binding assigning a pod to a node — used by the scheduler
/eviction polite eviction of a pod that respects the PodDisruptionBudget — a limit on the number of an application's pods that may be unavailable at the same time (part 14)
/ephemeralcontainers adding a debug container to a running pod (kubectl debug, part 32)

Permissions on a subresource are granted separately: a role with pods doesn't give pods/exec and pods/log, and that's a good thing — reading the description of pods and running commands in them are completely different powers.

The server provides the list of groups, resources and subresources itself (this is called discovery), and kubectl builds its "type → path" mapping from it:

$ kubectl api-resources --api-group=apps
NAME                  SHORTNAMES   APIVERSION   NAMESPACED   KIND
controllerrevisions                apps/v1      true         ControllerRevision
daemonsets            ds           apps/v1      true         DaemonSet
deployments           deploy       apps/v1      true         Deployment
replicasets           rs           apps/v1      true         ReplicaSet
statefulsets          sts          apps/v1      true         StatefulSet

$ kubectl get --raw /apis/apps/v1 | jq -r '.resources[] | select(.name|startswith("deployments")) | "\(.name)  \(.kind)  \(.verbs|join(","))"'
deployments  Deployment  create,delete,deletecollection,get,list,patch,update,watch
deployments/scale  Scale  get,patch,update
deployments/status  Deployment  get,patch,update

kubectl api-resources without arguments shows all the cluster's types, including those added by Calico; the NAMESPACED column tells whether the type lives in a namespace.

kubectl explain: the schema without documentation in a browser

The server publishes the schema of all types in the OpenAPI format — a standard machine-readable description of an HTTP API (/openapi/v3), and kubectl explain reads it, with field descriptions for the version running in the cluster:

$ kubectl explain deployment.spec.strategy
GROUP:      apps
KIND:       Deployment
VERSION:    v1

FIELD: strategy <DeploymentStrategy>

DESCRIPTION:
    The deployment strategy to use to replace existing pods with new ones.
    DeploymentStrategy describes how to replace existing pods with new ones.

FIELDS:
  rollingUpdate <RollingUpdateDeployment>
    Rolling update config params. Present only if DeploymentStrategyType =
    RollingUpdate.

  type  <string>
  enum: Recreate, RollingUpdate
    Type of deployment. Can be "Recreate" or "RollingUpdate". Default is
    RollingUpdate.

    Possible enum values:
     - `"Recreate"` Kill all existing pods before creating new ones.
     - `"RollingUpdate"` Replace the old ReplicaSets by new one using rolling
    update i.e gradually scale down the old ReplicaSets and scale up the new
    one.

Useful variants: kubectl explain pod.spec --recursive — a tree of all fields without descriptions, kubectl explain ippools --api-version=crd.projectcalico.org/v1 — the schema of a type from an add-on (if the author of the CRD — CustomResourceDefinition, the description of a custom object type, covered in part 29 — has defined a schema). This is faster and more accurate than searching the internet: documentation online may be for a different version, while explain shows exactly your cluster.

How kubectl talks to the server

The -v flag (log verbosity level) shows kubectl's requests: -v=6 — the method, URL and response code, -v=8 — also the headers and the beginning of the body, -v=9 — the whole body.

$ kubectl get deployment web -v=8
I1021 12:20:31.101874   48211 loader.go:407] Config loaded from file:  /home/admin/.kube/config
I1021 12:20:31.102331   48211 envvar.go:195] "Feature gate default state" feature="InformerResourceVersion" enabled=true
I1021 12:20:31.102331   48211 envvar.go:195] "Feature gate default state" feature="WatchListClient" enabled=true
...
I1021 12:20:31.104876   48211 helper.go:105] "Request Body" body=""
I1021 12:20:31.104876   48211 round_trippers.go:527] "Request" verb="GET" url="https://k8s-api.example.com:6443/apis/apps/v1/namespaces/default/deployments/web" headers=<
    Accept: application/json;as=Table;v=v1;g=meta.k8s.io,application/json;as=Table;v=v1beta1;g=meta.k8s.io,application/json
    User-Agent: kubectl/v1.36.5 (linux/amd64) kubernetes/4c1b3e8
 >
I1021 12:20:31.118302   48211 round_trippers.go:632] "Response" status="200 OK" headers=<
    Audit-Id: 9b2f6d1e-3c4a-4f5b-8e7d-0a1c2b3d4e5f
    Cache-Control: no-cache, private
    Content-Type: application/json
    Date: Wed, 21 Oct 2026 09:20:31 GMT
    X-Kubernetes-Pf-Flowschema-Uid: 0d4e2c1b-7a6f-4b3e-9c8d-1e2f3a4b5c6d
    X-Kubernetes-Pf-Prioritylevel-Uid: 5a6b7c8d-9e0f-4a1b-8c2d-3e4f5a6b7c8d
 > milliseconds=13
I1021 12:20:31.118911   48211 helper.go:105] "Response Body" body="{\"kind\":\"Table\",\"apiVersion\":\"meta.k8s.io/v1\",\"metadata\":{\"resourceVersion\":\"1843590\"},\"columnDefinitions\":[{\"name\":\"Name\",\"type\":\"string\",\"format\":\"name\",\"description\":\"Name must be unique within a namespace. ... [truncated 3539 chars]"
NAME   READY   UP-TO-DATE   AVAILABLE   AGE
web    3/3     3            3           8m

A few observations:

  • The Feature gate default state lines show the feature gates of the client-go library itself; they have nothing to do with the request, so only two of them are kept here, and the rest, like the middle of the long response body, are replaced with an ellipsis.
  • kubectl get asks the server not for the object but for a table (Accept: application/json;as=Table): the READY and UP-TO-DATE columns are computed by the server, which is why an older kubectl correctly displays types it knows nothing about. With -o yaml or -o json the header is just application/json.
  • Audit-Id is the request's identifier in the kube-apiserver audit log, if auditing is enabled.
  • X-Kubernetes-Pf-* says which flow schema and priority level of API Priority and Fairness (the mechanism that shares the server's capacity among clients) the request landed in.

Any API path can be requested with kubectl get --raw, with the same credentials as ordinary commands:

$ kubectl get --raw /api
{"kind":"APIVersions","versions":["v1"],"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"192.168.10.17:6443"}]}

$ kubectl get --raw /apis/apps/v1/namespaces/default/deployments/web/scale | jq .
{
  "kind": "Scale",
  "apiVersion": "autoscaling/v1",
  "metadata": {
    "name": "web",
    "namespace": "default",
    "uid": "7d1e9c2a-4b3f-4e8a-9c6d-2f0b1a5e8c34",
    "resourceVersion": "1843217",
    "creationTimestamp": "2026-10-21T09:12:40Z"
  },
  "spec": {
    "replicas": 3
  },
  "status": {
    "replicas": 3,
    "selector": "app=web"
  }
}

The /scale subresource returned an object of a different kind — Scale from the autoscaling group — with a single field in spec. This way autoscaling doesn't need to know how a Deployment is built. Its uid and resourceVersion are the Deployment's own: Scale isn't stored anywhere, the server assembles it from the Deployment on every request, and writing spec.replicas of the Scale changes spec.replicas of the Deployment.

For requests kubectl can't make (for example, with the PUT method), kubectl proxy is handy: it opens a local HTTP port on the workstation and proxies it to the API with your credentials. We'll use it below. The proxy listens only on 127.0.0.1, and it must not be opened to other addresses: anyone who can reach the port gets your permissions in the cluster.

The path of a request inside kube-apiserver

 request over HTTPS
   │
   ▼
 authentication: who is it? ───────────────────────► 401 Unauthorized
   │
   ▼
 authorization (Node, RBAC): is it allowed? ───────► 403 Forbidden
   │
   ▼
 mutating admission: fill in or modify the object
   │
   ▼
 schema and field validation ──────────────────────► 422 Invalid
   │
   ▼
 validating admission: accept or reject ───────────► rejection with a message
   │
   ▼
 write to etcd with a resourceVersion check ───────► 409 Conflict
   │
   ▼
 response 200 OK or 201 Created, event to subscribers

Admission consists of modules that see the object after authorization and before the write. Mutating modules can change it, validating ones can only accept or reject it. The sources:

  • built-in plugins enabled in kube-apiserver: NamespaceLifecycle (objects can't be created in a namespace being deleted), LimitRanger and ResourceQuota (default values and quotas, part 11), ServiceAccount (the pod's service account and token), DefaultStorageClass, NodeRestriction (kubelet changes only its own node), PodSecurity (pod security standards, part 31) and others;
  • policies in CEL (Common Expression Language — a small expression language that kube-apiserver runs itself): ValidatingAdmissionPolicy and, stable since 1.36, MutatingAdmissionPolicy. They don't require a separate service;
  • webhooks: ValidatingWebhookConfiguration and MutatingWebhookConfiguration — kube-apiserver sends the object over HTTPS to an external service (usually a pod in the cluster) and waits for a decision. This is how cert-manager, Calico and policy engines like Kyverno (checking and filling in objects according to rules written in YAML) work.

Webhooks have a price: if the webhook service is unavailable and its configuration has failurePolicy: Fail, kube-apiserver rejects all requests the webhook is supposed to check. A broken webhook on pods, and no pod can be created in the cluster, including the webhook's own pod. That's why webhooks are restricted with namespace selectors (and kube-system is excluded), and where possible they're replaced with CEL policies: kube-apiserver runs those itself, so there's nothing there to fail.

An example of a useful policy is forbidding images without a tag or with the latest tag. The latest tag doesn't say which version is running, and when a pod restarts on another node a different build may arrive; why version tags, and better still digests, are the right way is covered in part 6.

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: require-image-tag
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["pods"]
  validations:
    - expression: >-
        object.spec.containers.all(c,
          c.image.contains(':') && !c.image.endsWith(':latest'))
      message: "the image must be specified with a version tag, not latest"
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: require-image-tag
spec:
  policyName: require-image-tag
  validationActions: [Deny]
  matchResources:
    namespaceSelector:
      matchLabels:
        kubernetes.io/metadata.name: default

The policy describes the rule, the binding says where and how to apply it: Deny rejects the request, Warn only warns the client, Audit writes to the audit log. It's sensible to start a new policy with Warn to see what it would break, and only then switch to Deny. The check in the example is simplified (it doesn't look at init containers and treats the colon in a registry address with a port as a tag), but it shows the idea:

$ kubectl apply -f require-image-tag.yaml
validatingadmissionpolicy.admissionregistration.k8s.io/require-image-tag created
validatingadmissionpolicybinding.admissionregistration.k8s.io/require-image-tag created

$ kubectl run bad --image=nginx
The pods "bad" is invalid: : ValidatingAdmissionPolicy 'require-image-tag' with binding 'require-image-tag' denied request: the image must be specified with a version tag, not latest

$ kubectl run good --image=nginx:1.29 --dry-run=server
pod/good created (server dry run)

--dry-run=server puts the request through the whole path — authentication, authorization, admission, schema validation — but doesn't write it to etcd. This is the best way to check a manifest: --dry-run=client only prints the object and checks nothing on the server.

resourceVersion and optimistic locking

Two administrators open the same object at the same time, each changes their own field and saves. Without protection, the second one overwrites the first one's change. Kubernetes solves this with optimistic locking: nobody locks anything, but every write checks that the object hasn't changed since it was read.

The tool is metadata.resourceVersion. It's an opaque string: right now it matches the etcd revision at which the object was last written, but clients must not parse it or compare it as "greater or less" — only pass it back. A PUT request with an object that has resourceVersion set succeeds only if etcd holds exactly that version; otherwise the response is 409 Conflict.

Let's check. Create a ConfigMap, read it into a file, change the object "from another window" and try to write the stale copy:

$ kubectl create configmap demo --from-literal=color=blue
configmap/demo created

$ kubectl proxy --port=8001 &
Starting to serve on 127.0.0.1:8001

$ curl -s localhost:8001/api/v1/namespaces/default/configmaps/demo > demo.json
$ jq .metadata.resourceVersion demo.json
"1843702"

$ kubectl patch configmap demo -p '{"data":{"size":"large"}}'
configmap/demo patched

$ jq '.data.color = "green"' demo.json | curl -s -X PUT -H 'Content-Type: application/json' \
    --data-binary @- localhost:8001/api/v1/namespaces/default/configmaps/demo | jq '{code, reason, message}'
{
  "code": 409,
  "reason": "Conflict",
  "message": "Operation cannot be fulfilled on configmaps \"demo\": the object has been modified; please apply your changes to the latest version and try again"
}

The right client reaction to a 409 is to read the object again, reapply its change and try once more. Controllers do this all the time, and the lines the object has been modified in their logs are normal, not an error. kubectl replace -f with a file that still has an old resourceVersion gets the same conflict, and kubectl edit on a conflict saves your edit to a temporary file and offers to try again.

Not every write checks the version. A PATCH without resourceVersion in the body is applied to the current object, whatever it is: the kubectl patch above went through without a conflict. This is convenient when one field changes, and dangerous when the change depends on the value read (for example, "add an element to the list if it isn't there"): in such cases resourceVersion is added to the patch explicitly.

Watch: subscribing to changes

A controller doesn't poll the API, it subscribes. The request is the same list GET with the watch=1 parameter and the version to start from:

$ curl -s localhost:8001/api/v1/namespaces/default/pods | jq .metadata.resourceVersion
"1843755"

$ curl -sN 'localhost:8001/api/v1/namespaces/default/pods?watch=1&resourceVersion=1843755&allowWatchBookmarks=true' \
    | jq -c '{type, name: .object.metadata.name, rv: .object.metadata.resourceVersion, phase: .object.status.phase}'
{"type":"ADDED","name":"test","rv":"1843760","phase":"Pending"}
{"type":"MODIFIED","name":"test","rv":"1843761","phase":"Pending"}
{"type":"MODIFIED","name":"test","rv":"1843763","phase":"Pending"}
{"type":"MODIFIED","name":"test","rv":"1843768","phase":"Pending"}
{"type":"MODIFIED","name":"test","rv":"1843770","phase":"Pending"}
{"type":"MODIFIED","name":"test","rv":"1843774","phase":"Running"}
{"type":"BOOKMARK","name":null,"rv":"1844105","phase":null}
{"type":"MODIFIED","name":"test","rv":"1844228","phase":"Running"}
{"type":"MODIFIED","name":"test","rv":"1844230","phase":"Running"}
{"type":"MODIFIED","name":"test","rv":"1844231","phase":"Running"}
{"type":"MODIFIED","name":"test","rv":"1844235","phase":"Succeeded"}
{"type":"MODIFIED","name":"test","rv":"1844238","phase":"Succeeded"}
{"type":"DELETED","name":"test","rv":"1844239","phase":"Succeeded"}

Meanwhile, kubectl run test --image=nginx:1.29 and kubectl delete pod test were run in a neighboring terminal. The response is an endless stream of JSON objects, one per event: ADDED, MODIFIED, DELETED with the full object after the change. The first MODIFIED is the scheduler assigning a node. The next three are kubelet updating the pod's status step by step — conditions, the address, the container state while the image is being pulled; the fifth is the container started, phase Running. On deletion, the first MODIFIED is the pod getting a deletionTimestamp; then kubelet stops the container and reports each step: nginx exited cleanly on SIGTERM, so the phase became Succeeded; with the last change kubelet confirms that no containers are left, and only then does DELETED arrive — the record is gone.

The "list, then watch from the list's version" scheme guarantees that nothing is lost between reading and subscribing. But it has a limit: kube-apiserver remembers the history of changes for a limited time (on the order of a few minutes), and etcd periodically compacts old revisions. A client that reconnects with a long-outdated version gets:

{"type":"ERROR","object":{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"too old resource version: 1200000 (1843500)","reason":"Expired","code":410}}

410 Gone means "you can't continue from this point, read the list again". Re-reading the list in a large cluster is expensive, and bookmarks protect against it. If the client asked for allowWatchBookmarks=true, the server sends a BOOKMARK event from time to time — it has no object, only a fresh resourceVersion: "you've already received everything up to this version". In the example above the bookmark arrived when nothing was happening in the namespace, but the cluster went on living. Having reconnected after a break, the client continues from the bookmark's version rather than from the old version of the last event in its namespace, and most likely won't get a 410. The server gives no guarantees: bookmarks are an optimization, and the client must also be able to re-read the list.

For the first read of large lists there's one more mechanism — the streaming list: a watch with sendInitialEvents=true sends the current state as a series of ADDED events, then a bookmark, after which the ordinary changes begin. This way the server doesn't assemble a huge list response in memory. The mechanism is in beta and enabled by default; client libraries use it on their own.

kubectl can subscribe too: kubectl get pods -w is a watch, and kubectl get pods -w --output-watch-events also shows the event types.

generation and observedGeneration

resourceVersion changes with any write — including when a controller updates status or someone adds a label. To tell changes of the desired state apart, objects with a status subresource have metadata.generation: the server increments it only when spec changes. A controller, having processed the next version of spec, writes its number to status.observedGeneration.

$ kubectl get deploy web -o jsonpath='{.metadata.generation} {.status.observedGeneration}{"\n"}'
1 1

$ kubectl label deployment web tier=frontend
deployment.apps/web labeled
$ kubectl get deploy web -o jsonpath='{.metadata.generation} {.status.observedGeneration}{"\n"}'
1 1

$ kubectl patch deployment web -p '{"spec":{"replicas":4}}'
deployment.apps/web patched
$ kubectl get deploy web -o jsonpath='{.metadata.generation} {.status.observedGeneration}{"\n"}'
2 2

A label is a metadata change, and generation is untouched. The number of replicas is a spec change, generation became 2, and the controller confirmed almost immediately that it had seen it. While observedGeneration < generation, the status describes the old version of the desired state and can't be trusted: "3/3 ready" may mean that the controller hasn't started rolling out the new image yet. This is exactly what kubectl rollout status checks, as does any rollout script that wants to wait for the result rather than see the status of a past success.

Deletion: finalizers and ownerReferences

Deletion in Kubernetes isn't instant erasure of a record but a process.

Finalizers. If an object's metadata.finalizers contains at least one string, DELETE doesn't delete it but only sets deletionTimestamp. The object is visible and marked as being deleted. It can still be changed, but no new finalizer can be added — existing ones can only be removed; the server answers metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted. Each finalizer is a promise by some controller to do work before deletion: for example, kubernetes.io/pvc-protection doesn't let a volume claim be deleted while a pod is using it, and the finalizer of an external load balancer controller first deletes the load balancer in the cloud. Having done the work, the controller removes its string; when the list is empty, kube-apiserver deletes the object.

$ kubectl patch configmap demo --type=merge -p '{"metadata":{"finalizers":["example.com/cleanup"]}}'
configmap/demo patched

$ kubectl delete configmap demo --wait=false
configmap "demo" deleted from default namespace

$ kubectl get configmap demo -o jsonpath='{.metadata.deletionTimestamp} {.metadata.finalizers}{"\n"}'
2026-10-21T10:05:12Z ["example.com/cleanup"]

$ kubectl patch configmap demo --type=json -p '[{"op":"remove","path":"/metadata/finalizers"}]'
configmap/demo patched

$ kubectl get configmap demo
Error from server (NotFound): configmaps "demo" not found

kubectl honestly wrote deleted, even though the object kept existing: the deletion was requested, not performed. Without --wait=false the command would wait forever — nobody processes the example.com/cleanup finalizer. This is the classic "namespace stuck in Terminating" picture: some object inside holds a finalizer of a controller that no longer exists (the operator was deleted before its resources). Removing a finalizer by hand, as in the example, is a last resort: first you need to understand what the controller was supposed to do and do it yourself, otherwise resources outside the cluster will be left hanging.

ownerReferences. Objects created by a controller know their owner:

$ kubectl get rs -l app=web -o jsonpath='{.items[0].metadata.ownerReferences}' | jq .
[
  {
    "apiVersion": "apps/v1",
    "blockOwnerDeletion": true,
    "controller": true,
    "kind": "Deployment",
    "name": "web",
    "uid": "7d1e9c2a-4b3f-4e8a-9c6d-2f0b1a5e8c34"
  }
]

The reference points to the uid, not just the name: a new Deployment with the same name is a different owner. controller: true means this owner manages the object (an object can have only one such owner). The whole Deployment → ReplicaSet → Pod chain is built on these references, and the garbage collector (in kube-controller-manager) deletes objects whose owner no longer exists.

How dependents are deleted is set by the request's propagationPolicy, in kubectl by the --cascade flag:

--cascade Policy What happens
background (default) Background the owner is deleted at once, the garbage collector deletes the dependents afterwards
foreground Foreground the owner gets the foregroundDeletion finalizer and waits until all dependents with blockOwnerDeletion: true are deleted
orphan Orphan the owner is deleted, the reference to it is removed from the dependents — they're left as "orphans"

orphan is useful when you need to recreate the owner object without touching the running pods — for example, to change a Deployment's immutable selector field:

$ kubectl delete deployment web --cascade=orphan
deployment.apps "web" deleted from default namespace

$ kubectl get rs,pods -l app=web
NAME                             DESIRED   CURRENT   READY   AGE
replicaset.apps/web-7c5ddbdf54   4         4         4       52m

NAME                       READY   STATUS    RESTARTS   AGE
pod/web-7c5ddbdf54-4xkqp   1/1     Running   0          52m
...

$ kubectl apply -f web.yaml
deployment.apps/web created

The new Deployment finds the ReplicaSet with a matching selector and the same template and adopts it — writes itself in as the owner without restarting the pods, and brings the number of replicas to the three from the manifest.

apply: client-side and server-side

There are three ways to change an object: PUT (replace it entirely, with a resourceVersion check), PATCH (change part of it) and apply. Patches come in several kinds, and the type is set by the Content-Type header:

Type Content-Type How it works
JSON Patch application/json-patch+json a list of add, remove, replace operations on paths, RFC 6902
JSON Merge Patch application/merge-patch+json a partial object, fields are merged, lists are replaced entirely, RFC 7386
Strategic Merge Patch application/strategic-merge-patch+json like merge, but lists are merged by key (containers by name); only for built-in types
Apply application/apply-patch+yaml server-side apply

Client-side apply (kubectl apply by default) works in kubectl. It keeps the last applied manifest in the last-applied-configuration annotation and on the next run makes a three-way comparison: the previous manifest, the new manifest and the current object. Fields that were in the previous manifest and disappeared from the new one are deleted; fields that were never in the manifests (for example, set by a controller) aren't touched. Hence its limitations: the knowledge of "whose" a field is exists with only one client and only in the annotation; an annotation with a large manifest runs into the annotation size limit (256 KiB) — that's why large CRD sets, Calico's for example, are installed with kubectl create rather than apply; two tools applying the same object know nothing about each other.

Server-side apply (SSA) moves the logic to the server. The client sends a PATCH with the application/apply-patch+yaml type and the name of a field manager, and the server keeps a record for each object of which fields were set by which manager. This record is metadata.managedFields. So as not to mix the modes on one object, let's start with a clean slate:

$ kubectl delete deployment web
deployment.apps "web" deleted from default namespace

$ kubectl apply --server-side --field-manager=team-a -f web.yaml
deployment.apps/web serverside-applied

$ kubectl get deployment web --show-managed-fields -o yaml | yq '.metadata.managedFields'
- apiVersion: apps/v1
  fieldsType: FieldsV1
  fieldsV1:
    f:metadata:
      f:labels:
        f:app: {}
    f:spec:
      f:replicas: {}
      f:selector: {}
      f:template:
        f:metadata:
          f:labels:
            f:app: {}
        f:spec:
          f:containers:
            k:{"name":"nginx"}:
              .: {}
              f:image: {}
              f:name: {}
              f:ports:
                k:{"containerPort":80,"protocol":"TCP"}:
                  .: {}
                  f:containerPort: {}
  manager: team-a
  operation: Apply
  time: "2026-10-21T10:21:07Z"
- apiVersion: apps/v1
  fieldsType: FieldsV1
  fieldsV1:
    f:metadata:
      f:annotations:
        .: {}
        f:deployment.kubernetes.io/revision: {}
    f:status:
      f:availableReplicas: {}
      f:conditions:
        .: {}
        k:{"type":"Available"}:
          .: {}
          f:lastTransitionTime: {}
          f:lastUpdateTime: {}
          f:message: {}
          f:reason: {}
          f:status: {}
          f:type: {}
        k:{"type":"Progressing"}:
          .: {}
          f:lastTransitionTime: {}
          f:lastUpdateTime: {}
          f:message: {}
          f:reason: {}
          f:status: {}
          f:type: {}
      f:observedGeneration: {}
      f:readyReplicas: {}
      f:replicas: {}
      f:terminatingReplicas: {}
      f:updatedReplicas: {}
  manager: kube-controller-manager
  operation: Update
  subresource: status
  time: "2026-10-21T10:21:09Z"

The team-a manager owns the fields from its manifest, kube-controller-manager owns the revision annotation and the status: it made both writes through the status subresource, so there is a single entry for it, with subresource: status. List elements are addressed by key — containers by name (k:{"name":"nginx"}), conditions by type (k:{"type":"Available"}) — so two managers can own different containers of the same pod.

Now another manager tries to change a field owned by the first one. For SSA a manifest with only the fields it wants to manage is enough:

# replicas.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 5

$ kubectl apply --server-side --field-manager=team-b -f replicas.yaml
error: Apply failed with 1 conflict: conflict with "team-a": .spec.replicas
Please review the fields above--they currently have other managers. Here
are the ways you can resolve this warning:
* If you intend to manage all of these fields, please re-run the apply
  command with the `--force-conflicts` flag.
* If you do not intend to manage all of the fields, please edit your
  manifest to remove references to the fields that should keep their
  current managers.
* You may co-own fields by updating your manifest to match the existing
  value; in this case, you'll become the manager if the other manager(s)
  stop managing the field (remove it from their configuration).
See https://kubernetes.io/docs/reference/using-api/server-side-apply/#conflicts

The server didn't let someone else's field be silently overwritten. There are three ways out, and they're listed in the message: take the field over (--force-conflicts), remove it from your manifest, or agree with the current value and become a co-owner. This is the main advantage of SSA: conflicts between tools are visible, rather than discovered a week later as "someone rolled back my setting".

A typical real conflict is autoscaling. If the number of replicas is managed by a HorizontalPodAutoscaler (part 33), the replicas field must not be left in the Deployment manifest: with every apply it will reset the number of replicas to the value from the file. The right way is to remove replicas from the manifest and leave the field to autoscaling.

The difference in deleting fields. With SSA, a field that a manager stopped specifying in its manifest is deleted only if nobody else owns it; with client-side apply, according to the annotation. You can switch from client-side to server-side at any time: kubectl apply --server-side on an object created with client-side apply transfers ownership of the fields from the annotation to the kubectl manager. Going back and mixing the two modes on one object lead to confusion, so one way is chosen for an object. For new projects it's sensible to use --server-side from the start: Flux (a GitOps controller that applies manifests from Git) and Helm 4 (a package manager for manifests, part 28) work with it by default, and in Argo CD it's enabled with a sync option.

kubectl diff before apply shows what will change. It sends the object to the server with dryRun=All — with all the default values, admission mutations and field ownership taken into account — and compares the result with the current object. Exit code 0 means no differences, 1 means there are some, greater than 1 means an error; this is convenient in CI (a system for automated building and testing). With --server-side, diff follows the SSA rules. The comparison program can be replaced with the KUBECTL_EXTERNAL_DIFF variable, for example with diff -u --color.

Hands-on

Let's find our Deployment in etcd. First, the record's service fields:

$ kubectl exec -n kube-system etcd-k8s-master-1 -- etcdctl \
    --endpoints=https://127.0.0.1:2379 \
    --cacert=/etc/kubernetes/pki/etcd/ca.crt \
    --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
    --key=/etc/kubernetes/pki/etcd/healthcheck-client.key \
    get /registry/deployments/default/web -w json | jq '.kvs[0] | {create_revision, mod_revision, version}'
{
  "create_revision": 1844489,
  "mod_revision": 1844517,
  "version": 6
}

$ kubectl get deployment web -o jsonpath='{.metadata.resourceVersion}{"\n"}'
1844517

mod_revision, the etcd revision of the key's last write, matches the object's resourceVersion, and version is how many times the key has been written since it was created. The value itself is the object in Protobuf (a binary serialization format), and it's convenient to look at it with strings (here and below the certificate flags are replaced with ...):

$ kubectl exec -n kube-system etcd-k8s-master-1 -- etcdctl ... \
    get /registry/deployments/default/web --print-value-only | strings -n 3 | head -6
k8s
apps/v1
Deployment
web
default
*$5e3c8a1f-2b7d-4f6e-9a0c-8d4b1e7f3a62

The k8s prefix and the "version–type" pair are the Kubernetes serialization header; after them come the object's fields (the *$ characters before the uid are Protobuf service bytes: the field number and the string length). resourceVersion isn't stored in the value: kube-apiserver fills it in from the etcd revision on read. Secrets lie here the same way, as plain Protobuf, unless encryption is enabled — more on that in part 13.

How many requests creating an object costs with the two kinds of apply. Let's delete the Deployment and create it with client-side apply, filtering the requests to the object itself out of the log:

$ kubectl delete deployment web
deployment.apps "web" deleted from default namespace

$ kubectl apply -f web.yaml -v=6 2>&1 | grep -o '"Response" verb="[A-Z]*" url="[^"]*deployments[^"]*" status="[^"]*"'
"Response" verb="GET" url="https://k8s-api.example.com:6443/apis/apps/v1/namespaces/default/deployments/web" status="404 Not Found"
"Response" verb="POST" url="https://k8s-api.example.com:6443/apis/apps/v1/namespaces/default/deployments?fieldManager=kubectl-client-side-apply&fieldValidation=Strict" status="201 Created"

Client-side apply first reads the object: if it isn't there, it creates it with a POST carrying the last-applied-configuration annotation; if it is, it computes a patch and sends a PATCH. The fieldValidation=Strict parameter asks the server to reject a manifest with unknown or duplicate fields — the typo replcas: 3 gives an error instead of being silently ignored. Even client-side apply records its manager in managedFields — kubectl-client-side-apply, with the Update operation.

Now the same with server-side apply:

$ kubectl delete deployment web
deployment.apps "web" deleted from default namespace

$ kubectl apply -f web.yaml --server-side -v=6 2>&1 | grep -o '"Response" verb="[A-Z]*" url="[^"]*deployments[^"]*" status="[^"]*"'
"Response" verb="PATCH" url="https://k8s-api.example.com:6443/apis/apps/v1/namespaces/default/deployments/web?fieldManager=kubectl&fieldValidation=Strict&force=false" status="201 Created"

A single PATCH of the apply type, which created the object (201 Created): deciding whether to create or change is now the server's business.

And the last thing: how to see the list of all the server's subscriptions by type. kube-apiserver exposes metrics, and among them is the number of open watches:

$ kubectl get --raw /metrics | grep '^apiserver_longrunning_requests{.*verb="WATCH"' | sort -t' ' -k2 -nr | head -5
apiserver_longrunning_requests{component="apiserver",group="",resource="configmaps",scope="resource",subresource="",verb="WATCH",version="v1"} 12
apiserver_longrunning_requests{component="apiserver",group="",resource="pods",scope="cluster",subresource="",verb="WATCH",version="v1"} 9
apiserver_longrunning_requests{component="apiserver",group="",resource="nodes",scope="cluster",subresource="",verb="WATCH",version="v1"} 7
apiserver_longrunning_requests{component="apiserver",group="",resource="services",scope="cluster",subresource="",verb="WATCH",version="v1"} 7
apiserver_longrunning_requests{component="apiserver",group="discovery.k8s.io",resource="endpointslices",scope="cluster",subresource="",verb="WATCH",version="v1"} 5

Subscriptions to pods are held by the kubelet of every node, the scheduler, the controller manager and Calico; to services and EndpointSlices, by kube-proxy and CoreDNS. The line with scope="resource" counts subscriptions to individual objects: kubelet watches every ConfigMap and Secret mounted into the pods of its node — at least kube-root-ca.crt in every namespace that has pods. This is the "everyone is an API client" from the second part, in numbers. The counters belong to the kube-apiserver instance that answered the request: in a cluster with several masters, each instance has its own subscriptions.

Let's clean up after ourselves: kubectl delete deployment web, kubectl delete -f require-image-tag.yaml, and stop kubectl proxy (kill %1).

Sources

Next — we install the lab cluster from scratch: preparing three Ubuntu nodes, containerd, kubeadm init with a configuration file and a walk-through of each of its phases, Calico through the operator, and joining the worker nodes.

Only registered users can leave a comment.

Sign in or register to leave a comment.

© 2006-2026 Vitaliy Leschenko