Kubernetes for beginners. Part 26: How does Kustomize work?
All parts of the series “Kubernetes for beginners”Over twenty-five parts the hello application has collected a dozen manifests: a Deployment from part 7, a Service from part 15, an Ingress with a certificate from parts 18 and 19, a ConfigMap and a Secret from part 12, a PodDisruptionBudget from part 13, a NetworkPolicy from part 20. While there is one environment, kubectl apply -f is enough. But as soon as a second one appears — a test environment with one replica and small limits next to production with three replicas and a different host name — the question is how to keep two nearly identical sets of files without letting them drift apart. This part opens the packaging block, and its first tool is Kustomize: it is built into kubectl and works with the very same manifests without changing them.
The problem and two approaches
The differences between environments are usually small: the number of replicas, requests and limits, the image tag, the host name in the Ingress, the namespace, a couple of configuration parameters, secrets. Copying the directory and editing by hand works exactly until the first change in the shared part that somebody forgets to carry over to the second copy.
There are two fundamentally different ways to get rid of copying.
Templates. A manifest becomes text with substitutions: replicas: {{ .Values.replicas }}, and the values come from a separate file. This is how Helm works (part 27). Powerful, but the manifest stops being YAML: you cannot apply it, validate it or open it in an editor with a schema until the template engine has run, and the logic with conditions and loops lives inside the text.
Overlays. The manifests stay ordinary valid Kubernetes objects — the base. For each environment you write an overlay: "take the base, change this field here". The result is computed structurally, on the YAML tree, not by text substitution. This is how Kustomize works — a project of SIG CLI (the Kubernetes community group responsible for kubectl), built into kubectl since 2019 as kubectl apply -k and kubectl kustomize.
Kustomize installs nothing in the cluster and stores nothing there. It is a function "directory with kustomization.yaml → stream of YAML documents": it reads files, applies transformations and prints the result. The result is then applied by the same kubectl apply as before, and the API server knows nothing about Kustomize. So everything said in earlier parts about server-side apply, resourceVersion and the managedFields (part 3) holds without reservations.
kustomization.yaml
A directory becomes a "kustomization" when it contains a kustomization.yaml file. The minimal version for the existing hello manifests:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
- service.yaml
- ingress.yaml
apiVersion and kind are a formality here: the file is never sent to the API, it is the configuration of Kustomize itself. resources lists what goes into the result: files with manifests (one file may contain several documents separated by ---), directories with their own kustomization.yaml, and remote addresses like https://github.com/org/repo//deploy?ref=v1.4.0. A remote base is always pinned to a tag or commit in ref: without it every build takes the current state of the branch, and the result changes without a single edit in your repository.
The order in which Kustomize processes the file is fixed and does not depend on the order in which the fields are written:
resources, components ← load objects (recursively build bases)
│
▼
generators ← configMapGenerator, secretGenerator
│
▼
transformers ← namespace, namePrefix/nameSuffix, labels, annotations,
│ patches, images, replicas
▼
replacements ← copy values from some fields into others
│
▼
reference fix-up ← renamed ConfigMaps, Secrets, Services
│ are substituted where they are referenced
▼
output (YAML stream)
The last step matters: Kustomize knows in which fields of which objects names are referenced — configMapRef and secretKeyRef in a container, volumes[].configMap.name, backend.service.name in an Ingress, scaleTargetRef in a HorizontalPodAutoscaler (the autoscaling object, part 32) and a few dozen more. If an object's name changed because of a prefix or a hash, the references to it are fixed automatically.
Directory layout
Everything this part uses, in full:
hello/
├── base/
│ ├── kustomization.yaml
│ ├── deployment.yaml
│ ├── service.yaml
│ ├── ingress.yaml
│ └── config/
│ └── app.env
├── components/
│ └── ha/
│ ├── kustomization.yaml
│ ├── pdb.yaml
│ └── patch-anti-affinity.yaml
└── overlays/
├── dev/
│ ├── kustomization.yaml
│ └── patch-resources.yaml
└── prod/
├── kustomization.yaml
├── patch-resources.yaml
├── patch-ingress-host.yaml
└── config/
└── app.env
base/ is what all environments share: ordinary manifests that can be applied even without Kustomize. overlays/dev and overlays/prod are one directory per environment, each holding only the differences. components/ha is a reusable piece "make it highly available" that plugs into any overlay; more on it near the end.
The environments on the lab: dev lives in the demo-dev namespace and answers on hello-dev.apps.example.com, prod lives in the lab namespace demo on hello.apps.example.com.
The base
# base/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: hello
template:
metadata:
labels:
app.kubernetes.io/name: hello
spec:
containers:
- name: hello
image: ghcr.io/example/hello:1.0
ports:
- name: http
containerPort: 8080
envFrom:
- configMapRef:
name: hello-config
env:
- name: PUBLIC_HOST
value: set-by-kustomize
readinessProbe:
httpGet:
path: /healthz
port: http
resources:
requests:
cpu: 50m
memory: 32Mi
limits:
memory: 64Mi
# base/service.yaml
apiVersion: v1
kind: Service
metadata:
name: hello
spec:
selector:
app.kubernetes.io/name: hello
ports:
- name: http
port: 80
targetPort: http
# base/ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: hello
annotations:
cert-manager.io/cluster-issuer: letsencrypt-staging
spec:
ingressClassName: traefik
rules:
- host: hello-dev.apps.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: hello
port:
name: http
tls:
- hosts:
- hello-dev.apps.example.com
secretName: hello-tls
# base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
- service.yaml
- ingress.yaml
labels:
- pairs:
app.kubernetes.io/part-of: hello
includeTemplates: true
configMapGenerator:
- name: hello-config
envs:
- config/app.env
# base/config/app.env
LOG_LEVEL=debug
GREETING=hello
Note that the base has no namespace. The namespace is a property of the environment, and the overlay sets it. A base with the namespace written into every manifest would force the overlay to rewrite it with patches.
Labels: labels, not commonLabels
Kustomize can add labels and annotations to all objects. Historically this was the commonLabels field, and you still meet it in half of the examples on the internet. It is deprecated: the Kustomize documentation says so and points to labels. The reason is not syntax but behaviour.
commonLabels adds the label everywhere, including selectors: into spec.selector.matchLabels of a Deployment and into spec.selector of a Service. A Deployment's selector is an immutable field (part 7): if half a year later someone adds a new label to commonLabels, kubectl apply fails with field is immutable, and the Deployment has to be deleted and recreated — with downtime. And it does not fail all at once: objects are applied one by one, and by that moment the Service already has the new selector that no pod matches, so traffic is gone before anyone notices the error (what it looks like is shown in "Hands-on"). Besides, the label in a Service selector narrows it, and a pod created outside this kustomization stops receiving traffic. commonLabels has been deprecated since Kustomize v5.0.0, and kubectl kustomize prints a warning on every build that uses it: # Warning: 'commonLabels' is deprecated. Please use 'labels' instead.
labels by default changes only the metadata.labels of the objects themselves and controls the rest explicitly:
Field in a labels item |
Default | What it adds |
|---|---|---|
pairs |
— | the labels themselves |
includeTemplates |
false |
also into spec.template.metadata.labels — labels reach the pods, selectors are not touched |
includeSelectors |
false |
also into selectors and templates — the commonLabels behaviour |
fields |
— | your own list of paths (path, kind, create) where to put the labels as well |
The rule is simple: descriptive labels (part-of, environment, team) go through labels with includeTemplates: true, so that pods can be found by them; into selectors goes only what will never change, and that is better written into the base by hand, like app.kubernetes.io/name above. For annotations there is commonAnnotations; annotations are not part of any selector, and this field is not deprecated.
Old fields are rewritten into new ones by kustomize edit fix of the standalone Kustomize binary: it replaces deprecated constructs in the kustomization.yaml of the current directory with their modern equivalents. It does so faithfully, preserving the behaviour:
$ kustomize edit fix
# Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
Fixed fields:
patchesJson6902 -> patches
patchesStrategicMerge -> patches
commonLabels -> labels
…
$ cat kustomization.yaml
resources:
- deployment.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
labels:
- includeSelectors: true
pairs:
team: web
commonLabels: {team: web} became labels with includeSelectors: true — the label still goes into selectors, and for objects already running that is correct: removing it from the selector is the same change of an immutable field as adding it. So includeSelectors: false is set from the very beginning, while there are no objects in the cluster yet.
The dev overlay
# overlays/dev/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: demo-dev
nameSuffix: -dev
resources:
- ../../base
labels:
- pairs:
app.kubernetes.io/instance: hello-dev
environment: dev
includeTemplates: true
images:
- name: ghcr.io/example/hello
newTag: "1.1"
patches:
- path: patch-resources.yaml
# overlays/dev/patch-resources.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello
spec:
template:
spec:
containers:
- name: hello
resources:
limits:
memory: 48Mi
What happens here, field by field.
resources: [../../base] — the overlay refers to the base as a directory. Kustomize builds the whole base (with its generators and labels) and then applies everything else to the result. A base can itself be another overlay — that is how hierarchies like base → prod → prod-eu are built, but more than two levels are usually hard to read.
namespace sets metadata.namespace on every object that has one by nature (cluster-scoped resources like ClusterRole are left alone) and fixes subjects[].namespace of ServiceAccounts in RoleBindings. It does not create the namespace itself: either add a Namespace manifest to resources or create it separately — for the lab, kubectl create namespace demo-dev.
namePrefix and nameSuffix add a string to the names of all objects: hello becomes hello-dev, hello-config becomes hello-config-dev (and then gets a hash). References are fixed: the Ingress will point to Service hello-dev, the Deployment to ConfigMap hello-config-dev-…. A suffix is needed when two environments share a namespace; if the namespaces differ, as here, it is optional, but with it the kubectl get output shows at once what comes from where.
images changes the image in the containers of all workloads without touching the manifests. The match is by image name without the tag, and three things can be set:
images:
- name: ghcr.io/example/hello # what to look for
newName: ghcr.io/example/hello # what to replace the name with (another registry)
newTag: "1.1" # new tag
- name: busybox
digest: sha256:3fbc6327… # pin to a digest instead of a tag
The quotes around the tag are not a formality: without them YAML reads 1.1 as a number, while newTag is a string field, and the build stops with invalid Kustomization: json: cannot unmarshal number into Go struct field Image.images.newTag of type string. An error here is better than a silent conversion: if the number were quietly turned into a string, tag 1.10 would become 1.1. For production digest is better: a tag can be overwritten in the registry, a digest is the content of the image and does not change (part 5). The command kustomize edit set image ghcr.io/example/hello:1.1 does the same — CI usually calls it after building the image.
patches is the list of patches; they get their own section.
Patches
A patch describes a change to an object that is already loaded. Kustomize understands two formats, and both are set by one field, patches (there used to be separate patchesStrategicMerge and patchesJson6902, now deprecated).
Strategic merge
A patch in the form of a partial manifest, like patch-resources.yaml above: apiVersion, kind and metadata.name identify the target, and the remaining fields are laid over it. "Strategic" because the merge knows the Kubernetes schema: the containers list is merged neither by position nor replaced as a whole, but by the name key. So it is enough to give name: hello and a single limits.memory field in the patch — the other fields of the container and the other containers of the pod stay as they were. The merge keys of lists are defined in the API schema: containers and initContainers by name, ports by containerPort, env by name, volumes by name.
A field can be removed with a special directive:
spec:
template:
spec:
containers:
- name: hello
readinessProbe:
$patch: delete
A limitation: strategic merge works for built-in types. For custom resources (CRD, part 28) Kustomize does not know the schema, lists in them are replaced as a whole, and the second format is more convenient there.
JSON Patch (RFC 6902)
A list of operations on paths in the document: add, remove, replace, move, copy, test. A path is written as a JSON Pointer — segments separated by /, list elements by index. A JSON Patch has no target inside it, so the target is given in the target field:
# overlays/prod/patch-ingress-host.yaml
- op: replace
path: /spec/rules/0/host
value: hello.apps.example.com
- op: replace
path: /spec/tls/0/hosts/0
value: hello.apps.example.com
- op: replace
path: /metadata/annotations/cert-manager.io~1cluster-issuer
value: letsencrypt-prod
~1 in the last path is an escaped / from the annotation name (and ~0 is an escaped tilde), as RFC 6901 requires. Indexes are the weak spot of the format: if the Ingress rules in the base swap places, the patch silently replaces the wrong host. So JSON Patch is used where strategic merge cannot cope: for list elements without a merge key, to remove an element by position, for CRDs.
target: one patch for many objects
target also works for strategic merge, and then the patch is applied to every matching object. Objects can be selected by group, version, kind, name (a regular expression), namespace, labelSelector and annotationSelector:
patches:
- path: patch-ingress-host.yaml
target:
kind: Ingress
name: hello
- target:
kind: Deployment
labelSelector: app.kubernetes.io/part-of=hello
patch: |-
- op: add
path: /spec/template/spec/securityContext
value:
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
The second patch is written right in kustomization.yaml in the patch field, without a separate file: handy for short changes. It adds a securityContext to all Deployments of the application — those fields are covered in detail in part 30.
ConfigMap and Secret generators and the hash in the name
In the base the ConfigMap is not written by hand but generated from config/app.env. The generator takes three sources, which can be mixed:
configMapGenerator:
- name: hello-config
envs: # a KEY=VALUE file, each line is a key
- config/app.env
files: # the whole file becomes the value of a key
- config/settings.yaml # key settings.yaml
- motd=config/banner.txt # key motd
literals: # right here
- FEATURE_X=on
The main thing the generator does is append a hash suffix of the content to the name: hello-config becomes hello-config-k675bbfg64. The hash is the first ten characters of SHA-256 over the kind, the generator's name and the object's data, with some characters substituted so that there are no vowels and no word can form by accident. Labels, namespace, prefixes and suffixes are not part of the hash: hello-config-dev-bg9h65b2f2 in dev and hello-config-bg9h65b2f2 built from the same base without the suffix have the same hash. The reference in the Deployment (configMapRef.name: hello-config) is rewritten to the new name.
Why this matters. In part 12 we saw that environment variables from a ConfigMap are read once when the container starts, and changing the ConfigMap does not restart pods; Reloader was installed for that. With a generator the restart happens by itself:
edit config/app.env
│
▼
new hash → new ConfigMap name: hello-config-bg9h65b2f2
│
▼
spec.template of the Deployment changes (reference to the new name)
│
▼
pod template changed → Deployment creates a new ReplicaSet
│
▼
rolling update: pods are recreated one by one with the new configuration
You also get a configuration rollback: the old ConfigMap with the previous hash is still in the cluster, and kubectl rollout undo brings the pod back to it, because the previous ReplicaSet refers to the old name:
$ kubectl -n demo-dev rollout undo deploy/hello-dev
Warning: resource deployments/hello-dev was previously managed with 'kubectl apply'. Rolling back will not update the kubectl.kubernetes.io/last-applied-configuration annotation, which may cause unexpected behavior on future 'kubectl apply' operations. Consider using 'kubectl apply' with your previous configuration file instead.
deployment.apps/hello-dev rolled back
$ kubectl -n demo-dev get deploy hello-dev -o jsonpath='{.spec.template.spec.containers[0].envFrom[0].configMapRef.name}'
hello-config-dev-k675bbfg64
The warning is to the point: such a rollback is a measure for minutes. The files in the repository are still the new ones, and the next kubectl apply -k brings the new configuration back; a permanent rollback is reverting the commit and building again.
The downside is that old ConfigMaps pile up: kubectl apply -k does not delete objects that disappeared from the output. They are cleaned up by hand, with kubectl apply --prune and a label selector (it deletes objects with those labels that are not in the input), or, better, by a GitOps tool — a controller that applies the content of a Git repository to the cluster itself and remembers which objects it created (part 41).
The behaviour is tuned with generatorOptions (for all generators of the file) or options of an individual generator:
generatorOptions:
disableNameSuffixHash: false # true — no hash (and then no automatic restart)
labels:
app.kubernetes.io/component: config
immutable: true # the ConfigMap becomes immutable
immutable: true goes well with the hash: every version of the configuration is a separate immutable object, and kubelet does not spend requests watching immutable ConfigMaps and Secrets for changes.
In an overlay a generator can not only be created but can also change the base one, with the behavior field:
# overlays/prod/kustomization.yaml (fragment)
configMapGenerator:
- name: hello-config
behavior: merge # create (default) | replace | merge
envs:
- config/app.env # LOG_LEVEL=info
merge adds and overrides keys from the overlay's file and keeps the rest from the base; replace replaces the data entirely.
secretGenerator works the same way and creates a Secret with the same hash; it also has type (default Opaque, can be kubernetes.io/tls with keys tls.crt and tls.key). But the generator takes values from files next to kustomization.yaml, which means those files sit in the repository in plain text — base64 in a Secret is not encryption, as discussed in part 12. Ways to keep a plain secret out of Git: the generator's file is created in CI from a secret store and never committed (.gitignore); or SOPS (a tool that encrypts values in a YAML file with an age, PGP or KMS key) is used through the KSOPS generator plugin; or the secret is not described in Kustomize at all and comes from an external secret store through External Secrets Operator — a controller that, following a description in the cluster, reads the value from the store and creates an ordinary Secret from it. More on secrets in GitOps in part 41.
replacements
Sometimes the value of one field has to be copied into another. Example: the application needs the public address it is reachable at, to build absolute links. The host is already in the Ingress, and keeping it in two places means one day only one of them gets edited. For this the base Deployment has a PUBLIC_HOST variable with a placeholder, and Kustomize puts the value in:
# base/kustomization.yaml (addition)
replacements:
- source:
kind: Ingress
name: hello
fieldPath: spec.rules.0.host
targets:
- select:
kind: Deployment
name: hello
fieldPaths:
- spec.template.spec.containers.[name=hello].env.[name=PUBLIC_HOST].value
source says where to take the value, targets where to put it. The path is written with dots, and in lists an element can be selected by the value of a field: containers.[name=hello]. A target has options: create: true creates the field if it is missing, and delimiter and index let you replace only part of a string (for example, the tag inside image). It is more reliable, as here, to keep a placeholder field in the base: then a typo in the path gives a build error rather than a silently created extra field.
A subtlety: replacements from the base run when the base is built, before the overlay's patches. Here that means prod would get hello-dev.apps.example.com — the host from the base, not the one fixed by the patch. So in prod the same replacement is repeated in the overlay: it runs after the host patch and overwrites the value with the final one. The general rule: a replacement belongs at the level where its source already has its final value. This is a common mistake, and it is caught only by reading the output of kubectl kustomize: if you remove replacements from the prod overlay below, the build succeeds without a single warning, and the Deployment ends up with value: hello-dev.apps.example.com.
The vars field with $(VAR) substitutions used to solve a similar problem; it is deprecated, and replacements replaces it.
Components
An overlay inherits a base, and inheritance is single: you cannot say "prod is the base plus high availability plus monitoring" if high availability is also needed in stage and monitoring also in dev. Such cross-cutting pieces are components — directories with kind: Component that are mixed into any overlay:
# components/ha/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component
resources:
- pdb.yaml
patches:
- path: patch-anti-affinity.yaml
target:
kind: Deployment
name: hello
# components/ha/pdb.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: hello
spec:
minAvailable: 2
selector:
matchLabels:
app.kubernetes.io/name: hello
# components/ha/patch-anti-affinity.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello
spec:
replicas: 3
template:
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchLabels:
app.kubernetes.io/name: hello
The difference from a base: a component is applied inside the overlay that includes it, after its resources are loaded, and its patches see the overlay's objects. So a component can patch what it does not contain itself. A component's apiVersion is v1alpha1, although the feature has long been stable in use; this is a quirk of Kustomize versioning, not a sign of an experiment. PodDisruptionBudget and spreading pods across nodes are covered in part 13.
The prod overlay
# overlays/prod/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: demo
resources:
- ../../base
components:
- ../../components/ha
labels:
- pairs:
app.kubernetes.io/instance: hello-prod
environment: prod
includeTemplates: true
images:
- name: ghcr.io/example/hello
digest: sha256:2d8774f801594f2b1b115963e54ec54b1b49f95c2dece1d6b0fd1e59ab447c7b
configMapGenerator:
- name: hello-config
behavior: merge
envs:
- config/app.env
patches:
- path: patch-resources.yaml
- path: patch-ingress-host.yaml
target:
kind: Ingress
name: hello
replacements:
- source:
kind: Ingress
name: hello
fieldPath: spec.rules.0.host
targets:
- select:
kind: Deployment
name: hello
fieldPaths:
- spec.template.spec.containers.[name=hello].env.[name=PUBLIC_HOST].value
# overlays/prod/patch-resources.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello
spec:
template:
spec:
containers:
- name: hello
resources:
requests:
cpu: 100m
memory: 64Mi
limits:
memory: 128Mi
# overlays/prod/config/app.env
LOG_LEVEL=info
The number of replicas in prod comes from the ha component. If that were all that was needed, the replicas field — a short form of a patch — would do:
replicas:
- name: hello
count: 3
kubectl and standalone kustomize
Two kubectl commands:
kubectl kustomize <dir>— build and print the result without sending anything to the cluster. Equivalent tokustomize build <dir>.kubectl apply -k <dir>— build and apply.kubectl diff -k,kubectl delete -k,kubectl get -kwork the same way.
Inside kubectl is the Kustomize library of a specific version, and it is updated together with kubectl, that is every four months and with a lag. kubectl 1.36 embeds Kustomize v5.8.1, while the standalone binary is at v5.8.3 by the end of October 2026. It is a single file from the project's releases page on GitHub, it is released more often and can do more: kustomize edit commands (set image, add resource, fix) to edit kustomization.yaml from scripts, kustomize create, kustomize cfg, kustomize localize. For example, kustomize edit set image rewrites the images block:
$ kustomize edit set image ghcr.io/example/hello:1.0
$ sed -n '/^images/,/^patches/p' kustomization.yaml
images:
- name: ghcr.io/example/hello
newTag: "1.0"
patches:
Along the way the command reformats the whole file (list indentation, field order), so the Git diff after it is more than one line. If CI builds with the standalone binary while people check by hand with kubectl, align the versions: there are differences between versions in the output, especially in the order of objects and in how deprecated fields are handled.
Helm inside Kustomize. The helmCharts field in kustomization.yaml lets you inflate a Helm chart as a resource and then patch the result. It is off by default because it runs the external program helm: without the flag the build stops with must specify --enable-helm. The flag exists for kustomize build and kubectl kustomize (along with --helm-command if helm is not in PATH); kubectl apply -k does not have it, so the build is done in two steps: kubectl kustomize --enable-helm dir | kubectl apply -f -. Kustomize stores the downloaded chart in a charts/ subdirectory next to kustomization.yaml and takes it from there next time. The reverse combination — Kustomize as a post-processor of Helm output — is covered in part 27.
Load restrictions. By default a kustomization cannot read files above its directory, except through the resources of other kustomizations (LoadRestrictionsRootOnly). So a configMapGenerator in an overlay cannot refer to ../../base/config/app.env directly — the build stops with an error like security; file '…/hello/base/config/app.env' is not in or below '…/hello/overlays/prod'. The --load-restrictor LoadRestrictionsNone flag lifts the restriction but breaks the portability of the directory; it is better to keep the file next to the kustomization.yaml that uses it.
Hands-on
The version of the embedded Kustomize:
$ kubectl version --client
Client Version: v1.36.5
Kustomize Version: v5.8.1
Build dev and look at the result:
$ kubectl kustomize hello/overlays/dev
apiVersion: v1
data:
GREETING: hello
LOG_LEVEL: debug
kind: ConfigMap
metadata:
labels:
app.kubernetes.io/instance: hello-dev
app.kubernetes.io/part-of: hello
environment: dev
name: hello-config-dev-k675bbfg64
namespace: demo-dev
---
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/instance: hello-dev
app.kubernetes.io/part-of: hello
environment: dev
name: hello-dev
namespace: demo-dev
spec:
ports:
- name: http
port: 80
targetPort: http
selector:
app.kubernetes.io/name: hello
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app.kubernetes.io/instance: hello-dev
app.kubernetes.io/part-of: hello
environment: dev
name: hello-dev
namespace: demo-dev
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: hello
template:
metadata:
labels:
app.kubernetes.io/instance: hello-dev
app.kubernetes.io/name: hello
app.kubernetes.io/part-of: hello
environment: dev
spec:
containers:
- env:
- name: PUBLIC_HOST
value: hello-dev.apps.example.com
envFrom:
- configMapRef:
name: hello-config-dev-k675bbfg64
image: ghcr.io/example/hello:1.1
name: hello
…
Everything is visible at once: the -dev suffix on all names, the hash on the ConfigMap and the new reference to it in envFrom, the namespace, tag 1.1 instead of 1.0, labels in metadata and in the pod template, but not in selector. Objects are output in a particular order: first Namespace, CRD, ServiceAccount, roles, ConfigMap and Secret, then Service, then workloads, and everything else (PodDisruptionBudget, Ingress) after them, webhooks at the very end. That way, when the stream is applied top to bottom, dependencies already exist.
A handy trick is to compare two environments as a whole:
$ diff <(kubectl kustomize hello/overlays/dev) <(kubectl kustomize hello/overlays/prod) | head -16
4c4
< LOG_LEVEL: debug
---
> LOG_LEVEL: info
8c8
< app.kubernetes.io/instance: hello-dev
---
> app.kubernetes.io/instance: hello-prod
10,12c10,12
< environment: dev
< name: hello-config-dev-k675bbfg64
< namespace: demo-dev
---
> environment: prod
> name: hello-config-758kt8k87t
> namespace: demo
Now apply dev. The hello image lives in a private registry, so the new namespace first needs the ghcr-pull Secret from part 5, attached to the default ServiceAccount. The simplest way is to copy it from default, stripping the service fields with jq (a tool that parses JSON and builds new JSON from it by an expression):
$ kubectl create namespace demo-dev
namespace/demo-dev created
$ kubectl get secret ghcr-pull -n default -o json \
| jq '{apiVersion, kind, type, data, metadata: {name: .metadata.name, namespace: "demo-dev"}}' \
| kubectl apply -f -
secret/ghcr-pull created
$ kubectl -n demo-dev patch serviceaccount default -p '{"imagePullSecrets":[{"name":"ghcr-pull"}]}'
serviceaccount/default patched
$ kubectl apply -k hello/overlays/dev
configmap/hello-config-dev-k675bbfg64 created
service/hello-dev created
deployment.apps/hello-dev created
ingress.networking.k8s.io/hello-dev created
$ kubectl -n demo-dev get deploy,rs,cm -l app.kubernetes.io/part-of=hello
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/hello-dev 1/1 1 1 20s
NAME DESIRED CURRENT READY AGE
replicaset.apps/hello-dev-b5554cfcd 1 1 1 20s
NAME DATA AGE
configmap/hello-config-dev-k675bbfg64 2 20s
The objects created by kubectl apply -k can also be listed through the same kustomization: kubectl get -k hello/overlays/dev builds it and shows the live objects with those names.
Change the configuration: LOG_LEVEL=warn in base/config/app.env. First see what will change — kubectl diff -k runs the result through a server-side dry run and shows the difference from the live objects:
$ kubectl diff -k hello/overlays/dev
diff -u -N /tmp/LIVE-2213282542/apps.v1.Deployment.demo-dev.hello-dev /tmp/MERGED-2826293843/apps.v1.Deployment.demo-dev.hello-dev
--- /tmp/LIVE-2213282542/apps.v1.Deployment.demo-dev.hello-dev 2026-10-31 04:37:55.015820213 +0000
+++ /tmp/MERGED-2826293843/apps.v1.Deployment.demo-dev.hello-dev 2026-10-31 04:37:55.015820213 +0000
@@ -6,7 +6,7 @@
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"apps/v1","kind":"Deployment",…}
creationTimestamp: "2026-10-31T04:37:30Z"
- generation: 1
+ generation: 2
labels:
app.kubernetes.io/instance: hello-dev
app.kubernetes.io/part-of: hello
@@ -41,7 +41,7 @@
value: hello-dev.apps.example.com
envFrom:
- configMapRef:
- name: hello-config-dev-k675bbfg64
+ name: hello-config-dev-bg9h65b2f2
image: ghcr.io/example/hello:1.1
imagePullPolicy: IfNotPresent
name: hello
diff -u -N /tmp/LIVE-2213282542/v1.ConfigMap.demo-dev.hello-config-dev-bg9h65b2f2 /tmp/MERGED-2826293843/v1.ConfigMap.demo-dev.hello-config-dev-bg9h65b2f2
--- /tmp/LIVE-2213282542/v1.ConfigMap.demo-dev.hello-config-dev-bg9h65b2f2 2026-10-31 04:37:54.940817253 +0000
+++ /tmp/MERGED-2826293843/v1.ConfigMap.demo-dev.hello-config-dev-bg9h65b2f2 2026-10-31 04:37:54.940817253 +0000
@@ -0,0 +1,14 @@
+apiVersion: v1
+data:
+ GREETING: hello
+ LOG_LEVEL: warn
+kind: ConfigMap
…
One line in the pod template (plus generation, which the server increments when spec changes) and the whole new ConfigMap — for diff it "appears out of nowhere". The exit code of kubectl diff is 1 when there is a difference and 0 when there is none: CI checks are built on that. Apply and see how the Deployment reacted:
$ kubectl apply -k hello/overlays/dev
configmap/hello-config-dev-bg9h65b2f2 created
service/hello-dev unchanged
deployment.apps/hello-dev configured
ingress.networking.k8s.io/hello-dev unchanged
$ kubectl -n demo-dev rollout status deploy/hello-dev
Waiting for deployment spec update to be observed...
Waiting for deployment spec update to be observed...
Waiting for deployment "hello-dev" rollout to finish: 0 out of 1 new replicas have been updated...
Waiting for deployment "hello-dev" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "hello-dev" rollout to finish: 1 old replicas are pending termination...
deployment "hello-dev" successfully rolled out
$ kubectl -n demo-dev get rs,cm -l app.kubernetes.io/part-of=hello
NAME DESIRED CURRENT READY AGE
replicaset.apps/hello-dev-7df889c657 1 1 1 6s
replicaset.apps/hello-dev-b5554cfcd 0 0 0 37s
NAME DATA AGE
configmap/hello-config-dev-bg9h65b2f2 2 6s
configmap/hello-config-dev-k675bbfg64 2 37s
A new ReplicaSet, the pod was recreated without Reloader, the old ConfigMap is still there — the rollback relies on it. If old versions are no longer needed:
$ kubectl -n demo-dev delete cm hello-config-dev-k675bbfg64
configmap "hello-config-dev-k675bbfg64" deleted from demo-dev namespace
Now, what makes commonLabels dangerous. Add the deprecated field with a new label to the dev overlay and apply:
$ printf 'commonLabels:\n team: web\n' >> hello/overlays/dev/kustomization.yaml
$ kubectl apply -k hello/overlays/dev
# Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
configmap/hello-config-dev-bg9h65b2f2 configured
service/hello-dev configured
ingress.networking.k8s.io/hello-dev configured
The Deployment "hello-dev" is invalid: spec.selector: Invalid value: {"matchLabels":{"app.kubernetes.io/name":"hello","team":"web"}}: field is immutable
$ kubectl -n demo-dev get endpointslice -l kubernetes.io/service-name=hello-dev
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
hello-dev-6rxtn IPv4 <unset> <unset> 71s
The Deployment was rejected, but the ConfigMap, Service and Ingress were already applied: kubectl apply is not a transaction but a sequence of separate requests. The Service selector now contains team: web, the pods have no such label, and not a single address is left in the EndpointSlice — the application stopped receiving traffic although the pods are running as before. Note the ConfigMap name too: the hash did not change, labels are not part of it. Put kustomization.yaml back as it was, apply again — and the pod's address returns to the EndpointSlice:
$ kubectl -n demo-dev get endpointslice -l kubernetes.io/service-name=hello-dev
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
hello-dev-6rxtn IPv4 8080 172.20.211.3 72s
Prod with the component and replacements:
$ kubectl kustomize hello/overlays/prod | grep -E 'replicas|image:|PUBLIC_HOST' -A1
replicas: 3
selector:
--
- name: PUBLIC_HOST
value: hello.apps.example.com
--
image: ghcr.io/example/hello@sha256:2d8774f801594f2b1b115963e54ec54b1b49f95c2dece1d6b0fd1e59ab447c7b
name: hello
Three replicas from the component, the image by digest, the host from the already fixed Ingress. Apply prod (the hello objects created in demo by hand in earlier parts were deleted first, so that old labels and selectors do not get in the way) and see what the cluster knows about Kustomize — nothing, apart from what kubectl recorded as the field manager:
$ kubectl apply -k hello/overlays/prod
configmap/hello-config-758kt8k87t created
service/hello created
deployment.apps/hello created
poddisruptionbudget.policy/hello created
ingress.networking.k8s.io/hello created
$ kubectl -n demo get deploy hello -o jsonpath='{.metadata.managedFields[*].manager}'
kubectl-client-side-apply kube-controller-manager
There are no "kustomization" objects and no annotations about its origin by default. If you want to see which file an object came from, add buildMetadata: [originAnnotations] to kustomization.yaml, and Kustomize sets the config.kubernetes.io/origin annotation with the path to the source file.
When Kustomize and when not
Kustomize is good when you have your own manifests and several environments with predictable differences: everything stays plain YAML, a diff review in Git shows exactly the change that will go to the cluster, and the build is deterministic and needs nothing but kubectl. It is also handy as the "last mile" on top of someone else's set of manifests: take the official YAML of some component as a resource by URL with a tag and lay your patches over it without forking it.
It is inconvenient where logic is needed: "if monitoring is enabled, create a ServiceMonitor" (the object with which Prometheus Operator describes where to scrape metrics from, part 32), "a separate Ingress for every list item", parameters set by a consumer who does not know how the manifests are built. That is the niche of templates, and for distributing an application to people who will not read its manifests a packaging of its own has emerged — the chart.
Sources
- Kubernetes: Declarative Management of Kubernetes Objects Using Kustomize —
kubectl kustomize,kubectl apply -k, generators, bases and overlays. - Kustomize: kustomization.yaml reference — all fields; separately labels and commonLabels (deprecated since v5.0.0), patches, images, configMapGenerator, secretGenerator, generatorOptions, replacements, buildMetadata, helmCharts.
- Kustomize: Components — components and the order in which they are applied.
- kustomize v5.8.3 release — the standalone binary and checksums.
- Strategic Merge Patch and Kubernetes: Update API Objects in Place Using kubectl patch — list merge keys, the
$patchdirective. - RFC 6902: JSON Patch and RFC 6901: JSON Pointer — operations and the
~0,~1escapes. - Kubernetes: ConfigMaps — Immutable ConfigMaps — immutable ConfigMaps and the load on the API server.
- Kubernetes: Declarative Management of Kubernetes Objects Using Configuration Files —
last-applied-configuration,--prune; kubectl diff and kubectl kustomize — flags and exit codes. - SOPS, KSOPS and External Secrets Operator — keeping secrets out of plain text in Git.
Next — Helm: charts, templates and values, releases and where Helm keeps their history, install, upgrade and rollback, our own chart for hello, an OCI registry for charts and what changed in Helm 4.
Only registered users can leave a comment.
Sign in or register to leave a comment.