Kubernetes for beginners. Part 27: How does Helm work?

In part 26 we split the hello manifests into a base and overlays and built environments with Kustomize without changing the manifests themselves. That works well for your own applications, but Kustomize has no notion of "parameters": whoever installs the application has to know its manifests to patch them. Helm solves a different problem — it packages an application so that it can be installed with one command by setting a dozen understandable parameters, upgraded, rolled back and removed as a whole. We installed csi-driver-nfs in part 23 and csi-driver-smb in part 24 with release manifests and mentioned their Helm charts in a single sentence. Today we look at it properly: what a chart is, how the template engine works, where Helm keeps its state, what changed in Helm 4, and we write a complete chart for hello.

---

Helm 4

Helm is a CNCF project (the Cloud Native Computing Foundation, home of Kubernetes, containerd, etcd and others), a command-line tool without a server side. Version 2 installed a Tiller component with broad rights into the cluster; since version 3 there is none, and helm works as the user with their kubeconfig and their RBAC rights (part 29) — exactly like kubectl.

Helm 4, the first major version in six years, came out in November 2025. The lab runs the current 4.3.1 (released on 15 October 2026; the Helm 3 branch still gets fixes, the latest is 3.22.1). Charts of the previous format (apiVersion: v2) work unchanged, and the differences are mostly in how commands behave:

What	Helm 3	Helm 4
How changes reach the cluster	client-side: Helm computes the patch itself from the old manifest, the new one and the live object	server-side apply for new releases (flag --server-side, default auto: new installs use SSA, upgrades use the method the release was installed with)
Rollback on failure	--atomic	--rollback-on-failure (the old name works with a warning)
Recreating objects	--force	--force-replace (the old name works with a warning)
Waiting for readiness	--wait: its own checks per object type	--wait takes a strategy: watcher (default with --wait, via kstatus), legacy (as in Helm 3), hookOnly (default without the flag — wait only for hooks)
Post-processing	--post-renderer — a path to any program	--post-renderer — the name of a post-renderer plugin
Plugins	external programs only	plus WebAssembly plugins (portable bytecode executed in an isolated environment); installation verifies the signature
helm registry login	accepted a URL	accepts only a host name: https://… gives invalid registry

kstatus is a library of the Kubernetes SIGs project that decides from the status and metadata.generation fields of any object (custom ones included) whether it has reached its desired state: a Deployment when all replicas are updated and available, a Job when it has completed, a CRD object by the Ready condition in status.conditions. The old Helm 3 check knew only built-in types.

Helm 4 reads and upgrades releases installed by Helm 3: the storage format is the same. But they stay on client-side apply until --server-side=true is passed explicitly on upgrade: the method is recorded in the release itself (the apply_method field, ssa for new installs), and auto takes it from there. The old flag names work with a warning like Flag --atomic has been deprecated, use --rollback-on-failure instead.

---

The chart

A chart is a directory (or a .tgz archive of it) with a defined layout:

hello/
├── Chart.yaml            ← name, version, dependencies
├── Chart.lock            ← exact dependency versions (created by helm dependency update)
├── values.yaml           ← default parameters
├── values.schema.json    ← JSON Schema to validate parameters (optional)
├── .helmignore           ← what not to put into the archive
├── charts/               ← downloaded subchart dependencies
├── crds/                 ← CRD definitions (a special case, see below)
└── templates/
    ├── _helpers.tpl      ← named templates, create no objects themselves
    ├── NOTES.txt         ← text printed after installation
    ├── deployment.yaml
    ├── service.yaml
    ├── ...
    └── tests/
        └── test-connection.yaml

Chart.yaml describes the chart:

apiVersion: v2              # chart format of Helm 3 and 4
name: hello
description: Demo HTTP application
type: application           # or library — only named templates for other charts
version: 0.1.0              # CHART version, SemVer (MAJOR.MINOR.PATCH); changes with any chart edit
appVersion: "1.0"           # APPLICATION version inside, informational
kubeVersion: ">=1.33.0-0"   # which clusters the chart agrees to be installed on
dependencies:
  - name: redis
    version: "~1.4.0"
    repository: oci://ghcr.io/example/charts
    condition: redis.enabled

The two versions are a frequent source of confusion. version is the version of the package: you changed a template or a default value — you bump version, otherwise the registry will not accept the new archive under the same number, and users end up with two different 0.1.0 charts. appVersion is the version of the program inside, by convention used as the default image tag. kubeVersion is checked before rendering: on a cluster older than required Helm refuses right away — Error: chart requires kubeVersion: >=1.33.0-0 which is incompatible with Kubernetes v1.32.0.

Dependencies are other charts installed together with this one as its part (subcharts). The example above is a hypothetical chart of Redis (an in-memory key-value store) from our registry, enabled by the redis.enabled parameter. helm dependency update downloads them into charts/ and writes exact versions and digests into Chart.lock; helm dependency build restores charts/ strictly from Chart.lock. Parameters of a subchart are set in the parent's values.yaml under a key with its name (redis:), and the global: key is visible to all subcharts.

The easiest way to start a new chart is the helm create hello scaffold: it creates a working chart for nginx with all the typical templates, which you then adapt. We will write the chart from scratch so that every line is understood.

---

Templates

Files in templates/ are templates of Go text/template (Go's standard template engine: substitutions in {{ }}, conditions, loops, variables) with the Sprig function library (about a hundred functions for strings, lists, dictionaries, dates, hashes) and a few functions of Helm itself. Helm runs each file through the template engine, concatenates the result, parses it as YAML and sends it to the API.

Built-in objects available in a template:

Object	What it contains
.Values	the final parameters (after merging all sources)
.Release.Name, .Release.Namespace	release name and installation namespace
.Release.IsInstall, .Release.IsUpgrade, .Release.Revision	what is happening now and the revision number
.Chart.Name, .Chart.Version, .Chart.AppVersion	fields of Chart.yaml
.Capabilities.KubeVersion, .Capabilities.APIVersions	cluster version and available APIs (for helm template — set by flags)
.Files	access to the other chart files (.Files.Get "config/app.conf")
.Template.Name, .Template.BasePath	path of the current template

Syntax you meet all the time:

{{ .Values.replicaCount }}                     substitution
{{ .Values.image.tag | default .Chart.AppVersion }}   pipeline: default value
{{ .Values.host | quote }}                     in quotes
{{- if .Values.ingress.enabled }} ... {{- end }}      condition; "-" eats whitespace and the newline
{{- range .Values.extraEnv }} ... {{- end }}          loop
{{- with .Values.nodeSelector }} ... {{- end }}       switch "." to the value if it is not empty
{{ include "hello.labels" . | nindent 4 }}     named template indented by 4 with a newline
{{ toYaml .Values.resources | nindent 12 }}    insert a piece of values as YAML
{{ required "host is required" .Values.host }}  stop rendering with an error if empty

Almost all chart errors are indentation errors: the template engine knows nothing about YAML and inserts text as is. So blocks are always inserted with nindent N (a newline plus N spaces), and the result is checked with helm template.

include calls a named template — a {{ define "name" }}…{{ end }} block, by convention kept in _helpers.tpl (files starting with _ are not output by Helm as manifests). Unlike Go's built-in template, the result of include can be passed further down a pipeline, for example into nindent, so charts use only include.

---

Our own chart for hello

The whole chart. Parameters first:

# hello/values.yaml
replicaCount: 2

image:
  repository: ghcr.io/example/hello
  tag: ""                 # empty — appVersion from Chart.yaml is used
  digest: ""              # if set, takes precedence over the tag
  pullPolicy: IfNotPresent

imagePullSecrets: []

config:
  LOG_LEVEL: info
  GREETING: hello

service:
  port: 80

ingress:
  enabled: true
  className: traefik
  host: hello.apps.example.com
  clusterIssuer: letsencrypt-prod

resources:
  requests:
    cpu: 50m
    memory: 32Mi
  limits:
    memory: 64Mi

podDisruptionBudget:
  enabled: true
  minAvailable: 1

nodeSelector: {}
tolerations: []

Named templates: the full object name and label sets. The app.kubernetes.io/* labels are the ones recommended by Kubernetes; the selector is built only from the immutable ones (name and instance) so that a chart upgrade never changes the immutable selector of a Deployment.

{{/* hello/templates/_helpers.tpl */}}

{{- define "hello.fullname" -}}
{{- if contains .Chart.Name .Release.Name -}}
{{- .Release.Name | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- printf "%s-%s" .Release.Name .Chart.Name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{- end -}}

{{- define "hello.selectorLabels" -}}
app.kubernetes.io/name: {{ .Chart.Name }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end -}}

{{- define "hello.labels" -}}
{{ include "hello.selectorLabels" . }}
app.kubernetes.io/version: {{ .Values.image.tag | default .Chart.AppVersion | quote }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
helm.sh/chart: {{ printf "%s-%s" .Chart.Name .Chart.Version }}
{{- end -}}

{{- define "hello.image" -}}
{{- if .Values.image.digest -}}
{{ .Values.image.repository }}@{{ .Values.image.digest }}
{{- else -}}
{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}
{{- end -}}
{{- end -}}

trunc 63 is there because an object name that ends up in labels and DNS is limited to 63 characters. The contains check avoids names like hello-hello when the release has the same name as the chart.

# hello/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ include "hello.fullname" . }}
  labels:
    {{- include "hello.labels" . | nindent 4 }}
data:
  {{- range $key, $value := .Values.config }}
  {{ $key }}: {{ $value | quote }}
  {{- end }}

# hello/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "hello.fullname" . }}
  labels:
    {{- include "hello.labels" . | nindent 4 }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      {{- include "hello.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      labels:
        {{- include "hello.labels" . | nindent 8 }}
      annotations:
        checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
    spec:
      {{- with .Values.imagePullSecrets }}
      imagePullSecrets:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      automountServiceAccountToken: false
      securityContext:
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: hello
          image: {{ include "hello.image" . | quote }}
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          ports:
            - name: http
              containerPort: 8080
          envFrom:
            - configMapRef:
                name: {{ include "hello.fullname" . }}
          readinessProbe:
            httpGet:
              path: /healthz
              port: http
          livenessProbe:
            httpGet:
              path: /healthz
              port: http
            periodSeconds: 20
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
          resources:
            {{- toYaml .Values.resources | nindent 12 }}
      {{- with .Values.nodeSelector }}
      nodeSelector:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      {{- with .Values.tolerations }}
      tolerations:
        {{- toYaml . | nindent 8 }}
      {{- end }}

The checksum/config annotation is the same trick as the hash in the ConfigMap name in Kustomize, only done differently: the SHA-256 of the rendered ConfigMap is put into the pod template. A parameter in config changed — the annotation changed — the pod template changed — the Deployment rolled out new pods. The securityContext fields are covered in detail in part 30, automountServiceAccountToken: false in part 29: the application needs no API access, and the token is not mounted into the pod.

# hello/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: {{ include "hello.fullname" . }}
  labels:
    {{- include "hello.labels" . | nindent 4 }}
spec:
  selector:
    {{- include "hello.selectorLabels" . | nindent 4 }}
  ports:
    - name: http
      port: {{ .Values.service.port }}
      targetPort: http

# hello/templates/ingress.yaml
{{- if .Values.ingress.enabled }}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: {{ include "hello.fullname" . }}
  labels:
    {{- include "hello.labels" . | nindent 4 }}
  annotations:
    cert-manager.io/cluster-issuer: {{ .Values.ingress.clusterIssuer | quote }}
spec:
  ingressClassName: {{ .Values.ingress.className }}
  rules:
    - host: {{ required "ingress.host is required" .Values.ingress.host | quote }}
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: {{ include "hello.fullname" . }}
                port:
                  name: http
  tls:
    - hosts:
        - {{ .Values.ingress.host | quote }}
      secretName: {{ include "hello.fullname" . }}-tls
{{- end }}

# hello/templates/pdb.yaml
{{- if and .Values.podDisruptionBudget.enabled (gt (int .Values.replicaCount) 1) }}
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: {{ include "hello.fullname" . }}
  labels:
    {{- include "hello.labels" . | nindent 4 }}
spec:
  minAvailable: {{ .Values.podDisruptionBudget.minAvailable }}
  selector:
    matchLabels:
      {{- include "hello.selectorLabels" . | nindent 6 }}
{{- end }}

The PodDisruptionBudget is created only with two replicas or more: with one replica and minAvailable: 1 it would forbid kubectl drain of the node forever (part 13).

# hello/templates/tests/test-connection.yaml
apiVersion: v1
kind: Pod
metadata:
  name: {{ include "hello.fullname" . }}-test
  labels:
    {{- include "hello.labels" . | nindent 4 }}
  annotations:
    helm.sh/hook: test
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
spec:
  restartPolicy: Never
  automountServiceAccountToken: false
  securityContext:
    runAsNonRoot: true
    runAsUser: 65534
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: wget
      image: busybox:1.37
      command: ["wget", "-qO-", "http://{{ include "hello.fullname" . }}:{{ .Values.service.port }}/healthz"]
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop: ["ALL"]

{{/* hello/templates/NOTES.txt */}}
hello {{ .Chart.AppVersion }} installed as release {{ .Release.Name }} in {{ .Release.Namespace }}.
{{- if .Values.ingress.enabled }}
URL: https://{{ .Values.ingress.host }}/
{{- else }}
Check: kubectl -n {{ .Release.Namespace }} port-forward svc/{{ include "hello.fullname" . }} 8080:{{ .Values.service.port }}
{{- end }}

And the parameter schema — it is checked on install, upgrade, template and lint, and a typo in -f or --set gives a clear error instead of a silently ignored key:

{
  "$schema": "https://json-schema.org/draft-07/schema#",
  "type": "object",
  "required": ["image", "ingress"],
  "properties": {
    "replicaCount": { "type": "integer", "minimum": 1 },
    "image": {
      "type": "object",
      "required": ["repository"],
      "properties": {
        "repository": { "type": "string" },
        "tag": { "type": "string" },
        "digest": { "type": "string", "pattern": "^(sha256:[a-f0-9]{64})?$" }
      }
    },
    "ingress": {
      "type": "object",
      "properties": {
        "enabled": { "type": "boolean" },
        "host": { "type": "string" }
      }
    }
  }
}

---

Where the values come from

The final .Values are assembled from several sources, each overriding the previous one:

 the chart's values.yaml (and subcharts')
          │
          ▼
 -f / --values file1, file2 ...      ← in command-line order
          │
          ▼
 --set, --set-string, --set-json, --set-file   ← point edits
          │
          ▼
 .Values in templates

Dictionaries are merged by key, lists are replaced as a whole. In practice environment values are kept in files — values-dev.yaml, values-prod.yaml in Git next to the deployment description — and --set is left for one-off changes like --set image.tag=1.1 from CI. The reason is simple: --set leaves no trace except the release itself, and in a month nobody will remember why prod has three replicas.

--set has its own grammar: a dot means nesting (ingress.host=…), a comma separates pairs, an index in square brackets is a list element (tolerations[0].key=…). Values are converted to types: true and false become booleans, integers become numbers, and everything else stays a string. So --set image.tag=1.10 gives the string "1.10", while --set image.tag=2 gives the number 2, and our schema (see below) stops the installation:

$ helm template hello ./hello -f values-prod.yaml --set image.tag=2
Error: values don't meet the specifications of the schema(s) in the following chart(s):
hello:
- at '/image/tag': got number, want string

Without the schema the number would silently go into the template. So tags and everything that must stay a string are passed with --set-string.

On helm upgrade the default is the chart's values plus what is passed in this command; the values of the previous release are not carried over. The --reuse-values flag takes the user's previous values instead of the chart's values and thereby ignores new defaults of the updated chart, while --reset-then-reuse-values takes the new chart's values and lays the previous user values on top. The most reliable approach is to rely on neither and always pass the full set of files.

---

The release and where Helm keeps its state

A release is an installed instance of a chart with a name unique within the namespace. One chart can be installed many times under different names. Every install, upgrade and rollback creates a new revision of the release.

Helm keeps no database and no server: the state lives in the cluster itself, by default in a Secret in the release's namespace, one per revision. The name is sh.helm.release.v1.<release>.v<revision>, the type is helm.sh/release.v1. Inside is the release field: a JSON description of the release, gzip-compressed and base64-encoded (and the API adds its own base64 to all Secret data). The JSON holds everything needed for a rollback without the original chart: the templates, values.yaml and schema of the chart (chart), the user values (config), the rendered manifest (manifest), hooks (hooks), status and time (info) and the apply method (apply_method).

 helm upgrade hello ./hello -f values-prod.yaml
     │
     ├─ 1. read the current revision from Secret sh.helm.release.v1.hello.v3
     ├─ 2. assemble values, render templates
     ├─ 3. Secret ...v4 with status pending-upgrade
     ├─ 4. pre-upgrade hooks
     ├─ 5. apply objects (SSA: PATCH application/apply-patch+yaml,
     │     fieldManager=helm; delete objects gone from the manifest)
     ├─ 6. --wait: wait for readiness (kstatus)
     ├─ 7. post-upgrade hooks
     └─ 8. v4 → deployed, v3 → superseded

Consequences of this design:

• Rights to a release are rights to Secrets in the namespace. Whoever can read Secrets sees all values of the release, including passwords, if they were passed through values.
• At most 10 revisions per release are kept (the --history-max flag), older ones are deleted.
• If helm was interrupted in the middle (laptop closed, network dropped), the release stays in pending-install or pending-upgrade, and the next operation refuses with UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress. The cure is helm rollback to the last good revision. On the lab helm upgrade --wait was killed after 5 seconds — the history kept a pending-upgrade revision described as Preparing upgrade, the next upgrade got that error, and helm rollback went through and recorded a new deployed revision.
• If you delete the release's Secret, you "forget" the release, but the objects in the cluster stay. This is sometimes used to hand objects over to another release; since Helm 3.17 there is the --take-ownership flag for that, which takes over existing objects without deleting them.

Helm puts the label app.kubernetes.io/managed-by: Helm and the annotations meta.helm.sh/release-name and meta.helm.sh/release-namespace on every object. By them it refuses to touch an object with the same name that belongs to another release or was created by hand.

---

Install, upgrade, rollback, uninstall

helm install   hello ./hello -n demo -f values-prod.yaml          install
helm upgrade   hello ./hello -n demo -f values-prod.yaml          upgrade
helm upgrade --install hello ./hello -n demo -f values-prod.yaml  install or upgrade (for CI)
helm rollback  hello 2 -n demo                                    go back to revision 2 (creates a new one)
helm uninstall hello -n demo                                      remove objects and history

Useful flags:

• --wait — do not return until the objects are ready, but no longer than --timeout (5 minutes by default). Without it Helm considers the installation successful as soon as the API accepted the objects, even if the pods do not start.
• --rollback-on-failure (formerly --atomic) — roll back on failure: on install remove what was installed, on upgrade return the previous revision. It turns on --wait by itself.
• --cleanup-on-fail — on a failed upgrade delete the objects created by this upgrade.
• --create-namespace — create the namespace if it does not exist. Convenient, but then the namespace does not belong to the release and does not get your Pod Security labels (part 30) — better create it separately.
• --version — the chart version from the repository; without it the latest is taken, which is a bad idea for production.
• --dry-run=server — everything except writing: templates are rendered, objects go through validation and admission on the API server.

A rollback in Helm is not an "undo": helm rollback hello 2 takes the manifest of revision 2, applies it and records it as a new revision. The history is linear and shows what happened. helm uninstall removes the release's objects (except those annotated helm.sh/resource-policy: keep, hook resources and CRDs) and the history Secrets; --keep-history keeps the history, and the release can be rolled back later.

About server-side apply in Helm 4 one consequence matters. Helm registers itself as the owner (fieldManager) of the fields present in the chart. If someone changed such a field by hand with kubectl edit or kubectl scale, the next helm upgrade gets a field ownership conflict from the API server, and Helm stops:

$ kubectl -n demo scale deploy hello --replicas=4
deployment.apps/hello scaled
$ helm upgrade hello ./hello -n demo -f values-prod.yaml --set-string image.tag=1.1
level=WARN msg="upgrade failed" name=hello error="conflict occurred while applying object demo/hello apps/v1, Kind=Deployment: Apply failed with 1 conflict: conflict with \"kubectl\" with subresource \"scale\" using apps/v1: .spec.replicas"
Error: UPGRADE FAILED: conflict occurred while applying object demo/hello apps/v1, Kind=Deployment: Apply failed with 1 conflict: conflict with "kubectl" with subresource "scale" using apps/v1: .spec.replicas

A revision with status failed appears in the release history. --force-conflicts takes the fields over (a retry with it brought replicas: 3 back from values); it is better to find out who changes them. Fields that are not in the chart — for example replicas if autoscaling manages it — Helm no longer touches at all, and that is more convenient than the three-way merge of Helm 3, where such fields had to be removed from templates on purpose.

---

Checking before applying: lint, template, diff

helm lint ./hello checks the chart structure, template syntax and the value schema. helm template renders the chart locally and prints the manifests without contacting the cluster:

$ helm template hello ./hello -n demo -f values-prod.yaml --kube-version 1.36.0

Because of that the lookup function (reading a live object from the cluster) returns nothing in helm template, and .Capabilities is filled with defaults or the --kube-version and --api-versions flags. The output of helm template can be fed to kubectl apply -f - or put into Git, and that is a perfectly workable scheme for those who want only the template engine without stored releases.

For "what will change in the cluster" you need the helm-diff plugin: it renders the new version and compares it with the manifest of the current release (or, with --three-way-merge, with the live objects, to see manual edits too). In Helm 4 plugins are installed with signature verification, and the helm-diff author publishes signed archives (each with a .prov file next to it) and his public key pubkey.asc. Verification uses a GnuPG keyring (GnuPG is a free implementation of the OpenPGP signing and encryption standard), and without it the installation fails:

$ helm plugin install https://github.com/databus23/helm-diff/releases/latest/download/helm-diff-linux-amd64.tgz
Verifying plugin signature...
Error: plugin verification failed: open /home/user/.gnupg/pubring.gpg: no such file or directory

The author's key is downloaded from the same release page and converted from text form into a binary keyring (gpg --dearmor) that Helm can read; gpg --show-keys shows beforehand whose key it is:

$ curl -sSLO https://github.com/databus23/helm-diff/releases/latest/download/pubkey.asc
$ gpg --show-keys pubkey.asc | head -3
pub   rsa4096 2026-06-15 [SC] [expires: 2028-06-14]
      C5645EF47482257A1F806D2BEA17A2A206AFF8CD
uid                      Fabian Ruff (helm-diff release key) <fabian@progra.de>
$ gpg --dearmor < pubkey.asc > helm-diff.gpg
$ helm plugin install https://github.com/databus23/helm-diff/releases/latest/download/helm-diff-linux-amd64.tgz --keyring ./helm-diff.gpg
Verifying plugin signature...
Using Key With Fingerprint: C5645EF47482257A1F806D2BEA17A2A206AFF8CD
Plugin Hash Verified: sha256:504e99410611daa70702362e1b10dc74929df8a912cb3f6b5d21d1ee9d0cb7d9
Installed plugin: diff
$ helm plugin list
NAME    VERSION  TYPE    APIVERSION  PROVENANCE  SOURCE
diff    3.15.15  cli/v1  legacy      unknown     unknown

--verify=false skips the check, but then an unverified executable gets into Helm. Compare a release on 1.0 with an upgrade to 1.1 (output trimmed to the Deployment):

$ helm diff upgrade hello ./hello -n demo -f values-prod.yaml --set-string image.tag=1.1
…
demo, hello, Deployment (apps) has changed:
  # Source: hello/templates/deployment.yaml
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: hello
    labels:
      app.kubernetes.io/name: hello
      app.kubernetes.io/instance: hello
-     app.kubernetes.io/version: "1.0"
+     app.kubernetes.io/version: "1.1"
      app.kubernetes.io/managed-by: Helm
      helm.sh/chart: hello-0.1.0
…
        annotations:
-         checksum/config: ab2f9bbf23579c4ac7887de51dac11673fb2c6513f1355444e4e379dd4c8dc7d
+         checksum/config: 3d144dda669f1e95d4553b25f616f923469148ecc258d833e75c808fe995f7d6
…
-           image: "ghcr.io/example/hello:1.0"
+           image: "ghcr.io/example/hello:1.1"
…

The app.kubernetes.io/version label changes on all five objects of the chart, and with it checksum/config: the hash is computed over the whole rendered ConfigMap, labels included, not only over data. Here that does no harm, the pods restart anyway because of the new image, but if something that changes often gets into the ConfigMap's labels, pods will restart without any configuration change. Running helm diff upgrade with the same arguments before any helm upgrade of a production release is a good habit.

---

Hooks and tests

A hook is an object from templates/ with the helm.sh/hook annotation. Helm does not apply it together with the others, but creates it at a particular point of the release lifecycle and waits for it to finish (for a Job — successful completion, for a pod — the Succeeded phase):

Hook	When
pre-install, post-install	before and after creating objects on install
pre-upgrade, post-upgrade	before and after an upgrade
pre-rollback, post-rollback	before and after a rollback
pre-delete, post-delete	before and after uninstall
test	only on helm test

A typical example is a database schema migration as a Job in pre-upgrade: if it fails, the upgrade does not start. The order of several hooks is set by helm.sh/hook-weight (a string with a number, smaller first), and what happens to the object after it runs by helm.sh/hook-delete-policy: before-hook-creation (delete the previous instance before creating a new one, the default), hook-succeeded, hook-failed.

Two properties of hooks that people learn in practice. They are not part of the release: helm uninstall does not delete them, and helm diff and helm get manifest do not show them. And Helm waits for them up to --timeout: a hook Job stuck in Pending for lack of resources holds up the whole upgrade.

Tests are test hooks: pods that check that what was installed works. Our test-connection.yaml makes a request to /healthz through the Service. They are run separately, after installation:

$ helm test hello -n demo
NAME: hello
LAST DEPLOYED: Sat Oct 31 04:49:07 2026
NAMESPACE: demo
STATUS: deployed
REVISION: 4
DESCRIPTION: Rollback to 2
TEST SUITE:     hello-test
Last Started:   Sat Oct 31 04:49:15 2026
Last Completed: Sat Oct 31 04:49:23 2026
Phase:          Succeeded

After success the test pod is deleted (hook-succeeded), so kubectl get pod hello-test will not find it anymore; on failure it stays, and its log can be read.

---

Charts in an OCI registry

Historically charts were distributed by Helm repositories — an HTTP server with an index.yaml file (a list of all charts and versions) and the archives next to it. They are added with helm repo add, and most public charts are distributed this way. The modern way is to keep a chart in an ordinary OCI registry, the same one used for images (part 5): the chart becomes an OCI artifact with a manifest, a configuration of type application/vnd.cncf.helm.config.v1+json and one layer — the chart archive. No index.yaml, rights and authentication are the same as for images, versions are tags.

$ helm package ./hello
Successfully packaged chart and saved it to: /home/user/hello-0.1.0.tgz

$ helm registry login ghcr.io -u example
Password:
Login Succeeded

$ helm push hello-0.1.0.tgz oci://ghcr.io/example/charts
Pushed: ghcr.io/example/charts/hello:0.1.0
Digest: sha256:afdb173bddd50518e7fccf77906afcd549376dc53ab384c446f7bd6844babb5e

$ helm install hello oci://ghcr.io/example/charts/hello --version 0.1.0 -n demo -f values-prod.yaml
Pulled: ghcr.io/example/charts/hello:0.1.0
Digest: sha256:afdb173bddd50518e7fccf77906afcd549376dc53ab384c446f7bd6844babb5e
NAME: hello
…

On the lab the chart was pushed not to ghcr.io but to a temporary CNCF Distribution registry (an open-source image registry, formerly Docker Registry; image registry:3) running as a pod in the cluster and reachable through kubectl port-forward; the commands there had the --plain-http flag, because that registry ran without TLS. The manifest of the pushed artifact, read from the registry directly:

$ curl -s -H 'Accept: application/vnd.oci.image.manifest.v1+json' \
    localhost:5000/v2/charts/hello/manifests/0.1.0 | jq '{config: .config.mediaType, layers: [.layers[].mediaType], annotations}'
{
  "config": "application/vnd.cncf.helm.config.v1+json",
  "layers": [
    "application/vnd.cncf.helm.chart.content.v1.tar+gzip"
  ],
  "annotations": {
    "org.opencontainers.image.created": "2026-10-31T04:52:42Z",
    "org.opencontainers.image.description": "Demo HTTP application",
    "org.opencontainers.image.title": "hello",
    "org.opencontainers.image.version": "0.1.0"
  }
}

helm push takes the name and version from the archive, the path in the address is a "directory" in the registry. GitHub accepts nested paths like ghcr.io/<owner>/charts/<chart>: this is how prometheus-community publishes its charts, for example (helm show chart oci://ghcr.io/prometheus-community/charts/kube-prometheus-stack pulls it without logging in). In Helm 4 helm registry login accepts only the host name: for https://ghcr.io or a path in the registry it answers invalid registry. helm registry login stores the credentials in ~/.config/helm/registry/config.json. The password here is a GitHub token with the write:packages scope, the same one used to push images in part 5; for CI a separate token is created and kept in the pipeline's secrets rather than using a personal one. The integrity and origin of a chart can also be confirmed with a signature: helm package --sign creates a .prov file next to it, and helm install --verify checks it; for OCI artifacts signing with Cosign is also common — a tool of the Sigstore project that stores the signature in the same registry next to the artifact.

---

Post-renderer: Kustomize on top of Helm

Sometimes someone else's chart almost fits, but it has no parameter for the field you need: say, a label or a toleration for one DaemonSet, or a container securityContext. Forking the chart means maintaining the fork later. A post-renderer is a program to which Helm hands the rendered manifests on standard input and takes the result from standard output before applying; that result is also what gets recorded in the release. The natural candidate for this role is Kustomize.

In Helm 3 --post-renderer took a path to any executable. In Helm 4 it is a plugin of type postrenderer/v1. A plugin is a directory with plugin.yaml and a script:

# kustomize-pr/plugin.yaml
apiVersion: v1
type: postrenderer/v1
name: kustomize-pr
version: 0.1.0
runtime: subprocess
runtimeConfig:
  platformCommand:
    - command: ${HELM_PLUGIN_DIR}/render.sh

#!/bin/sh
# kustomize-pr/render.sh: $1 — a directory with kustomization.yaml and patches
set -eu
work=$(mktemp -d)
trap 'rm -rf "$work"' EXIT
cp -r "$1"/. "$work"/
cat > "$work/helm-output.yaml"
kubectl kustomize "$work"

The directory with patches lives next to the environment's values, and its kustomization.yaml refers to the file the script creates from Helm's output:

# deploy/post-render/kustomization.yaml
resources:
  - helm-output.yaml
patches:
  - target:
      kind: DaemonSet
      name: csi-nfs-node
    patch: |-
      apiVersion: apps/v1
      kind: DaemonSet
      metadata:
        name: csi-nfs-node
      spec:
        template:
          metadata:
            labels:
              team: storage

Here the patch adds a label to the DaemonSet's pods for which the chart has no parameter; a toleration, an annotation, an environment variable or a resource limit are added the same way.

$ helm plugin install ./kustomize-pr
Installing plugin from local directory (development mode)
Installed plugin: kustomize-pr
$ helm plugin list
NAME          VERSION  TYPE             APIVERSION  PROVENANCE  SOURCE
diff          3.15.15  cli/v1           legacy      unknown     unknown
kustomize-pr  0.1.0    postrenderer/v1  v1          local dev   unknown
$ helm repo add csi-driver-nfs https://raw.githubusercontent.com/kubernetes-csi/csi-driver-nfs/master/charts
"csi-driver-nfs" has been added to your repositories
$ helm upgrade csi-driver-nfs csi-driver-nfs/csi-driver-nfs -n kube-system --version 4.13.6 \
    --post-renderer kustomize-pr --post-renderer-args "$PWD/deploy/post-render"

The same post-renderer works with helm template too, so the result can be inspected without installing anything:

$ helm template csi-driver-nfs csi-driver-nfs/csi-driver-nfs -n kube-system --version 4.13.6 \
    --post-renderer kustomize-pr --post-renderer-args "$PWD/deploy/post-render" \
    | sed -n '/^kind: DaemonSet/,/containers:/p' | sed -n '/template:/,$p'
  template:
    metadata:
      labels:
        app: csi-nfs-node
        app.kubernetes.io/instance: csi-driver-nfs
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: csi-driver-nfs
        app.kubernetes.io/version: 4.13.6
        helm.sh/chart: csi-driver-nfs-4.13.6
        team: storage
    spec:
      containers:

A plugin from a local directory is installed in development mode without signature verification (PROVENANCE local dev); for a team it is better to package and sign it like any other. Two rules when working with a post-renderer. Everyone who upgrades this release must call it the same way: the post-renderer's output is recorded in the release manifest, and the next upgrade without it silently removes the patch — on the lab a label added this way to the hello pods disappeared from them after a plain helm upgrade. And the post-renderer sees all manifests, including Secrets with passwords, so only something you trust is used in this role.

---

Pitfalls

CRDs. Custom resource definitions (part 28) in the crds/ directory are applied by Helm only on the first install, before the templates, without templating, and are never upgraded or deleted. This is a deliberate decision: deleting a CRD deletes all objects of that type in the whole cluster, and an accidental helm uninstall must not destroy, say, all cert-manager certificates. The consequence is that when an operator's chart is upgraded, new CRD fields do not appear by themselves. So large projects either put CRDs into templates/ behind a separate parameter with the helm.sh/resource-policy: keep annotation, or ask you to apply the CRDs separately with kubectl apply --server-side -f crds.yaml before helm upgrade. Before upgrading any operator, read which path its chart has chosen.

Immutable fields. If a new chart version changes spec.selector of a Deployment, volumeClaimTemplates of a StatefulSet, spec.clusterIP of a Service or fields of a Job, helm upgrade fails with field is immutable. There are three ways out: do not allow such changes in your own charts (the selector only from immutable labels, as in _helpers.tpl above); for someone else's — delete the object by hand before upgrade (for a StatefulSet — kubectl delete sts --cascade=orphan, so that pods and volumes stay); or --force-replace, which deletes and recreates objects — with downtime.

Secrets in values. A password in values-prod.yaml in Git is the same plain secret as in a manifest; besides, it settles in the release history Secret. Options: the helm-secrets plugin decrypts a values file encrypted with SOPS (a tool that encrypts the values in YAML with an age, PGP or cloud KMS key while keeping the keys readable) right before calling Helm (helm secrets upgrade … -f secrets.yaml); or the chart does not accept the password at all and refers to a Secret created by External Secrets Operator — a controller that copies values from an external secret store into ordinary Secrets. The second option is better: the password does not pass through Helm and does not get into the release history.

latest and floating versions. helm install without --version and dependencies with a ~1.4.0 range without Chart.lock give different results on different days. In production environments the chart version is pinned and Chart.lock is committed.

Someone else's chart is someone else's code. Templates can create anything, including a ClusterRoleBinding to cluster-admin. Before installing an unfamiliar chart it is worth reading its helm template output at least for RBAC and privileged pods.

---

Hands-on

The production values live in a separate file next to the deployment description:

# values-prod.yaml
replicaCount: 3
ingress:
  host: hello.apps.example.com

$ helm version
version.BuildInfo{Version:"v4.3.1", GitCommit:"c0d08d0d6d9424d4a14febb5e8a3e51b0dd00dff", GitTreeState:"clean", GoVersion:"go1.27.2", KubeClientVersion:"v1.37"}

$ helm lint ./hello -f values-prod.yaml
==> Linting ./hello
[INFO] Chart.yaml: icon is recommended

1 chart(s) linted, 0 chart(s) failed

The schema catches an error in values before the cluster is even contacted:

$ helm template hello ./hello --set replicaCount=0
Error: values don't meet the specifications of the schema(s) in the following chart(s):
hello:
- at '/replicaCount': minimum: got 0, want 1

Install with waiting and rollback on failure. The hello image is private, so, as in part 26, the demo namespace already holds the ghcr-pull Secret attached to the default ServiceAccount (part 5).:

$ helm install hello ./hello -n demo -f values-prod.yaml --wait --rollback-on-failure
NAME: hello
LAST DEPLOYED: Sat Oct 31 04:46:22 2026
NAMESPACE: demo
STATUS: deployed
REVISION: 1
DESCRIPTION: Install complete
NOTES:
hello 1.0 installed as release hello in demo.
URL: https://hello.apps.example.com/

$ helm list -n demo
NAME    NAMESPACE  REVISION  UPDATED                                  STATUS    CHART        APP VERSION
hello   demo       1         2026-10-31 04:46:22.689654138 +0000 UTC  deployed  hello-0.1.0  1.0

What is in the cluster:

$ kubectl -n demo get secret -l owner=helm
NAME                          TYPE                 DATA   AGE
sh.helm.release.v1.hello.v1   helm.sh/release.v1   1      17s

$ kubectl -n demo get secret sh.helm.release.v1.hello.v1 --show-labels
NAME                          TYPE                 DATA   AGE   LABELS
sh.helm.release.v1.hello.v1   helm.sh/release.v1   1      17s   modifiedAt=1793421992,name=hello,owner=helm,status=deployed,version=1

$ kubectl -n demo get secret sh.helm.release.v1.hello.v1 -o jsonpath='{.data.release}' \
    | base64 -d | base64 -d | gunzip | jq '{name, version, namespace, info: .info.status, chart: .chart.metadata.version, config}'
{
  "name": "hello",
  "version": 1,
  "namespace": "demo",
  "info": "deployed",
  "chart": "0.1.0",
  "config": {
    "ingress": {
      "host": "hello.apps.example.com"
    },
    "replicaCount": 3
  }
}

Two base64 -d — the first removes the API's encoding, the second Helm's own. config is exactly what came from the user (-f and --set), without the chart's defaults. helm get values hello -n demo, helm get manifest hello -n demo and helm get all hello -n demo show the same without manual decoding.

Who owns the fields after a Helm 4 install:

$ kubectl -n demo get deploy hello -o jsonpath='{range .metadata.managedFields[*]}{.manager}{"\t"}{.operation}{"\n"}{end}'
helm	Apply
kube-controller-manager	Update

Apply means server-side apply; releases of Helm 3 would show helm Update here. Upgrade with a new tag and then a rollback:

$ helm upgrade hello ./hello -n demo -f values-prod.yaml --set-string image.tag=1.1 --wait
Release "hello" has been upgraded. Happy Helming!
NAME: hello
LAST DEPLOYED: Sat Oct 31 04:46:45 2026
NAMESPACE: demo
STATUS: deployed
REVISION: 2
DESCRIPTION: Upgrade complete
…
$ helm upgrade hello ./hello -n demo -f values-prod.yaml --set-string image.tag=1.2 --wait --timeout 2m --rollback-on-failure
level=WARN msg="upgrade failed" name=hello error="resource Deployment/demo/hello not ready. status: InProgress, message: Updated: 1/3\ncontext deadline exceeded"
Error: UPGRADE FAILED: release hello failed, and has been rolled back due to rollback-on-failure being set: resource Deployment/demo/hello not ready. status: InProgress, message: Updated: 1/3
context deadline exceeded

$ helm history hello -n demo
REVISION  UPDATED                   STATUS      CHART        APP VERSION  DESCRIPTION
1         Sat Oct 31 04:46:22 2026  superseded  hello-0.1.0  1.0          Install complete
2         Sat Oct 31 04:46:45 2026  superseded  hello-0.1.0  1.0          Upgrade complete
3         Sat Oct 31 04:47:06 2026  failed      hello-0.1.0  1.0          Upgrade "hello" failed: resource Deployment/demo/hello not ready. status: InProgress, message: Updated: ...
4         Sat Oct 31 04:49:07 2026  deployed    hello-0.1.0  1.0          Rollback to 2

There is no tag 1.2 in the registry, the new pod went into ImagePullBackOff, the Deployment did not become ready within two minutes — kstatus says exactly that: status: InProgress, message: Updated: 1/3 — and Helm brought revision 2 back by itself as a new revision 4. APP VERSION is 1.0 in every row: it is the chart's appVersion, not the tag passed with --set-string. The old pods were not affected: a rolling update does not remove old replicas until the new ones are ready (part 7), so users did not notice the failure. Without --rollback-on-failure the release would have stayed failed with a hanging new ReplicaSet, and the rollback would have been manual: helm rollback hello 2 -n demo.

Uninstall:

$ helm uninstall hello -n demo
release "hello" uninstalled
$ kubectl -n demo get secret -l owner=helm,name=hello
No resources found in demo namespace.

---

Helm or Kustomize

	Kustomize	Helm
Input	valid manifests	templates that are not YAML until rendered
Parameters	none, only patches	values.yaml with a schema
Logic (conditions, loops)	no	yes
State in the cluster	none	release and revision history in Secrets
Rollback	through Git and re-applying	helm rollback
Removing stale objects	no (apply does not delete)	yes, by the previous revision's manifest
Distribution	a directory in Git	an archive in a repository or an OCI registry
Built into kubectl	yes	no

The rule that works in most teams: your own applications — Kustomize (or a simple chart if there really are many parameters); third-party components — their official Helm chart with a values file in Git, plus a post-renderer if the chart cannot do something. The tools do not compete: Argo CD — a GitOps controller that applies the content of a Git repository to the cluster itself (part 41) — handles both.

---

Sources

• Helm 4 Overview and release v4.3.1 — what changed in Helm 4: server-side apply, kstatus, new flag names, plugins.
• Helm: Charts — chart layout, Chart.yaml, version and appVersion, kubeVersion, dependencies, values.schema.json, the crds/ directory.
• Helm: Chart Template Guide, Built-in Objects and Values Files — templates, built-in objects, the order in which values are merged; Go text/template and Sprig — the template language and the function library.
• Helm: Using Helm, helm install and helm upgrade — releases, --set and its types, the --wait, --rollback-on-failure, --server-side, --take-ownership flags.
• Helm: Chart Hooks and Chart Tests — hooks, weights and delete policies, helm test.
• Helm: Use OCI-based registries — helm registry login, helm push, the media types of a chart OCI artifact; Helm: Provenance and Integrity — chart signing and .prov files.
• Helm: Advanced Techniques and Helm Plugins — post-renderer and Helm 4 plugins, WebAssembly ones included.
• Helm: Custom Resource Definitions — why CRDs from crds/ are neither upgraded nor deleted.
• Kubernetes: Server-Side Apply — field owners and conflicts; kstatus — how an object's readiness is computed; Recommended Labels — the app.kubernetes.io/* labels.
• helm-diff and helm-secrets — plugins for comparison and for encrypted values; CNCF Distribution — the registry the chart push was tested with.

---

Next — CRDs and operators: how to add your own object type to the API, how the controller that turns it into pods and services works, and what the PostgreSQL and RabbitMQ operators do in place of an administrator.