Kubernetes for beginners. Part 12: How do ConfigMap and Secret work?

The image from part 5 is the same for all environments: the same ghcr.io/example/hello:1.0 runs both in the lab and in production. That means everything that differs between environments — addresses, log levels, feature flags, passwords and keys — has to come into the container from outside, at startup. In part 9 we already took the database password for a dump from a Secret object without explaining what it is. Today we look at the two configuration objects — ConfigMap and Secret: how to pass them into a pod, what happens when they change, why a Secret is not encryption, who can read it, how to encrypt secrets where they really live — in etcd — and how to keep them outside the cluster. All examples live in the demo namespace from the previous part.

---

ConfigMap

A ConfigMap is a namespaced object that stores key-value pairs: short strings or whole files.

apiVersion: v1
kind: ConfigMap
metadata:
  name: hello-config
  namespace: demo
  labels:
    app.kubernetes.io/name: hello
data:
  LOG_LEVEL: info
  GREETING: hello
  app.yaml: |
    listen: ":8080"
    upstream:
      url: http://backend.demo.svc.cluster.local:8080
      timeout: 5s
binaryData:
  logo.png: iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNkYAAAAAYAAjCB0C8AAAAASUVORK5CYII=

data holds UTF-8 strings, binaryData holds binary data in base64 (here a one-pixel PNG image). Keys are names of letters, digits, -, _ and .: they become file names if the ConfigMap is mounted as files. The object's data is limited to 1 MiB (1,048,576 bytes): a ConfigMap is stored in etcd as a whole and sent to everyone watching it, so large files and application data don't belong in it.

A ConfigMap can also be created with a command — from literals, files, a directory or a file of KEY=value lines:

$ kubectl create configmap hello-config --from-literal=LOG_LEVEL=info --from-file=app.yaml
configmap/hello-config created
$ kubectl create configmap nginx-conf --from-file=./conf.d/
configmap/nginx-conf created

$ kubectl get configmap nginx-conf -o yaml | head -7
apiVersion: v1
data:
  default.conf: |
    server { listen 80; }
  gzip.conf: |
    gzip on;
kind: ConfigMap

From a directory, each file becomes a key named after the file. Further on hello-config is created from a manifest, so after this experiment we delete both objects: kubectl delete configmap hello-config nginx-conf. --from-env-file parses the file line by line: KEY=value lines become keys, comments are skipped, and the value is taken as is, spaces included. A key with an invalid character is rejected by kubectl itself, and exceeding the limit by the API server:

$ kubectl create configmap badkey --from-literal='bad key=1'
error: "bad key" is not a valid key name for a ConfigMap: a valid config key must consist of alphanumeric characters, '-', '_' or '.' (e.g. 'key.name',  or 'KEY_NAME',  or 'key-name', regex used for validation is '[-._a-zA-Z0-9]+')

$ head -c 1100000 /dev/zero | tr '\0' a > big.txt
$ kubectl create configmap big --from-file=big.txt
error: failed to create configmap: ConfigMap "big" is invalid: []: Too long: may not be more than 1048576 bytes

A ConfigMap does nothing by itself: it is data in the API. It starts working when the kubelet puts it into a pod.

---

How to pass configuration into a pod

There are four ways, and they behave differently on changes.

A single variable from a key:

env:
- name: LOG_LEVEL
  valueFrom:
    configMapKeyRef:
      name: hello-config
      key: LOG_LEVEL

All keys at once as variables (envFrom), optionally with a prefix:

envFrom:
- configMapRef:
    name: hello-config
  prefix: HELLO_          # HELLO_LOG_LEVEL, HELLO_GREETING

Since 1.34 a variable name may consist of any printable ASCII characters except =, so envFrom turns the app.yaml key into a variable too — HELLO_app.yaml with the whole file content. Any valid ConfigMap key is now a valid variable name: even 1badkey, which the documentation gives as an example of a key skipped with an InvalidEnvironmentVariableNames event, becomes a variable on a 1.36 cluster without any events. Keys from binaryData never become variables — logo.png from our ConfigMap is silently skipped. It is safer to keep variables and files in separate ConfigMaps.

A volume — each key becomes a file:

volumes:
- name: config
  configMap:
    name: hello-config
    items:                       # optional: only these keys
    - key: app.yaml
      path: app.yaml
containers:
- name: hello
  volumeMounts:
  - name: config
    mountPath: /etc/hello
    readOnly: true

One file into an existing directory via subPath:

volumeMounts:
- name: config
  mountPath: /etc/nginx/conf.d/default.conf
  subPath: default.conf

subPath is handy when the image's directory already holds the files you need and only one has to be replaced: a regular volume would hide the whole directory. But it has a serious drawback — more on it below.

Two more volume types collect data from several sources: projected combines in one directory keys from several ConfigMaps, Secrets, Downward API information (the pod's data about itself — name, labels, resources) and a ServiceAccount token; this is exactly how the kubelet mounts an API access token into every pod, covered in detail in part 29.

A reference to a non-existent ConfigMap or key stops the pod from starting until the object appears. How exactly depends on the way: a variable from a missing key gives CreateContainerConfigError, a volume from a missing ConfigMap keeps the pod in ContainerCreating. If the absence is acceptable, the reference is marked optional: true — then the variable is empty and the volume directory is empty:

$ kubectl get pods missing-key missing-cm missing-optional
NAME               READY   STATUS                       RESTARTS   AGE
missing-key        0/1     CreateContainerConfigError   0          26s
missing-cm         0/1     ContainerCreating            0          26s
missing-optional   1/1     Running                      0          25s

$ kubectl get events --field-selector involvedObject.name=missing-key -o custom-columns=REASON:.reason,MSG:.message | tail -1
Failed      Error: couldn't find key DB_URL in ConfigMap demo/hello-config

$ kubectl get events --field-selector involvedObject.name=missing-cm -o custom-columns=REASON:.reason,MSG:.message | tail -1
FailedMount   MountVolume.SetUp failed for volume "extra" : configmap "extra-config" not found

---

What happens on a change: volumes

You changed a ConfigMap — what will a running pod see? It depends on the way.

Volumes are updated. The kubelet watches the ConfigMaps and Secrets used by the pods on its node and rewrites the files when they change. It does this so that the application never sees half old files and half new ones. The volume directory holds not the files themselves but symbolic links:

/etc/hello/
 ├─ app.yaml        → ..data/app.yaml
 ├─ GREETING        → ..data/GREETING
 ├─ ..data          → ..2026_10_28_15_07_41.507995263
 └─ ..2026_10_28_15_07_41.507995263/
      ├─ app.yaml
      └─ GREETING

On a change the kubelet creates a new timestamped directory, writes all files there, then creates a ..data_tmp link to the new directory and renames it to ..data. A rename in Linux is atomic: at any moment ..data points either to the whole old set or to the new one. The old directory is deleted afterwards.

Practical consequences:

• an application that watches a file through inotify (the kernel mechanism for file change notifications) must watch the directory or ..data: app.yaml itself is a link that doesn't change; what changes is where ..data points;
• an application that keeps the file open will keep reading the old version from the deleted directory; configuration must be re-read by opening the file again;
• the delay is about a minute or more. The kubelet re-checks the pod's volumes every syncFrequency (one minute by default), plus the delay of the change reaching its cache; by default the kubelet subscribes to changes (the Watch strategy) and this addition is small, but the documentation warns that the total can reach two minutes. On our lab the file was updated 77 seconds after kubectl patch. Don't count on instant application.

subPath is never updated. A file mounted via subPath is a bind mount of a specific file in a specific timestamped directory, not of the ..data link. The kubelet creates a new directory, switches ..data and deletes the old one, while the subPath mount keeps showing the old, already deleted file until the container restarts. If you need both to replace one file in the image's directory and to update it on the fly, mount the volume into a separate directory and point the application to that path.

---

Variables are not updated

A process's environment variables are set once — at its start, by the execve system call. Neither the kubelet nor the kernel can change the environment of a running process. So values from env and envFrom are fixed at container start and do not change until the container restarts. You changed LOG_LEVEL in the ConfigMap — the pods keep running with the old value, while new pods created later (on scaling or re-creation after a failure) get the new one. You end up with a mix of versions, which is worse than either of them.

For a change to reach everyone, the pods have to be restarted. By hand:

$ kubectl rollout restart deployment/hello
deployment.apps/hello restarted

rollout restart restarts nothing directly: it adds the kubectl.kubernetes.io/restartedAt annotation with the current time to the pod template. The template changed — the Deployment rolls out a new ReplicaSet with the usual rolling update from part 7, with maxSurge, maxUnavailable and probes. The same trick underlies automation: change something in the template when the configuration changes. Charts of the Helm package manager (part 27) conventionally put a hash of the ConfigMap content into a template annotation; Kustomize (part 26) appends a hash to the ConfigMap's name, so a new configuration is a new object and a new reference in the template. And for objects applied directly there is Reloader.

---

Reloader

Reloader is a small controller that watches ConfigMaps and Secrets and, when they change, triggers a rollout of the Deployments, StatefulSets and DaemonSets that use them. It doesn't restart pods itself: it changes the template, and then the regular rollout mechanism does the work with all its guarantees.

It is installed with the official Helm chart in one command (Helm in detail — in part 27). The project also has a ready-made manifest, but it creates everything in the default namespace, and a controller allowed to read the secrets of the whole cluster belongs in a namespace of its own:

$ helm repo add stakater https://stakater.github.io/stakater-charts
$ helm install reloader stakater/reloader --version 2.2.19 \
    --namespace reloader --create-namespace

Chart 2.2.19 installs Reloader v1.4.23 — the reloader-reloader Deployment with one replica, a ServiceAccount and a ClusterRole with get, list, watch on ConfigMaps and Secrets of the whole cluster. By default it watches all namespaces and reacts only to objects with annotations:

Annotation on the workload	What it does
reloader.stakater.com/auto: "true"	restart when any ConfigMap or Secret referenced by the template changes
configmap.reloader.stakater.com/auto: "true"	the same, for ConfigMaps only
secret.reloader.stakater.com/auto: "true"	the same, for Secrets only
configmap.reloader.stakater.com/reload: "hello-config"	restart when exactly these ConfigMaps change (comma-separated), even if the template doesn't reference them
secret.reloader.stakater.com/reload: "db-credentials"	the same for Secrets
reloader.stakater.com/search: "true"	restart only when objects change that carry the reloader.stakater.com/match: "true" annotation on the ConfigMap/Secret itself

The annotation goes into metadata.annotations of the Deployment itself, not of the pod template:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello
  namespace: demo
  annotations:
    reloader.stakater.com/auto: "true"

How Reloader changes the template is set by the strategy. By default (env-vars) it adds to the containers using the changed object a variable like STAKATER_HELLO_CONFIG_CONFIGMAP holding a hash of the content: the template changed — a rollout starts. The annotations strategy writes the reloader.stakater.com/last-reloaded-from annotation into the pod template instead; it is preferred with delivery from Git (part 41), so that the sync tool doesn't see a "foreign" variable as drift from the repository.

Reloader is convenient, but keep two things in mind. A configuration error is now rolled out automatically — and just as automatically breaks all pods; good readiness probes and maxUnavailable from part 7 stop the rollout at the first broken pod. And a ConfigMap isn't versioned: rolling a Deployment back to a previous revision doesn't roll the configuration back. The Kustomize approach with a hash in the name solves this too: the old revision references the old ConfigMap.

---

Immutable ConfigMaps and Secrets

The immutable: true field (stable since 1.21) makes a ConfigMap or Secret immutable:

apiVersion: v1
kind: ConfigMap
metadata:
  name: hello-config-v3
immutable: true
data:
  LOG_LEVEL: info

The data of such an object can't be changed, and immutable can't be removed either: only delete and re-create. Metadata — labels and annotations — changes as usual:

$ kubectl patch configmap hello-config-v3 --type=merge -p '{"data":{"LOG_LEVEL":"debug"}}'
The ConfigMap "hello-config-v3" is invalid: data: Forbidden: field is immutable when `immutable` is set

$ kubectl patch configmap hello-config-v3 --type=merge -p '{"immutable":false}'
The ConfigMap "hello-config-v3" is invalid: immutable: Forbidden: field is immutable when `immutable` is set

$ kubectl label configmap hello-config-v3 version=v3
configmap/hello-config-v3 labeled

Why it is needed:

• protection from accidental edits — the configuration running pods rely on won't change under their feet;
• load on the API server — the kubelet stops watching immutable objects. In a cluster with tens of thousands of pods and mounted ConfigMaps this noticeably offloads the API server.

Together with a version in the name (hello-config-v3) this is the right scheme: a new configuration is a new object, the Deployment switches to it with a regular rollout, and the old revision stays working for a rollback. Old versions are deleted later.

---

Secret

A Secret is built almost like a ConfigMap: key-value pairs in a namespace, up to 1 MiB, passed into a pod as variables or a volume. The differences are in how it is handled:

• values in data are written in base64; for convenience there is the stringData field where you can write plain text — the API server re-encodes it into data on write and never returns stringData in responses;
• a Secret has a type by which the API checks the set of keys;
• the kubelet puts Secret volumes into tmpfs — an in-memory filesystem — so that the secret doesn't land on the node's disk;
• the kubelet receives only the Secrets of pods scheduled to its node (the Node authorization mode of the API server enforces this), not all secrets of the cluster;
• kubectl describe shows only the sizes of a Secret's values, not the values themselves.

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
  namespace: demo
type: Opaque
stringData:
  username: app
  password: Zq3vN8rT1wX6pL0s

stringData has a trap. kubectl apply saves the applied manifest as a whole in the kubectl.kubernetes.io/last-applied-configuration annotation — for a Secret too, with the plain stringData:

$ kubectl get secret db-credentials -o yaml
apiVersion: v1
data:
  password: WnEzdk44clQxd1g2cEwwcw==
  username: YXBw
kind: Secret
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Secret","metadata":{"annotations":{},"name":"db-credentials","namespace":"demo"},"stringData":{"password":"Zq3vN8rT1wX6pL0s","username":"app"},"type":"Opaque"}
  creationTimestamp: "2026-10-28T15:07:41Z"
  name: db-credentials
  namespace: demo
  resourceVersion: "774947"
  uid: 1bee8f75-9deb-460d-b5c0-50e5027b79b8
type: Opaque

kubectl describe secret doesn't show this annotation (Annotations: <none>), but it is there in the API and in etcd. A Secret created with kubectl create secret or kubectl apply --server-side doesn't get it.

Types:

Type	Purpose	Required keys
Opaque	arbitrary data (default)	—
kubernetes.io/tls	TLS certificate and key	tls.crt, tls.key
kubernetes.io/dockerconfigjson	image registry access (imagePullSecrets from part 5)	.dockerconfigjson
kubernetes.io/dockercfg	the same in the old format	.dockercfg
kubernetes.io/basic-auth	login and password	username or password (at least one)
kubernetes.io/ssh-auth	SSH private key	ssh-privatekey
kubernetes.io/service-account-token	long-lived ServiceAccount token	filled in by a controller
bootstrap.kubernetes.io/token	node join token that kubeadm used in part 4	filled in by kubeadm

Long-lived ServiceAccount tokens in Secrets are a legacy: since 1.24 they are not created automatically, and unused old ones are removed. Pods get short-lived tokens through a projected volume (part 29). Create a Secret of this type by hand only for external systems that can't do otherwise.

A Secret is passed into a pod the same way as a ConfigMap: secretKeyRef for a single variable, envFrom with secretRef, a volume with secret.secretName. For volumes it helps to narrow file permissions: by default files are created with mode 0644, readable by any user in the container. defaultMode: 0440 together with the pod's fsGroup (part 30) leaves read access to the application process only.

A file is better than a variable. Environment variables are visible in /proc/<pid>/environ of the process and its descendants, are inherited by all child processes, and often end up in crash reports and in logs during debugging ("let's print the environment"). A file in tmpfs with narrow permissions is visible only to whoever opens it. Another advantage of files is on-the-fly updates: a changed password appears in the file, and an application that can re-read it switches without a restart. Variables are an acceptable compromise for programs that can't do otherwise, but not the default choice.

---

base64 is not encryption

Secret values in the API look unreadable, and that creates a false sense of security:

$ kubectl get secret db-credentials -o jsonpath='{.data.password}'
WnEzdk44clQxd1g2cEwwcw==
$ kubectl get secret db-credentials -o jsonpath='{.data.password}' | base64 -d
Zq3vN8rT1wX6pL0s

base64 is a way to write arbitrary bytes as printable characters so that binary data (keys, DER certificates) fits into JSON and YAML. It is reversible without any key. A Secret is protected exactly as well as these are:

• API access — who can read Secrets through kubectl;
• storage — etcd and its backups;
• nodes — where the kubelet puts secrets for pods;
• manifests — wherever the YAML the Secret was created from lives.

Hence two common mistakes. The first is committing a Secret manifest to Git, "it's base64 anyway". That's a plain-text password in the repository history forever. The second is putting passwords into a ConfigMap: a ConfigMap has neither tmpfs nor special access rules, it is often readable by everyone, and it shows up in describe output. Secret things go only into Secrets, and Secrets only from a protected source (more on that at the end).

---

Who can read a Secret

API access is controlled by RBAC — rules of the form "a subject may perform these actions on these resources"; in detail — in part 29. For Secrets it matters which permissions actually mean "can read a secret":

• get on secrets — read a specific Secret by name;
• list and watch on secrets — get all Secrets of a namespace with their values. The "see the list" permission here equals reading every secret;
• create on pods (or on Deployments, Jobs and any other objects that create pods) in a namespace — indirect access to all Secrets of that namespace: you can create a pod that mounts any of them and read it from inside. The same goes for pods/exec into a pod that already has the secret mounted;
• cluster-level permissions (a ClusterRole with secrets through a ClusterRoleBinding) — access to the secrets of all namespaces, including kube-system.

kubectl auth can-i shows who can do what. The default ServiceAccount in demo doesn't see secrets, while Reloader from the section above sees them across the whole cluster — that's the price of its work:

$ kubectl auth can-i list secrets -n demo --as=system:serviceaccount:demo:default
no
$ kubectl auth can-i list secrets -n kube-system --as=system:serviceaccount:reloader:reloader-reloader
yes

The rule: grant permissions on secrets only to those who really need them, and remember that the right to deploy into a namespace is the right to read its secrets.

---

Encrypting secrets in etcd

The API server stores objects in etcd in the binary protobuf format — and without encryption by default. A Secret's value lies in etcd, in etcd snapshots and in all backups as plain text. Anyone who gets a snapshot file or access to the control plane node's disk gets all secrets of the cluster.

Encryption at rest fixes this: the API server encrypts selected object types before writing them to etcd and decrypts them on read. Nothing changes for API clients. It is configured with an EncryptionConfiguration file and the --encryption-provider-config flag of kube-apiserver:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
  - secrets
  providers:
  - secretbox:
      keys:
      - name: key1
        secret: <32 random bytes in base64>
  - identity: {}

The provider list is ordered: the first one is used for writing, all of them for reading (by the prefix written at the start of the value). identity means "no encryption": at the end of the list it is needed to read what was written before encryption was enabled.

Providers:

Provider	Algorithm	Documentation's recommendation
identity	none	no encryption at all
aescbc	AES-CBC with PKCS#7 padding	not recommended: vulnerable to padding oracle attacks (recovering plaintext from answers about padding correctness)
aesgcm	AES-GCM with a random nonce	only with automatic key rotation: the key must be rotated every 200,000 writes
secretbox	XSalsa20 and Poly1305	strong and fast; the key, like with the other local providers, lives on the control plane node
kms v2	envelope encryption through an external key service	the strongest option, stable since 1.29
kms v1	the same, older version	deprecated since 1.28 and disabled by default since 1.29

The fundamental difference between the local providers and KMS is where the key is. For secretbox, aescbc, aesgcm the key lies in the configuration file on the control plane node, next to etcd. This protects against leaked etcd snapshots and backups, but not against someone who got root on that node: they will read the key too. KMS (Key Management Service — an external key management service: a cloud one, a hardware HSM module or HashiCorp Vault, a store for secrets and keys) uses envelope encryption: data is encrypted with a data encryption key (DEK) using AES-GCM, and the DEK itself with a key encryption key (KEK) that never leaves the KMS. etcd holds the encrypted value and the encrypted DEK; to read a secret you need both etcd and access to the KMS. With KMS v2 the API server derives DEKs from a secret seed that changes when the KEK changes, so it doesn't need to call the KMS on every write, and that's fast. The API server talks to the KMS through a plugin — a separate process on the control plane node listening on a Unix socket.

The practical choice: with a KMS available — kms v2; for a training or small cluster without an external key service — secretbox; don't use aescbc in new installations.

The steps on a kubeadm cluster:

1. generate a key (head -c 32 /dev/urandom | base64) and put the configuration into /etc/kubernetes/enc/enc.yaml with mode 0600 for root — this is the key to all secrets;

2. add the flag and the volume to the API server. In the static pod /etc/kubernetes/manifests/kube-apiserver.yaml this takes effect at once, but kubeadm upgrade regenerates the manifest, so the same has to be written into kubeadm's ClusterConfiguration:

   apiServer:
     extraArgs:
     - name: encryption-provider-config
       value: /etc/kubernetes/enc/enc.yaml
     extraVolumes:
     - name: enc
       hostPath: /etc/kubernetes/enc
       mountPath: /etc/kubernetes/enc
       readOnly: true
       pathType: DirectoryOrCreate
   

3. wait for the API server to restart: the kubelet watches the manifests directory and re-creates the static pod when the file changes;

4. rewrite existing secrets: encryption applies only on write, and the old values stay in plain text. The command from the documentation reads all Secrets and writes them back unchanged: kubectl get secrets -A -o json | kubectl replace -f -.

With several control plane nodes (as in the series about a highly available cluster) the file must be identical on all of them, otherwise one API server can't read what another wrote.

Key rotation takes three steps so that at no moment does data become unreadable: add the new key second and restart all API servers (now all of them can read it); move it first and restart again (now it's used for writing); rewrite all secrets and remove the old key. The --encryption-provider-config-automatic-reload=true flag lets the API server pick up file changes without a restart: it re-reads the file every minute.

And one last thing people forget: the encryption key must be backed up separately from etcd snapshots. A snapshot with encrypted secrets is unrecoverable without the key, and a snapshot stored together with the key defeats the point of encryption. etcd snapshots and restore — in part 34.

---

Secrets outside the cluster

Encryption in etcd protects secrets inside the cluster. The question remains where they come from: a Secret manifest can't be kept in Git in the open, and creating secrets by hand with kubectl create secret means having neither history nor reproducibility. There are several established tools for this.

• External Secrets Operator — a controller that takes values from an external secret store (HashiCorp Vault, cloud secret managers) and creates regular Secrets from them in the cluster. Git holds only an ExternalSecret object with a reference "which key from the store goes into which Secret"; the secret itself is stored and rotated in the external system, and the controller syncs it periodically.
• Sealed Secrets — a controller in the cluster holds a key pair; the kubeseal utility encrypts a Secret with the controller's public key into a SealedSecret object that can be committed to Git. Only the controller of that cluster can decrypt it, and it creates a regular Secret from it.
• SOPS — a utility that encrypts values in YAML and JSON files while keeping keys readable — with age keys (a simple modern encryption tool), OpenPGP or a cloud KMS. The encrypted manifest is kept in Git and decrypted at deployment — by a plugin of the delivery tool (part 41) or on a workstation.
• Secrets Store CSI Driver — a volume driver that mounts secrets from an external store straight into a pod as files, bypassing the Secret object entirely.

The choice depends on whether you have an external secret store. If you do — External Secrets Operator: the secret lives in one place, and rotating it doesn't require a commit. If not — Sealed Secrets or SOPS: secrets in Git, but encrypted. How this fits into delivery from Git — in part 41.

---

Hands-on

Let's create the ConfigMap and Secret from the examples above and a training pod that uses them in every way. The hello image is minimal and has no shell, so we'll look from busybox:

apiVersion: v1
kind: Pod
metadata:
  name: cfg-demo
  namespace: demo
spec:
  securityContext:
    fsGroup: 2000
  containers:
  - name: shell
    image: busybox:1.37
    command: ["sleep", "infinity"]
    env:
    - name: LOG_LEVEL
      valueFrom:
        configMapKeyRef: {name: hello-config, key: LOG_LEVEL}
    volumeMounts:
    - {name: config, mountPath: /etc/hello, readOnly: true}
    - {name: config, mountPath: /etc/single/app.yaml, subPath: app.yaml, readOnly: true}
    - {name: creds, mountPath: /run/secrets/db, readOnly: true}
  volumes:
  - name: config
    configMap: {name: hello-config}
  - name: creds
    secret: {secretName: db-credentials, defaultMode: 0440}

$ kubectl apply -f hello-config.yaml -f db-credentials.yaml -f cfg-demo.yaml
configmap/hello-config created
secret/db-credentials created
pod/cfg-demo created

$ kubectl exec cfg-demo -- ls -la /etc/hello
total 12
drwxrwsrwx    3 root     2000          4096 Oct 28 15:07 .
drwxr-xr-x    1 root     root          4096 Oct 28 15:07 ..
drwxr-sr-x    2 root     2000          4096 Oct 28 15:07 ..2026_10_28_15_07_41.507995263
lrwxrwxrwx    1 root     2000            31 Oct 28 15:07 ..data -> ..2026_10_28_15_07_41.507995263
lrwxrwxrwx    1 root     2000            15 Oct 28 15:07 GREETING -> ..data/GREETING
lrwxrwxrwx    1 root     2000            16 Oct 28 15:07 LOG_LEVEL -> ..data/LOG_LEVEL
lrwxrwxrwx    1 root     2000            15 Oct 28 15:07 app.yaml -> ..data/app.yaml
lrwxrwxrwx    1 root     2000            15 Oct 28 15:07 logo.png -> ..data/logo.png

$ kubectl exec cfg-demo -- ls -l /run/secrets/db/..data/
total 8
-r--r-----    1 root     2000            16 Oct 28 15:07 password
-r--r-----    1 root     2000             3 Oct 28 15:07 username

The links and the timestamped directory are as in the diagram; the time inside busybox is UTC, and the timestamp in the directory name is the moment of writing. The secret files have mode 0440 and the group from fsGroup. Now let's change the ConfigMap and read the file once a second until it updates:

$ kubectl patch configmap hello-config --type=merge -p '{"data":{"LOG_LEVEL":"debug","app.yaml":"listen: \":8080\"\nupstream:\n  url: http://backend.demo.svc.cluster.local:8080\n  timeout: 10s\n"}}'
configmap/hello-config patched

$ s=$(date +%s); for i in $(seq 1 120); do [ "$(kubectl exec cfg-demo -- cat /etc/hello/LOG_LEVEL)" = debug ] && break; sleep 1; done; echo "updated after $(( $(date +%s)-s )) s"
updated after 77 s

$ kubectl exec cfg-demo -- sh -c 'echo "env: $LOG_LEVEL"; echo "file: $(cat /etc/hello/LOG_LEVEL)"; grep timeout /etc/hello/app.yaml /etc/single/app.yaml; readlink /etc/hello/..data'
env: info
file: debug
/etc/hello/app.yaml:  timeout: 10s
/etc/single/app.yaml:  timeout: 5s
..2026_10_28_15_09_14.3053548814

Three behaviours at once: the variable stayed old, the file in the volume was updated (..data now points to the new directory), and the file mounted via subPath is old.

Where it all lives on the node. The pod is on k8s-node-2, its UID comes from kubectl get pod cfg-demo -o jsonpath='{.metadata.uid}':

user@k8s-node-2:~$ sudo ls /var/lib/kubelet/pods/f8b9521e-5015-4691-beb0-9587f57f21ac/volumes/
kubernetes.io~configmap
kubernetes.io~projected
kubernetes.io~secret

user@k8s-node-2:~$ findmnt -t tmpfs | grep f8b9521e
/var/lib/kubelet/pods/f8b9521e-5015-4691-beb0-9587f57f21ac/volumes/kubernetes.io~secret/creds                              tmpfs  tmpfs  rw,relatime,size=131072k,inode64,noswap
/var/lib/kubelet/pods/f8b9521e-5015-4691-beb0-9587f57f21ac/volumes/kubernetes.io~projected/kube-api-access-mz542           tmpfs  tmpfs  rw,relatime,size=131072k,inode64,noswap

user@k8s-node-2:~$ findmnt | grep f8b9521e | grep -v tmpfs
└─/var/lib/kubelet/pods/f8b9521e-5015-4691-beb0-9587f57f21ac/volume-subpaths/config/shell/1                                      /dev/sda2[/var/lib/kubelet/pods/f8b9521e-5015-4691-beb0-9587f57f21ac/volumes/kubernetes.io~configmap/config/..2026_10_28_15_07_41.507995263/app.yaml//deleted]        ext4        rw,relatime

The Secret volume and the projected volume with the ServiceAccount token are in tmpfs (noswap, size 128 MiB — the pod's memory limit from the LimitRange of the demo namespace), the ConfigMap is on the regular disk in /var/lib/kubelet. Secrets don't land on the node's disk. And the last line is subPath itself: the kubelet bind-mounted app.yaml from the directory of the first version, that directory is already deleted (//deleted), and the container keeps seeing the old file.

Other ways to create a Secret are the kubectl create secret command with the generic, docker-registry and tls subcommands. tls needs a certificate; for an experiment a self-signed one issued with openssl will do:

$ kubectl create secret generic api-token --from-literal=token=s3cr3t-t0ken
secret/api-token created
$ kubectl create secret generic db-env --from-env-file=db.env
secret/db-env created

$ kubectl create secret docker-registry regcred --docker-server=ghcr.io \
    --docker-username=example --docker-password=ghp_example --dry-run=client -o yaml
apiVersion: v1
data:
  .dockerconfigjson: eyJhdXRocyI6eyJnaGNyLmlvIjp7InVzZXJuYW1lIjoiZXhhbXBsZSIsInBhc3N3b3JkIjoiZ2hwX2V4YW1wbGUiLCJhdXRoIjoiWlhoaGJYQnNaVHBuYUhCZlpYaGhiWEJzWlE9PSJ9fX0=
kind: Secret
metadata:
  name: regcred
type: kubernetes.io/dockerconfigjson

$ openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes \
    -keyout tls.key -out tls.crt -days 30 -subj "/CN=hello.example.com"
$ kubectl create secret tls hello-tls --cert=tls.crt --key=tls.key
secret/hello-tls created

$ kubectl get secrets
NAME             TYPE                             DATA   AGE
api-token        Opaque                           1      1s
db-credentials   Opaque                           2      4m32s
db-env           Opaque                           2      1s
ghcr-pull        kubernetes.io/dockerconfigjson   1      6m41s
hello-tls        kubernetes.io/tls                2      0s

$ kubectl describe secret hello-tls | sed -n '/^Type/,$p'
Type:  kubernetes.io/tls

Data
====
tls.crt:  599 bytes
tls.key:  241 bytes

docker-registry builds the same JSON that docker login writes to ~/.docker/config.json: under the base64 it is {"auths":{"ghcr.io":{"username":"example","password":"ghp_example","auth":"…"}}}. The type is checked by the API server, and kubectl create secret tls also parses the PEM itself:

$ kubectl create secret generic bad-tls --type=kubernetes.io/tls --from-file=tls.crt
error: failed to create secret Secret "bad-tls" is invalid: data[tls.key]: Required value

$ kubectl create secret generic bad-basic --type=kubernetes.io/basic-auth --from-literal=user=app
error: failed to create secret Secret "bad-basic" is invalid: [data[username]: Required value, data[password]: Required value]

$ kubectl create secret tls bad-pair --cert=tls.crt --key=db.env
error: tls: failed to find any PEM data in key input

Reloader. Let's install it, put the annotation on hello from the previous part, switch LOG_LEVEL to the ConfigMap and change the ConfigMap once more:

$ kubectl annotate deployment hello reloader.stakater.com/auto=true
deployment.apps/hello annotated
$ kubectl set env deployment/hello --from=configmap/hello-config --keys=LOG_LEVEL
deployment.apps/hello env updated
$ kubectl rollout status deployment/hello
Waiting for deployment spec update to be observed...
…
deployment "hello" successfully rolled out

$ kubectl patch configmap hello-config --type=merge -p '{"data":{"LOG_LEVEL":"warn"}}'
configmap/hello-config patched

$ kubectl get deployment hello -o jsonpath='{range .spec.template.spec.containers[0].env[*]}{.name}={.value}{"\n"}{end}'
STAKATER_HELLO_CONFIG_CONFIGMAP=92f99426d763ed6778fb609e85204f8a50781b25
LOG_LEVEL=

$ kubectl -n reloader logs deploy/reloader-reloader | tail -1
time="2026-10-28T15:10:21Z" level=info msg="Changes detected in 'hello-config' of type 'CONFIGMAP' in namespace 'demo'; updated 'hello' of type 'Deployment' in namespace 'demo'"

$ kubectl get rs -l app.kubernetes.io/name=hello
NAME               DESIRED   CURRENT   READY   AGE
hello-6b76bd67d4   3         3         3       18s
hello-6d5c6db6b4   0         0         0       5m5s
hello-7c87cc4dc9   0         0         0       42s

LOG_LEVEL= with an empty value is the variable from configMapKeyRef: it has no value, the kubelet substitutes it. Reloader added the variable with the hash at the top of the list, the template changed, and the Deployment rolled out a new ReplicaSet 6b76bd67d4 in the same second the ConfigMap changed — without the minute-long delay of volumes. The new pods started with LOG_LEVEL=warn already. Three ReplicaSets mean three templates: the original 6d5c6db6b4 from the previous part, 7c87cc4dc9 after set env and 6b76bd67d4 after the ConfigMap change.

Finally, the Secret in etcd. In the lab etcd runs as a static pod on the control plane node, and its image contains the etcdctl client; we run it through crictl (the client of CRI — the interface through which the kubelet drives containerd) and examine the output on the node with hexdump:

user@k8s-master-1:~$ E() { sudo crictl exec "$(sudo crictl ps --name etcd -q)" etcdctl \
    --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt \
    --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key "$@"; }

user@k8s-master-1:~$ E get /registry/secrets/demo/db-credentials | hexdump -C
00000000  2f 72 65 67 69 73 74 72  79 2f 73 65 63 72 65 74  |/registry/secret|
00000010  73 2f 64 65 6d 6f 2f 64  62 2d 63 72 65 64 65 6e  |s/demo/db-creden|
00000020  74 69 61 6c 73 0a 6b 38  73 00 0a 0c 0a 02 76 31  |tials.k8s.....v1|
00000030  12 06 53 65 63 72 65 74  12 e1 04 0a a7 04 0a 0e  |..Secret........|
00000040  64 62 2d 63 72 65 64 65  6e 74 69 61 6c 73 12 00  |db-credentials..|
...
00000130  22 73 74 72 69 6e 67 44  61 74 61 22 3a 7b 22 70  |"stringData":{"p|
00000140  61 73 73 77 6f 72 64 22  3a 22 5a 71 33 76 4e 38  |assword":"Zq3vN8|
00000150  72 54 31 77 58 36 70 4c  30 73 22 2c 22 75 73 65  |rT1wX6pL0s","use|
...
00000260  7b 7d 7d 42 00 12 1c 0a  08 70 61 73 73 77 6f 72  |{}}B.....passwor|
00000270  64 12 10 5a 71 33 76 4e  38 72 54 31 77 58 36 70  |d..Zq3vN8rT1wX6p|
00000280  4c 30 73 12 0f 0a 08 75  73 65 72 6e 61 6d 65 12  |L0s....username.|
00000290  03 61 70 70 1a 06 4f 70  61 71 75 65 1a 00 22 00  |.app..Opaque..".|
000002a0  0a                                                |.|
000002a1

k8s\0 is the API server's protobuf format marker, and after it everything is in the open, twice: in the last-applied-configuration annotation (the very stringData from the manifest) and in the data itself — password, Zq3vN8rT1wX6pL0s. Not even base64 — etcd holds the raw bytes. Let's enable secretbox encryption following the steps above and rewrite the secrets:

user@k8s-master-1:~$ sudo install -d -m 0700 /etc/kubernetes/enc
user@k8s-master-1:~$ KEY=$(head -c 32 /dev/urandom | base64)
user@k8s-master-1:~$ sudo tee /etc/kubernetes/enc/enc.yaml > /dev/null <<EOF
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources: [secrets]
  providers:
  - secretbox:
      keys:
      - name: key1
        secret: ${KEY}
  - identity: {}
EOF
user@k8s-master-1:~$ unset KEY; sudo chmod 0600 /etc/kubernetes/enc/enc.yaml
user@k8s-master-1:~$ sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml ~/kube-apiserver.yaml.orig
user@k8s-master-1:~$ sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml   # the flag and the volume, as above

We keep the manifest copy outside the manifests directory: the kubelet runs every file there as a static pod. The API server is unavailable for about half a minute while the kubelet re-creates the pod; for another ten seconds or so the pod shows 0/1 Running until it passes its readiness check:

$ kubectl -n kube-system get pod -l component=kube-apiserver
NAME                          READY   STATUS    RESTARTS   AGE
kube-apiserver-k8s-master-1   1/1     Running   0          36s

$ kubectl get secrets -A -o json | kubectl replace -f -
secret/calico-apiserver-certs replaced
secret/node-certs replaced
secret/typha-certs replaced
secret/typha-certs-noncluster-host replaced
secret/ghcr-pull replaced
secret/api-token replaced
secret/db-credentials replaced
…

user@k8s-master-1:~$ E get /registry/secrets/demo/db-credentials | hexdump -C | head -8
00000000  2f 72 65 67 69 73 74 72  79 2f 73 65 63 72 65 74  |/registry/secret|
00000010  73 2f 64 65 6d 6f 2f 64  62 2d 63 72 65 64 65 6e  |s/demo/db-creden|
00000020  74 69 61 6c 73 0a 6b 38  73 3a 65 6e 63 3a 73 65  |tials.k8s:enc:se|
00000030  63 72 65 74 62 6f 78 3a  76 31 3a 6b 65 79 31 3a  |cretbox:v1:key1:|
00000040  fb 8b 0b 92 8d 47 31 ee  7c a5 d2 85 f2 56 bc 4e  |.....G1.|....V.N|
00000050  96 b7 db 37 7e 3e 0c 7b  80 81 c4 b0 78 73 c4 2e  |...7~>.{....xs..|
00000060  7f 63 46 3a 2c 9c a1 b9  af 64 a7 1b d9 31 79 17  |.cF:,....d...1y.|
00000070  55 e1 c0 28 f8 0f 3e 9f  08 31 24 0b 5b c0 67 39  |U..(..>..1$.[.g9|

$ kubectl get secret db-credentials -o jsonpath='{.data.password}' | base64 -d
Zq3vN8rT1wX6pL0s

Now etcd has the prefix k8s:enc:secretbox:v1:key1: after the key: the provider, the version and the name of the key the API server will decrypt with — followed by ciphertext; the annotation with stringData is encrypted together with the whole object. The ConfigMap is still stored in the open (k8s\0 at the start of the value): the configuration lists only secrets. Nothing changed for API clients: kubectl gets the value as before. All secrets have to be rewritten, not just new ones: without replace the old ones would have stayed in plain text, readable thanks to identity at the end of the list — the prefix tells the API server which provider to read each value with.

Don't forget to add the same to kubeadm's ClusterConfiguration and to back up /etc/kubernetes/enc/enc.yaml separately from etcd snapshots. If this was an experiment on a training cluster and encryption has to be turned off, the order is reversed and the key is removed last: put identity first and secretbox with the same key second, restart the API server, run kubectl get secrets -A -o json | kubectl replace -f - again (now values are written in the open, and every key in etcd starts with k8s\0 again), restore the manifest from the copy, and only then delete /etc/kubernetes/enc. The training objects: kubectl delete pod cfg-demo, kubectl delete configmap hello-config-v3, kubectl delete secret api-token db-env hello-tls; Reloader — helm uninstall reloader -n reloader and kubectl delete namespace reloader.

---

Sources

• ConfigMaps — data and binaryData keys, the 1 MiB limit, volumes and variables, immutable ConfigMaps.
• Configure a Pod to Use a ConfigMap — --from-file, --from-env-file, envFrom, optional, the volume update delay and subPath without updates.
• Secrets — Secret types and their keys, stringData, tmpfs, immutable, long-lived ServiceAccount tokens.
• Managing Secrets using kubectl — kubectl create secret and reading values.
• Good practices for Kubernetes Secrets — encryption, RBAC permissions, indirect access through creating pods.
• Projected Volumes — the projected volume, Downward API and the ServiceAccount token.
• Encrypting Confidential Data at Rest — EncryptionConfiguration, the provider table, rewriting secrets, key rotation, automatic reload, turning encryption off.
• Using a KMS provider for data encryption — KMS v1 and v2, envelope encryption, the plugin.
• Authorization — kubectl auth can-i and authorization modes, including Node.
• Reloader — annotations, the env-vars and annotations strategies, installation.

---

The configuration is in place and the application starts, but we never asked which node a pod lands on and why exactly there. Taints, tolerations and drain came up in the examples, and the DaemonSet in part 8 tied pods to nodes through affinity.

Next — the scheduler: how kube-scheduler picks a node, nodeSelector, affinity and anti-affinity, taints and tolerations, spreading across zones and nodes, PodDisruptionBudget, and what kubectl cordon and kubectl drain really do.