Kubernetes for beginners. Part 30: How do you secure pods?

In the previous part we ended on RBAC deciding whether a subject may create a pod, but not which pod it creates. A container is an ordinary process on the node, isolated by namespaces, cgroups, a reduced set of capabilities and seccomp and AppArmor profiles (part 1, part 6). Each of these boundaries is controlled by manifest fields, and each can be removed with one line of YAML. Today — which fields these are, what the kubelet and containerd do with them, which values are right, and how to make a whole namespace follow them: the Pod Security standards, the Pod Security Admission controller and custom rules in CEL. At the end of the part the demo namespace moves to the restricted level.

Everything in this part was tested on the lab cluster: securityContext fields on real pods (checked on the node through /proc), user namespaces, Pod Security Admission with namespace labels and with cluster-level configuration, ValidatingAdmissionPolicy. Left untested are Localhost seccomp and AppArmor profiles, SELinux, MutatingAdmissionPolicy, Kyverno and OPA Gatekeeper — those are described from the documentation.

---

securityContext: what goes where

Security fields are set in securityContext — at the pod level (applying to all containers) and at the container level (overriding the pod). Some fields exist in only one place:

Field	Pod	Container	What it does
runAsUser, runAsGroup	yes	yes	the process UID and primary GID
runAsNonRoot	yes	yes	the kubelet refuses to start the container if the UID turns out to be 0
supplementaryGroups, fsGroup	yes	—	supplementary groups; the group owning volumes
seccompProfile	yes	yes	system call filter
appArmorProfile	yes	yes	AppArmor profile
seLinuxOptions	yes	yes	SELinux label
sysctls	yes	—	kernel parameters in the pod's namespaces
allowPrivilegeEscalation	—	yes	forbid gaining privileges through setuid files
capabilities	—	yes	add and drop capabilities
privileged	—	yes	remove almost all isolation
readOnlyRootFilesystem	—	yes	read-only container root filesystem
procMount	—	yes	whether to mask parts of /proc

Separately, not in securityContext but directly in the pod spec: hostNetwork, hostPID, hostIPC, hostUsers and hostPath volumes.

Kubernetes defaults are chosen for compatibility, not security: if you specify nothing, the container runs as the user recorded in the image (often root), with containerd's default set of capabilities, can gain privileges through setuid files and runs without a system call filter unless the kubelet has seccompDefault enabled. Let's go through the fields one by one.

---

The user: runAsUser, runAsGroup, runAsNonRoot

A process in a container with UID 0 is root of the node's kernel, limited only by what was taken away from it. If the isolation turns out to be leaky (a bug in the kernel, in runc, a carelessly mounted volume), a root process ends up root on the node. A process with UID 10001 ends up as a user that owns nothing on the node.

securityContext:
  runAsNonRoot: true
  runAsUser: 10001
  runAsGroup: 10001

runAsUser and runAsGroup replace the user from the image (USER in the Dockerfile, part 5). runAsNonRoot: true is a check, not a setting: before starting the container the kubelet looks at which UID it will run with and refuses if it's zero. If the UID isn't set in the manifest and the image records a user name (USER curl_user), the kubelet can't verify that it isn't 0 and refuses as well. Both refusals are a CreateContainerConfigError state with the reason in the message:

$ kubectl -n demo get pod nonroot-check -o jsonpath='{.status.containerStatuses[0].state.waiting.message}{"\n"}'
container has runAsNonRoot and image will run as root (pod: "nonroot-check_demo(41f90b68-f083-48d8-8355-13fd488955cd)", container: c)
$ kubectl -n demo get pod named-user -o jsonpath='{.status.containerStatuses[0].state.waiting.message}{"\n"}'
container has runAsNonRoot and image has non-numeric user (curl_user), cannot verify user is non-root (pod: "named-user_demo(979e565d-87cd-40c5-a3a4-355fb4063cbe)", container: c)

The first pod is busybox:1.37 (its user is root), the second is curlimages/curl:8.17.0 with USER curl_user. The fix is a numeric USER in the image or runAsUser in the manifest.

Best practice is to build the image with an unprivileged user from the start (USER 10001:10001) and still set runAsNonRoot: true, so that an accidental image change doesn't start root. Our hello is built this way: the distroless base image (a minimal image without a shell or package manager) has a ready-made nonroot user, and the image config records "User": "65532:65532" — as a number, so the runAsNonRoot check passes even without runAsUser.

---

Capabilities

Root's privileges in Linux are split into capabilities — about forty separate privileges: CAP_NET_ADMIN (configure the network), CAP_SYS_ADMIN (mount and dozens of other operations), CAP_NET_BIND_SERVICE (listen on ports below 1024) and so on. By default containerd leaves a container 14 of them:

Capability	What it allows
CHOWN, FOWNER, FSETID, DAC_OVERRIDE	change file ownership, bypass file permission checks
SETUID, SETGID, SETFCAP, SETPCAP	change one's own UID and GID, set capabilities on files
KILL	send signals to any process in its PID namespace
NET_BIND_SERVICE	open ports below 1024
NET_RAW	raw sockets: ping, but also crafting arbitrary packets
MKNOD	create device files
SYS_CHROOT	chroot
AUDIT_WRITE	write to the kernel audit log

Only root in the container gets them in full. A process with a non-zero UID has an empty effective set, and the 14 capabilities remain only in the bounding set (CapBnd) — the ceiling above which the process can't rise even through a setuid file. Compare busybox running as root and our hello running as UID 65532, both without securityContext:

$ kubectl -n demo exec root-plain -- grep -E 'Cap(Eff|Bnd)' /proc/1/status
CapEff:	00000000a80425fb
CapBnd:	00000000a80425fb
user@k8s-node-1:~$ grep -E 'Cap(Eff|Bnd)' /proc/$PID/status    # the /hello process
CapEff:	0000000000000000
CapBnd:	00000000a80425fb

The mask a80425fb is exactly those 14 capabilities (capsh --decode=00000000a80425fb lists them by name). Almost none of them is needed by an ordinary application. NET_RAW stayed in the set historically for the sake of ping, and DAC_OVERRIDE lets root in a container read any file on mounted volumes regardless of its permissions.

The right way is to drop everything and add only what's necessary:

securityContext:
  capabilities:
    drop: ["ALL"]
    add: ["NET_BIND_SERVICE"]   # only if really needed

A nuance with add: capabilities added this way work for root in the container. For a process with a non-zero UID, add: [NET_BIND_SERVICE] ends up only in the bounding set (CapBnd: 0000000000000400, bit 10), while the effective set stays empty — unless the executable has a file capability (set with setcap when building the image). And on containerd 2.x ports below 1024 in the pod network are available without it anyway: by default containerd sets net.ipv4.ip_unprivileged_port_start=0 in the pod's network namespace (details in part 22). A process with UID 10001 and drop: [ALL] listens on port 80 just fine:

$ kubectl -n demo exec caps -- sh -c 'nc -l -s 0.0.0.0 -p 80 & sleep 1; netstat -ltn; kill $!'
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN

So an application in a pod usually doesn't need NET_BIND_SERVICE at all, and the simplest thing is to listen on 8080 and ask for nothing.

privileged: true isn't "even more capabilities" but the removal of almost all isolation: all capabilities, all node devices in /dev, no seccomp or AppArmor, writable /sys. A privileged container is as capable as root on the node. Only system components that manage the node need it: calico-node configures routes and netfilter rules, CSI drivers mount volumes (part 21). An application — never.

allowPrivilegeEscalation: false sets the kernel's no_new_privs flag on the process: neither setuid files nor file capabilities give any privileges beyond those it started with. Even if su or sudo remained in the image, they're useless. The field is incompatible with privileged: true, and the API won't accept such a pod:

The Pod "priv-ape" is invalid: spec.containers[0].securityContext: Invalid value: {…}: cannot set `allowPrivilegeEscalation` to false and `privileged` to true

The documentation warns that the flag doesn't take effect with CAP_SYS_ADMIN either. On containerd 2.2 it is still set in that case (NoNewPrivs: 1), but it forbids little to a process with CAP_SYS_ADMIN, so don't rely on it.

---

seccomp and AppArmor

seccomp is a system call filter in the kernel: the process is told in advance which of several hundred calls it may use. containerd's RuntimeDefault profile forbids a few dozen dangerous and rarely needed ones — mounting, loading modules and a new kernel, some calls for creating namespaces and debugging. This closes whole classes of kernel vulnerabilities reachable only through rare calls. Ordinary applications don't notice the difference.

securityContext:
  seccompProfile:
    type: RuntimeDefault        # or Localhost with localhostProfile: a path relative to /var/lib/kubelet/seccomp

Without an explicit profile the pod runs without a filter (Unconfined): /proc/<pid>/status of such a process shows Seccomp: 0. This can be changed for a whole node: in KubeletConfiguration, seccompDefault: true makes RuntimeDefault the default for all pods of the node. A good setting for a new cluster; on an existing one, check first that nothing breaks. It isn't enabled on our lab cluster.

AppArmor is a kernel security module enabled in Ubuntu by default: a profile restricts the process's access to files, network and capabilities by path. On nodes with AppArmor, containerd applies its own cri-containerd.apparmor.d profile to containers, which, for example, forbids writing to /proc/sys and mounting — and does so even without an explicit field: in any container on the lab cluster /proc/1/attr/current shows cri-containerd.apparmor.d (enforce). It's set with the appArmorProfile field (stable since 1.30; before that it was annotations):

securityContext:
  appArmorProfile:
    type: RuntimeDefault        # Localhost with localhostProfile: the name of a profile loaded on the node; Unconfined

SELinux is the counterpart of AppArmor with labels instead of paths, the main one in RHEL and derivatives; on Ubuntu it usually isn't used, and seLinuxOptions is left empty.

---

The filesystem: readOnlyRootFilesystem

The container layer is writable by default. Whoever gets code execution in the application will first try to save their files somewhere in /tmp or change the application's files. readOnlyRootFilesystem: true makes the container root filesystem read-only — /tmp included:

$ kubectl -n demo exec secure -- touch /tmp/x
touch: /tmp/x: Read-only file system
command terminated with exit code 1

Directories the application really needs to write to are mounted separately:

containers:
  - name: hello
    securityContext:
      readOnlyRootFilesystem: true
    volumeMounts:
      - name: tmp
        mountPath: /tmp
volumes:
  - name: tmp
    emptyDir:
      sizeLimit: 64Mi

It also shows where the application writes: if something breaks after enabling it, the Read-only file system error points to the path.

---

Fields that remove isolation: hostNetwork, hostPID, hostIPC, hostPath

Four fields in the pod spec, each removing one of the walls between the container and the node:

• hostNetwork: true — the pod is in the node's network namespace: it sees all of the node's interfaces and traffic, listens on the node's addresses, goes out with the node's address. Calico pod policies don't apply to it (part 20): to the network it's the node's own traffic. Needed by kube-proxy, calico-node, the MetalLB speaker (part 17).
• hostPID: true — the pod sees all processes of the node, including their command lines and environment variables, where passwords often lie; with root privileges it can interfere with them.
• hostIPC: true — shared memory and System V message queues shared with the node.
• hostPath — a node directory in the container. A node directory with root privileges in the container means access to whatever lies there: in /var/lib/kubelet — the kubelet's keys and the mounted volumes and tokens of all pods on the node, in /etc — the node's configuration. Even a "harmless" hostPath: /var/log gives more than it seems.

Together with privileged or root privileges, any of these fields turns a pod into administrative access to the node, and through the kubelet's keys and the neighbouring pods' tokens — to part of the cluster. So the main conclusion of part 29 is worth repeating: the create pods permission without restrictions on pod contents means root on the nodes. Restrictions on the contents are the subject of the rest of this part.

---

User namespaces: hostUsers: false

Everything above reduces what root in a container can do. User namespaces make root in a container not root on the node at all. With hostUsers: false the kubelet creates a separate user namespace for the pod and allocates it a range of 65536 node UIDs and GIDs that doesn't overlap with other pods: UID 0 inside the pod is, for example, UID 675741696 outside, which owns not a single file and has not a single privilege on the node. Capabilities inside apply only to the pod's own resources.

spec:
  hostUsers: false
  containers:
    - name: app
      image: ghcr.io/example/legacy-app:2.3   # an image that needs root inside

What it looks like inside and outside — a busybox pod running as root:

$ kubectl -n demo exec userns -- sh -c 'id; cat /proc/self/uid_map'
uid=0(root) gid=0(root) groups=0(root),10(wheel)
         0  675741696      65536
user@k8s-node-2:~$ ps -o uid,gid,cmd -p "$PID"
  UID   GID CMD
675741696 675741696 sleep infinity

uid_map reads like this: UID 0 inside corresponds to UID 675741696 outside, and so on for 65536 in a row.

In Kubernetes 1.36 user namespaces became stable (enabled by default since 1.33). Requirements: Linux kernel 6.3 or newer (met on Ubuntu Server 26.04 with kernel 7.0), containerd 2.0 and runc 1.2 or newer, and support for idmapped mounts (mapping file owners at mount time) by the filesystems of the pod directory and volumes — ext4, xfs, btrfs, tmpfs, overlayfs have it. The limitations are worth knowing in advance:

• with hostUsers: false you can't use hostNetwork, hostPID, hostIPC or volumeDevices block volumes — the API rejects such a pod immediately: spec.hostNetwork: Forbidden: when hostUsers is false;
• NFS volumes don't mount in such pods: the Linux kernel NFS client doesn't support idmapped mounts. The pod is created, the kubelet mounts NFS on the node, but the container doesn't start:

Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: failed to fulfil mount request: failed to set MOUNT_ATTR_IDMAP on /var/lib/kubelet/pods/84466711-6518-4a89-8a48-75a020530347/volumes/kubernetes.io~nfs/data: invalid argument (maybe the filesystem used doesn't support idmap mounts on this kernel?)

The same volume works in an ordinary pod. For applications with volumes from csi-driver-nfs (part 23) this mechanism isn't available today.

Pod Security (below) takes user namespaces into account: for a pod with hostUsers: false the runAsNonRoot and runAsUser checks don't apply even in restricted, because root inside isn't root outside. This is the main path for images that can't be rebuilt for an unprivileged user.

---

A secure pod as a whole

Putting everything together — this is a hello pod that passes the strictest level of checks:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello
  namespace: demo
spec:
  replicas: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: hello
  template:
    metadata:
      labels:
        app.kubernetes.io/name: hello
    spec:
      serviceAccountName: hello
      automountServiceAccountToken: false
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        runAsGroup: 10001
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: hello
          image: ghcr.io/example/hello:1.0
          ports:
            - name: http
              containerPort: 8080
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
          volumeMounts:
            - name: tmp
              mountPath: /tmp
      volumes:
        - name: tmp
          emptyDir:
            sizeLimit: 64Mi

The hello ServiceAccount without a token comes from part 29; to pull the private image, the ghcr-pull Secret is attached to it (imagePullSecrets in the ServiceAccount). The chart from part 27 already contains these fields. The patch from part 26, which added only runAsNonRoot and seccompProfile, doesn't: without allowPrivilegeEscalation and drop: [ALL] the pod won't be let into restricted, as we'll see below.

---

Pod Security Standards

Expecting every team to remember a dozen fields is pointless; a check at creation time is needed. The Kubernetes project defined three levels — the Pod Security Standards:

Level	For whom	What's forbidden
privileged	system components of nodes	nothing
baseline	any application without changes	known ways to escape isolation
restricted	applications written with security in mind	everything an ordinary application doesn't need

baseline forbids: privileged, hostNetwork, hostPID, hostIPC, hostPath volumes, hostPort; adding capabilities beyond the default set (NET_RAW is not allowed to be added explicitly, let alone SYS_ADMIN and the like); an explicit Unconfined in seccomp and AppArmor; non-standard SELinux types and procMount; sysctls outside a short list of safe ones; the host field in probes and lifecycle hooks (since 1.34). A pod breaking three rules at once gets all of them in one refusal:

pods "b" is forbidden: violates PodSecurity "baseline:latest": forbidden AppArmor profile (pod must not set AppArmor profile type to "Unconfined"), non-default capabilities (container "c" must not include "NET_RAW" in securityContext.capabilities.add), probe or lifecycle host (container "c" uses probe or lifecycle host "192.0.2.10")

restricted adds:

• runAsNonRoot: true is required (except for pods with hostUsers: false);
• allowPrivilegeEscalation: false is required;
• capabilities.drop must contain ALL, and only NET_BIND_SERVICE is allowed in add;
• seccompProfile.type must be RuntimeDefault or Localhost (not empty);
• volumes only from a safe list: configMap, secret, downwardAPI, projected, emptyDir, ephemeral, persistentVolumeClaim, csi — that is, no hostPath.

The standards are a description in the documentation, not an object in the cluster. An admission controller enforces them.

---

Pod Security Admission

Pod Security Admission (PSA) is a built-in admission controller (the third link from part 29) that checks every pod being created against a level. It replaced the PodSecurityPolicy mechanism removed in 1.25 and works more simply: the level is set by namespace labels, with no separate objects or bindings.

A label has three modes plus a level and a version:

pod-security.kubernetes.io/enforce: restricted      reject a violating pod
pod-security.kubernetes.io/enforce-version: v1.36
pod-security.kubernetes.io/audit: restricted        allow, but record in the audit log
pod-security.kubernetes.io/audit-version: v1.36
pod-security.kubernetes.io/warn: restricted         allow, but warn whoever creates it
pod-security.kubernetes.io/warn-version: v1.36

• enforce is a hard ban: a pod that fails the level isn't created. Only the pod is checked; if a Deployment template violates the level, the Deployment itself is created but its pods aren't, and this shows up in the ReplicaSet's events, not as a kubectl apply error. That's why warn is set together with enforce: the warning arrives immediately on kubectl apply of the Deployment.
• audit is a record in the apiserver audit log; the user notices nothing.
• warn is a warning in the API response to whoever creates the object (Deployments included).

The version (enforce-version) pins which Kubernetes version's rules to apply: the levels' rules tighten over time, and pinning keeps a cluster upgrade from suddenly rejecting pods that used to pass. Set it to the cluster's current version.

A typical approach is a soft rollout: first the whole cluster warns and audits on restricted without forbidding anything, violators are found from the logs, and enforce is turned on when none are left. At the cluster level, defaults are set in the PSA configuration (an AdmissionConfiguration file passed to the apiserver), together with the list of exempted namespaces (kube-system is always exempted: it holds system pods with hostNetwork and privileges).

# AdmissionConfiguration, passed to the apiserver via --admission-control-config-file
apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
  - name: PodSecurity
    configuration:
      apiVersion: pod-security.admission.config.k8s.io/v1
      kind: PodSecurityConfiguration
      defaults:
        enforce: baseline
        enforce-version: latest
        warn: restricted
        audit: restricted
      exemptions:
        namespaces: [kube-system]

Here the whole cluster must comply with baseline by default, and restricted is applied as warnings and audit. This is a reasonable starting point: baseline cuts off the crude stuff (privileged pods, hostPath, hostNetwork) without breaking most ready-made images. In kubeadm the file is wired in the same way as AuthenticationConfiguration in part 29: a flag in extraArgs and a directory in extraVolumes. With this configuration a pod with hostPath isn't created in an unlabelled namespace, an ordinary one is created with a warning, and in kube-system everything passes:

$ kubectl -n default apply --dry-run=server -f logreader.yaml
Error from server (Forbidden): error when creating "logreader.yaml": pods "logreader" is forbidden: violates PodSecurity "baseline:latest": hostPath volumes (volume "l")
$ kubectl -n kube-system apply --dry-run=server -f logreader.yaml
pod/logreader created (server dry run)

Namespace labels take precedence over the defaults, but only the labels that are set: in the calico-system namespace, where the Tigera operator set a single enforce=privileged label, the cluster's warn and audit on restricted keep working, and every pod there is created with a warning.

---

Moving demo to restricted

Let's move the demo namespace (part 11) to the strictest level. First let's see who would violate it, without forbidding anything — warn and audit:

$ kubectl label namespace demo \
    pod-security.kubernetes.io/warn=restricted \
    pod-security.kubernetes.io/warn-version=v1.36 \
    pod-security.kubernetes.io/audit=restricted \
    pod-security.kubernetes.io/audit-version=v1.36
namespace/demo labeled

We restart the old hello Deployment (without securityContext) and immediately see a warning:

$ kubectl -n demo rollout restart deploy/hello
Warning: would violate PodSecurity "restricted:v1.36": allowPrivilegeEscalation != false (container "hello" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "hello" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "hello" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "hello" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
deployment.apps/hello restarted

The Deployment rolls out (warn doesn't forbid), and the warning listed exactly the four missing fields. Note runAsNonRoot != true: the hello image already runs as UID 65532, but PSA looks only at the manifest, not at the image. Turning on enforcement — and PSA immediately reports pods already running in violation of the new level:

$ kubectl label namespace demo \
    pod-security.kubernetes.io/enforce=restricted \
    pod-security.kubernetes.io/enforce-version=v1.36
Warning: existing pods in namespace "demo" violate the new PodSecurity enforce level "restricted:v1.36"
Warning: hello-54c6c68779-qdhrp (and 1 other pod): allowPrivilegeEscalation != false, unrestricted capabilities, runAsNonRoot != true, seccompProfile
namespace/demo labeled

PSA doesn't touch running pods — it checks only creation. But the next rollout of the old Deployment gets stuck: kubectl gets the same warning, the Deployment changes, and the new ReplicaSet can't create a single pod. The reason is in its events:

$ kubectl -n demo rollout restart deploy/hello
Warning: would violate PodSecurity "restricted:v1.36": …
deployment.apps/hello restarted
$ kubectl -n demo get rs
NAME               DESIRED   CURRENT   READY   AGE
hello-54c6c68779   2         2         2       56s
hello-688cb9b6f7   0         0         0       19m
hello-d99998cf4    1         0         0       12s
$ kubectl -n demo describe rs hello-d99998cf4 | tail -1
  Warning  FailedCreate  1s (x3 over 9s)  replicaset-controller  (combined from similar events): Error creating: pods "hello-d99998cf4-mbchp" is forbidden: violates PodSecurity "restricted:v1.36": allowPrivilegeEscalation != false (container "hello" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "hello" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "hello" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "hello" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

The old pods keep running meanwhile — the rollout simply doesn't progress. A server-side dry run (the apiserver runs the object through all of admission and writes nothing) of a Deployment also shows only a warning: PSA checks the pod, not the template. To get a real refusal in advance, run the dry run on a pod manifest — an example is below in "Hands-on".

And our secure manifest from the section above passes without warnings:

$ kubectl -n demo apply -f deploy-hello-secure.yaml
deployment.apps/hello configured
$ kubectl -n demo get pods -l app.kubernetes.io/name=hello
NAME                     READY   STATUS    RESTARTS   AGE
hello-68f476f876-8dlgk   1/1     Running   0          36s
hello-68f476f876-vptvv   1/1     Running   0          42s

What breaks most often when moving to restricted:

• images that run as root — many official database and tool images: they need runAsUser with an unprivileged UID, and sometimes a rebuilt image (the standard nginx image listens on 80 and writes to /var/cache/nginx as root; for Kubernetes there's the nginx-unprivileged variant on port 8080);
• applications that write to the root filesystem — restricted itself doesn't require readOnlyRootFilesystem, but if you enable it, you need emptyDir for /tmp and working directories;
• things that really need root or a capability — system agents, old applications: for them — a separate namespace with the needed level, not weakening the common one.

---

Exceptions

Not everything can be reworked for restricted. The right way to make an exception is a separate namespace with softer labels, not weakening the common rule:

$ kubectl create namespace infra-agents
namespace/infra-agents created
$ kubectl label namespace infra-agents \
    pod-security.kubernetes.io/enforce=privileged \
    pod-security.kubernetes.io/warn=privileged \
    pod-security.kubernetes.io/audit=privileged
namespace/infra-agents labeled

All three labels — so that the cluster's warn and audit defaults don't produce a warning for every agent pod. This way a privileged agent (for example, a log collector with hostPath: /var/log, part 33) lives in its own namespace under privileged, while applications stay in demo under restricted. Access to the infra-agents namespace is restricted with RBAC: the create pods permission there effectively means root on the nodes (part 29), and it's given to a handful of people and system ServiceAccounts.

Fine-grained cluster-level exemptions (exemptions in AdmissionConfiguration) — by namespace, by user or by RuntimeClass — exist, but are used sparingly: an exempted subject isn't checked at all, and that's easy to forget.

---

Custom rules: ValidatingAdmissionPolicy

Pod Security covers isolation security, but teams have their own requirements: forbid images with the latest tag (part 5), require a team label, allow images only from your own registry, require requests and limits (part 10). For this there's ValidatingAdmissionPolicy — a built-in admission controller that checks objects with CEL rules right in the apiserver, without an external service. Stable since 1.30.

A policy is two objects. ValidatingAdmissionPolicy describes what to check and with which expression, ValidatingAdmissionPolicyBinding — which namespaces to apply it to and what to do on a violation (Deny, Warn, Audit):

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: disallow-latest-tag
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: ["apps"]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["deployments", "statefulsets", "daemonsets"]
  validations:
    - expression: >-
        object.spec.template.spec.containers.all(c,
          c.image.contains('@') ||
          (c.image.matches(':[^/:]+$') && !c.image.endsWith(':latest')))
      message: "the image must not use the latest tag; specify a version or a digest"
      reason: Invalid
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: disallow-latest-tag
spec:
  policyName: disallow-latest-tag
  validationActions: [Deny]
  matchResources:
    namespaceSelector:
      matchLabels:
        kubernetes.io/metadata.name: demo

The expression lets through an image with a digest (@sha256:…), and without a digest requires a tag — a colon followed by neither / nor : up to the end of the string — and requires that tag not to be latest. This rejects both ghcr.io/example/hello and registry.example.com:5000/hello without a tag (the colon before the registry port doesn't count as a tag), while registry.example.com:5000/hello:1.0 passes. Only containers are checked; initContainers need a second expression of the same kind.

$ kubectl -n demo set image deploy/hello hello=ghcr.io/example/hello:latest
error: failed to patch image update to pod template: deployments.apps "hello" is forbidden: ValidatingAdmissionPolicy 'disallow-latest-tag' with binding 'disallow-latest-tag' denied request: the image must not use the latest tag; specify a version or a digest

CEL expressions have access to object (the new object), oldObject (the previous one, on update), request (who is doing what), namespaceObject and params (parameters from a separate resource, if the policy is parameterised). Since this is the same CEL as in CRD validation (part 28) and in authentication checks (part 29), the set of expressions in the cluster is uniform. The apiserver catches an error in an expression when the policy is created: the policy isn't created, and the message points to the line and position.

For rules that need not only to check but to fill in (set a missing label, add a securityContext), the companion MutatingAdmissionPolicy became stable in 1.36 — the same in CEL, but changing the object, a replacement for mutating webhooks in simple cases.

When there are many rules and complex scenarios are needed (generating objects, checks against external data, violation reports across the cluster), a separate policy engine is installed: Kyverno — an operator with rules as YAML resources, no programming, or OPA Gatekeeper — in the Rego language on top of Open Policy Agent. Both work as admission webhooks; when assessing them, everything from part 28 about webhooks applies, including the risk of locking up the cluster with failurePolicy: Fail and the engine down. For simple rules on one cluster, the built-in ValidatingAdmissionPolicy is simpler and more reliable: there's nothing to go down.

---

The link to part 22

In part 22, while examining the NFS server, we reached an unpleasant conclusion: any process in a pod — even non-root and without capabilities — can open a connection to the NFS server from a port below 1024 and build an AUTH_SYS credential itself (the UID and GID the client writes into every request and which a sec=sys server simply trusts). restricted doesn't fix this: that part demonstrated it with a pod running exactly by its requirements. What it does give: the pod can't mount NFS with the kernel (that needs CAP_SYS_ADMIN), can't become root on the node and can't get hostNetwork, which means it can't bypass, through the node's network, the Calico policy that keeps pods away from the server's address. The main layer against pods is that network policy, and restricted keeps it from being bypassed. No single layer is enough on its own, but together they close the path. The same applies to the SMB server from part 24.

---

Hands-on

The test pods in this part are busybox with various securityContext settings, all created in demo before the Pod Security labels were set: root-plain with no fields, caps with UID 10001 and drop: [ALL], add: [NET_BIND_SERVICE], secure with all the fields of the secure pod. The namespace labels after the move:

$ kubectl get namespace demo --show-labels
NAME   STATUS   AGE     LABELS
demo   Active   19m     kubernetes.io/metadata.name=demo,pod-security.kubernetes.io/audit-version=v1.36,pod-security.kubernetes.io/audit=restricted,pod-security.kubernetes.io/enforce-version=v1.36,pod-security.kubernetes.io/enforce=restricted,pod-security.kubernetes.io/warn-version=v1.36,pod-security.kubernetes.io/warn=restricted

Let's check that a privileged pod really won't pass in demo — with a server-side dry run of a pod manifest, without creating it:

$ cat <<'EOF' | kubectl -n demo apply --dry-run=server -f -
apiVersion: v1
kind: Pod
metadata:
  name: probe
spec:
  containers:
    - name: c
      image: busybox:1.37
      securityContext:
        privileged: true
EOF
Error from server (Forbidden): error when creating "STDIN": pods "probe" is forbidden: violates PodSecurity "restricted:v1.36": privileged (container "c" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "c" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "c" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "c" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "c" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

While a pod with hostUsers: false passes restricted without runAsNonRoot — this is how the user namespace exception works:

$ cat <<'EOF' | kubectl -n demo apply --dry-run=server -f -
apiVersion: v1
kind: Pod
metadata:
  name: legacy
spec:
  hostUsers: false
  containers:
    - name: c
      image: busybox:1.37
      command: [sleep, infinity]
      securityContext:
        allowPrivilegeEscalation: false
        capabilities: {drop: [ALL]}
        seccompProfile: {type: RuntimeDefault}
EOF
pod/legacy created (server dry run)

Without the hostUsers: false line the same pod gets violates PodSecurity "restricted:v1.36": runAsNonRoot != true.

What the pod looks like on the node when the fields are applied. The process UID is the one set, not root:

user@k8s-node-1:~$ CID=$(sudo crictl ps --name hello -q | head -1)
user@k8s-node-1:~$ PID=$(sudo crictl inspect --output go-template --template '{{.info.pid}}' "$CID")
user@k8s-node-1:~$ ps -o uid,gid,cmd -p "$PID"
  UID   GID CMD
10001 10001 /hello

The remaining capabilities and the no_new_privs flag are in the process's /proc:

user@k8s-node-1:~$ grep -E 'Cap(Eff|Bnd)|NoNewPrivs' /proc/$PID/status
CapEff:	0000000000000000
CapBnd:	0000000000000000
NoNewPrivs:	1

CapEff and CapBnd are zero — not a single capability is left, even in the ceiling (drop: [ALL]), NoNewPrivs: 1 — privilege escalation is forbidden (allowPrivilegeEscalation: false). The same hello without securityContext had CapEff: 0 (not root), but CapBnd: 00000000a80425fb and NoNewPrivs: 0: a setuid file in the image could have raised the process to all 14 capabilities.

The seccomp profile is visible too:

user@k8s-node-1:~$ grep Seccomp /proc/$PID/status
Seccomp:	2
Seccomp_filters:	1

Seccomp: 2 is filter mode (not 0, disabled), and the one loaded filter is RuntimeDefault, applied by containerd.

And the exception namespace for a system agent among the rest:

$ kubectl get ns -L pod-security.kubernetes.io/enforce
NAME              STATUS   AGE   ENFORCE
calico-system     Active   8d    privileged
default           Active   8d
demo              Active   13m   restricted
infra-agents      Active   51s   privileged
kube-node-lease   Active   8d
kube-public       Active   8d
kube-system       Active   8d
tigera-operator   Active   8d    privileged

demo is under strict enforcement, infra-agents under privileged for agents, calico-system and tigera-operator were labelled by the Tigera operator, the rest have no label. On a cluster without AdmissionConfiguration an unlabelled namespace means privileged, that is, no checks at all: default should either not be used or be labelled too, and kube-system should be exempted in the PSA configuration, as in the example above.

Cleanup: kubectl delete namespace infra-agents, kubectl delete validatingadmissionpolicybinding,validatingadmissionpolicy disallow-latest-tag, the test pods root-plain, caps, secure — kubectl -n demo delete pod. We keep the labels on demo: from here on the series works in it under restricted.

---

This concludes the security block: part 29 defined who can do what with the API, this one — what ends up inside a pod and on the node. Next, we need to be able to fix the cluster when it breaks after all.

---

Sources

• Configure a Security Context for a Pod or Container — securityContext fields, capabilities, seccomp, AppArmor, allowPrivilegeEscalation.
• Linux kernel security constraints for Pods and containers — capabilities, seccomp, AppArmor, SELinux, no_new_privs.
• Restrict a Container's Syscalls with seccomp — profiles, seccompDefault.
• Restrict a Container's Access to Resources with AppArmor — the appArmorProfile field.
• User Namespaces — hostUsers: false, requirements and limitations.
• Pod Security Standards — the privileged, baseline and restricted levels.
• Pod Security Admission and Enforce Pod Security Standards with Namespace Labels — modes, versions, labels.
• Enforce Pod Security Standards by Configuring the Built-in Admission Controller — AdmissionConfiguration and exemptions.
• Validating Admission Policy and Mutating Admission Policy — CEL policies.
• CEL in Kubernetes — string functions and matches.
• containerd: CRI Plugin Config Guide — enable_unprivileged_ports, disable_apparmor.
• Kyverno and OPA Gatekeeper — external policy engines.

---

Next — troubleshooting: how to read pod events and states, deal with Pending, CrashLoopBackOff and ImagePullBackOff, use kubectl logs, exec and kubectl debug, and what to look at on the node itself with crictl when the API no longer helps.