Kubernetes for beginners. Part 21: How do volumes and CSI work?

The networking block ended in the previous part, and storage begins. So far all our pods have been stateless: hello writes nothing to disk, and if the pod is recreated, nothing is lost. A database cannot work like that. In part 6 we saw that a container's file system is the image layers plus a writable overlayfs layer that lives exactly as long as the container; in part 12 we mounted ConfigMaps and Secrets as files; in part 8 a StatefulSet asked for volumes through volumeClaimTemplates. Today we look at how all of this works: what kinds of volumes there are, how a PersistentVolumeClaim finds its PersistentVolume, who creates volumes on demand, and what a CSI driver does to make a mounted directory appear on the node. Network storage — NFS and SMB — comes in the next parts; here the mechanics are shown on the nodes' local disks.

The experiments run in the demo namespace from part 11 — with its LimitRange, which gives every container without resources a 128 MiB memory limit. That will matter when we talk about tmpfs.

---

Pod volumes

A volume in Kubernetes is a directory that kubelet prepares before the containers start and mounts into them at the paths from volumeMounts. A volume is described in the pod's spec.volumes and lives either as long as the pod (ephemeral volumes) or independently of it (persistent ones). Unlike the container's writable layer, a volume survives a container restart inside the pod and is visible to every container of the pod that mounts it.

kubelet keeps all of a pod's volumes in the directory /var/lib/kubelet/pods/<pod uid>/volumes/<plugin>/<volume name>, and they get into the container by a bind mount of that directory at the required path. The plugin in the directory name is written with a ~: kubernetes.io~empty-dir, kubernetes.io~projected, kubernetes.io~csi.

emptyDir

An empty directory that is created when the pod starts on a node and deleted together with the pod. It suits temporary files, caches and file exchange between the pod's containers (for example, an init container downloads data and the main one reads it). A container crash does not wipe the data: the pod stays on the node, and the restarted container sees the same files.

volumes:
  - name: cache
    emptyDir:
      sizeLimit: 1Gi
  - name: scratch
    emptyDir:
      medium: Memory
      sizeLimit: 256Mi

A regular emptyDir lives on the node's disk, in kubelet's directory, and takes up the node's ephemeral storage — the same space where container logs and writable layers go. sizeLimit is not a file system quota: inside the pod the whole node disk is visible and you can write as much as you like, while kubelet periodically measures the directory size and evicts the pod when it exceeds the limit. That is why the limit is mandatory: a pod without one can fill the node's disk and get its neighbours evicted.

medium: Memory mounts a tmpfs — a file system in RAM. It is fast, but everything written to it counts towards the memory of the container that wrote it: in the cgroup these bytes show up as shmem, and they count against the limit like the process's ordinary memory. kubelet picks the size of the tmpfs itself as the smallest of three: sizeLimit, the pod's memory limit and the node's allocatable memory. Nothing can be written beyond the tmpfs size — No space left on device; and the files in it are freed neither by the OOM killer nor by eviction while the pod exists. It is a good place for secrets that must not reach the disk and for small hot files, but not for large volumes of data.

configMap, secret, downwardAPI and projected

These volumes present data from the API as files: ConfigMap and Secret keys, pod metadata (downwardAPI — name, namespace, labels, annotations, resource requests and limits). projected gathers several such sources into one directory and adds its own: serviceAccountToken — a short-lived ServiceAccount token with a given audience and lifetime (at least 10 minutes, an hour by default) that kubelet renews by itself, and clusterTrustBundle — a set of trust certificates from cluster-wide ClusterTrustBundle objects. The latter is a beta feature in 1.36, off by default (the ClusterTrustBundleProjection gate), and our cluster does not have it.

volumes:
  - name: app-config
    projected:
      sources:
        - configMap:
            name: hello-config
        - secret:
            name: hello-db
            items:
              - key: password
                path: db-password
        - downwardAPI:
            items:
              - path: pod-name
                fieldRef:
                  fieldPath: metadata.name
        - serviceAccountToken:
            audience: vault
            expirationSeconds: 3600
            path: vault-token

The update mechanics we discussed in part 12: kubelet writes the files into a hidden timestamped directory, and the ..data symbolic link is switched atomically to the new one. The application never sees a half-updated set of files. The exception is a mount with subPath: it is tied to a specific file and gets no updates. kubelet puts secret and projected volumes on tmpfs, so secrets never reach the node's disk. Every pod gets such a volume even without asking: kube-api-access-<suffix> with the ServiceAccount token, the cluster CA certificate and the namespace name.

image

Since Kubernetes 1.36 one more source is stable — image: kubelet pulls an OCI image (or another OCI artifact) the same way as a container image and mounts its contents into the pod read-only. This is how models, dictionaries and other large data are delivered when it is convenient to version them with a tag in a registry without baking them into the application image. The pull policy is set with pullPolicy, as for a container.

---

hostPath and why it is dangerous

hostPath mounts a file or directory of the node's file system into the pod:

volumes:
  - name: host-logs
    hostPath:
      path: /var/log
      type: Directory

The type field checks what is at the path: Directory, File, Socket, CharDevice, BlockDevice — must exist; DirectoryOrCreate, FileOrCreate — create it if missing; an empty value — check nothing.

The Kubernetes documentation says it plainly: if you can avoid hostPath, avoid it. The reasons:

• It is a hole in isolation. A pod with write access to /, /etc, /var/lib/kubelet or the containerd socket effectively gets root on the node: it can replace configuration, read other pods' volumes and secrets, start a container bypassing Kubernetes. Even a "only these directories are allowed" restriction holds only if the mount is read-only: with write access, symbolic links let you escape from an allowed directory.
• The data is tied to the node. The pod moved to another node — the data stayed on the old one, and on the new one it sees an empty (or someone else's) directory.
• No accounting. Writes to hostPath do not count as the pod's ephemeral storage: Kubernetes does not know the size of this data, does not evict for it and does not clean it up.

Where hostPath is legitimate is in system DaemonSets that need the node by their nature: log collectors read /var/log, monitoring agents /proc and /sys, CSI node plugins (more on them below) kubelet's directories. Applications do not use it. Pod Security Admission, covered in part 30, forbids hostPath entirely in the baseline profile. If an application really needs a node's local disk, the right way is a local PersistentVolume bound to the node or a driver that provisions such volumes itself.

---

PersistentVolume and PersistentVolumeClaim

Persistent storage in Kubernetes is split into two objects, and the split is the same as with Gateway and HTTPRoute: one describes the infrastructure, the other the application's need.

A PersistentVolume (PV) is a cluster-wide object: "here is a specific volume of this size, with these access modes, and this is how to connect to it". An administrator creates it by hand, or a provisioner creates it automatically.

A PersistentVolumeClaim (PVC) is a namespaced object: "the application needs a volume of at least 10 GiB, writable from one node, of this class". A pod refers to the PVC, not to the PV, and knows nothing about whether it is NFS, a local disk or a cloud volume.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: manual-pv-1
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: manual
  hostPath:
    path: /srv/volumes/manual-pv-1
    type: DirectoryOrCreate
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
                - k8s-node-1
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data
  namespace: demo
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi

The PV here is a directory on k8s-node-1, and nodeAffinity tells the scheduler that a pod with this volume can only run there. This is a learning example of the mechanics; for real data a hostPath PV is unsuitable for the reasons from the previous section. There is no StorageClass object named manual in the cluster, and none is needed: with static binding, storageClassName is just a label that has to match on the PV and the PVC.

Binding

Binding is done by the PersistentVolume controller in kube-controller-manager. It watches PVCs without a volume and looks for a suitable PV:

• the same storageClassName (the empty string "" means "only PVs without a class"; a missing field means "the default class", more on that below);
• the PV has all the requested access modes;
• the PV is not smaller than requested — the smallest of the suitable ones is chosen (on the lab, out of PVs of 3 and 5 GiB a 2 GiB PVC got the 3 GiB one);
• the PV's labels match the selector if the PVC sets one, or volumeName if the PVC asks for a specific PV.

Having found a pair, the controller writes the reference both ways: into the PV — spec.claimRef (namespace, name and UID of the claim), into the PVC — spec.volumeName. The binding is one-to-one and exclusive: a 5 GiB PV bound to a 2 GiB PVC belongs entirely to that PVC, and the remaining 3 GiB go to nobody. The PVC reports the real size in status.capacity — 5 GiB. The controller goes through PVCs both on events and periodically, every 15 seconds, so binding sometimes does not happen instantly.

Phases:

Object	Phase	Meaning
PV	Available	free
PV	Bound	bound to a PVC
PV	Released	the PVC is deleted, but the data is still there; not bound again
PV	Failed	automatic reclamation failed
PVC	Pending	no suitable PV (or waiting for the first consumer)
PVC	Bound	bound
PVC	Lost	the bound PV has disappeared

Finalizers protect against accidental deletion: kubernetes.io/pvc-protection does not let a PVC be deleted while at least one pod uses it (the object hangs in Terminating), kubernetes.io/pv-protection does the same for a bound PV.

---

StorageClass and dynamic provisioning

Creating a PV by hand for every request is inconvenient. A StorageClass describes a "kind" of storage and who can create volumes of that kind:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-hostpath
provisioner: hostpath.csi.k8s.io
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

• provisioner — the name of the driver that creates the volumes; for CSI it is the driver name from the CSIDriver object;
• parameters — parameters passed to the driver as is (server address, disk type, path on the server) that Kubernetes does not interpret; the demo driver below does not need them;
• reclaimPolicy — the reclaim policy the created PVs will get;
• volumeBindingMode — when to create the volume;
• allowVolumeExpansion — whether volumes can be grown;
• mountOptions — mount options for all PVs of the class (needed for NFS and SMB);
• allowedTopologies — where volumes may be created.

Dynamic provisioning: the PVC refers to a class and there is no PV — the PV controller sets the volume.kubernetes.io/storage-provisioner annotation with the provisioner's name on the PVC, the provisioner sees "its" PVC, creates a volume in the storage and a PV object already bound to that PVC. The name of such a PV is pvc-<claim UID>.

The default class is marked with the annotation storageclass.kubernetes.io/is-default-class: "true". A PVC without a storageClassName field gets it automatically (the DefaultStorageClass admission plugin does that), and retroactively too: a PVC created while there was no default class hangs in Pending, and when such a class appears, the PV controller writes its name into the PVC. Convenient, but treacherous: a typo in the class name or a forgotten field — and the data silently goes to the default storage. There should be exactly one default class, and it is worth choosing it deliberately. Our cluster has none at all after installation.

WaitForFirstConsumer versus Immediate

Immediate creates the volume as soon as the PVC appears. The provisioner does not know where the pod will run and creates the volume wherever it decides. For network storage reachable from all nodes this does not matter. For topology-constrained storage — a node's local disk, a disk in one cloud zone — it does: the volume is created on k8s-node-2, but because of a lack of memory or taints the pod can only run on k8s-node-1. The pod stays Pending forever.

WaitForFirstConsumer postpones provisioning until a pod that uses this PVC appears. The scheduler picks a node taking everything into account — resources, affinity, taints and storage topology (and, for drivers that publish CSIStorageCapacity objects, free space as well) — and writes its choice into the PVC annotation volume.kubernetes.io/selected-node. The provisioner creates the volume right there, and the PV gets a nodeAffinity. While there is no pod, the PVC hangs in Pending with a WaitForFirstConsumer event — that is normal.

The rule: for any storage with a topology — WaitForFirstConsumer. For network storage Immediate is acceptable, but WaitForFirstConsumer breaks nothing there either.

---

Access modes and reclaim policies

Mode	Abbreviation	Meaning
ReadWriteOnce	RWO	read and write from one node (several pods on that node are fine)
ReadOnlyMany	ROX	read-only from many nodes
ReadWriteMany	RWX	read and write from many nodes
ReadWriteOncePod	RWOP	read and write by one pod in the whole cluster

The modes are promises of the driver and a binding criterion, not a locking mechanism inside the file system. RWO restricts the node, not the pod: two pods on one node will happily mount the same RWO volume and may corrupt each other's files. If you need a guarantee of "exactly one writer", there is RWOP (stable since 1.29, supported by CSI drivers only): a second pod with the same PVC will not be scheduled. RWX is supported by network file systems such as NFS and SMB; block volumes are usually RWO only.

The reclaim policy (persistentVolumeReclaimPolicy) decides what happens to the data when the PVC is deleted:

• Delete — the provisioner deletes the volume in the storage together with the PV. The default for dynamically provisioned volumes. Deleted the PVC by mistake — the data is gone.
• Retain — the PV goes to Released, the data stays. It can only be brought back by hand: remove claimRef from the PV and bind it to a new PVC through volumeName. Such a PV can be deleted, but the volume itself stays in the storage — the administrator cleans it up.
• Recycle — deprecated: it used to run rm -rf on the volume and return it to Available. Not used; dynamic provisioning replaces it.

For important data a class with reclaimPolicy: Retain is the better choice: an accidental kubectl delete pvc or namespace deletion will not destroy the database. The policy can also be changed on an existing PV — kubectl patch pv <name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'.

---

Expansion, snapshots and generic ephemeral volumes

Expansion. If the class has allowVolumeExpansion: true, a volume is grown by editing spec.resources.requests.storage in the PVC. Then there are two steps: the controller part of the driver grows the volume itself in the storage, then kubelet on the node asks the driver to grow the file system — for most drivers on the mounted volume, without stopping the pod. Until the second step is done, the PV already has the new size, while the PVC carries the FileSystemResizePending condition and the old status.capacity. Volumes cannot be shrunk. If the storage cannot provide the requested size, the request can be repeated with a smaller value, but still larger than the current one.

Snapshots are point-in-time copies of a volume made by the storage itself. They are three CRDs that are not built into Kubernetes and are installed separately, from the external-snapshotter project:

• VolumeSnapshotClass — the StorageClass counterpart for snapshots: the driver and deletionPolicy (Delete or Retain);
• VolumeSnapshot — a snapshot request in a namespace: "snapshot the data PVC";
• VolumeSnapshotContent — a cluster-wide object, the snapshot itself in the storage, the PV counterpart.

They are served by the snapshot-controller — a Deployment in kube-system that binds VolumeSnapshot to VolumeSnapshotContent, while the actual command to the storage is issued by the driver's sidecar (more on it below). A new PVC is created from a snapshot by naming it in dataSource; a clone is made the same way, with another PVC in dataSource. Since Kubernetes 1.36 VolumeGroupSnapshot is stable as well — a consistent snapshot of several volumes at once (a database's data and log at the same moment), if the storage supports it; these are three more CRDs of the same project.

A snapshot taken on the same storage is not a backup: if the storage is lost, both the volume and the snapshots go with it. It is good for a quick rollback before an upgrade and as a source for copying elsewhere.

Generic ephemeral volumes are a PVC that is created together with the pod and deleted together with it:

volumes:
  - name: work
    ephemeral:
      volumeClaimTemplate:
        spec:
          accessModes: ["ReadWriteOnce"]
          storageClassName: csi-hostpath
          resources:
            requests:
              storage: 1Gi

The controller creates a PVC named <pod>-<volume> with the pod as its owner; when the pod is deleted, the garbage collector deletes the PVC too. It is like emptyDir, but on any storage with any features — size, class, snapshots — and without consuming the node's ephemeral storage. It suits large temporary data.

---

How a CSI driver works

CSI (Container Storage Interface) is a specification of the interface between an orchestrator and a storage system. Storage drivers used to live right in the Kubernetes code (in-tree), and adding a new storage system or fixing a bug in an old one was only possible with a Kubernetes release. CSI moved them out: a driver is a separate program that implements gRPC services (gRPC is a remote procedure call framework: the client calls a server method by name, and the request and response travel as Protocol Buffers messages over HTTP/2) and listens on a unix socket. The old in-tree plugins for cloud disks have been migrated to CSI drivers.

The core of the specification is three services:

Service	Calls	Who calls
Identity	GetPluginInfo, GetPluginCapabilities, Probe	everyone: who are you, what can you do, are you alive
Controller	CreateVolume, DeleteVolume, ControllerPublishVolume, ControllerUnpublishVolume, ControllerExpandVolume, ControllerModifyVolume, CreateSnapshot, DeleteSnapshot	the controller plugin's sidecar containers
Node	NodeStageVolume, NodeUnstageVolume, NodePublishVolume, NodeUnpublishVolume, NodeExpandVolume, NodeGetInfo, NodeGetCapabilities	kubelet

Controller calls work with the storage "from afar" and can run anywhere; Node calls mount the volume on a specific node and therefore run on it. Besides these, the specification has a GroupController service for group snapshots and SnapshotMetadata for getting the list of blocks changed between snapshots (backup systems use it to make incremental copies). Which calls a driver supports it reports through capabilities: GetPluginCapabilities, ControllerGetCapabilities, NodeGetCapabilities — and the sidecars and kubelet adapt to the answer.

Hence the typical layout of a driver:

 ┌──────────────── controller plugin (Deployment, 1–2 replicas) ────────────────┐
 │  external-provisioner   PVC → CreateVolume / DeleteVolume → PV object        │
 │  external-attacher      VolumeAttachment → ControllerPublishVolume           │
 │  external-resizer       larger PVC → ControllerExpandVolume                  │
 │  external-snapshotter   VolumeSnapshotContent → CreateSnapshot               │
 │  livenessprobe          Probe → /healthz                                     │
 │            │ unix socket (emptyDir shared by the pod's containers)           │
 │            ▼                                                                 │
 │  driver: Identity + Controller                                               │
 └──────────────────────────────────────────────────────────────────────────────┘

 ┌──────────────── node plugin (DaemonSet, on every node) ──────────────────────┐
 │  node-driver-registrar  registers the driver with kubelet                    │
 │  livenessprobe          Probe → /healthz                                     │
 │            │ /var/lib/kubelet/plugins/<driver>/csi.sock (hostPath)           │
 │            ▼                                                                 │
 │  driver: Identity + Node  ◄── kubelet: NodeStage / NodePublish               │
 │  privileged, mounts into /var/lib/kubelet with Bidirectional propagation     │
 └──────────────────────────────────────────────────────────────────────────────┘

Sidecar containers are programs of the kubernetes-csi project shared by all drivers. The driver does not need to know anything about the Kubernetes API: a sidecar watches objects and turns their changes into gRPC calls.

• external-provisioner watches PVCs of its class, calls CreateVolume and creates the PV; when a PV with the Delete policy is deleted — DeleteVolume.
• external-attacher watches VolumeAttachment objects and calls ControllerPublishVolume — "attach the volume to the node" (for a cloud disk — attach it to the virtual machine). If the driver lacks the PUBLISH_UNPUBLISH_VOLUME capability, the attacher calls nothing and simply marks the VolumeAttachment as attached.
• external-resizer watches for size increases in PVCs and calls ControllerExpandVolume.
• external-snapshotter (the csi-snapshotter sidecar) watches its driver's VolumeSnapshotContents and calls CreateSnapshot.
• node-driver-registrar asks the driver for its name at startup and puts a registration socket into /var/lib/kubelet/plugins_registry/; kubelet watches that directory, finds the new driver and from then on knows which socket to call it on.
• livenessprobe turns the gRPC Probe call into an HTTP /healthz endpoint for the pod's liveness probes.

The node plugin runs privileged: it has to run mount in the node's mount namespace. kubelet's directories — /var/lib/kubelet/pods and /var/lib/kubelet/plugins — are mounted into it with mountPropagation: Bidirectional: that way mounts made inside the container become visible to kubelet and to other pods on the node (mount propagation is the kernel's shared subtree mechanism for mount namespaces).

Our cluster already has one CSI driver, even though we did not install it: csi.tigera.io from Calico. Its node plugin is the csi-node-driver DaemonSet in calico-system with the calico-csi and csi-node-driver-registrar containers, and there is no controller part at all: the driver serves only inline ephemeral volumes, through which a pod can get a socket for talking to the Calico agent on the node. No pod of ours uses it, but the driver's registration shows up in the objects below.

CSI objects in the API

CSIDriver is a cluster-wide object through which a driver tells Kubernetes about its peculiarities. For the demo driver we will install below it looks like this (without labels and comments):

apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
  name: hostpath.csi.k8s.io
spec:
  volumeLifecycleModes:
  - Persistent
  - Ephemeral
  podInfoOnMount: true
  fsGroupPolicy: File

• attachRequired — whether the attach step is needed (VolumeAttachment and ControllerPublishVolume); not set means true. For NFS and SMB there is nothing to attach to a node, such drivers have false, and the step is skipped.
• podInfoOnMount — whether to pass the pod's name, namespace and UID to the driver in NodePublishVolume.
• volumeLifecycleModes — persistent volumes (Persistent) and/or CSI ephemeral volumes inline in the pod (Ephemeral).
• fsGroupPolicy — how kubelet treats the pod's fsGroup on this volume. Three values, and the difference between them will matter in the NFS and SMB parts:

fsGroupPolicy	What kubelet does
ReadWriteOnceWithFSType (default)	changes the owning group and permissions only if the volume has an fsType set and its access modes include ReadWriteOnce. Leaves network volumes without fsType and RWX volumes alone
File	always changes ownership and permissions, regardless of the file system type and access mode
None	never changes them; the volume is mounted as is, permissions are the storage's business

The logic of the default is this: a block volume with a file system mounted on one node belongs to the pod entirely, and changing permissions is safe; a network volume may be used by other clients at the same time, and a recursive ownership change on it interferes with someone else's data. There is a fourth option, not through this field: a driver with the VOLUME_MOUNT_GROUP capability receives fsGroup in NodeStageVolume/NodePublishVolume requests and applies it itself — for example, with the gid= mount option. Then kubelet does not change permissions at all, and fsGroupPolicy and fsGroupChangePolicy are ignored.

CSINode is a per-node object created by kubelet: which drivers are registered on the node, under which ID the node is known to the driver (nodeID), what topology keys it has, and how many of the driver's volumes can be attached to the node (allocatable.count). kubelet also puts the topology keys onto the Node object as labels — that is how the provisioner and the scheduler know where a volume is available.

VolumeAttachment is "volume X is attached to node Y". It is created by the AttachDetach controller in kube-controller-manager when a pod with the volume is scheduled to a node, and external-attacher carries it out and sets status.attached: true. It only appears for drivers with attachRequired: true.

The path of a volume from PVC to a directory in the container

 PVC created ──► external-provisioner: CreateVolume ──► PV (pvc-<uid>), Bound
                  (with WaitForFirstConsumer — after the scheduler picks a node)
      │
 pod scheduled to k8s-node-2
      │
      ├──► AttachDetach controller: VolumeAttachment ──► external-attacher:
      │       ControllerPublishVolume ──► status.attached: true
      │
      ▼ kubelet on k8s-node-2
 NodeStageVolume    once per node and volume:
      │             /var/lib/kubelet/plugins/kubernetes.io/csi/<driver>/<sha256>/globalmount
      ▼
 NodePublishVolume  for every pod:
      │             /var/lib/kubelet/pods/<pod uid>/volumes/kubernetes.io~csi/<pv>/mount
      ▼
 containerd: bind mount of .../mount into the container at the volumeMounts path

NodeStageVolume prepares the volume on the node once, no matter how many pods of the node use it: for a block disk it finds the device, creates a file system if needed and mounts it at globalmount; for SMB it connects the share. The <sha256> in the path is the SHA-256 of the volume's ID in the storage (volumeHandle). The step is optional: a driver declares it with the STAGE_UNSTAGE_VOLUME capability, and without it kubelet goes straight to publishing.

NodePublishVolume makes the volume visible to a specific pod — usually with a bind mount from globalmount into the pod's directory. Next to the mount directory kubelet puts vol_data.json with what it needs to unmount after a restart.

Teardown goes in reverse order: NodeUnpublishVolume when the pod is deleted, NodeUnstageVolume when no pods with this volume are left on the node, ControllerUnpublishVolume to detach from the node, DeleteVolume when a PV with the Delete policy is deleted.

---

fsGroup and fsGroupChangePolicy

A process in a container runs as some UID and GID, and it needs permissions on the volume. The pod's securityContext.fsGroup sets a supplementary group that all processes of the pod get; kubelet, if the driver's fsGroupPolicy allows it, recursively makes this group the owner of all files on the volume, gives the group read and write permissions and sets the setgid bit on directories so that new files inherit the group.

spec:
  securityContext:
    runAsUser: 1000
    runAsGroup: 1000
    fsGroup: 2000
    fsGroupChangePolicy: OnRootMismatch

The recursive walk is a problem on large volumes: a million files on network storage take minutes to walk, and the pod does not start all that time. fsGroupChangePolicy controls this:

• Always (default) — walk the whole volume on every mount;
• OnRootMismatch — check only the root of the volume: if the owner and permissions of the root are already right, do nothing. For volumes that only this application writes to, it is the right choice.

This is also the source of the questions we will come back to in parts 23 and 24: on NFS the ownership change is done as the node's root, and a server with root_squash will not allow it; with SMB file permissions are set by mount options, not by chown. That is why for network volumes it matters which fsGroupPolicy the driver sets and whether it supports VOLUME_MOUNT_GROUP.

---

The demo driver

To see the CSI mechanics in full, we need a driver with all the sidecar containers, and real network storage only appears in the next parts. Let us take csi-driver-host-path — the kubernetes-csi project's demo driver, which creates volumes as directories on the node in /var/lib/csi-hostpath-data/, supports expansion and snapshots and is used in Kubernetes' own tests. It is no good for data: all volumes live on one node — where its single StatefulSet pod runs, which combines both the controller and the node part. For it, a volume's size is only a record in the object: the directory is not limited in any way. In real drivers, such as csi-driver-nfs in part 23, the controller part is a Deployment and the node part is a DaemonSet.

For snapshots we first install the CRDs and the snapshot-controller from the official external-snapshotter v8.7.0 manifests. I downloaded the files on the workstation and copied them to the master into ~/p21, and ran kubectl from there:

$ V=v8.7.0
$ for f in client/config/crd/snapshot.storage.k8s.io_volumesnapshotclasses.yaml client/config/crd/snapshot.storage.k8s.io_volumesnapshotcontents.yaml client/config/crd/snapshot.storage.k8s.io_volumesnapshots.yaml deploy/kubernetes/snapshot-controller/rbac-snapshot-controller.yaml deploy/kubernetes/snapshot-controller/setup-snapshot-controller.yaml; do curl -sLO https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/$V/$f; done
$ kubectl apply -f snapshot.storage.k8s.io_volumesnapshotclasses.yaml -f snapshot.storage.k8s.io_volumesnapshotcontents.yaml -f snapshot.storage.k8s.io_volumesnapshots.yaml
customresourcedefinition.apiextensions.k8s.io/volumesnapshotclasses.snapshot.storage.k8s.io created
customresourcedefinition.apiextensions.k8s.io/volumesnapshotcontents.snapshot.storage.k8s.io created
customresourcedefinition.apiextensions.k8s.io/volumesnapshots.snapshot.storage.k8s.io created
$ kubectl apply -f rbac-snapshot-controller.yaml -f setup-snapshot-controller.yaml
serviceaccount/snapshot-controller created
clusterrole.rbac.authorization.k8s.io/snapshot-controller-runner created
clusterrolebinding.rbac.authorization.k8s.io/snapshot-controller-role created
role.rbac.authorization.k8s.io/snapshot-controller-leaderelection created
rolebinding.rbac.authorization.k8s.io/snapshot-controller-leaderelection created
deployment.apps/snapshot-controller created
$ kubectl -n kube-system get pods -l app.kubernetes.io/name=snapshot-controller -o wide
NAME                                   READY   STATUS    RESTARTS   AGE   IP              NODE         NOMINATED NODE   READINESS GATES
snapshot-controller-7fd654b88d-7pkfr   1/1     Running   0          71s   172.20.172.56   k8s-node-2   <none>           <none>
snapshot-controller-7fd654b88d-wk69v   1/1     Running   0          71s   172.20.211.42   k8s-node-1   <none>           <none>

Two replicas with leader election; in the v8.7.0 release manifest the controller image is snapshot-controller:v8.6.0.

The driver itself is installed by a script from the repository that applies a set of manifests: the RBAC of each sidecar (the script downloads it from the sidecars' repositories at the version named in the manifest), the CSIDriver, the StatefulSet and a VolumeSnapshotClass. Let us take release v1.19.0. The hostpath directory also contains csi-hostpath-testing.yaml — a pod with socat that exposes the driver's socket through a NodePort for test utilities; we do not need it, and there is no reason to expose the driver's socket to the network, so we delete the file. We clone the repository on the workstation as well, copy it to the master into ~/p21 and run the script where kubectl is:

$ git clone --depth 1 --branch v1.19.0 https://github.com/kubernetes-csi/csi-driver-host-path.git
$ cd ~/p21/csi-driver-host-path
$ rm deploy/kubernetes-latest/hostpath/csi-hostpath-testing.yaml
$ deploy/kubernetes-latest/deploy.sh
applying RBAC rules
curl https://raw.githubusercontent.com/kubernetes-csi/external-provisioner/v6.4.0/deploy/kubernetes/rbac.yaml --output /tmp/tmp.5DbD3MoFpP/rbac.yaml --silent --location
kubectl apply --kustomize /tmp/tmp.5DbD3MoFpP
# Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
serviceaccount/csi-provisioner created
role.rbac.authorization.k8s.io/external-provisioner-cfg created
clusterrole.rbac.authorization.k8s.io/external-provisioner-runner created
rolebinding.rbac.authorization.k8s.io/csi-provisioner-role-cfg created
clusterrolebinding.rbac.authorization.k8s.io/csi-provisioner-role created
...
deploying hostpath components
   /home/user/p21/csi-driver-host-path/deploy/kubernetes-latest/hostpath/csi-hostpath-driverinfo.yaml
csidriver.storage.k8s.io/hostpath.csi.k8s.io created
   /home/user/p21/csi-driver-host-path/deploy/kubernetes-latest/hostpath/csi-hostpath-plugin.yaml
        using           image: registry.k8s.io/sig-storage/hostpathplugin:v1.18.0
        using           image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.18.0
        using           image: registry.k8s.io/sig-storage/livenessprobe:v2.20.0
        using           image: registry.k8s.io/sig-storage/csi-attacher:v4.13.0
        using           image: registry.k8s.io/sig-storage/csi-provisioner:v6.4.0
        using           image: registry.k8s.io/sig-storage/csi-resizer:v2.3.0
        using           image: registry.k8s.io/sig-storage/csi-snapshotter:v8.7.0
serviceaccount/csi-hostpathplugin-sa created
clusterrolebinding.rbac.authorization.k8s.io/csi-hostpathplugin-attacher-cluster-role created
clusterrolebinding.rbac.authorization.k8s.io/csi-hostpathplugin-provisioner-cluster-role created
clusterrolebinding.rbac.authorization.k8s.io/csi-hostpathplugin-resizer-cluster-role created
clusterrolebinding.rbac.authorization.k8s.io/csi-hostpathplugin-snapshotter-cluster-role created
clusterrolebinding.rbac.authorization.k8s.io/csi-hostpathplugin-snapshot-metadata-cluster-role created
rolebinding.rbac.authorization.k8s.io/csi-hostpathplugin-attacher-role created
rolebinding.rbac.authorization.k8s.io/csi-hostpathplugin-provisioner-role created
rolebinding.rbac.authorization.k8s.io/csi-hostpathplugin-resizer-role created
rolebinding.rbac.authorization.k8s.io/csi-hostpathplugin-snapshotter-role created
Warning: spec.template.spec.containers[1].ports[0]: duplicate port name "healthz" with spec.template.spec.containers[0].ports[0], services and probes that select ports by name will use spec.template.spec.containers[0].ports[0]
statefulset.apps/csi-hostpathplugin created
   /home/user/p21/csi-driver-host-path/deploy/kubernetes-latest/hostpath/csi-hostpath-snapshotclass.yaml
volumesnapshotclass.snapshot.storage.k8s.io/csi-hostpath-snapclass created
17:42:25 waiting for hostpath deployment to complete, attempt #0
...
17:43:36 waiting for hostpath deployment to complete, attempt #7

The script installs the driver into the default namespace and waits until the StatefulSet pod is ready (about 80 seconds for us, while the images were pulled). The driver itself in release v1.19.0 is the hostpathplugin:v1.18.0 image. The warning about the duplicate healthz port name is a quirk of the manifest and does not affect operation. The last step is the csi-hostpath class from the StorageClass section: kubectl apply -f sc.yaml.

---

Hands-on

emptyDir and tmpfs

A pod with the two volumes from the example above — cache on disk with sizeLimit: 1Gi and scratch in memory with sizeLimit: 256Mi:

apiVersion: v1
kind: Pod
metadata:
  name: vols
  namespace: demo
spec:
  containers:
    - name: app
      image: busybox:1.37
      command: ["sleep", "1d"]
      volumeMounts:
        - name: cache
          mountPath: /cache
        - name: scratch
          mountPath: /scratch
  volumes:
    - name: cache
      emptyDir:
        sizeLimit: 1Gi
    - name: scratch
      emptyDir:
        medium: Memory
        sizeLimit: 256Mi

$ kubectl -n demo exec vols -- df -h /cache /scratch
Filesystem                Size      Used Available Use% Mounted on
/dev/sda2                30.3G      5.0G     23.7G  17% /cache
tmpfs                   128.0M         0    128.0M   0% /scratch
$ kubectl -n demo get pod vols -o jsonpath='{.spec.containers[0].resources}'
{"limits":{"memory":"128Mi"},"requests":{"cpu":"100m","memory":"128Mi"}}

Both peculiarities are visible right away. cache is simply the node's root disk: sizeLimit: 1Gi is not visible inside the pod in any way. And the tmpfs got 128 MiB, not 256: the LimitRange of the demo namespace gave the container a 128 MiB memory limit, and kubelet took the smaller of sizeLimit and the pod's limit. The tmpfs-nolimit pod with a single medium: Memory volume without sizeLimit in the default namespace, where there is no LimitRange and the pod has no memory limit either, gets a tmpfs the size of the node's allocatable memory:

$ kubectl -n default exec tmpfs-nolimit -- df -h /scratch
Filesystem                Size      Used Available Use% Mounted on
tmpfs                     3.2G         0      3.2G   0% /scratch
$ kubectl get node -o custom-columns=NAME:.metadata.name,ALLOC:.status.allocatable.memory
NAME           ALLOC
k8s-master-1   3377260Ki
k8s-node-1     3377260Ki
k8s-node-2     3377252Ki

Let us write 8 MiB to the tmpfs and look at the container's memory in its cgroup:

$ kubectl -n demo exec vols -- sh -c 'grep -E "^(anon|file|shmem) " /sys/fs/cgroup/memory.stat; cat /sys/fs/cgroup/memory.current; dd if=/dev/zero of=/scratch/blob bs=1M count=8; grep -E "^(anon|file|shmem) " /sys/fs/cgroup/memory.stat; cat /sys/fs/cgroup/memory.current; df -h /scratch'
anon 413696
file 0
shmem 0
1556480
8+0 records in
8+0 records out
8388608 bytes (8.0MB) copied, 0.002622 seconds, 3.0GB/s
anon 417792
file 8388608
shmem 8388608
9879552
Filesystem                Size      Used Available Use% Mounted on
tmpfs                   128.0M      8.0M    120.0M   6% /scratch

The file in the tmpfs is exactly 8,388,608 bytes of shmem in the container's memory, and memory.current grew by the same amount. What happens when you hit the tmpfs size is shown by the tmpfs-small pod with sizeLimit: 16Mi (writing 32 MiB):

$ kubectl -n demo exec tmpfs-small -- sh -c "dd if=/dev/zero of=/scratch/blob bs=1M count=32; df -h /scratch"
dd: error writing '/scratch/blob': No space left on device
17+0 records in
16+0 records out
16777216 bytes (16.0MB) copied, 0.003415 seconds, 4.6GB/s
Filesystem                Size      Used Available Use% Mounted on
tmpfs                    16.0M     16.0M         0 100% /scratch

Eviction by sizeLimit

With a volume on disk it is different: the file system does not stop the writes, kubelet keeps watch. The filler pod writes 20 MiB into an emptyDir with sizeLimit: 10Mi:

apiVersion: v1
kind: Pod
metadata:
  name: filler
  namespace: demo
spec:
  restartPolicy: Never
  containers:
    - name: app
      image: busybox:1.37
      command: ["sh", "-c", "dd if=/dev/zero of=/cache/blob bs=1M count=20; sleep 1d"]
      volumeMounts:
        - name: cache
          mountPath: /cache
  volumes:
    - name: cache
      emptyDir:
        sizeLimit: 10Mi

The write goes through at once, and 55 seconds after the pod was created kubelet evicts it:

$ kubectl -n demo get pod filler
NAME     READY   STATUS                   RESTARTS   AGE
filler   0/1     ContainerStatusUnknown   1          61s
$ kubectl -n demo get events --field-selector involvedObject.name=filler -o custom-columns=TYPE:.type,REASON:.reason,FROM:.source.component,MESSAGE:.message
TYPE      REASON      FROM      MESSAGE
Normal    Scheduled   <none>    Successfully assigned demo/filler to k8s-node-1
Normal    Pulled      kubelet   Container image "busybox:1.37" already present on machine and can be accessed by the pod
Normal    Created     kubelet   Container created
Normal    Started     kubelet   Container started
Warning   Evicted     kubelet   Usage of EmptyDir volume "cache" exceeds the limit "10Mi". 
Normal    Killing     kubelet   Stopping container app

The pod's phase is Failed, the reason is Evicted, and the STATUS column shows ContainerStatusUnknown: the container was killed during eviction, and kubelet did not keep its exit code. kubelet measures directory sizes periodically, so the eviction happens not at the moment of writing but at the next check; in that time a pod without a limit could have written much more.

Volume directories on the node

The vols pod runs on k8s-node-1. We find its UID and look at kubelet's directory:

$ kubectl -n demo get pod vols -o jsonpath='{.metadata.uid}'
5e7efaf0-5e5f-42f3-a394-a995822ef630
user@k8s-node-1:~$ sudo find /var/lib/kubelet/pods/5e7efaf0-5e5f-42f3-a394-a995822ef630/volumes -maxdepth 2
/var/lib/kubelet/pods/5e7efaf0-5e5f-42f3-a394-a995822ef630/volumes
/var/lib/kubelet/pods/5e7efaf0-5e5f-42f3-a394-a995822ef630/volumes/kubernetes.io~empty-dir
/var/lib/kubelet/pods/5e7efaf0-5e5f-42f3-a394-a995822ef630/volumes/kubernetes.io~empty-dir/cache
/var/lib/kubelet/pods/5e7efaf0-5e5f-42f3-a394-a995822ef630/volumes/kubernetes.io~empty-dir/scratch
/var/lib/kubelet/pods/5e7efaf0-5e5f-42f3-a394-a995822ef630/volumes/kubernetes.io~projected
/var/lib/kubelet/pods/5e7efaf0-5e5f-42f3-a394-a995822ef630/volumes/kubernetes.io~projected/kube-api-access-bvh4v
user@k8s-node-1:~$ sudo findmnt -o TARGET,SOURCE,FSTYPE -T /var/lib/kubelet/pods/5e7efaf0-5e5f-42f3-a394-a995822ef630/volumes/kubernetes.io~empty-dir/cache
TARGET SOURCE    FSTYPE
/      /dev/sda2 ext4
user@k8s-node-1:~$ findmnt -o TARGET,SOURCE,FSTYPE,OPTIONS -T /var/lib/kubelet/pods/5e7efaf0-5e5f-42f3-a394-a995822ef630/volumes/kubernetes.io~empty-dir/scratch
TARGET                                                                                             SOURCE FSTYPE OPTIONS
/var/lib/kubelet/pods/5e7efaf0-5e5f-42f3-a394-a995822ef630/volumes/kubernetes.io~empty-dir/scratch tmpfs  tmpfs  rw,relatime,size=131072k,inode64,noswap
user@k8s-node-1:~$ sudo ls -l /var/lib/kubelet/pods/5e7efaf0-5e5f-42f3-a394-a995822ef630/volumes/kubernetes.io~empty-dir/scratch
total 8192
-rw-r--r-- 1 root root 8388608 Oct 29 17:34 blob

cache is an ordinary directory on the node's root file system, without a mount of its own. scratch is a separate tmpfs with size=131072k (128 MiB) and noswap: even if swap is enabled on the node, the volume's pages will not go there. Next to them is kube-api-access-bvh4v — the projected volume every pod gets. Inside the container both volumes show up as mounts:

$ kubectl -n demo exec vols -- grep -E ' /(cache|scratch) ' /proc/mounts
/dev/sda2 /cache ext4 rw,relatime 0 0
tmpfs /scratch tmpfs rw,relatime,size=131072k,inode64,noswap 0 0

projected

The proj pod with the volume from the example above (the hello-config ConfigMap with the LOG_LEVEL key, the hello-db Secret, the pod name and a token for the vault audience), mounted at /etc/app:

$ kubectl -n demo exec proj -- ls -la /etc/app
total 4
drwxrwxrwt    3 root     root           160 Oct 29 14:36 .
drwxr-xr-x    1 root     root          4096 Oct 29 14:36 ..
drwxr-xr-x    2 root     root           120 Oct 29 14:36 ..2026_10_29_14_36_35.2804868689
lrwxrwxrwx    1 root     root            32 Oct 29 14:36 ..data -> ..2026_10_29_14_36_35.2804868689
lrwxrwxrwx    1 root     root            16 Oct 29 14:36 LOG_LEVEL -> ..data/LOG_LEVEL
lrwxrwxrwx    1 root     root            18 Oct 29 14:36 db-password -> ..data/db-password
lrwxrwxrwx    1 root     root            15 Oct 29 14:36 pod-name -> ..data/pod-name
lrwxrwxrwx    1 root     root            18 Oct 29 14:36 vault-token -> ..data/vault-token
$ kubectl -n demo exec proj -- cat /etc/app/pod-name
proj
$ kubectl -n demo exec proj -- cat /etc/app/vault-token | cut -d. -f2 | tr "_-" "/+" | base64 -d 2>/dev/null | jq -c "{aud,exp,iat,sub}"
{"aud":["vault"],"exp":1793288195,"iat":1793284595,"sub":"system:serviceaccount:demo:default"}
$ kubectl -n demo exec proj -- grep /etc/app /proc/mounts
tmpfs /etc/app tmpfs ro,relatime,size=131072k,inode64,noswap 0 0

Four sources in one directory, all through ..data. The token is issued for the vault audience for exactly an hour (exp − iat = 3600) on behalf of the default ServiceAccount of the demo namespace; kubelet will refresh the file when 80% of the lifetime has passed. The volume is a read-only tmpfs, and its size again equals the pod's memory limit.

hostPath

The hostlogs pod with the /var/log volume from the example above, mounted read-only at /host/var/log:

$ kubectl -n demo exec hostlogs -- ls /host/var/log | head -8
README
alternatives.log
apport.log
apt
auth.log
bootstrap.log
btmp
calico
$ kubectl -n demo exec hostlogs -- touch /host/var/log/x
touch: /host/var/log/x: Read-only file system
command terminated with exit code 1

The node's logs are visible to the pod as they are. The type check fires at mount time — the hostmissing pod asks for type: Directory for a non-existent /srv/no-such-dir:

$ kubectl -n demo get pod hostmissing -o wide
NAME          READY   STATUS              RESTARTS   AGE   IP       NODE         NOMINATED NODE   READINESS GATES
hostmissing   0/1     ContainerCreating   0          12s   <none>   k8s-node-1   <none>           <none>
$ kubectl -n demo get events --field-selector involvedObject.name=hostmissing -o custom-columns=TYPE:.type,REASON:.reason,MESSAGE:.message | tail -2
Normal    Scheduled     Successfully assigned demo/hostmissing to k8s-node-1
Warning   FailedMount   MountVolume.SetUp failed for volume "data" : hostPath type check failed: /srv/no-such-dir is not a directory

And in a namespace with the baseline profile such a pod is not created at all. A namespace with the Pod Security Admission label:

apiVersion: v1
kind: Namespace
metadata:
  name: baseline-demo
  labels:
    pod-security.kubernetes.io/enforce: baseline

And the same hostlogs, only in this namespace:

$ kubectl apply -f baseline-ns.yaml
namespace/baseline-demo created
$ kubectl apply -f hostlogs-baseline.yaml
Error from server (Forbidden): error when creating "hostlogs-baseline.yaml": pods "hostlogs" is forbidden: violates PodSecurity "baseline:latest": hostPath volumes (volume "host-logs")

A static PV

We apply manual-pv-1 and the data PVC from one file:

$ kubectl apply -f static-pv.yaml
persistentvolume/manual-pv-1 created
persistentvolumeclaim/data created
$ kubectl get pv manual-pv-1
NAME          CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM       STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE
manual-pv-1   5Gi        RWO            Retain           Bound    demo/data   manual         <unset>                          25s
$ kubectl -n demo get pvc data
NAME   STATUS   VOLUME        CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
data   Bound    manual-pv-1   5Gi        RWO            manual         <unset>                 25s
$ kubectl get pv manual-pv-1 -o jsonpath='{.spec.claimRef}' | jq -c
{"apiVersion":"v1","kind":"PersistentVolumeClaim","name":"data","namespace":"demo","resourceVersion":"964210","uid":"072a2dcd-6a49-486b-9366-6d9a4c12c7da"}
$ kubectl -n demo describe pvc data | tail -6
VolumeMode:    Filesystem
Used By:       <none>
Events:
  Type     Reason              Age   From                         Message
  ----     ------              ----  ----                         -------
  Warning  ProvisioningFailed  25s   persistentvolume-controller  storageclass.storage.k8s.io "manual" not found

We asked for 2 GiB and got all 5: the binding is exclusive. The ProvisioningFailed event is a trace of the first attempt: the PVC reached the controller before the PV got into its cache, no suitable volume was found, and the controller tried to provision a volume dynamically for the manual class, which does not exist. On the next pass it found the PV and bound them: 3 seconds after apply the PVC was still Pending, by the 25th second it was Bound. The VOLUMEATTRIBUTESCLASS column is the class of mutable volume attributes (VolumeAttributesClass, stable since 1.34): it is used to change parameters such as IOPS and throughput of an existing volume, and the driver gets a ControllerModifyVolume call. We do not set it.

The pv-writer pod writes the creation date into the volume (date > /data/created.txt) and lands, as nodeAffinity promises, on k8s-node-1. We delete the PVC while the pod is running:

$ kubectl -n demo delete pvc data --wait=false
persistentvolumeclaim "data" deleted from demo namespace
$ kubectl -n demo get pvc data
NAME   STATUS        VOLUME        CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
data   Terminating   manual-pv-1   5Gi        RWO            manual         <unset>                 40s
$ kubectl -n demo describe pvc data | grep -E "Status|Used By|Finalizers"
Status:        Terminating (lasts 2s)
Finalizers:    [kubernetes.io/pvc-protection]
Used By:       pv-writer

kubectl honestly prints deleted, but the kubernetes.io/pvc-protection finalizer holds the object while pv-writer uses it. We delete the pod — the PVC disappears, the PV goes to Released, and the data stays on the node:

$ kubectl -n demo delete pod pv-writer
pod "pv-writer" deleted from demo namespace
$ kubectl -n demo get pvc
No resources found in demo namespace.
$ kubectl get pv manual-pv-1
NAME          CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM       STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE
manual-pv-1   5Gi        RWO            Retain           Released   demo/data   manual         <unset>                          82s
user@k8s-node-1:~$ ls -l /srv/volumes/manual-pv-1; cat /srv/volumes/manual-pv-1/created.txt
total 4
-rw-r--r-- 1 root root 29 Oct 29 17:37 created.txt
Thu Oct 29 14:37:54 UTC 2026

The /srv/volumes/manual-pv-1 directory was created by kubelet at mount time (type: DirectoryOrCreate). Let us give the data back to the application: a new PVC data-again asks for this PV by name (volumeName: manual-pv-1, the same class, 5 GiB), and the pv-reader pod mounts it:

$ kubectl -n demo get pvc data-again
NAME         STATUS    VOLUME        CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
data-again   Pending   manual-pv-1   0                         manual         <unset>                 20s
$ kubectl -n demo describe pvc data-again | sed -n '/Events/,$p'
Events:
  Type     Reason         Age                From                         Message
  ----     ------         ----               ----                         -------
  Warning  FailedBinding  12s (x2 over 21s)  persistentvolume-controller  volume "manual-pv-1" already bound to a different claim.
$ kubectl -n demo get events --field-selector involvedObject.name=pv-reader -o custom-columns=REASON:.reason,MESSAGE:.message | tail -2
REASON             MESSAGE
FailedScheduling   0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found

The PV still has a claimRef to the deleted demo/data with its UID — for the controller the volume is taken. We remove the reference:

$ kubectl patch pv manual-pv-1 --type json -p '[{"op":"remove","path":"/spec/claimRef"}]'
persistentvolume/manual-pv-1 patched
$ kubectl get pv manual-pv-1
NAME          CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM             STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE
manual-pv-1   5Gi        RWO            Retain           Bound    demo/data-again   manual         <unset>                          2m17s
$ kubectl -n demo exec pv-reader -- cat /data/created.txt
Thu Oct 29 14:37:54 UTC 2026

Right after the patch the PV is still Released without a CLAIM, and 20 seconds later it is already bound to data-again, and the pod sees the old file. This is what manually restoring a Retain volume looks like. After the experiment the PV was deleted, and the /srv/volumes directory on the node had to be removed by hand — Kubernetes does not clean up after hostPath.

A PVC without a class

While the cluster has no default class, a PVC without storageClassName and without suitable PVs just waits:

$ kubectl -n demo get pvc nosc
NAME   STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
nosc   Pending                                                     <unset>                 6s
$ kubectl -n demo describe pvc nosc | sed -n '/Events/,$p'
Events:
  Type    Reason         Age   From                         Message
  ----    ------         ----  ----                         -------
  Normal  FailedBinding  5s    persistentvolume-controller  no persistent volumes available for this claim and no storage class is set

We will come back to it when the driver appears.

The driver in place

The driver's pod has seven containers: the driver itself and the sidecars:

$ kubectl get pod -o wide
NAME                   READY   STATUS    RESTARTS   AGE   IP              NODE         NOMINATED NODE   READINESS GATES
csi-hostpathplugin-0   7/7     Running   0          90s   172.20.172.61   k8s-node-2   <none>           <none>
$ kubectl get pod csi-hostpathplugin-0 -o jsonpath='{range .spec.containers[*]}{.name}{"\n"}{end}'
hostpath
node-driver-registrar
liveness-probe
csi-attacher
csi-provisioner
csi-resizer
csi-snapshotter

$ kubectl get csidriver
NAME                  ATTACHREQUIRED   PODINFOONMOUNT   STORAGECAPACITY   TOKENREQUESTS   REQUIRESREPUBLISH   MODES                  AGE
csi.tigera.io         true             true             false             <unset>         false               Ephemeral              5d4h
hostpath.csi.k8s.io   true             true             false             <unset>         false               Persistent,Ephemeral   92s

Next to our driver is csi.tigera.io from Calico, for ephemeral volumes only. Both are registered in CSINode:

$ kubectl get csinode
NAME           DRIVERS   AGE
k8s-master-1   1         5d4h
k8s-node-1     1         5d4h
k8s-node-2     2         5d3h
$ kubectl get csinode k8s-node-2 -o jsonpath='{.spec.drivers}' | jq
[
  {
    "name": "csi.tigera.io",
    "nodeID": "k8s-node-2",
    "topologyKeys": null
  },
  {
    "name": "hostpath.csi.k8s.io",
    "nodeID": "k8s-node-2",
    "topologyKeys": [
      "topology.hostpath.csi/node"
    ]
  }
]
$ kubectl get node k8s-node-2 --show-labels | tr "," "\n" | grep -i topo
topology.hostpath.csi/node=k8s-node-2

The demo driver declares a "node" topology: only k8s-node-2, where its pod runs, has the topology.hostpath.csi/node key, and kubelet gave the node a label of the same name. On the node there are two driver sockets: the registration one, which kubelet reads, and the working one:

user@k8s-node-2:~$ sudo ls -l /var/lib/kubelet/plugins_registry/ /var/lib/kubelet/plugins/csi-hostpath/ /var/lib/kubelet/plugins/
/var/lib/kubelet/plugins/:
total 8
drwxr-xr-x 2 root root 4096 Oct 29 17:42 csi-hostpath
drwxr-xr-x 2 root root 4096 Oct 28 16:33 csi.tigera.io

/var/lib/kubelet/plugins/csi-hostpath/:
total 0
srwxr-xr-x 1 root root 0 Oct 29 17:42 csi.sock

/var/lib/kubelet/plugins_registry/:
total 0
srwx------ 1 root root 0 Oct 28 16:33 csi.tigera.io-reg.sock
srwx------ 1 root root 0 Oct 29 17:42 hostpath.csi.k8s.io-reg.sock

What the driver can do is visible in the sidecars' logs at startup:

$ kubectl logs csi-hostpathplugin-0 -c csi-attacher | grep -i -E "capabilit|trivial|attach" | head -8
...
I1029 14:43:08.096659       1 main.go:260] "CSI driver does not support ControllerPublishUnpublish, using trivial handler" driver="hostpath.csi.k8s.io"
...
$ kubectl logs csi-hostpathplugin-0 -c csi-provisioner | grep -E "CreateVolume|provision" | head -6
...
I1029 14:43:22.446461       1 csi-provisioner.go:317] CSI driver does not support PUBLISH_UNPUBLISH_VOLUME, not watching VolumeAttachments
...

The driver has attachRequired: true, so Kubernetes will create VolumeAttachments, but it does not implement ControllerPublishVolume, and the attacher marks the attachment itself, without calling the driver.

A dynamic volume with WaitForFirstConsumer

The hp-data PVC for 1 GiB of class csi-hostpath without a pod:

$ kubectl -n demo get pvc hp-data
NAME      STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
hp-data   Pending                                      csi-hostpath   <unset>                 21s
$ kubectl -n demo describe pvc hp-data | sed -n '/Events/,$p'
Events:
  Type    Reason                Age                From                         Message
  ----    ------                ----               ----                         -------
  Normal  WaitForFirstConsumer  14s (x2 over 21s)  persistentvolume-controller  waiting for first consumer to be created before binding

The consumer pod writer with fsGroup: 2000 (as in the fsGroup section), at first without pinning to a node:

apiVersion: v1
kind: Pod
metadata:
  name: writer
  namespace: demo
spec:
  securityContext:
    runAsUser: 1000
    runAsGroup: 1000
    fsGroup: 2000
    fsGroupChangePolicy: OnRootMismatch
  containers:
    - name: writer
      image: busybox:1.37
      command: ["sh", "-c", "echo hello > /data/hello.txt; sleep 1d"]
      volumeMounts:
        - name: data
          mountPath: /data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: hp-data

$ kubectl -n demo get pod writer -o wide
NAME     READY   STATUS    RESTARTS   AGE   IP       NODE     NOMINATED NODE   READINESS GATES
writer   0/1     Pending   0          25s   <none>   <none>   k8s-node-1       <none>
$ kubectl -n demo describe pvc hp-data | sed -n '/Events/,$p'
Events:
  Type     Reason                Age                From                                                                           Message
  ----     ------                ----               ----                                                                           -------
  Normal   WaitForFirstConsumer  30s (x3 over 52s)  persistentvolume-controller                                                    waiting for first consumer to be created before binding
  Normal   ExternalProvisioning  15s (x3 over 25s)  persistentvolume-controller                                                    Waiting for a volume to be created either by the external provisioner 'hostpath.csi.k8s.io' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  Normal   Provisioning          10s (x5 over 25s)  hostpath.csi.k8s.io_csi-hostpathplugin-0_d11f0230-32e1-4686-897c-fd0e8af7bc28  External provisioner is provisioning volume for claim "demo/hp-data"
  Warning  ProvisioningFailed    10s (x5 over 25s)  hostpath.csi.k8s.io_csi-hostpathplugin-0_d11f0230-32e1-4686-897c-fd0e8af7bc28  error generating accessibility requirements: no topology key found for node k8s-node-1

This is the flip side of WaitForFirstConsumer with a driver that does not publish CSIStorageCapacity: the scheduler picked k8s-node-1 (NOMINATED NODE, the selected-node annotation) knowing nothing about the driver not being there, and the provisioner cannot create a volume on a node without the topology key. The pod waits for the volume and has no events. The driver's install script warns about this in a comment: with late binding, pods have to be forced onto the driver's node. We delete the pod and the PVC, create the PVC again, and add a nodeSelector with kubernetes.io/hostname: k8s-node-2 to the pod:

$ kubectl -n demo get pvc hp-data -o jsonpath='{.metadata.annotations}' | jq 'del(."kubectl.kubernetes.io/last-applied-configuration")'
{
  "pv.kubernetes.io/bind-completed": "yes",
  "pv.kubernetes.io/bound-by-controller": "yes",
  "volume.beta.kubernetes.io/storage-provisioner": "hostpath.csi.k8s.io",
  "volume.kubernetes.io/selected-node": "k8s-node-2",
  "volume.kubernetes.io/storage-provisioner": "hostpath.csi.k8s.io"
}
$ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM          STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE
pvc-8b17697d-7849-455d-af68-d08c71604723   1Gi        RWO            Delete           Bound    demo/hp-data   csi-hostpath   <unset>                          14s
$ kubectl get pv pvc-8b17697d-7849-455d-af68-d08c71604723 -o jsonpath='{.spec.csi}' | jq
{
  "driver": "hostpath.csi.k8s.io",
  "volumeAttributes": {
    "storage.kubernetes.io/csiProvisionerIdentity": "1793285002446-4963-hostpath.csi.k8s.io"
  },
  "volumeHandle": "665e490b-d3a7-11f1-8d4c-52a962bdf80a"
}
$ kubectl get pv pvc-8b17697d-7849-455d-af68-d08c71604723 -o jsonpath='{.spec.nodeAffinity}' | jq -c
{"required":{"nodeSelectorTerms":[{"matchExpressions":[{"key":"topology.hostpath.csi/node","operator":"In","values":["k8s-node-2"]}]}]}}
$ kubectl -n demo describe pvc hp-data | sed -n '/Events/,$p'
Events:
  Type    Reason                 Age                From                                                                           Message
  ----    ------                 ----               ----                                                                           -------
  Normal  WaitForFirstConsumer   17s                persistentvolume-controller                                                    waiting for first consumer to be created before binding
  Normal  Provisioning           14s                hostpath.csi.k8s.io_csi-hostpathplugin-0_d11f0230-32e1-4686-897c-fd0e8af7bc28  External provisioner is provisioning volume for claim "demo/hp-data"
  Normal  ExternalProvisioning   14s (x2 over 14s)  persistentvolume-controller                                                    Waiting for a volume to be created either by the external provisioner 'hostpath.csi.k8s.io' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  Normal  ProvisioningSucceeded  14s                hostpath.csi.k8s.io_csi-hostpathplugin-0_d11f0230-32e1-4686-897c-fd0e8af7bc28  Successfully provisioned volume pvc-8b17697d-7849-455d-af68-d08c71604723

The scheduler picked k8s-node-2 (selected-node), the provisioner created a PV with a volumeHandle — the volume's ID in the "storage" — and pinned it to the node through nodeAffinity with the driver's topology key. The attachment:

$ kubectl get volumeattachment
NAME                                                                   ATTACHER              PV                                         NODE         ATTACHED   AGE
csi-05efdba6b22aa93a17bad6253a0b59e6ad7918b264e8044ce160c70f1b4b5380   hostpath.csi.k8s.io   pvc-8b17697d-7849-455d-af68-d08c71604723   k8s-node-2   true       13s

What is on the node

user@k8s-node-2:~$ PV=pvc-8b17697d-7849-455d-af68-d08c71604723
user@k8s-node-2:~$ sudo find /var/lib/kubelet/plugins/kubernetes.io/csi/hostpath.csi.k8s.io -maxdepth 2
/var/lib/kubelet/plugins/kubernetes.io/csi/hostpath.csi.k8s.io
/var/lib/kubelet/plugins/kubernetes.io/csi/hostpath.csi.k8s.io/62cbbb6ca8ab41c5f487dcebe05d49f71f31c455abe62538185a0e1b932d92c7
/var/lib/kubelet/plugins/kubernetes.io/csi/hostpath.csi.k8s.io/62cbbb6ca8ab41c5f487dcebe05d49f71f31c455abe62538185a0e1b932d92c7/globalmount
/var/lib/kubelet/plugins/kubernetes.io/csi/hostpath.csi.k8s.io/62cbbb6ca8ab41c5f487dcebe05d49f71f31c455abe62538185a0e1b932d92c7/vol_data.json
user@k8s-node-2:~$ echo -n 665e490b-d3a7-11f1-8d4c-52a962bdf80a | sha256sum
62cbbb6ca8ab41c5f487dcebe05d49f71f31c455abe62538185a0e1b932d92c7  -
user@k8s-node-2:~$ POD_UID=$(sudo crictl inspectp $(sudo crictl pods --name writer -q) | jq -r .status.metadata.uid)
user@k8s-node-2:~$ sudo ls /var/lib/kubelet/pods/$POD_UID/volumes/kubernetes.io~csi/$PV/
mount
vol_data.json
user@k8s-node-2:~$ sudo findmnt -T /var/lib/kubelet/pods/$POD_UID/volumes/kubernetes.io~csi/$PV/mount
TARGET                                                                                                                              SOURCE                                                                     FSTYPE OPTIONS
/var/lib/kubelet/pods/838f3069-261c-4b47-8f6d-b4e838e81f43/volumes/kubernetes.io~csi/pvc-8b17697d-7849-455d-af68-d08c71604723/mount /dev/sda2[/var/lib/csi-hostpath-data/665e490b-d3a7-11f1-8d4c-52a962bdf80a] ext4   rw,relatime

The name of the staging directory is the SHA-256 of volumeHandle, as promised. The globalmount directory itself is empty and nothing is mounted on it:

user@k8s-node-2:~$ sudo ls -la /var/lib/kubelet/plugins/kubernetes.io/csi/hostpath.csi.k8s.io/62cbbb6ca8ab41c5f487dcebe05d49f71f31c455abe62538185a0e1b932d92c7/globalmount
total 8
drwxr-x--- 2 root root 4096 Oct 29 17:45 .
drwxr-x--- 3 root root 4096 Oct 29 17:45 ..
user@k8s-node-2:~$ findmnt | grep -c globalmount
0

The demo driver declares the stage step but does nothing in it, and mounts only at publish time — with a bind mount of its data directory /var/lib/csi-hostpath-data/<volumeHandle> into the pod's directory. With a block disk driver, globalmount would have the device's file system mounted on it, and the pod's directory would get a bind mount from globalmount. The square brackets in findmnt's SOURCE are exactly that bind mount: "such-and-such subdirectory of such-and-such file system".

vol_data.json is what kubelet remembered about the volume, in the pod's directory and in the staging directory:

user@k8s-node-2:~$ sudo cat /var/lib/kubelet/pods/$POD_UID/volumes/kubernetes.io~csi/$PV/vol_data.json | jq
{
  "attachmentID": "csi-05efdba6b22aa93a17bad6253a0b59e6ad7918b264e8044ce160c70f1b4b5380",
  "driverName": "hostpath.csi.k8s.io",
  "nodeName": "k8s-node-2",
  "specVolID": "pvc-8b17697d-7849-455d-af68-d08c71604723",
  "volumeHandle": "665e490b-d3a7-11f1-8d4c-52a962bdf80a",
  "volumeLifecycleMode": "Persistent"
}
user@k8s-node-2:~$ G=$(sudo ls /var/lib/kubelet/plugins/kubernetes.io/csi/hostpath.csi.k8s.io/); sudo cat /var/lib/kubelet/plugins/kubernetes.io/csi/hostpath.csi.k8s.io/$G/vol_data.json | jq
{
  "driverName": "hostpath.csi.k8s.io",
  "volumeHandle": "665e490b-d3a7-11f1-8d4c-52a962bdf80a"
}

The driver's calls

The driver runs with --v=5 and logs every gRPC call. A summary for the duration of the experiment:

$ kubectl logs csi-hostpathplugin-0 -c hostpath | grep -E "GRPC call" | sed -E "s/.*GRPC call: //" | sort | uniq -c
      6 /csi.v1.Controller/ControllerGetCapabilities
      1 /csi.v1.Controller/CreateVolume
      3 /csi.v1.Identity/GetPluginCapabilities
      6 /csi.v1.Identity/GetPluginInfo
     76 /csi.v1.Identity/Probe
      7 /csi.v1.Node/NodeGetCapabilities
      1 /csi.v1.Node/NodeGetInfo
      1 /csi.v1.Node/NodeGetVolumeStats
      1 /csi.v1.Node/NodePublishVolume
      1 /csi.v1.Node/NodeStageVolume

Exactly the sequence from the diagram: CreateVolume from the provisioner, then 11 seconds later, when the pod started, NodeStageVolume and NodePublishVolume from kubelet; there is no ControllerPublishVolume — the attacher did without it. Probe is livenessprobe, and NodeGetInfo is kubelet learning the nodeID and topology keys for CSINode when registering the driver. The failed attempts on k8s-node-1 never reached the driver: the provisioner refused earlier. The publish request shows what podInfoOnMount provided:

$ kubectl logs csi-hostpathplugin-0 -c hostpath | grep -A6 "Node/NodePublishVolume" | head -8
I1029 14:45:46.387768       1 server.go:98] GRPC call: /csi.v1.Node/NodePublishVolume
I1029 14:45:46.387783       1 server.go:102] GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/hostpath.csi.k8s.io/62cbbb6ca8ab41c5f487dcebe05d49f71f31c455abe62538185a0e1b932d92c7/globalmount","target_path":"/var/lib/kubelet/pods/838f3069-261c-4b47-8f6d-b4e838e81f43/volumes/kubernetes.io~csi/pvc-8b17697d-7849-455d-af68-d08c71604723/mount","volume_capability":{"access_mode":{"mode":"SINGLE_NODE_MULTI_WRITER"},"mount":{}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"writer","csi.storage.k8s.io/pod.namespace":"demo","csi.storage.k8s.io/pod.uid":"838f3069-261c-4b47-8f6d-b4e838e81f43","csi.storage.k8s.io/serviceAccount.name":"default","storage.kubernetes.io/csiProvisionerIdentity":"1793285002446-4963-hostpath.csi.k8s.io"},"volume_id":"665e490b-d3a7-11f1-8d4c-52a962bdf80a"}
...

The volume_context holds the pod's name, namespace, UID and ServiceAccount. ReadWriteOnce reached the driver as SINGLE_NODE_MULTI_WRITER — the CSI mode "one node, many writers": the driver declared the SINGLE_NODE_MULTI_WRITER capability, and kubelet passes RWO exactly that way.

fsGroup

The driver has fsGroupPolicy: File, so kubelet changed the group of the volume's root to 2000 and set setgid:

$ kubectl -n demo exec writer -- id
uid=1000 gid=1000 groups=1000,2000
$ kubectl -n demo exec writer -- ls -ln /data
total 4
-rw-r--r--    1 1000     2000             6 Oct 29 14:45 hello.txt
$ kubectl -n demo exec writer -- stat -c '%A %u:%g %n' /data
drwxrwsr-x 0:2000 /data
$ kubectl -n demo exec writer -- df -h /data
Filesystem                Size      Used Available Use% Mounted on
/dev/sda2                30.3G      6.5G     22.2G  23% /data

A file created by a process whose primary group is 1000 got group 2000 — that is how the s bit on a directory works. If the CSIDriver had None, the root would stay root:root with 0755 permissions, and a process running as UID 1000 could not write there. And df honestly shows that the "1 GiB volume" is a directory on the node's root disk.

Expansion

We grow the PVC to 2 GiB:

$ kubectl -n demo patch pvc hp-data -p '{"spec":{"resources":{"requests":{"storage":"2Gi"}}}}'
persistentvolumeclaim/hp-data patched
$ kubectl -n demo get pvc hp-data
NAME      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
hp-data   Bound    pvc-8b17697d-7849-455d-af68-d08c71604723   1Gi        RWO            csi-hostpath   <unset>                 80s
$ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM          STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE
pvc-8b17697d-7849-455d-af68-d08c71604723   2Gi        RWO            Delete           Bound    demo/hp-data   csi-hostpath   <unset>                          77s
$ kubectl -n demo get pvc hp-data -o jsonpath='{.status.conditions}{"\n"}{.status.allocatedResources}{"\n"}{.status.allocatedResourceStatuses}'
[{"lastProbeTime":null,"lastTransitionTime":"2026-10-29T14:46:37Z","status":"True","type":"Resizing"},{"lastProbeTime":null,"lastTransitionTime":"2026-10-29T14:46:37Z","message":"Waiting for user to (re-)start a pod to finish file system resize of volume on node.","status":"True","type":"FileSystemResizePending"}]
{"storage":"2Gi"}
{"storage":"NodeResizePending"}

The first step is done: the PV is already 2 GiB, while the PVC is still 1 GiB with the FileSystemResizePending condition. The condition text suggests restarting the pod, but the driver can expand a mounted volume, and waiting is enough:

$ kubectl -n demo get pvc hp-data
NAME      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
hp-data   Bound    pvc-8b17697d-7849-455d-af68-d08c71604723   2Gi        RWO            csi-hostpath   <unset>                 97s
$ kubectl -n demo describe pvc hp-data | sed -n '/Events/,$p' | grep -E "Type|----|Resiz|FileSystem"
  Type    Reason                      Age                From                                                                           Message
  ----    ------                      ----               ----                                                                           -------
  Normal  Resizing                    32s                external-resizer hostpath.csi.k8s.io                                           External resizer is resizing volume pvc-8b17697d-7849-455d-af68-d08c71604723
  Normal  FileSystemResizeRequired    32s                external-resizer hostpath.csi.k8s.io                                           Require file system resize of volume on node
  Normal  FileSystemResizeSuccessful  9s                 kubelet                                                                        MountVolume.NodeExpandVolume succeeded for volume "pvc-8b17697d-7849-455d-af68-d08c71604723" k8s-node-2
$ kubectl logs csi-hostpathplugin-0 -c hostpath | grep "GRPC call" | grep -i expand
I1029 14:46:37.367330       1 server.go:98] GRPC call: /csi.v1.Controller/ControllerExpandVolume
I1029 14:47:00.332855       1 server.go:98] GRPC call: /csi.v1.Node/NodeExpandVolume

Both steps are visible in the events and in the driver's log: ControllerExpandVolume from external-resizer and 23 seconds later NodeExpandVolume from kubelet, without restarting the pod.

Snapshot and restore

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: hp-data-snap-1
  namespace: demo
spec:
  volumeSnapshotClassName: csi-hostpath-snapclass
  source:
    persistentVolumeClaimName: hp-data
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: hp-data-restored
  namespace: demo
spec:
  storageClassName: csi-hostpath
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 2Gi
  dataSource:
    apiGroup: snapshot.storage.k8s.io
    kind: VolumeSnapshot
    name: hp-data-snap-1

First the snapshot, then a PVC from it and the reader pod with a nodeSelector on k8s-node-2 that mounts it:

$ kubectl -n demo get volumesnapshot
NAME             READYTOUSE   SOURCEPVC   SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS            SNAPSHOTCONTENT                                    CREATIONTIME   AGE
hp-data-snap-1   true         hp-data                             2Gi           csi-hostpath-snapclass   snapcontent-f2003d1e-d2c5-4618-b33b-494453ab263f   8s             8s

$ kubectl get volumesnapshotcontent
NAME                                               READYTOUSE   RESTORESIZE   DELETIONPOLICY   DRIVER                VOLUMESNAPSHOTCLASS      VOLUMESNAPSHOT   VOLUMESNAPSHOTNAMESPACE   AGE
snapcontent-f2003d1e-d2c5-4618-b33b-494453ab263f   true         2147483648    Delete           hostpath.csi.k8s.io   csi-hostpath-snapclass   hp-data-snap-1   demo                      8s
$ kubectl -n demo get pvc
NAME               STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
hp-data            Bound     pvc-8b17697d-7849-455d-af68-d08c71604723   2Gi        RWO            csi-hostpath   <unset>                 2m9s
hp-data-restored   Bound     pvc-acfbd774-c3ae-4bd9-ac78-d85b90acee5e   2Gi        RWO            csi-hostpath   <unset>                 10s
nosc               Pending                                                                                       <unset>                 7m26s
$ kubectl -n demo exec reader -- cat /data/hello.txt
hello
$ kubectl -n demo exec reader -- ls -ln /data
total 4
-rw-r--r--    1 1000     2000             6 Oct 29 14:45 hello.txt
$ kubectl logs csi-hostpathplugin-0 -c hostpath | grep "GRPC call" | grep -E "Snapshot|CreateVolume" | grep -v List
I1029 14:45:35.564950       1 server.go:98] GRPC call: /csi.v1.Controller/CreateVolume
I1029 14:47:23.563740       1 server.go:98] GRPC call: /csi.v1.Controller/CreateSnapshot
I1029 14:47:32.341470       1 server.go:98] GRPC call: /csi.v1.Controller/CreateVolume

The VolumeSnapshot and VolumeSnapshotContent pair mirrors the PVC and PV pair: a request in a namespace and a cluster-wide object with the snapshot itself. The second CreateVolume is the restore: the provisioner passed the snapshot to the driver as the data source, and the new volume contains hello.txt with the same owner and group. In the demo driver's "storage" a snapshot is an archive of the volume's directory:

user@k8s-node-2:~$ sudo ls -la /var/lib/csi-hostpath-data/
total 24
drwxr-xr-x  4 root root 4096 Oct 29 17:47 .
drwxr-xr-x 54 root root 4096 Oct 29 17:42 ..
drwxrwsr-x  2 root 2000 4096 Oct 29 17:45 665e490b-d3a7-11f1-8d4c-52a962bdf80a
-rw-r--r--  1 root root  157 Oct 29 17:47 a6bd954c-d3a7-11f1-8d4c-52a962bdf80a.snap
drwxrwsr-x  2 root 2000 4096 Oct 29 17:45 abf8f915-d3a7-11f1-8d4c-52a962bdf80a
-rw-------  1 root root 1648 Oct 29 17:47 state.json

ReadWriteOncePod and ReadWriteOnce

The single PVC with accessModes: ["ReadWriteOncePod"] and two pods using it, both with a nodeSelector on k8s-node-2:

$ kubectl -n demo get pod single-1 single-2 -o wide
NAME       READY   STATUS    RESTARTS   AGE   IP              NODE         NOMINATED NODE   READINESS GATES
single-1   1/1     Running   0          18s   172.20.172.48   k8s-node-2   <none>           <none>
single-2   0/1     Pending   0          10s   <none>          <none>       <none>           <none>
$ kubectl -n demo get events --field-selector involvedObject.name=single-2 -o custom-columns=REASON:.reason,FROM:.source.component,MESSAGE:.message
REASON             FROM     MESSAGE
FailedScheduling   <none>   0/3 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 1 node(s) had untolerated taint(s), 1 node(s) unavailable due to PersistentVolumeClaim with ReadWriteOncePod access mode already in-use by another pod. no new claims to deallocate, preemption: 0/3 nodes are available: 1 No preemption victims found for incoming pod, 2 Preemption is not helpful for scheduling.

The second pod is not scheduled: RWOP is checked by the scheduler. With plain RWO it is different — a second pod writer-2 with the same hp-data on the same node starts and reads the first one's file:

$ kubectl -n demo logs writer-2
hello
$ kubectl -n demo get pod writer writer-2 -o wide
NAME       READY   STATUS    RESTARTS   AGE     IP              NODE         NOMINATED NODE   READINESS GATES
writer     1/1     Running   0          2m58s   172.20.172.62   k8s-node-2   <none>           <none>
writer-2   1/1     Running   0          8s      172.20.172.4    k8s-node-2   <none>           <none>

A generic ephemeral volume

The scratchpod pod with the work volume from the example in the ephemeral volumes section (and a nodeSelector on k8s-node-2):

$ kubectl -n demo get pvc scratchpod-work
NAME              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
scratchpod-work   Bound    pvc-4b0e7599-23cf-4b29-a0a1-1da7e57e5f8f   1Gi        RWO            csi-hostpath   <unset>                 7s
$ kubectl -n demo get pvc scratchpod-work -o jsonpath='{.metadata.ownerReferences}' | jq -c
[{"apiVersion":"v1","blockOwnerDeletion":true,"controller":true,"kind":"Pod","name":"scratchpod","uid":"d7feaf57-7e0b-4547-91d6-a76c6700972c"}]
$ kubectl -n demo delete pod scratchpod
pod "scratchpod" deleted from demo namespace

Five seconds after the pod was deleted, neither the scratchpod-work PVC nor its PV are in the lists any more: the garbage collector deleted the PVC by its ownerReferences, and the Delete policy took care of the volume.

The default class, retroactively

The nosc PVC has been hanging in Pending without a class all this time. We make csi-hostpath the default class:

$ kubectl annotate storageclass csi-hostpath storageclass.kubernetes.io/is-default-class=true
storageclass.storage.k8s.io/csi-hostpath annotated
$ kubectl get sc
NAME                     PROVISIONER           RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
csi-hostpath (default)   hostpath.csi.k8s.io   Delete          WaitForFirstConsumer   true                   5m22s
$ kubectl -n demo get pvc nosc
NAME   STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
nosc   Pending                                      csi-hostpath   <unset>                 9m22s

20 seconds later the PVC created nine minutes earlier already has csi-hostpath; it is still Pending, but now waiting for its first consumer. This is exactly how data "silently goes" to the default class.

Reclaiming: Delete and Retain

We delete both writer pods and the hp-data PVC. The class has reclaimPolicy: Delete, so the VolumeAttachment, the PV and the data directory disappear:

$ kubectl -n demo delete pod writer writer-2 && kubectl -n demo delete pvc hp-data
pod "writer" deleted from demo namespace
pod "writer-2" deleted from demo namespace
persistentvolumeclaim "hp-data" deleted from demo namespace
$ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                   STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE
pvc-acfbd774-c3ae-4bd9-ac78-d85b90acee5e   2Gi        RWO            Delete           Bound    demo/hp-data-restored   csi-hostpath   <unset>                          2m52s
pvc-b187bcc5-1c6d-456b-ac0d-690d24975e71   1Gi        RWOP           Delete           Bound    demo/single             csi-hostpath   <unset>                          2m29s
$ kubectl logs csi-hostpathplugin-0 -c hostpath --since=2m | grep "GRPC call" | grep -v -E "Probe|GetCapabilities|GetPluginInfo|GetVolumeStats|NodeGetCapabilities|ControllerGetCapabilities"
...
I1029 14:50:18.400566       1 server.go:98] GRPC call: /csi.v1.Node/NodeUnpublishVolume
I1029 14:50:19.103586       1 server.go:98] GRPC call: /csi.v1.Node/NodeUnpublishVolume
I1029 14:50:19.205119       1 server.go:98] GRPC call: /csi.v1.Node/NodeUnstageVolume
I1029 14:50:19.964379       1 server.go:98] GRPC call: /csi.v1.Controller/DeleteVolume

Teardown in reverse order: two NodeUnpublishVolume calls — one per pod, one NodeUnstageVolume — when no pods with the volume are left on the node, and DeleteVolume after the PVC is deleted. The order of the commands matters: while the pod is alive, kubernetes.io/pvc-protection keeps the PVC in Terminating, and the deletion completes only after the pod stops.

Now the same with Retain: we switch the policy on the restored volume and delete its PVC, and then the PV itself:

$ kubectl patch pv pvc-acfbd774-c3ae-4bd9-ac78-d85b90acee5e -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
persistentvolume/pvc-acfbd774-c3ae-4bd9-ac78-d85b90acee5e patched
$ kubectl -n demo delete pod reader && kubectl -n demo delete pvc hp-data-restored
pod "reader" deleted from demo namespace
persistentvolumeclaim "hp-data-restored" deleted from demo namespace
$ kubectl get pv pvc-acfbd774-c3ae-4bd9-ac78-d85b90acee5e
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM                   STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE
pvc-acfbd774-c3ae-4bd9-ac78-d85b90acee5e   2Gi        RWO            Retain           Released   demo/hp-data-restored   csi-hostpath   <unset>                          3m55s
$ kubectl delete pv pvc-acfbd774-c3ae-4bd9-ac78-d85b90acee5e
persistentvolume "pvc-acfbd774-c3ae-4bd9-ac78-d85b90acee5e" deleted
$ kubectl logs csi-hostpathplugin-0 -c hostpath --since=1m | grep "GRPC call" | grep -E "Delete|Unstage|Unpublish"
I1029 14:51:21.579093       1 server.go:98] GRPC call: /csi.v1.Node/NodeUnpublishVolume
I1029 14:51:21.680692       1 server.go:98] GRPC call: /csi.v1.Node/NodeUnstageVolume
user@k8s-node-2:~$ sudo ls -la /var/lib/csi-hostpath-data/
total 24
drwxr-xr-x  4 root root 4096 Oct 29 17:50 .
drwxr-xr-x 54 root root 4096 Oct 29 17:42 ..
-rw-r--r--  1 root root  157 Oct 29 17:47 a6bd954c-d3a7-11f1-8d4c-52a962bdf80a.snap
drwxrwsr-x  2 root 2000 4096 Oct 29 17:45 abf8f915-d3a7-11f1-8d4c-52a962bdf80a
drwxr-xr-x  2 root root 4096 Oct 29 17:47 b98c7c72-d3a7-11f1-8d4c-52a962bdf80a
-rw-------  1 root root 1648 Oct 29 17:51 state.json

There was no DeleteVolume: the PV object is deleted, while the volume directory abf8f915-… stayed in the storage — now it belongs to nobody, and only the administrator can remove it. That is the price of Retain: data is not lost by accident, but it is not cleaned up by itself either.

Cleanup

The driver is removed by the destroy.sh script by the installation labels (StatefulSet, RBAC, CSIDriver), the snapshot-controller and the CRDs by the same manifests with kubectl delete -f:

$ deploy/kubernetes-latest/destroy.sh
...
csidriver.storage.k8s.io "hostpath.csi.k8s.io" deleted
$ kubectl delete volumesnapshotclass csi-hostpath-snapclass --ignore-not-found
volumesnapshotclass.snapshot.storage.k8s.io "csi-hostpath-snapclass" deleted
$ kubectl delete -f setup-snapshot-controller.yaml -f rbac-snapshot-controller.yaml
deployment.apps "snapshot-controller" deleted from kube-system namespace
serviceaccount "snapshot-controller" deleted from kube-system namespace
clusterrole.rbac.authorization.k8s.io "snapshot-controller-runner" deleted
clusterrolebinding.rbac.authorization.k8s.io "snapshot-controller-role" deleted
role.rbac.authorization.k8s.io "snapshot-controller-leaderelection" deleted from kube-system namespace
rolebinding.rbac.authorization.k8s.io "snapshot-controller-leaderelection" deleted from kube-system namespace
$ kubectl delete -f snapshot.storage.k8s.io_volumesnapshots.yaml -f snapshot.storage.k8s.io_volumesnapshotcontents.yaml -f snapshot.storage.k8s.io_volumesnapshotclasses.yaml
customresourcedefinition.apiextensions.k8s.io "volumesnapshots.snapshot.storage.k8s.io" deleted
customresourcedefinition.apiextensions.k8s.io "volumesnapshotcontents.snapshot.storage.k8s.io" deleted
customresourcedefinition.apiextensions.k8s.io "volumesnapshotclasses.snapshot.storage.k8s.io" deleted
$ kubectl get csinode -o custom-columns=NAME:.metadata.name,DRIVERS:.spec.drivers[*].name
NAME           DRIVERS
k8s-master-1   csi.tigera.io
k8s-node-1     csi.tigera.io
k8s-node-2     csi.tigera.io
$ kubectl get node --show-labels | tr "," "\n" | grep topology
topology.hostpath.csi/node=k8s-node-2

kubelet removed the driver from CSINode but left the topology label on the node — we remove it ourselves (kubectl label node k8s-node-2 topology.hostpath.csi/node-). The script also leaves the snapshot-controller-leader Lease in kube-system and the directories on the driver's node: /var/lib/csi-hostpath-data with the orphaned volume, /var/lib/kubelet/plugins/csi-hostpath and /var/lib/kubelet/plugins/kubernetes.io/csi/hostpath.csi.k8s.io are removed by hand, the sidecar images with crictl rmi by exact names.

---

Sources

• Kubernetes: Volumes — volume types, emptyDir and tmpfs, hostPath and its warnings, image, mount propagation.
• Kubernetes: Projected Volumes — projected sources, ServiceAccount token lifetime, clusterTrustBundle.
• Kubernetes: Ephemeral Volumes — generic ephemeral volumes, CSI inline ephemeral volumes.
• Kubernetes: Local ephemeral storage — the node's ephemeral storage and emptyDir limits.
• Kubernetes: Persistent Volumes — binding, phases, finalizers, reclaim policies, access modes, expansion, retroactive default class.
• Kubernetes: Storage Classes and Dynamic Volume Provisioning — StorageClass fields, volumeBindingMode, the default class.
• Kubernetes: Storage Capacity — CSIStorageCapacity and free space in scheduling.
• Kubernetes: Volume Snapshots, Volume Snapshot Classes and Kubernetes v1.36: Moving Volume Group Snapshots to GA — snapshots, restore, group snapshots.
• Kubernetes: Volume Attributes Classes — mutable volume parameters and ModifyVolume.
• Kubernetes: Configure a Security Context for a Pod or Container — fsGroup, fsGroupChangePolicy.
• Kubernetes: Pod Security Standards — hostPath forbidden in the baseline profile.
• Kubernetes CSI Developer Documentation: Sidecar Containers, external-provisioner, external-attacher, external-resizer, external-snapshotter, node-driver-registrar, livenessprobe — what each sidecar does.
• Kubernetes CSI: CSIDriver Object, CSINode Object, Topology and fsGroup Support — CSIDriver fields, fsGroupPolicy, VOLUME_MOUNT_GROUP, topology keys.
• Kubernetes CSI: Snapshot Controller and the external-snapshotter repository (release v8.7.0) — snapshot CRDs and the snapshot-controller.
• csi-driver-host-path (release v1.19.0, installation guide) — the demo driver and its limitations.
• Container Storage Interface Specification — the Identity, Controller, Node, GroupController and SnapshotMetadata services, capabilities and the volume lifecycle.
• gRPC: Introduction — remote procedure calls and Protocol Buffers.
• Linux kernel: Shared Subtrees — mount propagation between mount namespaces.
• Calico: pod2daemon — connecting a pod to the Calico agent on the node through a volume.

---

The storage mechanics in Kubernetes are the same for all drivers: PVC, class, provisioner, attach, stage, publish, permissions through fsGroup. Only what exactly the driver does at each step and where the data lives changes.

Next — an NFS server for the cluster: how to open an export to the nodes correctly and securely, why restricting by node addresses does not protect against pods, and what to use instead of the popular no_root_squash.