Kubernetes for beginners. Part 23: Connecting NFS with csi-driver-nfs
All parts of the series “Kubernetes for beginners”In the previous part we prepared nfs.example.com: NFSv4 only, the /srv/nfs/k8s export on a separate disk, given to the three cluster nodes with no_root_squash, manage-gids=n, a firewall and a Calico policy that keeps pods away from the server. Today it is the client side: csi-driver-nfs, the driver that turns this export into a StorageClass. How a CSI driver is built in general was covered in part 21 (controller and node plugins, external-provisioner, node-driver-registrar, NodePublish); here — what exactly this driver does on the server and on the node, which StorageClass parameters to choose and why, how to set permissions so the application does not run as root, how to turn on encryption in transit, and what happens to pods when the server disappears.
Installation
The driver is developed in the kubernetes-csi organization; the current release at the time of writing is v4.13.6. For every release the project publishes ready manifests in the repository's deploy/v4.13.6/ directory; the install-driver.sh script from the same directory does exactly this — four kubectl apply calls. We apply them ourselves to see what gets installed. The files were downloaded on the workstation and copied to the master into ~/p23:
$ V=v4.13.6
$ for f in rbac-csi-nfs.yaml csi-nfs-driverinfo.yaml csi-nfs-controller.yaml csi-nfs-node.yaml; do curl -sLO https://raw.githubusercontent.com/kubernetes-csi/csi-driver-nfs/$V/deploy/$V/$f; done
$ kubectl apply -f rbac-csi-nfs.yaml -f csi-nfs-driverinfo.yaml -f csi-nfs-controller.yaml -f csi-nfs-node.yaml
serviceaccount/csi-nfs-controller-sa created
serviceaccount/csi-nfs-node-sa created
clusterrole.rbac.authorization.k8s.io/nfs-external-provisioner-role created
clusterrolebinding.rbac.authorization.k8s.io/nfs-csi-provisioner-binding created
clusterrole.rbac.authorization.k8s.io/nfs-external-resizer-role created
clusterrolebinding.rbac.authorization.k8s.io/nfs-csi-resizer-role created
csidriver.storage.k8s.io/nfs.csi.k8s.io created
deployment.apps/csi-nfs-controller created
daemonset.apps/csi-nfs-node created
The version in the URL is given explicitly: the unversioned deploy/ directory on the master branch changes with every commit, and two clusters deployed a month apart would get different drivers. The same driver can also be installed with the Helm chart from the repository (helm install csi-driver-nfs csi-driver-nfs/csi-driver-nfs --namespace kube-system --version 4.13.6; since 4.11.0 the chart version has no v); Helm is the Kubernetes package manager, covered in part 27.
Snapshots need CRDs and snapshot-controller — the same ones as in part 21, from the external-snapshotter v8.7.0 release; there we removed them after the experiments, so we install them again with the same commands:
$ V=v8.7.0
$ for f in client/config/crd/snapshot.storage.k8s.io_volumesnapshotclasses.yaml client/config/crd/snapshot.storage.k8s.io_volumesnapshotcontents.yaml client/config/crd/snapshot.storage.k8s.io_volumesnapshots.yaml deploy/kubernetes/snapshot-controller/rbac-snapshot-controller.yaml deploy/kubernetes/snapshot-controller/setup-snapshot-controller.yaml; do curl -sLO https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/$V/$f; done
$ kubectl apply -f snapshot.storage.k8s.io_volumesnapshotclasses.yaml -f snapshot.storage.k8s.io_volumesnapshotcontents.yaml -f snapshot.storage.k8s.io_volumesnapshots.yaml
customresourcedefinition.apiextensions.k8s.io/volumesnapshotclasses.snapshot.storage.k8s.io created
customresourcedefinition.apiextensions.k8s.io/volumesnapshotcontents.snapshot.storage.k8s.io created
customresourcedefinition.apiextensions.k8s.io/volumesnapshots.snapshot.storage.k8s.io created
$ kubectl apply -f rbac-snapshot-controller.yaml -f setup-snapshot-controller.yaml
serviceaccount/snapshot-controller created
clusterrole.rbac.authorization.k8s.io/snapshot-controller-runner created
clusterrolebinding.rbac.authorization.k8s.io/snapshot-controller-role created
role.rbac.authorization.k8s.io/snapshot-controller-leaderelection created
rolebinding.rbac.authorization.k8s.io/snapshot-controller-leaderelection created
deployment.apps/snapshot-controller created
The driver's repository has its own snapshot CRD manifests too (crd-csi-snapshot.yaml), but they are noticeably older than the external-snapshotter release; if you do not need snapshots, skip this step. What appeared in the cluster:
$ kubectl -n kube-system get pods -l 'app in (csi-nfs-controller,csi-nfs-node)' -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
csi-nfs-controller-d5b8c6f76-tsdm7 5/5 Running 2 (14s ago) 2m31s 192.168.10.20 k8s-node-1 <none> <none>
csi-nfs-node-5w8tv 3/3 Running 0 2m31s 192.168.10.20 k8s-node-1 <none> <none>
csi-nfs-node-b592c 3/3 Running 0 2m31s 192.168.10.21 k8s-node-2 <none> <none>
csi-nfs-node-dht5v 3/3 Running 0 2m31s 192.168.10.17 k8s-master-1 <none> <none>
$ kubectl get csidriver nfs.csi.k8s.io -o yaml | sed -n '/^spec:/,$p'
spec:
attachRequired: false
fsGroupPolicy: File
podInfoOnMount: false
requiresRepublish: false
seLinuxMount: false
storageCapacity: false
volumeLifecycleModes:
- Persistent
csi-nfs-controller— a Deployment with one replica and five containers:csi-provisioner(external-provisioner v6.3.0, watches PVCs and callsCreateVolume/DeleteVolume),csi-resizer(volume expansion),csi-snapshotter(snapshots),liveness-probeand the driver itself,nfs(imagenfsplugin:v4.13.6). The pod runs inhostNetwork: its address is the node's address (the IP column), so the server lets it in by the rule for the nodes, and the Calico policy from the previous part does not see it. The controller has tolerations for the master's taints, so it may also land onk8s-master-1— which is why the master is in/etc/exports. The two restarts on first start are not an error: the sidecar containers started before the driver image was pulled, did not get its socket within the allotted time and were restarted by kubelet.csi-nfs-node— a DaemonSet on every node, the master included (it tolerates any taints), also inhostNetwork, with three containers:node-driver-registrarregisters the driver with kubelet through the socket/var/lib/kubelet/plugins_registry/nfs.csi.k8s.io-reg.sock,liveness-probe, andnfs, which runsmounton kubelet's command. The mount utilities (mount.nfsfrom thenfs-commonpackage, Debian 12) live in the driver image, so the nodes themselves do not need thenfs-commonpackage.- CSIDriver — the object through which kubelet learns the driver's properties.
attachRequired: false: NFS has no step of attaching a disk to a node, so there will be no VolumeAttachment objects (the record "volume attached to such and such node").fsGroupPolicy: File— kubelet itself changes the group of the volume's files to the pod'sfsGroup. The remaining fields are defaults filled in by the API server.
What happens when a volume is created
PVC demo/data ──► csi-provisioner ──► CreateVolume (nfs, controller)
│
│ mount nfs.example.com:/srv/nfs/k8s → /tmp/pvc-…
│ mkdir /tmp/pvc-…/demo-data-pvc-…
│ umount
▼
PV pvc-… (volumeHandle, server, share, subDir) ── Bound ── PVC
Pod on k8s-node-1 ──► kubelet ──► NodePublishVolume (nfs, csi-nfs-node)
│
│ mount -t nfs nfs.example.com:/srv/nfs/k8s/demo-data-pvc-…
│ /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~csi/pvc-…/mount
▼
kubelet: chgrp/chmod for fsGroup, bind into the container
The controller does not keep the export mounted: for every CreateVolume it mounts the export root into a temporary directory inside its container — with the same mountOptions as in the StorageClass — creates the volume subdirectory as root (which is why no_root_squash is needed) and unmounts. This is how it looks in the log of the controller's nfs container (-v=5 in the manifest):
$ kubectl -n kube-system logs deploy/csi-nfs-controller -c nfs | grep -A3 'Controller/CreateVolume'
I1030 11:29:45.243675 1 utils.go:121] GRPC call: /csi.v1.Controller/CreateVolume
I1030 11:29:45.243709 1 utils.go:122] GRPC request: {"capacity_range":{"required_bytes":5368709120},"name":"pvc-9640d114-1423-4c1e-9a9f-144d406e860a","parameters":{"csi.storage.k8s.io/pv/name":"pvc-9640d114-1423-4c1e-9a9f-144d406e860a","csi.storage.k8s.io/pvc/name":"data","csi.storage.k8s.io/pvc/namespace":"demo","onDelete":"archive","server":"nfs.example.com","share":"/srv/nfs/k8s","subDir":"${pvc.metadata.namespace}-${pvc.metadata.name}-${pv.metadata.name}"},"volume_capabilities":[{"AccessType":{"Mount":{"mount_flags":["nfsvers=4.2","hard","timeo=600","retrans=3","nconnect=4","nosuid","nodev"]}},"access_mode":{"mode":5}}]}
I1030 11:29:45.243918 1 controllerserver.go:570] internally mounting nfs.example.com:/srv/nfs/k8s at /tmp/pvc-9640d114-1423-4c1e-9a9f-144d406e860a
I1030 11:29:45.244041 1 nodeserver.go:176] NodePublishVolume: volumeID(nfs.example.com#srv/nfs/k8s#demo-data-pvc-9640d114-1423-4c1e-9a9f-144d406e860a#pvc-9640d114-1423-4c1e-9a9f-144d406e860a#archive) source(nfs.example.com:/srv/nfs/k8s) targetPath(/tmp/pvc-9640d114-1423-4c1e-9a9f-144d406e860a) mountflags([nfsvers=4.2 hard timeo=600 retrans=3 nconnect=4 nosuid nodev])
Internally the controller mounts the export with the same function as the node plugin (hence NodePublishVolume in the controller's log), and half a second later it unmounts and removes the temporary directory. access_mode 5 is MULTI_NODE_MULTI_WRITER, that is, RWX. DeleteVolume does the same, but with rm -rf or a rename of the directory.
On the node the driver has no NodeStage step: every pod gets its own mount of the subdirectory directly into the pod's volume directory (kubelet still creates an empty globalmount directory under /var/lib/kubelet/plugins/kubernetes.io/csi/nfs.csi.k8s.io/, as with the hostpath driver in part 21). The kernel does not open new TCP connections for every mount: all mounts of one server on a node share one NFS client and its connections — we will come back to this, because this economy has a flip side.
StorageClass
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: nfs
provisioner: nfs.csi.k8s.io
parameters:
server: nfs.example.com
share: /srv/nfs/k8s
subDir: ${pvc.metadata.namespace}-${pvc.metadata.name}-${pv.metadata.name}
onDelete: archive
reclaimPolicy: Retain
volumeBindingMode: Immediate
allowVolumeExpansion: true
mountOptions:
- nfsvers=4.2
- hard
- timeo=600
- retrans=3
- nconnect=4
- nosuid
- nodev
server and share
server is the server's name or address, share is the export path. A name is more convenient than an address: if the server moves, one DNS record changes. But the name ends up in every PV, and both plugins of the driver resolve it: the controller and the node plugin both run in the node's network with dnsPolicy: ClusterFirstWithHostNet, that is, they ask the cluster's CoreDNS (the nfs container's /etc/resolv.conf has nameserver 10.96.0.10 and the search domains kube-system.svc.cluster.local, svc.cluster.local, cluster.local). mount.nfs in the container resolves the name and hands the kernel an address. CoreDNS forwards external names to the node's DNS servers (part 16), so the server's name has to resolve there; and if CoreDNS is unavailable, new volumes are neither created nor mounted, even when the NFS server itself is fine.
subDir: readable and unique
Without subDir the driver names the volume directory after the PV: pvc-9640d114-…. Looking at the server, there is no way to tell whose data it is. The templates ${pvc.metadata.namespace}, ${pvc.metadata.name} and ${pv.metadata.name} make names readable, but there is a trap. If the directory already exists, the driver takes it without an error. So with subDir: ${pvc.metadata.namespace}-${pvc.metadata.name} and the directory kept after deletion, a new PVC with the same name in the same namespace gets the old data — possibly another application's data, if the namespace has been handed to another team in the meantime. Checked with a class with such a subDir and onDelete: retain: a pod wrote a file, the PVC was deleted and created again with the same name — and the new pod read the old file, although its PV was new. That is why the template always contains ${pv.metadata.name}: the PV name contains the claim's UID and never repeats. The result is demo-data-pvc-9640d114-… — you can see whose it is, and it will not collide with an old one.
The directory is flat, with hyphens, not demo/data/…. With onDelete: delete the driver removes the emptied parent directories after deleting the directory, but with archive it renames a nested path to archived-demo/data/…, and empty demo/data/ stays on the server; a flat name leaves no such leftovers.
onDelete and reclaimPolicy
These are two different levels.
reclaimPolicy— the Kubernetes level: what to do with the PV when the PVC is deleted.Delete— external-provisioner calls the driver'sDeleteVolumeand deletes the PV.Retain— the PV staysReleased, nobody calls the driver, the data is not touched; the administrator decides what happens next.onDelete— the driver level: whatDeleteVolumedoes with the directory.delete(the default) wipes it,retainkeeps it,archiverenames it toarchived-<subDir>.
An important consequence follows: with reclaimPolicy: Retain the onDelete parameter never fires, even if the administrator later deletes the PV by hand — DeleteVolume is called only for PVs with the Delete policy. That is why onDelete is written into every PV's volumeHandle (you can see it below, in "Hands-on"): it is needed when the administrator, after checking, switches a released PV to Delete so that the driver cleans up by itself. With archive nothing is wiped even then — the directory gets a clear name and can be removed by hand once it is definitely not needed.
For valuable data — reclaimPolicy: Retain: an accidental kubectl delete pvc, or a PVC deleted together with its namespace, destroys nothing. onDelete: archive is a second safety belt for the case when the PV is deleted through Delete after all. For temporary data (caches, intermediate files) — a separate nfs-scratch class with Delete and onDelete: delete, otherwise the server fills up with junk.
The reclaimPolicy of an existing PV can be changed on the fly — this is how you rescue a volume created by a class with Delete before a risky operation:
$ kubectl patch pv pvc-9640d114-1423-4c1e-9a9f-144d406e860a \
-p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
persistentvolume/pvc-9640d114-1423-4c1e-9a9f-144d406e860a patched (no change)
Our PV already has Retain from the class, hence (no change); a PV from a class with Delete will simply say patched.
mountPermissions: not 0777
The driver creates the volume directory as root, and its permissions end up as 0755 root:root. The mountPermissions parameter (default 0 — do nothing) makes the driver chmod the directory: when creating the volume and after every mount. Many examples set mountPermissions: "0777" so that an application not running as root can write. It works, but then any UID from any node can write to the volume. The right way is the pod's fsGroup (below): the volume's group becomes the application's group, and only that group gets the permissions. For pods without fsGroup the driver documentation also has uid and gid parameters: the controller chowns the new directory in CreateVolume; the pod's fsGroup, if set, still overwrites the group.
mountOptions
The options from the StorageClass are copied into every new PV (spec.mountOptions) and passed to mount on the node as is — both when the volume is mounted for a pod and for the controller's internal mount.
nfsvers=4.2— the version, explicitly. Without it the client negotiates by itself starting from the newest, but an explicit version makes the behaviour predictable and immediately shows an error if the server does not support it.hard— when the server is unavailable, requests are retried forever and the process waits.soft— afterretransretries the request fails withEIO.softlooks more convenient ("it won't hang"), but applications rarely expect an error onwriteorclose: the file ends up half-written while the application thinks everything is fine. Thenfs(5)documentation warns plainly thatsoftcan lead to silent data corruption. For data volumes —hardonly; what happens with it during an outage is shown below.timeo=600— the time to wait for a reply before the first retry, in tenths of a second (60 s, the default for TCP); after each retry the interval grows by anothertimeo, up to 600 seconds.retrans=3— how many retries before theserver not respondingmessage in the kernel log (the TCP default is 2; withhardthe request keeps being retried after that). Loweringtimeoon TCP is not worth it: TCP retransmits lost segments by itself, and a short RPC timeout only adds extra retries.nconnect=4— the number of TCP connections to the server (up to 16). One connection is limited by one queue and one CPU core on the server; several spread the load. For 1–10 Gbit/s, 2–4 are enough.nosuid,nodev— do not honour setuid files and do not open device files from this volume. Applications on a data volume do not need this, and a file like that written by one pod must not give privileges to another.
Connection options are shared per node. The kernel has one NFS client per "node — server" pair, and all mounts of that server on the node go through its connections. So the options that belong to the connection — nconnect and xprtsec (encryption, see below) — are set by the first mount, and later ones silently join it, whatever their own options say. On the test bench, a static volume without nconnect in its options, mounted on a node that already had a volume of this class, showed nconnect=4 in findmnt; and a mount without xprtsec that came after an encrypted one showed xprtsec=tls. Mount-level options (ro, retrans, noatime, nosuid) are each mount's own. The takeaway: the connection options must match in all classes and static PVs of one server, otherwise a node's behaviour depends on which pod arrived on it first.
What can be changed in a StorageClass
The provisioner, parameters, reclaimPolicy and volumeBindingMode fields of an existing StorageClass are immutable:
$ kubectl patch sc nfs -p '{"parameters":{"onDelete":"delete"}}'
The StorageClass "nfs" is invalid: parameters: Invalid value: {"onDelete":"delete","server":"nfs.example.com","share":"/srv/nfs/k8s","subDir":"${pvc.metadata.namespace}-${pvc.metadata.name}-${pv.metadata.name}"}: field is immutable
$ kubectl patch sc nfs -p '{"reclaimPolicy":"Delete"}'
The StorageClass "nfs" is invalid: reclaimPolicy: Invalid value: "Delete": field is immutable
They are changed like this: delete the class and create it again under the same name (existing PVs are not affected — they have their own copy of the parameters), or create a new class with a new name and move applications to it. The class's mountOptions and allowVolumeExpansion, however, can be changed with an ordinary kubectl patch or apply — but only for future PVs. New options are added to existing PVs separately; a PV's spec.mountOptions field is mutable:
$ kubectl patch pv pvc-9640d114-1423-4c1e-9a9f-144d406e860a --type=json \
-p '[{"op":"replace","path":"/spec/mountOptions","value":["nfsvers=4.2","hard","timeo=600","retrans=3","nconnect=4","nosuid","nodev","noatime"]}]'
persistentvolume/pvc-9640d114-1423-4c1e-9a9f-144d406e860a patched
Options take effect only on a new mount. Every pod mounts the volume itself, so it is enough to recreate the pods that use it (kubectl rollout restart): on the test bench, after a Deployment restart the new pod had noatime in findmnt, while the old one, not yet terminated, still had relatime. The exception is the connection options from the paragraph above: a new nconnect or xprtsec takes effect on a node only once no mount of that server is left there, that is, after a kubectl drain of the node.
A volume for an application, and permissions
A PVC and a Deployment of two replicas that write to one volume, in the demo namespace from part 11. The application here is busybox, which every 10 seconds appends the time and its pod's name to a file: the distroless hello image has neither a shell nor any disk writes, and we need to see who writes and how.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: data
namespace: demo
spec:
accessModes: ["ReadWriteMany"]
storageClassName: nfs
resources:
requests:
storage: 5Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: writer
namespace: demo
spec:
replicas: 2
selector:
matchLabels:
app.kubernetes.io/name: writer
template:
metadata:
labels:
app.kubernetes.io/name: writer
spec:
securityContext:
runAsNonRoot: true
runAsUser: 10001
runAsGroup: 10001
fsGroup: 10001
fsGroupChangePolicy: OnRootMismatch
seccompProfile:
type: RuntimeDefault
containers:
- name: writer
image: busybox:1.37
command: ["sh", "-c", "while true; do echo \"$(date -u +%FT%TZ) $HOSTNAME\" >> /data/visits.log; sleep 10; done"]
volumeMounts:
- name: data
mountPath: /data
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
volumes:
- name: data
persistentVolumeClaim:
claimName: data
$ kubectl apply -f writer.yaml
persistentvolumeclaim/data created
deployment.apps/writer created
$ kubectl -n demo get pvc data -w
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
data Pending nfs <unset> 0s
data Pending pvc-9640d114-1423-4c1e-9a9f-144d406e860a 0 nfs <unset> 1s
data Bound pvc-9640d114-1423-4c1e-9a9f-144d406e860a 5Gi RWX nfs <unset> 2s
$ kubectl -n demo get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
writer-7d568d48b4-jwtsz 1/1 Running 0 46s 172.20.211.48 k8s-node-1 <none> <none>
writer-7d568d48b4-k68j9 1/1 Running 0 46s 172.20.172.1 k8s-node-2 <none> <none>
The volume was created in two seconds, and the replicas went to the two workers.
ReadWriteMany (RWX) — the volume can be mounted read-write on many nodes at once; NFS does this naturally, and both replicas see the same files. But NFS does not coordinate for the application: if two replicas write the same file, they decide the order of writes themselves (with locks or by splitting files). Here both append lines at the end, and the lines interleave:
user@nfs:~$ head -3 /srv/nfs/k8s/demo-data-pvc-9640d114-1423-4c1e-9a9f-144d406e860a/visits.log
2026-10-30T11:29:52Z writer-7d568d48b4-jwtsz
2026-10-30T11:29:54Z writer-7d568d48b4-k68j9
2026-10-30T11:30:03Z writer-7d568d48b4-jwtsz
runAsUser and runAsGroup — the process runs as UID and GID 10001; on the server its files belong to 10001 (numeric UIDs, as in the previous part). fsGroup: 10001 — when mounting, kubelet as root changes the group of the volume root and of all files in it to 10001, gives the group read and write permissions and sets the setgid bit on directories so that new files inherit the group; on top of that the process gets 10001 among its supplementary groups. The result is a drwxrwsr-x root 10001 directory, and the application writes to it without being root and without 0777. This works under three conditions: the driver declared fsGroupPolicy: File; the export has no_root_squash; the server has manage-gids=n (otherwise, as we saw in the previous part, the server replaces the client's groups with its own, and group 10001 never reaches it).
What happens with root_squash was checked by temporarily switching the export: pod fsg (runAsUser, runAsGroup and fsGroup 20002, a volume of the same class mounted at /v) on k8s-node-2. kubelet's chown reaches the server as nobody and is rejected. kubelet writes the error to its log and starts the pod anyway — without write access to the volume:
user@k8s-node-2:~$ sudo journalctl -u kubelet | grep 'Lchown failed' | tail -1
Oct 30 11:36:13 k8s-node-2 kubelet[1270]: E1030 11:36:13.884814 1270 volume_linux.go:151] "Lchown failed" err="lchown /var/lib/kubelet/pods/e0f56626-56d0-4950-a987-d9e95513a941/volumes/kubernetes.io~csi/pvc-05a5de9e-5819-48b1-847c-c878043f7579/mount: operation not permitted" path="/var/lib/kubelet/pods/e0f56626-56d0-4950-a987-d9e95513a941/volumes/kubernetes.io~csi/pvc-05a5de9e-5819-48b1-847c-c878043f7579/mount"
$ kubectl -n demo exec fsg -- sh -c 'ls -lnd /v; echo x >> /v/visits.log'
drwxr-xr-x 2 0 0 24 Oct 30 11:29 /v
sh: can't create /v/visits.log: Permission denied
command terminated with exit code 1
The pod is Running, there are no events — the error is visible only in kubelet's log on the node and in the application failing. New volumes are not created with root_squash either: the controller's mkdir in the 0755 root:root root runs as nobody, and the PVC hangs in Pending with a ProvisioningFailed event: failed to make subdirectory: mkdir /tmp/pvc-…/demo-squashed-pvc-…: permission denied.
fsGroupChangePolicy: OnRootMismatch — without it (the value Always) kubelet walks the whole volume recursively on every mount, that is, on every pod start. On an NFS volume with a million files this takes minutes, during which the pod sits in ContainerCreating. With OnRootMismatch kubelet looks only at the volume root: if the group and permissions are already right, there is no walk. The flip side: files created inside the volume with a different group (for example, by hand on the server) will not be fixed. And one more thing: if pods with different fsGroup values mount one RWX volume, they will keep pulling the group from each other; one volume should have one group.
The PVC size is not a quota. storage: 5Gi is written into the PV but limits nothing: the application can write 50 GiB as long as the server's disk has room. df in the pod shows the whole export, not 5 GiB:
$ kubectl -n demo exec deploy/writer -- df -h /data
Filesystem Size Used Available Use% Mounted on
nfs.example.com:/srv/nfs/k8s/demo-data-pvc-9640d114-1423-4c1e-9a9f-144d406e860a
15.9G 344.8M 15.6G 2% /data
Real limits come from file system quotas on the server (the XFS project quotas from the previous part) and from monitoring used space.
A static PV for existing data
If the server already has data — a photo archive, reference data — it is attached without the controller: the administrator writes the PV by hand. The previous part suggested a separate export for such data, /srv/nfs/static, given to the workers read-only and with root_squash:
apiVersion: v1
kind: PersistentVolume
metadata:
name: photos-archive
spec:
capacity:
storage: 100Gi
accessModes: ["ReadOnlyMany"]
persistentVolumeReclaimPolicy: Retain
storageClassName: ""
mountOptions: ["nfsvers=4.2", "hard", "nosuid", "nodev", "ro"]
csi:
driver: nfs.csi.k8s.io
volumeHandle: nfs.example.com#srv/nfs/static/photos
readOnly: true
volumeAttributes:
server: nfs.example.com
share: /srv/nfs/static/photos
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: photos-archive
namespace: demo
spec:
accessModes: ["ReadOnlyMany"]
storageClassName: ""
volumeName: photos-archive
resources:
requests:
storage: 100Gi
volumeHandle is the volume's identifier, unique in the cluster; the driver suggests the format server#path#subdirectory, but for a static volume with Retain it is just a label: nobody will ever call DeleteVolume for it. kubelet treats two PVs with the same volumeHandle as one volume, so identifiers must not repeat. storageClassName: "" in both objects and volumeName in the PVC bind them directly, without dynamic provisioning. persistentVolumeReclaimPolicy: Retain is mandatory: nobody should delete a static volume through Kubernetes. Read-only appears three times here: the access mode, readOnly, ro in the options — and a fourth time on the server, in the export. The export is given only to the workers, so a pod with such a volume has to run on a worker.
A pod with this volume (readOnly: true both in volumeMounts and in persistentVolumeClaim) on k8s-node-2:
$ kubectl get pv photos-archive
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE
photos-archive 100Gi ROX Retain Bound demo/photos-archive <unset> 16s
$ kubectl -n demo exec photos -- sh -c 'cat /photos/0001.txt; touch /photos/x'
photo-0001
touch: /photos/x: Read-only file system
command terminated with exit code 1
user@k8s-node-2:~$ findmnt -t nfs4 -n -o TARGET,SOURCE,OPTIONS
/var/lib/kubelet/pods/907fd370-7c5e-4aad-a533-9fdf1235e845/volumes/kubernetes.io~csi/photos-archive/mount nfs.example.com:/srv/nfs/static/photos ro,nosuid,nodev,relatime,vers=4.2,rsize=262144,wsize=262144,namlen=255,hard,fatal_neterrors=none,proto=tcp,nconnect=4,timeo=600,retrans=2,sec=sys,clientaddr=192.168.10.21,local_lock=none,addr=192.168.10.30
/var/lib/kubelet/pods/82a68469-6370-4735-a128-2ba0766baca4/volumes/kubernetes.io~csi/pvc-9640d114-1423-4c1e-9a9f-144d406e860a/mount nfs.example.com:/srv/nfs/k8s/demo-data-pvc-9640d114-1423-4c1e-9a9f-144d406e860a rw,nosuid,nodev,noatime,vers=4.2,rsize=262144,wsize=262144,namlen=255,hard,fatal_neterrors=none,proto=tcp,nconnect=4,timeo=600,retrans=3,sec=sys,clientaddr=192.168.10.21,local_lock=none,addr=192.168.10.30
Here you can see what the options section talked about: the static volume has no nconnect in its options, yet findmnt shows it — the volume joined the NFS client that had already opened four connections for the writer volume. Its retrans=2 (the default) and ro, though, are its own.
Snapshots, clones and expansion
A snapshot in this driver is not an instant file system snapshot but an archive: on CreateSnapshot the controller packs the volume directory into a .tar.gz on the same server. This needs the CRDs and snapshot-controller (installed above) and a snapshot class:
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
name: nfs
driver: nfs.csi.k8s.io
deletionPolicy: Delete
---
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
name: data-2026-10-30
namespace: demo
spec:
volumeSnapshotClassName: nfs
source:
persistentVolumeClaimName: data
$ kubectl apply -f snap.yaml
volumesnapshotclass.snapshot.storage.k8s.io/nfs created
volumesnapshot.snapshot.storage.k8s.io/data-2026-10-30 created
$ kubectl -n demo get volumesnapshot
NAME READYTOUSE SOURCEPVC SOURCESNAPSHOTCONTENT RESTORESIZE SNAPSHOTCLASS SNAPSHOTCONTENT CREATIONTIME AGE
data-2026-10-30 true data 421 nfs snapcontent-568b66d7-402c-48ca-9ac1-d8f567956142 0s 1s
user@nfs:~$ sudo ls -l /srv/nfs/k8s/snapshot-568b66d7-402c-48ca-9ac1-d8f567956142
total 4
-rw-r--r-- 1 root root 421 Oct 30 11:34 pvc-9640d114-1423-4c1e-9a9f-144d406e860a.tar.gz
The archive lives in a snapshot-<uid> directory next to the volumes, and RESTORESIZE is its size in bytes (421), not the volume size. deletionPolicy: Delete means that deleting the VolumeSnapshot deletes the archive too; with Retain the VolumeSnapshotContent object and the file on the server stay.
Restoring is a new PVC with a dataSource pointing at the VolumeSnapshot; the driver unpacks the archive into the new volume's directory:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: data-restored
namespace: demo
spec:
accessModes: ["ReadWriteMany"]
storageClassName: nfs
dataSource:
apiGroup: snapshot.storage.k8s.io
kind: VolumeSnapshot
name: data-2026-10-30
resources:
requests:
storage: 10Gi
$ kubectl -n demo get pvc data-restored
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
data-restored Bound pvc-05a5de9e-5819-48b1-847c-c878043f7579 10Gi RWX nfs <unset> 2s
user@nfs:~$ ls -ln /srv/nfs/k8s/demo-data-restored-pvc-05a5de9e-5819-48b1-847c-c878043f7579
total 4
-rw-r--r-- 1 0 0 2970 Oct 30 11:35 visits.log
The restored files are owned by root (0 0), not 10001 as in the source volume: the controller unpacks the archive as root. A pod with fsGroup: 10001 brings back the group and the group's write permission (-rw-rw-r-- 0 10001), but the owner stays root; if the application needs the owner specifically, it is set with the class's uid/gid parameters or by hand.
Consequences of this design. The archive takes a long time to create, proportional to the volume, not instantly. It is not consistent: if the application writes during packing, the archive gets a mix of old and new files, so take snapshots with writes stopped. And it sits on the same server and the same disk — protection against an application mistake, but not a backup.
A clone is a PVC with a dataSource pointing at another PVC of the same class: the driver declares this capability (CLONE_VOLUME in the ControllerGetCapabilities response) and copies the source volume's directory into the new one. The limitations are the same as for a snapshot: the copy is neither instant nor consistent.
Volume expansion (allowVolumeExpansion: true, increase storage in the PVC) is instant: the driver's ControllerExpandVolume only returns the new size, nothing changes on disk — there was no limit to begin with, and there is no node step (NodeExpandVolume):
$ kubectl -n demo patch pvc data -p '{"spec":{"resources":{"requests":{"storage":"10Gi"}}}}'
persistentvolumeclaim/data patched
$ kubectl -n demo describe pvc data | tail -3
Normal ExternalExpanding 5s volume_expand waiting for an external controller to expand this PVC
Normal Resizing 5s external-resizer nfs.csi.k8s.io External resizer is resizing volume pvc-9640d114-1423-4c1e-9a9f-144d406e860a
Normal VolumeResizeSuccessful 5s external-resizer nfs.csi.k8s.io Resize volume succeeded
Encryption in transit: xprtsec through mountOptions
In the previous part NFS over TLS worked with a manual mount, and the question remained whether it goes through the driver. It does, under three conditions that were checked on the test bench.
1. tlshd is needed on every node. The driver's nfs container does the mounting, but the kernel hands the TLS handshake to the tlshd service over netlink in its own network namespace. The driver's plugins run in hostNetwork, that is, in the node's network namespace, and reach the node's tlshd. So every node needs the ktls-utils package, the CA certificate in /etc/tlshd/config ([authenticate.client], x509.truststore=, as in the previous part) and a running tlshd, and the class's mountOptions need xprtsec=tls. Without tlshd the mount fails with an error that is puzzling at first sight:
Warning FailedMount 13s (x6 over 29s) kubelet MountVolume.SetUp failed for volume "photos-tls" : rpc error: code = Internal desc = mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t nfs -o nfsvers=4.2,hard,nosuid,nodev,ro,xprtsec=tls,ro nfs.example.com:/srv/nfs/static/photos /var/lib/kubelet/pods/fba28700-48d9-40a0-916a-49a97d702c46/volumes/kubernetes.io~csi/photos-tls/mount
Output: mount.nfs: No such process
No such process means the kernel found nobody to hand the handshake to. With tlshd, its log shows Handshake with '…' succeeded, and findmnt shows xprtsec=tls. If the export requires TLS but the volume's options have no xprtsec, the server answers the same way as with a manual mount: mount.nfs: Operation not permitted.
2. The controller mounts too. On CreateVolume the csi-nfs-controller pod mounts the export with the class options, so tlshd is also needed on the node where it runs (and it can run on any of them, the master included). Deletion and snapshots are worse: the CSI specification passes no mount options in DeleteVolume and CreateSnapshot requests, and the driver mounts the export with no options at all. A server that requires TLS rejects that, and a PV with the Delete policy gets stuck in Released:
Warning VolumeFailedDelete 9s nfs.csi.k8s.io_k8s-node-1_fb0a5b52-01af-4dae-9c5c-bae0ce50e165 rpc error: code = Internal desc = failed to mount nfs server: rpc error: code = Internal desc = mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t nfs nfs.example.com:/srv/nfs/k8s /tmp/pvc-0380c1d0-e878-4a4e-8081-f950749307e5
Output: mount.nfs: rpc.statd is not running but is required for remote locking.
mount.nfs: Either use '-o nolock' to keep locks local, or start statd.
mount.nfs: Operation not permitted
Without a version mount.nfs first tried NFSv4 (refused), then fell back to NFSv3 — hence the lines about rpc.statd. The workaround is described in the driver documentation: a Secret with the options and a reference to it in the class parameters.
apiVersion: v1
kind: Secret
metadata:
name: nfs-mount-options
namespace: kube-system
stringData:
mountOptions: "nfsvers=4.2,xprtsec=tls"
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: nfs-tls
provisioner: nfs.csi.k8s.io
parameters:
server: nfs.example.com
share: /srv/nfs/k8s
subDir: ${pvc.metadata.namespace}-${pvc.metadata.name}-${pv.metadata.name}
onDelete: archive
csi.storage.k8s.io/provisioner-secret-name: nfs-mount-options
csi.storage.k8s.io/provisioner-secret-namespace: kube-system
reclaimPolicy: Retain
allowVolumeExpansion: true
mountOptions: ["nfsvers=4.2", "hard", "timeo=600", "retrans=3", "nconnect=4", "nosuid", "nodev", "xprtsec=tls"]
external-provisioner writes the Secret's name into the PV's annotations (volume.kubernetes.io/provisioner-deletion-secret-name and …-namespace) and passes it to the driver on deletion, and the driver mounts with nfsvers=4.2,xprtsec=tls. On the test bench this was checked with a class with reclaimPolicy: Delete and onDelete: delete: the controller's log shows Mounting cmd (mount) with arguments (-t nfs -o nfsvers=4.2,xprtsec=tls …), then removing subdirectory, and the PV was gone. With Retain, DeleteVolume is reached only when the administrator switches the PV to Delete — and then the Secret is needed as well. The annotation is set when the PV is created, so for volumes created by a class without the Secret the workaround does not apply retroactively. For snapshots the same is done with the mountOptions parameter of the VolumeSnapshotClass (or the csi.storage.k8s.io/snapshotter-secret-name Secret).
3. TLS for all volumes of a node at once. Because of the shared NFS client (see "Connection options are shared per node"), a node cannot keep both encrypted and plain mounts to one server: whatever arrived first defines the connection. So TLS is turned on for a whole node: xprtsec=tls in the node's line in /etc/exports, in all classes and static PVs of this server and in the Secret. On a running cluster the move goes node by node: kubectl drain, edit the node's line in /etc/exports, exportfs -ra, kubectl uncordon; the classes and PVs with xprtsec=tls must be ready before the first such node.
xprtsec=mtls (the client presents a certificate too) works the same way, only the node's /etc/tlshd/config also gets the node's certificate and key; it was not checked on the test bench. Kerberos (sec=krb5p) is also passed through the driver as a mount option, but it needs a keytab and rpc.gssd on every node and tickets for processes in pods — not checked on the test bench.
Locks and databases
NFSv4 supports byte-range locks in the protocol: the client sends an fcntl lock taken in a pod to the server with the LOCK operation, and all clients see it. Locks are tied to the client's lease (90 seconds); if a node lost contact with the server for longer than the lease, the server releases its locks, and after the connection comes back the process gets an error on file operations. On NFS in Linux, flock is emulated with the same byte-range lock on the whole file (local_lock=none in the mount options), so, unlike on a local disk, flock and fcntl on the same file conflict with each other. After a server restart there is a grace period (also 90 seconds), and new locks and opens wait for a while.
This makes it clear why databases are not put on NFS:
- Prometheus says in its documentation that non-POSIX-compliant file systems, NFS included, are not supported for its local storage and that unrecoverable corruption may happen: its time series database (TSDB) relies on the semantics of a local file system.
- etcd — every change waits for
fsync, and disk write latency directly defines the latency of the whole cluster; the etcd documentation requires a fast local disk, and a network layer adds latency to everyfsync. More in part 34. - PostgreSQL officially allows a data directory on NFS when mounted
hardand with synchronous writes on the server, but warns about the risks. In a cluster this also means: an RWX volume, two PostgreSQL pods on the same files in case of a deployment mistake, the latency of everyfsyncover the network. The practice is a local disk and replication by the DBMS itself (part 25). - SQLite, which many applications use silently, warns in its documentation that locking may not work correctly on network file systems, and the database gets corrupted. If an application keeps SQLite on a volume, that volume must not be NFS with several writers.
NFS is good for what really are files: user uploads, generated reports, resources shared by several replicas, static content.
When the server is unavailable
With hard, everything that touches the volume waits. Let's check: on the server we drop NFS packets from k8s-node-2 with a rule at the top of the INPUT chain (a ufw deny rule does not work here: ufw lets established connections through before its own rules), wait seven minutes and put things back. Each node has one writer replica, each writing every 10 seconds.
user@nfs:~$ sudo iptables -I INPUT 1 -s 192.168.10.21 -p tcp --dport 2049 -j DROP
After 25 seconds the pod's process on k8s-node-2 hangs in the kernel, and kubectl exec into that pod with ls /data does not return:
user@k8s-node-2:~$ ps -eo pid,stat,wchan:30,cmd | grep 'while true' | grep -v grep
309395 Ds - sh -c while true; do echo "$(date -u +%FT%TZ) $HOSTNAME" >> /data/visits.log; sleep 10; done
$ timeout 15 kubectl -n demo exec writer-7cddd4c9c7-f29ff -- ls -l /data; echo rc=$?
rc=124
State D is a sleep in the kernel that ordinary signals do not interrupt: the process is waiting for the reply to its write. Meanwhile the replica on k8s-node-1 keeps writing as if nothing happened: the problem is in one node's NFS client, not in the volume. Let's delete the stuck pod:
$ kubectl -n demo delete pod writer-7cddd4c9c7-f29ff --wait=false
pod "writer-7cddd4c9c7-f29ff" deleted from demo namespace
$ kubectl -n demo get pods -o wide # 30 seconds later
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
writer-7cddd4c9c7-67nfp 1/1 Running 0 2m40s 172.20.211.57 k8s-node-1 <none> <none>
writer-7cddd4c9c7-f29ff 0/1 Error 0 2m40s 172.20.172.8 k8s-node-2 <none> <none>
writer-7cddd4c9c7-t524g 0/1 ContainerCreating 0 34s <none> k8s-node-2 <none> <none>
The process died: after 30 seconds (the default terminationGracePeriodSeconds) kubelet sent SIGKILL, and the NFS wait let it through — since kernel 2.6.25 the Linux NFS client waits in a way that SIGKILL interrupts (which is why the old intr option is marked as ignored in nfs(5)). The pod, however, does not leave the API: kubelet cannot unmount the volume, and the node plugin's log repeats the same thing every two minutes:
E1030 11:52:50.324715 1 utils.go:126] GRPC error: rpc error: code = Internal desc = failed to unmount target "/var/lib/kubelet/pods/b7027368-976f-4c2f-a3f6-a7e0c86b2b0f/volumes/kubernetes.io~csi/pvc-9640d114-1423-4c1e-9a9f-144d406e860a/mount": unmount failed: exit status 16
Output: umount.nfs4: /var/lib/kubelet/pods/b7027368-976f-4c2f-a3f6-a7e0c86b2b0f/volumes/kubernetes.io~csi/pvc-9640d114-1423-4c1e-9a9f-144d406e860a/mount: device is busy
The replacement the Deployment created right away landed on the same node and hangs in ContainerCreating: the mount does not complete, the driver aborts it on its own timeout (mount volume nfs.example.com:/srv/nfs/k8s/demo-data-pvc-… to /var/lib/kubelet/pods/…/mount timeout after 110s), and kubelet tries again. The message about the server appeared in the node's kernel log only four minutes after the outage began — after retrans=3 retries with a growing interval:
user@k8s-node-2:~$ sudo dmesg -T | grep nfs
[Fri Oct 30 11:54:14 2026] nfs: server nfs.example.com not responding, still trying
[Fri Oct 30 11:55:03 2026] nfs: server nfs.example.com not responding, timed out
[Fri Oct 30 11:55:17 2026] nfs: server nfs.example.com not responding, still trying
still trying is a request on the hard volume that will keep being retried; timed out is a request with its own deadline, here the mount attempt aborted on timeout. We remove the rule on the server:
user@nfs:~$ sudo iptables -D INPUT -s 192.168.10.21 -p tcp --dport 2049 -j DROP
user@k8s-node-2:~$ sudo dmesg -T | grep nfs | tail -2
[Fri Oct 30 11:57:17 2026] nfs: server nfs.example.com OK
[Fri Oct 30 11:57:21 2026] nfs: server nfs.example.com OK
$ kubectl -n demo get pods -o wide # 40 seconds later
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
writer-7cddd4c9c7-67nfp 1/1 Running 0 9m11s 172.20.211.57 k8s-node-1 <none> <none>
writer-7cddd4c9c7-t524g 1/1 Running 0 7m5s 172.20.172.6 k8s-node-2 <none> <none>
Seconds after the server came back the old pod was unmounted and disappeared, and the new one mounted the volume and started — kubelet recorded its startup time as 6 minutes 53 seconds. The file shows how it looked for the data: the replica on k8s-node-2 wrote until 11:50:09, then for seven minutes only k8s-node-1, then both again. The write the process hung on is lost together with the process — write never returned, and the application never got to believe it succeeded.
What follows from this:
- pods with the volume stop responding,
livenessProbechecks that touch the volume fail, and kubelet restarts the container; SIGKILL will take the process down, but the new container hangs on the same volume; - pod deletion hangs: the object stays in the API (as
ErrororTerminating) until kubelet can unmount the volume; - replacements land on the same node and hang in
ContainerCreating; the scheduler knows nothing about the NFS problem; - any command that touches this mount waits too: on the test bench,
lsthroughkubectl exec, and on the nodedfand other utilities that query all mounted file systems will hang the same way.
How to get out. The best way out is to bring the server back. As soon as it answers, all requests complete, the mounts are released, and the pods recover by themselves — that is what hard is for. So fix the server or the network first, not the pods.
If the server will not come back (the disk died, a move to a new server), on every affected node:
kubectl cordonthe node so that no new pods with the volume land on it.- Find the mounts:
findmnt -t nfs4. - Force the unmount:
sudo umount -f <path>interrupts pending requests; if that fails,sudo umount -l <path>detaches the mount from the tree. - Delete the pods; if one hangs in
Terminating,kubectl delete pod --grace-period=0 --forceremoves the object from the API, but not the processes on the node — they stay until the kernel lets them go. - If something is left — reboot the node (
kubectl drain, then reboot). This is the only guaranteed way.
On the test bench steps 3–5 were not needed (the server came back); they follow the umount(8) and Kubernetes documentation. kubectl delete --force without unmounting is also dangerous because the controller may start a new pod on another node while the old process on the first one is still alive and, as soon as the server returns, finishes writing its data. For StatefulSet pods this is a straight road to two writers on one volume.
Hands-on
The PVC and the PV — right after creation, before the expansion and the option changes discussed above:
$ kubectl -n demo get pvc data
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
data Bound pvc-9640d114-1423-4c1e-9a9f-144d406e860a 5Gi RWX nfs <unset> 46s
$ kubectl get pv pvc-9640d114-1423-4c1e-9a9f-144d406e860a -o yaml
apiVersion: v1
kind: PersistentVolume
metadata:
annotations:
pv.kubernetes.io/provisioned-by: nfs.csi.k8s.io
volume.kubernetes.io/provisioner-deletion-secret-name: ""
volume.kubernetes.io/provisioner-deletion-secret-namespace: ""
creationTimestamp: "2026-10-30T11:29:46Z"
finalizers:
- kubernetes.io/pv-protection
name: pvc-9640d114-1423-4c1e-9a9f-144d406e860a
resourceVersion: "1123868"
uid: 452f009f-4330-481d-81a8-71c300ce0884
spec:
accessModes:
- ReadWriteMany
capacity:
storage: 5Gi
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: data
namespace: demo
resourceVersion: "1123842"
uid: 9640d114-1423-4c1e-9a9f-144d406e860a
csi:
driver: nfs.csi.k8s.io
volumeAttributes:
csi.storage.k8s.io/pv/name: pvc-9640d114-1423-4c1e-9a9f-144d406e860a
csi.storage.k8s.io/pvc/name: data
csi.storage.k8s.io/pvc/namespace: demo
onDelete: archive
server: nfs.example.com
share: /srv/nfs/k8s
storage.kubernetes.io/csiProvisionerIdentity: 1793359746544-2025-nfs.csi.k8s.io
subDir: demo-data-pvc-9640d114-1423-4c1e-9a9f-144d406e860a
volumeHandle: nfs.example.com#srv/nfs/k8s#demo-data-pvc-9640d114-1423-4c1e-9a9f-144d406e860a#pvc-9640d114-1423-4c1e-9a9f-144d406e860a#archive
mountOptions:
- nfsvers=4.2
- hard
- timeo=600
- retrans=3
- nconnect=4
- nosuid
- nodev
persistentVolumeReclaimPolicy: Retain
storageClassName: nfs
volumeMode: Filesystem
status:
lastPhaseTransitionTime: "2026-10-30T11:29:46Z"
phase: Bound
volumeHandle is assembled from the server, the export path, the subdirectory, the PV name and the deletion policy: from it the controller knows on DeleteVolume what to delete and how, even if the StorageClass has been deleted or recreated by then. mountOptions are copied from the class, the subDir template is already expanded, and the PV name is pvc- plus the claim's UID. The only finalizer is kubernetes.io/pv-protection: the provisioner puts its own external-provisioner.volume.kubernetes.io/finalizer only on PVs with the Delete policy, and with Retain it will never have to delete this one.
The directory on the server:
user@nfs:~$ ls -la /srv/nfs/k8s
total 4
drwxr-xr-x 3 root root 64 Oct 30 11:29 .
drwxr-xr-x 3 root root 4096 Oct 30 10:41 ..
drwxrwsr-x 2 root 10001 24 Oct 30 11:29 demo-data-pvc-9640d114-1423-4c1e-9a9f-144d406e860a
user@nfs:~$ ls -ln /srv/nfs/k8s/demo-data-pvc-9640d114-1423-4c1e-9a9f-144d406e860a
total 4
-rw-r--r-- 1 10001 10001 540 Oct 30 11:30 visits.log
The root is 0755 root:root, the volume directory was created by the controller as root, group 10001 and the s bit were set by kubelet for fsGroup, the application's file belongs to UID 10001 — without a single 0777.
On the node — the mount and the path in kubelet's directory:
user@k8s-node-1:~$ findmnt -t nfs4 -o TARGET,SOURCE,FSTYPE,OPTIONS
TARGET SOURCE FSTYPE OPTIONS
/var/lib/kubelet/pods/a122b643-da5f-4495-b4f0-ac8cb9066e1d/volumes/kubernetes.io~csi/pvc-9640d114-1423-4c1e-9a9f-144d406e860a/mount nfs.example.com:/srv/nfs/k8s/demo-data-pvc-9640d114-1423-4c1e-9a9f-144d406e860a nfs4 rw,nosuid,nodev,relatime,vers=4.2,rsize=262144,wsize=262144,namlen=255,hard,fatal_neterrors=none,proto=tcp,nconnect=4,timeo=600,retrans=3,sec=sys,clientaddr=192.168.10.20,local_lock=none,addr=192.168.10.30
user@k8s-node-1:~$ D=/var/lib/kubelet/pods/a122b643-da5f-4495-b4f0-ac8cb9066e1d/volumes/kubernetes.io~csi/pvc-9640d114-1423-4c1e-9a9f-144d406e860a
user@k8s-node-1:~$ sudo ls $D
mount vol_data.json
user@k8s-node-1:~$ sudo cat $D/vol_data.json
{"attachmentID":"csi-31a7b5c86a9cfa94c305f5b44219d9a216c5977b8b868f54088678a8ae59c966","driverName":"nfs.csi.k8s.io","nodeName":"k8s-node-1","specVolID":"pvc-9640d114-1423-4c1e-9a9f-144d406e860a","volumeHandle":"nfs.example.com#srv/nfs/k8s#demo-data-pvc-9640d114-1423-4c1e-9a9f-144d406e860a#pvc-9640d114-1423-4c1e-9a9f-144d406e860a#archive","volumeLifecycleMode":"Persistent"}
$ kubectl get volumeattachment
No resources found
/var/lib/kubelet/pods/<pod UID>/volumes/kubernetes.io~csi/<PV>/mount is the point that kubelet then passes into the container with a bind mount at /data. vol_data.json is kubelet's note to itself: which driver and which volume the directory belongs to, so that after a kubelet restart it unmounts it correctly. kubelet always computes attachmentID (a hash of the volume, the driver and the node), but there is no VolumeAttachment object with that name — attachRequired: false. All the options from mountOptions are in place, rsize/wsize of 256 KiB were chosen by the server (as in the previous part), fatal_neterrors=none is the default in kernel 7.0: network errors such as "network unreachable" are not treated as final, and the request is retried. clientaddr is the node's address.
The node's connections to the server with nconnect=4:
user@k8s-node-1:~$ sudo ss -tn -4 dst 192.168.10.30
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 0 192.168.10.20:720 192.168.10.30:2049
ESTAB 0 0 192.168.10.20:757 192.168.10.30:2049
ESTAB 0 0 192.168.10.20:918 192.168.10.30:2049
ESTAB 0 0 192.168.10.20:990 192.168.10.30:2049
Four connections for all volumes of this server on the node, all from privileged ports. After rollout restart the new pod got its own mount, while the connections stayed the same four.
The pod itself cannot reach the server — the Calico policy from the previous part is in place — yet its volume works:
$ kubectl -n demo exec deploy/writer -- nc -zv -w 3 192.168.10.30 2049
nc: 192.168.10.30 (192.168.10.30:2049): Connection timed out
command terminated with exit code 1
$ kubectl -n demo exec deploy/writer -- sh -c 'id; tail -2 /data/visits.log'
uid=10001 gid=10001 groups=10001
2026-10-30T11:31:44Z writer-7d568d48b4-jwtsz
2026-10-30T11:31:44Z writer-7d568d48b4-k68j9
NFS client counters. We did not install nfs-common on the nodes, but nfsstat and nfsiostat are in the driver image, and the node plugin's nfs container runs in the node's network and sees kubelet's mounts (/var/lib/kubelet/pods is passed into it with mountPropagation: Bidirectional), so you can look from there:
$ P=$(kubectl -n kube-system get pod -l app=csi-nfs-node --field-selector spec.nodeName=k8s-node-1 -o name)
$ kubectl -n kube-system exec $P -c nfs -- nfsstat -c -4 | head -3
Client rpc stats:
calls retrans authrefrsh
117 0 117
$ kubectl -n kube-system exec $P -c nfs -- nfsiostat 5 2 /var/lib/kubelet/pods/a122b643-da5f-4495-b4f0-ac8cb9066e1d/volumes/kubernetes.io~csi/pvc-9640d114-1423-4c1e-9a9f-144d406e860a/mount
nfs.example.com:/srv/nfs/k8s/demo-data-pvc-9640d114-1423-4c1e-9a9f-144d406e860a mounted on /var/lib/kubelet/pods/a122b643-da5f-4495-b4f0-ac8cb9066e1d/volumes/kubernetes.io~csi/pvc-9640d114-1423-4c1e-9a9f-144d406e860a/mount:
ops/s rpc bklog
0.208 0.000
read: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms) avg queue (ms) errors
0.000 0.000 0.000 0 (0.0%) 0.000 0.000 0.000 0 (0.0%)
write: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms) avg queue (ms) errors
0.104 0.045 0.435 0 (0.0%) 116.625 116.750 0.000 0 (0.0%)
...
The first nfsiostat block is averages since the mount, the following ones are per interval. retrans above zero is the first sign of network or server trouble, long before the application notices anything.
And finally, releasing the volume. We delete the application and the claim — the PV stays Released, the directory on the server is untouched:
$ kubectl -n demo delete deploy writer
deployment.apps "writer" deleted from demo namespace
$ kubectl -n demo delete pvc data
persistentvolumeclaim "data" deleted from demo namespace
$ kubectl get pv pvc-9640d114-1423-4c1e-9a9f-144d406e860a
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE
pvc-9640d114-1423-4c1e-9a9f-144d406e860a 10Gi RWX Retain Released demo/data nfs <unset> 30m
user@nfs:~$ ls -l /srv/nfs/k8s | grep demo-data-pvc
drwxrwsr-x 2 root 10001 24 Oct 30 11:29 demo-data-pvc-9640d114-1423-4c1e-9a9f-144d406e860a
The capacity is 10Gi because the PVC was expanded above. The data has been checked and is no longer needed — we switch the PV to Delete, and this is the first time onDelete: archive from the volumeHandle fires:
$ kubectl patch pv pvc-9640d114-1423-4c1e-9a9f-144d406e860a -p '{"spec":{"persistentVolumeReclaimPolicy":"Delete"}}'
persistentvolume/pvc-9640d114-1423-4c1e-9a9f-144d406e860a patched
$ kubectl get pv pvc-9640d114-1423-4c1e-9a9f-144d406e860a
Error from server (NotFound): persistentvolumes "pvc-9640d114-1423-4c1e-9a9f-144d406e860a" not found
$ kubectl -n kube-system logs deploy/csi-nfs-controller -c nfs | grep -E 'Controller/DeleteVolume|archived subdirectory' | tail -2
I1030 12:00:15.300056 1 utils.go:121] GRPC call: /csi.v1.Controller/DeleteVolume
I1030 12:00:15.897611 1 controllerserver.go:314] archived subdirectory /tmp/pvc-9640d114-1423-4c1e-9a9f-144d406e860a/demo-data-pvc-9640d114-1423-4c1e-9a9f-144d406e860a --> /tmp/pvc-9640d114-1423-4c1e-9a9f-144d406e860a/archived-demo-data-pvc-9640d114-1423-4c1e-9a9f-144d406e860a
user@nfs:~$ ls -l /srv/nfs/k8s | grep demo-data-pvc
drwxrwsr-x 2 root 10001 24 Oct 30 11:29 archived-demo-data-pvc-9640d114-1423-4c1e-9a9f-144d406e860a
The PV was gone within seconds, and the directory became archived-demo-data-pvc-…: to remove it for good — rm -rf on the server once it is definitely not needed. Between these log lines the controller mounts the export with mountflags([]): exactly the options that DeleteVolume does not get.
Sources
- csi-driver-nfs: repository, release v4.13.6, release manifests, installing v4.13.6 and the Helm chart — what the driver consists of, images, the chart version without
v. - csi-driver-nfs: driver parameters —
server,share,subDirand its templates,onDelete,mountPermissions,uid/gidandfsGroup, thevolumeHandleformat,mountOptionsforDeleteVolumethrough a Secret and for snapshots; examples — static PV, snapshot, clone. - Kubernetes CSI: CSIDriver Object and fsGroup Support —
attachRequired,fsGroupPolicy: File; external-provisioner and Secrets and Credentials — thecsi.storage.k8s.io/provisioner-secret-*parameters and the deletion annotations on the PV; Snapshot Controller and external-snapshotter v8.7.0 — snapshot CRDs and the controller. - Container Storage Interface Specification —
CreateVolumewithvolume_capabilities,DeleteVolumeandCreateSnapshotwithout mount options, controller capabilities (CLONE_VOLUME,EXPAND_VOLUME). - Kubernetes: Storage Classes — class fields,
reclaimPolicy,mountOptions,allowVolumeExpansion; Persistent Volumes — reclaiming (Retain,Delete), mount options, expansion; Volume Snapshots, Volume Snapshot Classes and CSI Volume Cloning — snapshots,deletionPolicy, clones throughdataSource. - Kubernetes: Configure a Security Context for a Pod or Container —
runAsUser,fsGroup,fsGroupChangePolicy: OnRootMismatch. - Linux: nfs(5) —
nfsvers,hardandsoft,timeo,retrans,nconnect,local_lock,xprtsec,intr; exports(5) —root_squash,xprtsecin an export; umount(8) —-ffor an unreachable NFS server and-l; nfsstat(8) and nfsiostat(8) — client counters. - Protocol: RFC 7530 and RFC 8881 — NFSv4 locks, leases and the grace period; RFC 9289 — RPC over TLS.
- NFS over TLS: ktls-utils, tlshd.conf(5) and In-Kernel TLS Handshake — the handshake service and how the kernel hands it the socket.
- Databases on NFS: Prometheus Storage — NFS is not supported, etcd: Hardware recommendations — disks and
fsynclatency, PostgreSQL: Creating a Database Cluster — a data directory on NFS andhard, SQLite: How To Corrupt An SQLite Database File and the SQLite FAQ — locking on network file systems.
Next — csi-driver-smb: volumes from a Windows server and Samba, SMB3 with signing and encryption, credentials in a Secret, cifs options line by line, and a real case where pods got Host is down because of how the client read the NAS's replies.
Only registered users can leave a comment.
Sign in or register to leave a comment.