Vitaliy Leschenko’s blog
Виталий Лещенко

Kubernetes for beginners. Part 22: Setting up an NFS server for the cluster

All parts of the series “Kubernetes for beginners”

In the previous part we went through PersistentVolume, PersistentVolumeClaim and StorageClass, looked at what a CSI driver is made of, and met the fsGroupPolicy field, with which a driver tells kubelet whether to change the group of the volume's files to the pod's fsGroup. The next two parts are about the most common storage for a cluster without a cloud: an NFS server from which the csi-driver-nfs driver carves volumes. Today it is the server side, and we will look at it through the eyes of an administrator responsible for keeping the data safe: how to set up nfs.example.com (192.168.10.30, Ubuntu Server 26.04, not part of the cluster) so that the cluster gets its volumes and nobody else gets anything. NFSv4 only, the export on a separate disk, /etc/exports line by line, a firewall, encryption in transit. And one non-obvious point: why restricting the export to the node addresses is necessary but not sufficient, and what the network policy from part 20 has to do with it.

An NFS server for a cluster is often set up from the first recipe found, and it runs for years as "everyone, read-write, root unrestricted". It really does work well, and that is exactly why nobody touches it. At the end of the part such a configuration is taken apart point by point, together with a plan for moving to a proper one without stopping the cluster.

NFS in brief

NFS (Network File System) is a protocol for accessing files over the network. A client mounts a remote directory, and the kernel turns every open, read, write, stat into a request to the server. Underneath NFS is ONC RPC (RFC 5531), a remote procedure call protocol: every request carries a program number (100003 for NFS), a version, a procedure number, a transaction ID and a credential — information about on whose behalf it is made. Today RPC runs over TCP (part 10 of the networking series): one connection carries records, each with a four-byte length prefix.

NFSv3 and its helpers

NFSv3 (RFC 1813) is a stateless protocol: every request is self-contained, and the server does not remember clients. Everything that needs state or lookups is moved into separate RPC programs with their own daemons and ports:

Program Daemon Port Purpose
portmapper (rpcbind) rpcbind 111 TCP and UDP directory: which port program N version V runs on
NFS kernel (nfsd) 2049 the file operations themselves
MOUNT rpc.mountd dynamic check whether the client is allowed the export and hand out the root file handle
NLM (locking) kernel (lockd) dynamic fcntl byte-range locks
NSM (status) rpc.statd dynamic tell clients that the server has rebooted so they take their locks again

The client first goes to 111, learns the ports of MOUNT, NLM and NSM, then talks to each of them. Dynamic ports are awkward for a firewall (they have to be pinned in the configuration), and an rpcbind reachable from foreign networks hands out the list of the server's services and was used for years as an amplifier for UDP DDoS attacks. The fewer ports a server listens on, the smaller its attack surface.

NFSv4: one port and state on the server

NFSv4 (RFC 7530) redid the protocol completely:

  • One port, 2049. The MOUNT program is not needed: the server gathers all exports into one pseudo-filesystem, and the client walks down from its root with ordinary LOOKUP requests. Locking is built into the protocol itself (the LOCK and LOCKU operations), so neither NLM, NSM nor rpcbind is needed. The server is left with one TCP connection per client and one port.
  • State on the server. The server remembers clients, open files and locks. A client holds a lease (90 seconds by default) and renews it with any request; if the client has been gone longer than the lease, the server releases its locks. After a restart the server declares a grace period (also 90 seconds) during which clients reclaim their opens and locks and no new ones are granted — which is why the first mount after systemctl restart nfs-server may wait up to a minute and a half.
  • Delegations. The server can hand a file over to a client to manage and cache locally, and recall it on a conflict. In version 4.0 the server opens a connection back to the client for the recall (a callback), and that back channel passes poorly through NAT and firewalls.
  • NFSv4.1 (RFC 8881) added sessions: the back channel runs inside the same connection the client opened, and a slot table guarantees that a request retried after a disconnect is not executed twice (exactly-once semantics). It also introduced pNFS — parallel access to data spread over several servers.
  • NFSv4.2 (RFC 7862) adds features on top of 4.1: server-side copy (COPY, the data does not travel through the client), sparse files (SEEK, ALLOCATE, DEALLOCATE), extended attributes (RFC 8276), and passing SELinux labels — SELinux being a mandatory access control system — so that server and client see the same label on a file.

For the cluster this means: on the server it is enough to enable 4.1 and 4.2 and disable 3 and 4.0. Clients on Ubuntu 26.04 and any modern Linux speak 4.2 and pick it by themselves.

User names and numeric UIDs

In NFSv3 a file's owner is sent as a number. In NFSv4 the owner and owner_group attributes are strings of the form user@domain, and translating names into UIDs on both sides is the job of the idmapping service (nfsidmap, rpc.idmapd, the domain in /etc/idmapd.conf). But with sec=sys the credential in the RPC request is numeric anyway, and by default the Linux kernel with sec=sys sends numbers in the attributes too: client and server exchange the strings "1000", not "alice@example.com" (the nfs4_disable_idmapping parameter of the nfs and nfsd modules, Y by default on both).

Hence the practical rule for a cluster: numeric UIDs and GIDs decide permissions. A pod with runAsUser: 1000 creates files that on the server belong to UID 1000, whoever that UID is in the server's /etc/passwd, and on a node ls -l shows whatever name UID 1000 has on that node. So application UIDs are chosen deliberately and kept apart from the UIDs of real users of the server.

The sec=sys trust model

The default credential is AUTH_SYS, written sec=sys in export and mount options. The client puts the process's UID, GID and up to 16 supplementary groups into every request, and the server does not verify them: there is nothing to verify with, the request carries no password or key. The server checks only the client's address — whether it is allowed in /etc/exports — and from then on believes what the client says about the user. All of NFS security without Kerberos rests on the assumption that only the kernels of allowed machines talk to the server and that they honestly report on whose behalf a process runs.

The only thing the server can do with a credential is replace it (squash):

Option What it does
root_squash (default) requests from UID 0 run as the anonymous user (anonuid, 65534 by default, nobody); other UIDs as they are
no_root_squash the client's UID 0 is root on the server: it can change the owner of any file in the export, read and write everything
all_squash all UIDs and GIDs, root included, are replaced with anonuid/anongid; permissions inside the export are effectively the same for everyone
anonuid, anongid which UID and GID to substitute for squashed ones, for example anonuid=2000,anongid=2000

root_squash protects against the most obvious thing: root on any allowed machine does not become root on the server. The permissions of ordinary UIDs remain the client's business.

Why the CSI driver asks for no_root_squash

csi-driver-nfs, the subject of the next part, works with the export as root twice.

  1. The controller creates and deletes volume directories. For every new PVC its container temporarily mounts the export root and runs mkdir for the volume's subdirectory, and on deletion rm -rf. It runs as root, in the host network (hostNetwork: true) and with SYS_ADMIN, which mount needs. With root_squash these operations run as nobody and fail if the export root belongs to root, and on deletion they cannot remove files that applications created under their own UIDs.
  2. kubelet sets permissions for fsGroup. The driver has fsGroupPolicy: File, and when it mounts a volume into a pod with securityContext.fsGroup: 2000, kubelet, as root, changes the group of the volume's files to 2000 and gives the group write permission. A chown to someone else's group is impossible with root_squash: the server sees nobody, not root. The driver's documentation states plainly that for fsGroup the export must allow the ownership change: no_root_squash or a squash UID that owns the directory.

It can be worked around, but at a cost. An export root handed to nobody lets the controller create directories even with root_squash, but fsGroup stops working, and write access has to be given through mountPermissions: 0777 — that is, to everyone. all_squash with the application's anonuid makes all files owned by one UID: a workable option for a single application, but not for a cluster's shared storage. So for an export behind a CSI driver the practical choice is no_root_squash, and security moves to the question who can reach the server at all. The rest of the part is about that.

A node address is not kubelet yet

The server checks the client's address, and the first idea is to allow the export to the node addresses only. That is right, but here is what to keep in mind.

 k8s-node-1, 192.168.10.20
 ┌──────────────────────────────────────────────────┐
 │  kubelet, csi-nfs-node     pod 172.20.211.46     │
 │  (host network)                   │              │
 │        │                  SNAT (natOutgoing)     │
 │        │                          │              │
 │        └── src 192.168.10.20 ─────┘              │
 │                  │                               │
 └──────────────────┼───────────────────────────────┘
                    ▼
        nfs.example.com:2049: client 192.168.10.20

Outgoing pod traffic to addresses outside the Calico pools goes through NAT: with natOutgoing: true the node replaces the pod address 172.20.x.x with its own (ordinary SNAT in netfilter, covered in part 6 of the networking series; we saw Calico's masquerading in nftables in part 14). To the NFS server a connection from a process in any pod looks exactly like a connection from kubelet: the same node address. The address filter in /etc/exports and in the firewall separates the cluster nodes from the rest of the network, but does not separate kubelet from the pods running on those nodes.

The secure option does not help either. It requires the client to use a privileged port (below 1024), and its point is that on an ordinary machine only root can take such a port. In pods that is not the case. containerd 2 by default sets net.ipv4.ip_unprivileged_port_start=0 for containers outside the host network (the enable_unprivileged_ports setting of CRI), so any process in a pod — non-root and without capabilities — can open a connection from port 700. And the node's NAT keeps the port class: a source port below 1024 is replaced with another port below 1024. Let us check with a pod that follows every security rule:

apiVersion: v1
kind: Pod
metadata:
  name: nonroot
  namespace: demo
spec:
  nodeName: k8s-node-1        # so we know which node the connection comes from
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    runAsGroup: 1000
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: busybox
      image: busybox:1.37
      command: ["sleep", "3600"]
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop: ["ALL"]

$ kubectl apply -f nonroot.yaml
pod/nonroot created
$ kubectl -n demo exec nonroot -- sh -c 'id; grep CapEff /proc/self/status; cat /proc/sys/net/ipv4/ip_unprivileged_port_start'
uid=1000 gid=1000 groups=1000
CapEff:	0000000000000000
0
$ kubectl -n demo exec nonroot -- nc -zv -w 3 -p 700 192.168.10.30 2049
192.168.10.30 (192.168.10.30:2049) open

And on the server at that moment:

user@nfs:~$ sudo tcpdump -ni ens18 -c 1 'tcp dst port 2049 and tcp[tcpflags] == tcp-syn'
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ens18, link-type EN10MB (Ethernet), snapshot length 262144 bytes
10:56:53.114900 IP 192.168.10.20.990 > 192.168.10.30.2049: Flags [S], seq 3462405522, win 64860, options [mss 1410,sackOK,TS val 731856867 ecr 0,nop,wscale 9], length 0
1 packet captured
1 packet received by filter
0 packets dropped by kernel

The node's address and privileged port 990 — to the server this is indistinguishable from the node's kernel. Then the sec=sys model takes over: the client assembles the credential in the request. The kernel honestly puts the process's UID there, but a program that speaks the NFS protocol itself, in user space (such libraries exist), writes whatever it wants into the credential, including UID 0, and it does not need root for that. With no_root_squash such a request gets root's rights on the export. In other words, with this configuration the trust boundary runs not along the nodes but around every workload in the cluster, including third-party images and vulnerable applications.

Conclusion: the address filter is necessary but not sufficient. A second layer is needed that separates pod traffic from node traffic. It exists: a Calico network policy, which applies to pod traffic before NAT, while the packet still carries the pod's address. Below are all the measures in order.

1. A separate disk for the export

NFS knows nothing about a PVC's size: a 1 GiB volume is just a directory, and an application can write as much into it as it likes, until the server's file system runs out. If the export lives on the root partition, one application overfilling its volume stops the whole server: logs, updates, nfsd itself. So the export gets a separate disk or partition, in our case the second disk /dev/sdb:

user@nfs:~$ sudo mkfs.xfs -L k8s-data /dev/sdb
user@nfs:~$ sudo mkdir -p /srv/nfs/k8s
user@nfs:~$ echo 'LABEL=k8s-data  /srv/nfs/k8s  xfs  defaults,nodev,nosuid  0 2' | sudo tee -a /etc/fstab
LABEL=k8s-data  /srv/nfs/k8s  xfs  defaults,nodev,nosuid  0 2
user@nfs:~$ sudo systemctl daemon-reload && sudo mount /srv/nfs/k8s
user@nfs:~$ sudo chown root:root /srv/nfs/k8s && sudo chmod 0755 /srv/nfs/k8s
user@nfs:~$ findmnt /srv/nfs/k8s
TARGET       SOURCE   FSTYPE OPTIONS
/srv/nfs/k8s /dev/sdb xfs    rw,nosuid,nodev,relatime,inode64,logbufs=8,logbsize=32k,noquota

nodev and nosuid forbid device files and setuid bits on this file system: clients write to the export, and nothing they write should take effect on the server itself. XFS is chosen because it has project quotas: a directory can be given a hard limit. csi-driver-nfs does not set them, but an administrator can do it by hand for individual large volumes (xfs_quota).

In /etc/exports this export will have the mp (mountpoint) option: export the directory only if it really is a mount point. If the disk one day fails to mount, without mp the server exports an empty directory on the root partition, the cluster starts creating volumes in it, and the root partition fills up. With mp the export is simply not given to clients: rpc.mountd logs Cannot export path '/srv/nfs/k8s': not a mountpoint, the client gets No such file or directory, and the error is visible at once. With this line in /etc/fstab, without nofail, a server with a missing disk will not even boot past emergency mode; if you add nofail so that it boots and stays reachable over SSH, it is mp that protects against the empty export.

2. NFSv4 only

The NFS server in Ubuntu is the nfs-kernel-server package; nfs-common, rpcbind, keyutils and libnfsidmap1 come with it:

user@nfs:~$ sudo apt install nfs-kernel-server

Right after installation all versions and all NFSv3 helpers are enabled:

user@nfs:~$ sudo cat /proc/fs/nfsd/versions
+3 +4 +4.1 +4.2
user@nfs:~$ rpcinfo -p | grep -v portmapper
   program vers proto   port  service
    100024    1   udp  59836  status
    100024    1   tcp  37909  status
    100005    1   udp  40287  mountd
    100005    1   tcp  36865  mountd
    100005    2   udp  35330  mountd
    100005    2   tcp  52015  mountd
    100005    3   udp  54414  mountd
    100005    3   tcp  47415  mountd
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    100227    3   tcp   2049  nfs_acl
    100021    1   udp  33818  nlockmgr
    100021    3   udp  33818  nlockmgr
    100021    4   udp  33818  nlockmgr
    100021    1   tcp  35093  nlockmgr
    100021    3   tcp  35093  nlockmgr
    100021    4   tcp  35093  nlockmgr

The whole table from the beginning of the part: portmapper itself on 111 (its lines are filtered out), status (NSM), mountd, nlockmgr (NLM) on random ports, plus nfs_acl — an auxiliary POSIX ACL protocol for NFSv3.

Ubuntu keeps the server settings in /etc/nfs.conf and in the /etc/nfs.conf.d/ directory, where it is convenient to keep your changes in a separate file:

# /etc/nfs.conf.d/k8s.conf
[nfsd]
vers3=n
vers4.0=n
vers4.1=y
vers4.2=y

[mountd]
manage-gids=n

vers3=n disables version 3, vers4.0=n the version without sessions, which needs a back channel from server to client. The vers4= line is left alone: it would disable the whole of version 4. The default number of nfsd threads is 16 (that is how nfsdctl, which the nfs-server service uses to start the server, launches them); for a large cluster it is raised with a threads= line in the same section.

manage-gids=n is the line without which fsGroup will not work. Ubuntu's /etc/nfs.conf has manage-gids=y: the server throws away the list of supplementary groups sent by the client and substitutes the groups of that UID from its own user database. This gets around the 16-group limit of AUTH_SYS and is handy when clients and server share their users. A cluster has no such shared users: a pod with runAsUser: 1000 and fsGroup: 2000 arrives with group 2000, while on the server UID 1000 is either nobody or someone unrelated, and has no group 2000. It was checked like this: into a root:2000 directory with mode 2775, a process with UID 1000, 1001 or 5000 and supplementary group 2000 got Permission denied with manage-gids=y and wrote its file with manage-gids=n.

Without NFSv3, rpcbind and rpc.statd are no longer needed. They are masked so that no dependency brings them back:

user@nfs:~$ sudo systemctl mask --now rpc-statd.service rpcbind.service rpcbind.socket
Created symlink '/etc/systemd/system/rpc-statd.service' → '/dev/null'.
Created symlink '/etc/systemd/system/rpcbind.service' → '/dev/null'.
Created symlink '/etc/systemd/system/rpcbind.socket' → '/dev/null'.

rpc.mountd stays: even with NFSv4 the kernel asks it whether a client is allowed and how an export maps (through /proc/net/rpc/, not over the network). But by default it also listens on network ports for NFSv3 clients — even after vers3=n it keeps listening sockets for MOUNT versions 1 and 2. This is switched off with the --no-tcp --no-udp flags through a systemd drop-in:

# /etc/systemd/system/nfs-mountd.service.d/v4only.conf
[Service]
ExecStart=
ExecStart=/usr/sbin/rpc.mountd --no-tcp --no-udp

user@nfs:~$ sudo systemctl daemon-reload
user@nfs:~$ sudo systemctl restart nfs-server
user@nfs:~$ sudo cat /proc/fs/nfsd/versions
-3 +4 -4.0 +4.1 +4.2
user@nfs:~$ sudo ss -tulnp -4 | grep -v 127.0.0
Netid State  Recv-Q Send-Q       Local Address:Port Peer Address:PortProcess
udp   UNCONN 0      0      192.168.10.30%ens18:68        0.0.0.0:*    users:(("systemd-network",pid=962,fd=33))
tcp   LISTEN 0      4096               0.0.0.0:2049      0.0.0.0:*
tcp   LISTEN 0      4096               0.0.0.0:22        0.0.0.0:*    users:(("sshd",pid=1076,fd=3),("systemd",pid=1,fd=262))

/proc/fs/nfsd/versions is what is really enabled in the kernel: minus for 3 and 4.0, plus for 4.1 and 4.2. In ss (without the local 127.0.0.x — the systemd-resolved DNS cache and chronyd) only 2049 remains (a kernel socket has no process, so the column is empty), SSH and the DHCP client on port 68, through which the server gets its address (part 11 of the networking series). No 111, no dynamic ports. rpc.mountd now logs mountd: No V2 or V3 listeners created! at every start — that is intended. The nfs-idmapd, nfsdcld (remembers NFSv4 clients on disk so that after a server restart it knows who may reclaim state) and fsidd services keep running but do not face the network.

3. /etc/exports line by line

A fresh /etc/exports contains only comments with examples. We replace it entirely:

# /etc/exports
# Cluster volumes for csi-driver-nfs: cluster nodes only, root is not squashed
/srv/nfs/k8s  192.168.10.17(rw,sync,no_subtree_check,secure,no_root_squash,mp,sec=sys) \
              192.168.10.20(rw,sync,no_subtree_check,secure,no_root_squash,mp,sec=sys) \
              192.168.10.21(rw,sync,no_subtree_check,secure,no_root_squash,mp,sec=sys)

To whom. Not * and not the whole 192.168.10.0/24 network, but the node addresses. The master 192.168.10.17 is on the list because the driver's controller is allowed on the control plane by default (it has tolerations for the master's taints) and may end up there. The compact form is the subnet 192.168.10.16/28, but in our network it covers .16–.31, which includes the NFS server itself and files.example.com. A subnet is appropriate only if it is reserved for nodes entirely; otherwise addresses one by one, and adding a node to the cluster includes editing /etc/exports and the firewall (an item on the node onboarding checklist).

rw — writing is allowed; by default an export is read-only.

sync versus async. With sync the server answers a write and a COMMIT only after the data is on disk. async answers at once and writes later; that is faster, but if the server loses power, the client considers written what is not on disk and has no way of finding out. For application volumes, sync only (it is the default, but it is written explicitly so that the reader has no doubt).

no_subtree_check — do not check on every request that the file lies inside the exported subdirectory. The check makes sense when a subdirectory of a file system is exported rather than the whole file system, and it misbehaves when open files are renamed. Here the export is the whole file system (mp), so the check is not needed; this is the default.

secure — accept requests only from privileged ports (below 1024). The Linux kernel NFS client connects from such a port anyway, so insecure, which often appears in recipes, is not needed by the cluster: only clients that do not run as root need it — programs with a user-space NFS client. secure is a weak check, and as we saw it does not protect against pods, but there is no reason to turn it off.

no_root_squash — only on this export and only because the CSI driver asks for it. If the server has other exports, they keep root_squash (item 7).

mp — described in item 1. sec=sys is the default, written explicitly so that when moving to Kerberos (item 8) it is clear what to change.

The export root's permissions are 0755 root:root: only root can write to it, that is, the driver's controller; it creates the volume directories inside by itself, and kubelet sets their permissions for the pod's fsGroup.

Apply without restarting the server and check what the kernel really sees:

user@nfs:~$ sudo exportfs -ra
user@nfs:~$ sudo exportfs -v
/srv/nfs/k8s  	192.168.10.17(sync,wdelay,hide,no_subtree_check,mountpoint,sec=sys,rw,secure,no_root_squash,no_all_squash)
/srv/nfs/k8s  	192.168.10.20(sync,wdelay,hide,no_subtree_check,mountpoint,sec=sys,rw,secure,no_root_squash,no_all_squash)
/srv/nfs/k8s  	192.168.10.21(sync,wdelay,hide,no_subtree_check,mountpoint,sec=sys,rw,secure,no_root_squash,no_all_squash)

exportfs -ra rereads the file and synchronizes the kernel's table: added clients gain access, removed ones lose it on their next request, and connected clients that are still allowed notice nothing. The -v output also shows the implicit values: wdelay (group writes close in time), hide, no_all_squash. The table does not show the mp check: the export is listed even with the disk unmounted; the refusal happens when a client comes.

4. A firewall on the server

/etc/exports decides whom to give the export to, but port 2049 itself is open to anyone who reaches the server, and behind it is kernel code parsing requests from any address. A firewall narrows the circle of those who can open a connection at all. On Ubuntu ufw is convenient — a front end to iptables, which in Ubuntu run on top of nftables (iptables -V shows nf_tables):

user@nfs:~$ sudo ufw default deny incoming
Default incoming policy changed to 'deny'
(be sure to update your rules accordingly)
user@nfs:~$ sudo ufw default allow outgoing
Default outgoing policy changed to 'allow'
(be sure to update your rules accordingly)
user@nfs:~$ sudo ufw allow from 192.168.10.240/28 to any port 22 proto tcp comment 'SSH: admin workstations'
Rules updated
user@nfs:~$ for ip in 192.168.10.17 192.168.10.20 192.168.10.21; do
>   sudo ufw allow from $ip to any port 2049 proto tcp comment 'NFSv4: k8s nodes'
> done
Rules updated
Rules updated
Rules updated
user@nfs:~$ sudo ufw enable
Command may disrupt existing ssh connections. Proceed with operation (y|n)? y
Firewall is active and enabled on system startup
user@nfs:~$ sudo ufw status numbered
Status: active

     To                         Action      From
     --                         ------      ----
[ 1] 22/tcp                     ALLOW IN    192.168.10.240/28          # SSH: admin workstations
[ 2] 2049/tcp                   ALLOW IN    192.168.10.17              # NFSv4: k8s nodes
[ 3] 2049/tcp                   ALLOW IN    192.168.10.20              # NFSv4: k8s nodes
[ 4] 2049/tcp                   ALLOW IN    192.168.10.21              # NFSv4: k8s nodes

192.168.10.240/28 here is a notional network of administrators' workstations; substitute your own. The SSH rule is added before ufw enable, otherwise you can lose access to the server. UDP is not needed for NFS: NFSv4 works over TCP only. ufw logs dropped packets to the kernel log marked [UFW BLOCK] (logging level low by default).

The two layers — /etc/exports and the firewall — are configured the same way but protect against different things: the first against a mistake in the firewall, the second against a vulnerability in request parsing and against someone one day widening the export to *.

5. Network policy: pods may not reach the server

Now the second layer against pods. Pods have no need to go to the NFS server directly: volumes are mounted by kubelet on the node, and the process in the pod sees an ordinary directory. So all pod traffic to 192.168.10.30 can be forbidden, and no application will notice. A Calico policy checks the packet on the pod's interface (the node side of the veth), before NAT: the packet still has the pod's address, and Calico knows exactly which pod it came from.

Calico policies, which we discussed in part 20, are grouped into tiers: ordered groups checked one after another, from lower order to higher; the built-in ones are kube-admin (1000), default (1,000,000), where ordinary NetworkPolicies live, and kube-baseline (10,000,000), plus calico-system (100), which the Calico operator creates for its own policies. Inside a tier, policies apply by order; an Allow or Deny rule ends the evaluation, and if the packet matched no rule of the selected policies, the tier's default action applies. Here is an important subtlety. If you put one policy with one Deny rule into the default tier, then for every pod it selected all other egress becomes forbidden too: the policy selected the pod, no allowing rule matched, and the tier ends the evaluation with an implicit deny. That is why the deny goes into a separate tier with defaultAction: Pass: everything that did not match the deny is passed on to the next tiers and to ordinary NetworkPolicies as if this tier did not exist.

apiVersion: projectcalico.org/v3
kind: Tier
metadata:
  name: storage-guard
spec:
  order: 200          # after calico-system (100), before kube-admin (1000) and default
  defaultAction: Pass
---
apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
  name: storage-guard.deny-nfs-server
spec:
  tier: storage-guard
  order: 10
  namespaceSelector: all()   # all namespaces, that is, pods only
  types:
    - Egress
  egress:
    - action: Deny
      destination:
        nets:
          - 192.168.10.30/32

A GlobalNetworkPolicy acts in all namespaces at once, so a new namespace will not be left unprotected. All traffic to the server is forbidden, not just port 2049: pods have no business with SSH or any other port there either. The tier comes before default, so an allowing NetworkPolicy in an application's namespace will not override it. Calico 3.32 does not require the storage-guard. prefix in the policy name, but with it kubectl get shows at once which tier the policy belongs to.

$ kubectl apply -f storage-guard.yaml
tier.projectcalico.org/storage-guard created
globalnetworkpolicy.projectcalico.org/storage-guard.deny-nfs-server created
$ kubectl get tiers.projectcalico.org
NAME            CREATED AT
calico-system   2026-10-24T10:36:50Z
default         2026-10-24T10:33:59Z
kube-admin      2026-10-24T10:33:59Z
kube-baseline   2026-10-24T10:33:59Z
storage-guard   2026-10-30T10:47:48Z

On the node Felix turns this into an nftables chain. The pod's egress chain (filter-cali-fw-<interface>) gets the start of the tier and a jump to the policy chain, and there is no "end of tier — drop" line — that is what Pass is:

user@k8s-node-1:~$ sudo nft list ruleset ip | grep -A3 'Start of tier storage-guard'
# Warning: table ip filter is managed by iptables-nft, do not touch!
		counter packets 4 bytes 240 meta mark set meta mark & 0xfffdffff comment "cali:H8y8Bg0SPeMX2TVV; Start of tier storage-guard"
		meta mark & 0x00020000 == 0x00000000 counter packets 4 bytes 240 jump filter-cali-po-gnp/storage-guard.deny-nfs-server comment "cali:I5XycXG0aqjdThzp;"
		meta mark & 0x00010000 == 0x00010000 counter packets 0 bytes 0 return comment "cali:Oa07qTVNuWtGvA2q; Return if policy accepted"
		counter packets 1 bytes 60 jump filter-cali-pro-kns.demo comment "cali:TJE0Mgsa5aM6ijZF;"
user@k8s-node-1:~$ sudo nft list chain ip calico 'filter-cali-po-gnp/storage-guard.deny-nfs-server'
table ip calico {
	chain filter-cali-po-gnp/storage-guard.deny-nfs-server {
		ip daddr 192.168.10.30 counter packets 3 bytes 180 meta mark set meta mark | 0x00040000 comment "cali:PrgeB-EQDo7dMLwl; GlobalNetworkPolicy storage-guard.deny-nfs-server egress"
		meta mark & 0x00040000 == 0x00040000 counter packets 3 bytes 180 drop comment "cali:ZtM_-7WTNjSkTrSx;"
	}
}

The first line is printed by nft itself: the filter table is managed by iptables-nft. A packet to the server gets the Deny bit (0x40000) and is dropped — the three packets here are the SYN from nc and its retransmissions; everything else goes on to the profile of the kns.demo namespace, which allows everything — as if the tier did not exist.

Why this does not break volumes. The policy selects workload endpoints — pod interfaces. kubelet and the driver's controller run in the host network (hostNetwork), they have no pod interface, and the policy does not see their traffic. Automatic host endpoints (policies for the node's own interfaces) are disabled in our cluster (autoCreate: Disabled in KubeControllersConfiguration); if you enable them, check that the policy's selector does not select the nodes.

What the policy does not cover. A pod started with hostNetwork: true also lives in the host network and bypasses it. Such pods should exist only for system components, and for everyone else they are forbidden by Pod Security Admission — the built-in admission controller that checks pods against security profiles; more on it in part 30.

The policy is removed in reverse order: first the policy, then the tier. A single kubectl delete -f storage-guard.yaml deletes the policy, but for the tier it answers Cannot delete a non-empty tier — the policy deletion has not finished by then; running the command again succeeds.

6. Pods not running as root

The third layer is what the pod itself does. A process with runAsUser: 1000, without capabilities and with allowPrivilegeEscalation: false cannot mount NFS through the kernel (mount needs CAP_SYS_ADMIN), does not become root on the node and does not get hostNetwork if the namespace runs under the restricted profile of Pod Security Admission. More in part 30.

But on its own, as we saw, it does not protect against talking to the NFS server directly: a port below 1024 is available to it, and a program assembles the AUTH_SYS credential itself. The port threshold can be brought back: in containerd with enable_unprivileged_ports = false in the CRI settings for all pods, and for a single pod with the net.ipv4.ip_unprivileged_port_start sysctl in securityContext.sysctls (it is on the safe list, so a pod can also set it itself, either way). That closes only the path through secure, not the path through the network. So the main layer against pods is the network policy from item 5, and non-root plus restricted are what keep it from being bypassed through the host network.

7. Different exports for different classes of data

One export, one trust level. Application volumes created by the CSI driver need no_root_squash; other data does not, and they are better not mixed.

# Reference data the cluster only reads: its own disk, read-only, root squashed
/srv/nfs/static  192.168.10.20(ro,sync,no_subtree_check,root_squash,mp) \
                 192.168.10.21(ro,sync,no_subtree_check,root_squash,mp)

# Backups: not to the cluster nodes, only to the backup server (the address is notional)
/srv/nfs/backup  192.168.10.40(rw,sync,no_subtree_check,root_squash,mp)

Backups matter most. If database backups sit on the same export as the volumes, anything that can corrupt the volumes can also delete the backups. An export that is not given to the cluster nodes at all survives both an application bug and a compromise of the cluster. Each export lives on its own file system (hence mp): one filling up does not stop the others, and NFS does not have to check subdirectory boundaries.

8. Authentication and encryption in transit

None of the above changes the main property of sec=sys: plaintext data and unsigned requests travel over the network between node and server. There are two mechanisms against this.

Kerberos: sec=krb5, krb5i, krb5p. Instead of a UID, the request carries a Kerberos ticket (RPCSEC_GSS, RFC 2203): the server verifies who is calling by key rather than taking their word for it. krb5 is authentication only, krb5i adds a signature on every request (protection against tampering in transit), krb5p adds encryption. On the server it is the export option sec=krb5p, on the client a mount option, plus a KDC (key distribution center, which issues tickets), a keytab (a file with principals' keys) with an nfs/<node name> principal on every node and on the server, the rpc.gssd service and synchronized clocks. The price in a cluster is higher than on ordinary machines: Kerberos authenticates users, and root on a node uses the machine key, while processes in pods have no tickets of their own — access will be denied to them unless every application is issued tickets. Kerberos support in csi-driver-nfs is not documented out of the box; it is possible (the driver passes mount options through as they are), but it needs testing on your own setup. Where Kerberos already exists, krb5p gives the best NFS protection there is; building it for one cluster is usually expensive.

NFS over TLS. RFC 9289 describes encrypting RPC with TLS 1.3 on the same port 2049: with its first request, a NULL with an AUTH_TLS credential, the client asks whether the server supports TLS, and if it does, performs the handshake on the same connection. In Linux the handshake is done not by the kernel but by the tlshd service from the ktls-utils package: the kernel hands it the socket, it performs the handshake and returns the keys, after which the kernel does the encryption itself (kTLS). The NFS server supports this since kernel 6.4, the client since 6.5. On the server it is the export option xprtsec=tls (encryption only) or xprtsec=mtls (the client presents a certificate too — which is also authentication of the node); several can be listed separated by colons (xprtsec=mtls:tls:none); on the client it is the mount option xprtsec=tls and the same tlshd. Certificates are issued the usual way, for example by an internal CA (TLS and certificates — in part 23 of the networking series).

Ubuntu 26.04 has everything needed: the kernel is built with CONFIG_NET_HANDSHAKE=y, ktls-utils 1.3.0 is in the repository, the settings live in /etc/tlshd/config. On the server, a certificate for the name nfs.example.com and its key:

# /etc/tlshd/config on the server
[authenticate.server]
x509.certificate= /etc/tlshd/server.crt
x509.private_key= /etc/tlshd/server.key

On the node, the CA certificate used to verify the server (otherwise the system trust store is used):

# /etc/tlshd/config on the node
[authenticate.client]
x509.truststore= /etc/tlshd/ca.crt

After systemctl restart tlshd on both sides and xprtsec=tls in the k8s-node-1 entry of /etc/exports:

user@k8s-node-1:~$ sudo mount -t nfs4 -o vers=4.2 nfs.example.com:/srv/nfs/k8s /mnt
mount.nfs4: Operation not permitted for nfs.example.com:/srv/nfs/k8s on /mnt
user@k8s-node-1:~$ sudo mount -t nfs4 -o vers=4.2,xprtsec=tls nfs.example.com:/srv/nfs/k8s /mnt
user@k8s-node-1:~$ findmnt -n /mnt
/mnt nfs.example.com:/srv/nfs/k8s nfs4 rw,relatime,vers=4.2,rsize=262144,wsize=262144,namlen=255,hard,fatal_neterrors=none,proto=tcp,timeo=600,retrans=2,sec=sys,xprtsec=tls,clientaddr=192.168.10.20,local_lock=none,addr=192.168.10.30

Without TLS the server refuses this node; with TLS the mount succeeds, and the options show xprtsec=tls. tcpdump on the server shows NFS request … null followed by TLS records instead of readable NFS operations. The server certificate is verified by name, so mount by the name in the certificate, not by address. Whether xprtsec gets through csi-driver-nfs mount options, where the driver's container does the mounting while tlshd is needed on the node itself, is a question for the next part. If it works, xprtsec=mtls closes both eavesdropping and connections from machines without a certificate — so the "node or not a node" question is decided by a key rather than an address; and pods with the node's address do not have the node's certificate.

If neither is available, physical isolation remains: the network of the nodes and the storage server is a separate VLAN (part 17 of the networking series), which workstations and guest devices do not get into.

How it goes in practice: an anti-example

A configuration found in practice that "just works" for years:

# /etc/exports
/data *(rw,insecure,sync,no_subtree_check,no_root_squash)

/data is a directory on the root partition with mode 0777, the firewall is off, NFSv3 is on, rpcbind listens on all interfaces. Piece by piece:

What Why it is bad
* the export is given to any address that reaches the server: workstations, VPN clients, the guest network
no_root_squash for * root on any of those machines is root on all the export's data
insecure even the weak privileged-port requirement is dropped; it never got in the way of Linux kernel clients
root partition one volume filling up stops the server; no mp — nothing to catch an unmounted disk
0777 on /data any UID from any allowed machine can write to the export root; permissions lose their meaning even with root_squash
firewall off nothing but /etc/exports limits who opens a connection to 2049 and the other ports
NFSv3 and rpcbind on all interfaces extra services with dynamic ports, a directory service open to everyone, separate locking protocols
no network policy any pod in the cluster talks to the server with the node's address

A migration plan without downtime

The order is chosen so that each step can be checked and rolled back on its own while the cluster keeps working.

  1. Inventory. Who is connected and with which version: ss -tn sport = :2049 (client addresses), ls /proc/fs/nfsd/clients/ and the info files in it (NFSv4 clients), nfsstat -s (are there non-zero Server nfs v3 counters). Anything that is not a cluster node — find out whose it is and whether it is needed.
  2. Firewall. Rules for administrators' SSH and for 2049 from the nodes (and from the other legitimate clients found, if any), then ufw enable. Check: volumes in the cluster work, ss -tn shows the same node connections.
  3. The Calico network policy from item 5. Check: a test pod cannot reach the server, pods with volumes work — their traffic to the server comes from kubelet.
  4. /etc/exports: replace * with the node addresses, remove insecure, exportfs -ra. Connected nodes notice nothing. Check: exportfs -v, a mount from a node.
  5. Export root permissions: chmod 0755 /data. Volume directories inside keep their permissions. Check: creating a new PVC works (the controller runs as root).
  6. Clients on NFSv4.2. In the mountOptions of the StorageClass and existing PVs — nfsvers=4.2 (how — in the next part). New options apply only on a new mount, so nodes are taken out of service one at a time (kubectl drain) and brought back. Check: on the nodes mount | grep nfs shows vers=4.2, on the server the v3 counters stop growing.
  7. Turn off v3, rpcbind, statd, the network ports of mountd (item 2). Check: /proc/fs/nfsd/versions, ss -tulnp.
  8. A separate disk. The only step that needs a short window: when data moves to another file system, the NFS file handles change, and mounted volumes get a Stale file handle error. The order: prepare the disk, copy the data with rsync -aHAX in advance, then stop the applications with volumes (scale to zero), run rsync again (copying the difference), mount the disk in place of /data (or at a new path with the PVs edited), add mp, exportfs -ra, bring the applications back. Check: findmnt, exportfs -v, pods with volumes running.

Separately, decide whether no_root_squash is needed on this export at all: if there are no clients besides the CSI driver, keep it — it is now restricted to the nodes and by the policy; if there are other clients, move their data to a separate export with root_squash (item 7).

Hands-on

A mount from an allowed address — from a node, by hand. For this the node needs the nfs-common package with mount.nfs (sudo apt install nfs-common); the driver from the next part will not need it on the node — the mount utilities are in its image.

user@k8s-node-1:~$ sudo mount -t nfs4 -o vers=4.2 nfs.example.com:/srv/nfs/k8s /mnt
user@k8s-node-1:~$ findmnt /mnt
TARGET SOURCE                       FSTYPE OPTIONS
/mnt   nfs.example.com:/srv/nfs/k8s nfs4   rw,relatime,vers=4.2,rsize=262144,wsize=262144,namlen=255,hard,fatal_neterrors=none,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.10.20,local_lock=none,addr=192.168.10.30

rsize and wsize are the largest size of a single read and write; the server picks its limit from its amount of memory (/proc/fs/nfsd/max_block_size), 256 KiB on ours with 2 GB. hard means that with the server unreachable requests are retried forever instead of returning an error to the application. Let us create a "volume" directory the way the controller and kubelet would: owner — the application's UID, group — the fsGroup, the setgid bit so that new files inherit the group:

user@k8s-node-1:~$ sudo mkdir /mnt/pvc-test
user@k8s-node-1:~$ sudo chown 1000:2000 /mnt/pvc-test && sudo chmod 2775 /mnt/pvc-test
user@k8s-node-1:~$ sudo setpriv --reuid=1000 --regid=1000 --groups=2000 \
    sh -c 'echo hello > /mnt/pvc-test/app.txt; dd if=/dev/zero of=/mnt/pvc-test/blob bs=1M count=32 status=none'
user@k8s-node-1:~$ sudo umount /mnt

The chown as root went through — that is no_root_squash. On the server the owner is the very same numbers:

user@nfs:~$ ls -ln /srv/nfs/k8s /srv/nfs/k8s/pvc-test
/srv/nfs/k8s:
total 0
drwxrwsr-x 2 1000 2000 33 Oct 30 10:49 pvc-test

/srv/nfs/k8s/pvc-test:
total 65476
-rw-r--r-- 1 1000 2000        6 Oct 30 10:49 app.txt
-rw-r--r-- 1 1000 2000 33554432 Oct 30 10:49 blob

total 65476 for a 32 MiB file — XFS allocated space ahead for a growing file (speculative preallocation) and will give the excess back later.

The NFSv4 pseudo-filesystem is visible if you mount the server's root: the client walks down to the export through ordinary directories.

user@k8s-node-1:~$ sudo mount -t nfs4 nfs.example.com:/ /mnt && find /mnt -maxdepth 3 && sudo umount /mnt
/mnt
/mnt/srv
/mnt/srv/nfs
/mnt/srv/nfs/k8s

Version 4.0 is off:

user@k8s-node-1:~$ sudo mount -t nfs4 -o vers=4.0 nfs.example.com:/srv/nfs/k8s /mnt
mount.nfs4: Protocol not supported for nfs.example.com:/srv/nfs/k8s on /mnt

From an address that is not in /etc/exports. The easiest check is from the server itself: the firewall lets local traffic through, but the address 192.168.10.30 is not in the export, so it is precisely the /etc/exports check that fires:

user@nfs:~$ sudo mount -t nfs4 nfs.example.com:/srv/nfs/k8s /mnt
mount.nfs4: access denied by server while mounting nfs.example.com:/srv/nfs/k8s

From the laptop 192.168.10.50, which the firewall does not let through to 2049, there is no answer at all — the connection is not established, and mount eventually ends with a timeout. On the server such attempts show up in the kernel log:

user@nfs:~$ sudo journalctl -k | grep 'UFW BLOCK' | grep 'DPT=2049' | tail -1
Oct 30 10:44:46 nfs kernel: [UFW BLOCK] IN=ens18 OUT= MAC=52:54:00:3a:10:30:52:54:00:3a:10:50:08:00 SRC=192.168.10.50 DST=192.168.10.30 LEN=52 TOS=0x00 PREC=0x00 TTL=128 ID=63550 DF PROTO=TCP SPT=52189 DPT=2049 WINDOW=65535 RES=0x00 SYN URGP=0

These are different layers, and both are worth checking.

A pod before and after the policy. A temporary busybox pod in the demo namespace on k8s-node-1:

$ kubectl -n demo run nettest --image=busybox:1.37 --restart=Never \
    --overrides='{"spec":{"nodeName":"k8s-node-1"}}' -- sleep 3600
pod/nettest created
$ kubectl -n demo exec nettest -- nc -zv -w 3 192.168.10.30 2049
192.168.10.30 (192.168.10.30:2049) open

The connection was established, and on the server it arrived from the node's address:

user@nfs:~$ sudo tcpdump -ni ens18 -c 2 'tcp port 2049 and tcp[tcpflags] & tcp-syn != 0'
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ens18, link-type EN10MB (Ethernet), snapshot length 262144 bytes
10:45:50.121387 IP 192.168.10.20.36220 > 192.168.10.30.2049: Flags [S], seq 3875367041, win 64860, options [mss 1410,sackOK,TS val 2778270908 ecr 0,nop,wscale 9], length 0
10:45:50.121425 IP 192.168.10.30.2049 > 192.168.10.20.36220: Flags [S.], seq 3403409359, ack 3875367042, win 65160, options [mss 1460,sackOK,TS val 4016543837 ecr 2778270908,nop,wscale 8], length 0
2 packets captured
2 packets received by filter
0 packets dropped by kernel

After kubectl apply -f storage-guard.yaml:

$ kubectl -n demo exec nettest -- nc -zv -w 3 192.168.10.30 2049
nc: 192.168.10.30 (192.168.10.30:2049): Connection timed out
command terminated with exit code 1
$ kubectl -n demo exec nettest -- nc -zv -w 3 192.168.10.30 22
nc: 192.168.10.30 (192.168.10.30:22): Connection timed out
command terminated with exit code 1
$ kubectl -n demo exec nettest -- nc -zv -w 3 192.168.10.17 6443
192.168.10.17 (192.168.10.17:6443) open
$ kubectl -n demo exec nettest -- nslookup kubernetes.default.svc.cluster.local
Server:		10.96.0.10
Address:	10.96.0.10:53


Name:	kubernetes.default.svc.cluster.local
Address: 10.96.0.1

The pod's packets to the server are dropped on its veth and never reach NAT — neither to 2049 nor to SSH; the rest of the traffic, DNS and the API included, flows as before.

And the volumes keep working. Let us check with a pod using the built-in nfs volume type — no CSI driver, kubelet mounts it itself through the node's mount.nfs:

apiVersion: v1
kind: Pod
metadata:
  name: nfs-vol
  namespace: demo
spec:
  nodeName: k8s-node-1
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    runAsGroup: 1000
    supplementalGroups: [2000]
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: app
      image: busybox:1.37
      command: ["sleep", "3600"]
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop: ["ALL"]
      volumeMounts:
        - name: data
          mountPath: /data
  volumes:
    - name: data
      nfs:
        server: nfs.example.com
        path: /srv/nfs/k8s/pvc-test

$ kubectl apply -f nfs-vol.yaml
pod/nfs-vol created
$ kubectl -n demo exec nfs-vol -- sh -c 'cat /data/app.txt; echo from-pod > /data/pod.txt && ls -ln /data'
hello
total 32776
-rw-r--r--    1 1000     2000             6 Oct 30 10:49 app.txt
-rw-r--r--    1 1000     2000      33554432 Oct 30 10:49 blob
-rw-r--r--    1 1000     2000             9 Oct 30 10:57 pod.txt
$ kubectl -n demo exec nfs-vol -- nc -zv -w 3 192.168.10.30 2049
nc: 192.168.10.30 (192.168.10.30:2049): Connection timed out
command terminated with exit code 1

The pod reads and writes a volume on the NFS server but cannot reach the server itself: kubelet did the mount from the host network, and the policy selects only the pod's interface. The new file got group 2000 from the directory (setgid).

What is connected to the server and from where:

user@nfs:~$ ss -tn sport = :2049
State Recv-Q Send-Q   Local Address:Port    Peer Address:Port
ESTAB 0      0        192.168.10.30:2049   192.168.10.20:915

The client port is below 1024 — that is how the Linux kernel connects with secure. One connection per node: all of a node's volumes from this server go through it (nodes without mounted volumes have no connection). There must be no addresses here other than nodes; a stranger's address is a reason to investigate, not just to close.

Which NFSv4 clients the server knows by name:

user@nfs:~$ sudo sh -c 'cat /proc/fs/nfsd/clients/*/info'
clientid: 0xb450d8186ae477e0
address: "192.168.10.20:915"
status: confirmed
seconds from last renew: 10
name: "Linux NFSv4.2 k8s-node-1"
minor version: 2
Implementation domain: "kernel.org"
Implementation name: "Linux 7.0.0-38-generic #38-Ubuntu SMP PREEMPT_DYNAMIC Fri Sep  4 09:10:14 UTC 2026 x86_64"
Implementation time: [0, 0]
callback state: UP
callback address: "192.168.10.20:0"
admin-revoked states: 0
session slots: 64
session target slots: 64

minor version: 2 — the client came with 4.2; callback address with port 0 — the back channel runs inside the session, and the server opens no separate connection to the client.

Server statistics: which versions and operations are actually used.

user@nfs:~$ nfsstat -s
Server rpc stats:
calls      badcalls   badfmt     badauth    badclnt
751        12         0          12         0

Server nfs v4:
null             compound
22        2%     729      97%

Server nfs v4 operations:
op0-unused       op1-unused       op2-future       access           close
0         0%     0         0%     0         0%     91        3%     0         0%
commit           create           delegpurge       delegreturn      getattr
1         0%     2         0%     0         0%     11        0%     450      19%
getfh            link             lock             lockt            locku
96        4%     0         0%     0         0%     0         0%     0         0%
lookup           lookup_root      nverify          open             openattr
86        3%     0         0%     0         0%     12        0%     0         0%
...
read             readdir          readlink         remove           rename
1         0%     13        0%     0         0%     0         0%     0         0%
...
setcltid         setcltidconf     verify           write            rellockowner
0         0%     0         0%     0         0%     130       5%     0         0%
bc_ctl           bind_conn        exchange_id      create_ses       destroy_ses
0         0%     0         0%     28        1%     14        0%     13        0%
...
layoutget        layoutreturn     secinfononam     sequence         set_ssv
0         0%     0         0%     14        0%     660      28%     0         0%
...

There is no Server nfs v3 section — version 3 is off. sequence opens almost every compound request — a sign of NFSv4.1 sessions and later; exchange_id and create_ses set up the client and the session at every mount, and setcltid (the way 4.0 did it) is zero. badauth and badclnt are requests with a rejected credential and from unknown clients; non-zero values deserve an explanation. Our 12 badauth are two mount attempts from the server itself, whose address is not in the export (6 refusals each).

The kernel's export table once more:

user@nfs:~$ sudo exportfs -v
/srv/nfs/k8s  	192.168.10.17(sync,wdelay,hide,no_subtree_check,mountpoint,sec=sys,rw,secure,no_root_squash,no_all_squash)
/srv/nfs/k8s  	192.168.10.20(sync,wdelay,hide,no_subtree_check,mountpoint,sec=sys,rw,secure,no_root_squash,no_all_squash)
/srv/nfs/k8s  	192.168.10.21(sync,wdelay,hide,no_subtree_check,mountpoint,sec=sys,rw,secure,no_root_squash,no_all_squash)

These four checks — exportfs -v, ss -tn sport = :2049, a test pod and nfsstat -s — are worth running after any change on the server and from time to time for no reason at all.

Sources

Next — csi-driver-nfs: how it is built and installed, a StorageClass with subDir, onDelete and proper mountOptions, volume permissions through fsGroup, static PVs and what happens to pods when the NFS server is unavailable.

Only registered users can leave a comment.

Sign in or register to leave a comment.

© 2006-2026 Vitaliy Leschenko