31 October 2026, 07:06Виталий Лещенко

Kubernetes for beginners. Part 24: Connecting SMB with csi-driver-smb

All parts of the series “Kubernetes for beginners”

In the previous part the cluster got volumes from an NFS server through csi-driver-nfs, and in part 22 we locked that server down to everyone except the nodes. The second common source of shared volumes is a Windows file server, Samba or a NAS that already serves shares to users over SMB. For it there is csi-driver-smb: the same PVs and StorageClasses as in part 21, but underneath them a protocol with authentication, signing and encryption — which means credentials that have to be stored somewhere, and a cifs client in the kernel with its own options and quirks. The server in this part is files.example.com (192.168.10.31), Linux with Samba; how the same is done on Windows Server is shown alongside, following Microsoft's documentation. At the end — a real case where pods periodically got Host is down because of how the client read the NAS's replies.

SMB in brief

SMB (Server Message Block) is the Windows file access protocol. The first version (SMB1, also known as CIFS) is obsolete and insecure: Windows Server has not installed it by default since version 1709, and Samba has not enabled it since 4.11. From here on we talk about SMB2 and SMB3 — one protocol with dialect numbers: 2.0.2, 2.1, 3.0, 3.0.2, 3.1.1.

Transport. SMB2/3 runs over TCP port 445 (TCP is covered in part 10 of the networking series). Every message in the stream is preceded by a four-byte header: a zero type byte and three bytes of length. This is a legacy of the NetBIOS session from RFC 1002, which is why the Linux client's logs call it that. The SMB2 message itself begins with a 64-byte header whose first four bytes are the signature FE 53 4D 42 ("\xFESMB"); an encrypted SMB3 message instead begins with a transform header with the signature FD 53 4D 42. These facts will come in handy in the case study.

Three levels of state.

Session — the result of authentication (SESSION_SETUP): the client proves who it is with NTLM (a challenge–response protocol where the server checks knowledge of the password without receiving it) or Kerberos (a ticket from a domain controller, covered in detail in part 6 of the Windows Server series), and both sides get a session key from which the signing and encryption keys are derived.
Tree (TREE_CONNECT) — a connection to a specific share inside the session: \\files.example.com\k8s.
Open file (CREATE) — a FileId handle used by READ, WRITE, CLOSE.

Dialects. The client sends a list, the server picks the highest common one. 2.1 added leases, 3.0 — encryption, multichannel (several TCP connections per session) and persistent handles for failover file server clusters, 3.0.2 — minor improvements, 3.1.1 — negotiation integrity (a SHA-512 hash of all messages before authentication, so a man in the middle cannot downgrade the dialect) and AES-GCM ciphers.

Credits. Flow control: each request spends credits, and the server grants new ones in responses. As many credits as the client has, that many requests it can keep in flight at once.

Compound requests: several commands in one message linked by the NextCommand field — for example, CREATE and CLOSE back to back to learn a file's attributes in a single round trip. The server may reply the same way, in one compound message.

Asynchronous responses. If an operation cannot complete right away, the server sends an interim response with status STATUS_PENDING and the final one later.

Oplocks and leases. The server can let a client cache a file: read from its cache (R), defer writes (W), keep the handle open after close (H). The old mechanism is called an oplock; since 2.1 it is a lease, tied to a key rather than a handle. When another client opens the same file in an incompatible way, the server sends the holder a break, waits for acknowledgement and only then replies to the second client — and while it waits, it answers that client with STATUS_PENDING.

SMB3 signing and encryption. Signing is an authentication code (AES-CMAC, and AES-GMAC with 3.1.1 in newer Windows) on every message: server and client detect tampering. Encryption is AES-CCM or AES-GCM over the whole message after authentication; with 3.1.1 AES-128-GCM is negotiated by default. An encrypted message is also protected against tampering, so separate signing on top of encryption is not needed. Both are turned on at the server for the whole server or for an individual share, and a client can require them itself.

The client in Linux

The SMB client in Linux is the cifs kernel module (sources in fs/smb/client); the name is left over from the first protocol version, but it speaks SMB2/3. It is mounted with the mount.cifs utility from the cifs-utils package: it parses options, fetches the password, and for Kerberos gets a ticket through a cifs.upcall call. The cluster driver does not need the package on the nodes — mount.cifs is in its image — and for a manual check we install it on k8s-node-1 temporarily:

user@k8s-node-1:~$ sudo apt install cifs-utils
user@k8s-node-1:~$ sudo mount -t cifs //files.example.com/k8s /mnt \
    -o vers=3,username=svc-k8s-smb,domain=EXAMPLE,seal,uid=10001,gid=10001,dir_mode=0770,file_mode=0660
Password for svc-k8s-smb@//files.example.com/k8s:
user@k8s-node-1:~$ findmnt -t cifs -o TARGET,SOURCE,FSTYPE,OPTIONS
TARGET SOURCE                  FSTYPE OPTIONS
/mnt   //files.example.com/k8s cifs   rw,relatime,vers=3.1.1,cache=strict,upcall_target=app,username=svc-k8s-smb,domain=EXAMPLE,uid=10001,forceuid,gid=10001,forcegid,addr=192.168.10.31,file_mode=0660,dir_mode=0770,seal,soft,nounix,serverino,mapposix,reparse=nfs,nativesocket,symlink=native,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=1,closetimeo=1
user@k8s-node-1:~$ ls /mnt
ls: cannot open directory '/mnt': Permission denied

vers=3 in the options, vers=3.1.1 as the result — the highest common dialect was negotiated. soft is the default, more on it below. domain=EXAMPLE is here as an example: Samba as a standalone server does not check the domain name, while a Windows server in a domain needs it.

With NFS, permissions are decided by the UID of the process, which the client passes to the server. With SMB it is different: all operations through one mount go as one account, the one given at mount time. Permissions on the server are checked for that account, while the owner and file mode that processes on the node see are emulated by the client from the uid, gid, file_mode, dir_mode options. Hence the Permission denied above: ls ran as a user with UID 1000, and the directory "belongs" to 10001 with mode 0770. A process with UID 10001 writes, a process with a different UID but group 10001 does too, and in both cases the server sees one service account:

user@k8s-node-1:~$ sudo setpriv --reuid=10001 --regid=10001 --clear-groups sh -c 'echo hello > /mnt/test.txt; mkdir /mnt/dir; ls -lan /mnt'
total 4
drwxrwx---  2 10001 10001    0 Oct 30 12:23 .
drwxr-xr-x 20 0     0     4096 Oct 26 22:11 ..
drwxrwx---  2 10001 10001    0 Oct 30 12:23 dir
-rw-rw----  1 10001 10001    6 Oct 30 12:23 test.txt
user@k8s-node-1:~$ cat /mnt/test.txt
cat: /mnt/test.txt: Permission denied
user@k8s-node-1:~$ sudo setpriv --reuid=2000 --regid=10001 --clear-groups sh -c 'cat /mnt/test.txt; echo more >> /mnt/test.txt && echo appended'
hello
appended
user@files:~$ sudo ls -lan /srv/samba/k8s
total 16
drwxrwx--- 3 999 982 4096 Oct 30 12:23 .
drwxr-xr-x 3 0   0   4096 Oct 30 12:22 ..
drwxrwx---+2 999 982 4096 Oct 30 12:23 dir
-rw-rw----+1 999 982   11 Oct 30 12:23 test.txt

On the server both files belong to svc-k8s-smb (UID 999, group 982), and their mode comes from the share settings described below. This is the main thing to remember about permissions on SMB volumes: the process UID never reaches the server, and all separation between processes on the node happens on the client.

Here is what refusals look like. A dialect lower than the server allows, and a wrong password (mount.cifs also takes the password from the PASSWD variable):

user@k8s-node-1:~$ sudo PASSWD="$SMB_PASS" mount -t cifs //files.example.com/k8s /mnt -o vers=2.1,username=svc-k8s-smb
mount error(95): Operation not supported
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs) and kernel log messages (dmesg)
user@k8s-node-1:~$ sudo PASSWD=wrong-password mount -t cifs //files.example.com/k8s /mnt -o vers=3,username=svc-k8s-smb
mount error(13): Permission denied
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs) and kernel log messages (dmesg)
user@k8s-node-1:~$ sudo dmesg -T | grep -i cifs | tail -5
[Fri Oct 30 12:23:35 2026] CIFS: VFS: \\files.example.com Dialect not supported by server. Consider  specifying vers=1.0 or vers=2.0 on mount for accessing older servers
[Fri Oct 30 12:23:35 2026] CIFS: VFS: cifs_mount failed w/return code = -95
[Fri Oct 30 12:23:35 2026] CIFS: Status code returned 0xc000006d STATUS_LOGON_FAILURE
[Fri Oct 30 12:23:35 2026] CIFS: VFS: \\files.example.com failed to create a new SMB session with Unknown: -13
[Fri Oct 30 12:23:35 2026] CIFS: VFS: cifs_mount failed w/return code = -13

After the check — sudo umount /mnt and sudo apt purge cifs-utils: from here on only the driver mounts.

The server: a share for the cluster

Samba

Samba is a free implementation of an SMB server for Linux. On files.example.com (Ubuntu 26.04) we install the samba package (Samba 4.23) and create a separate service account — for the cluster only, with no login, with a long random password. Samba as a standalone server keeps its passwords separately from the system ones (smbpasswd), but every Samba account must correspond to a Linux user — smbd works with files on its behalf:

user@files:~$ sudo apt install samba
user@files:~$ sudo useradd --system --no-create-home --shell /usr/sbin/nologin svc-k8s-smb
user@files:~$ id svc-k8s-smb
uid=999(svc-k8s-smb) gid=982(svc-k8s-smb) groups=982(svc-k8s-smb)
user@files:~$ sudo install -d -o svc-k8s-smb -g svc-k8s-smb -m 0770 /srv/samba/k8s
user@files:~$ sudo smbpasswd -a svc-k8s-smb
New SMB password:
Retype new SMB password:
Added user svc-k8s-smb.
user@files:~$ sudo pdbedit -L -v -u svc-k8s-smb | grep -E 'Unix username|Account Flags|Password must change'
Unix username:        svc-k8s-smb
Account Flags:        [U          ]
Password must change: never

svc-k8s-smb has no Linux password and its shell is nologin: this account cannot log in to the server over SSH or on the console, it exists only for Samba. The share directory belongs to it with mode 0770: the server's other users do not see the cluster's data.

The whole /etc/samba/smb.conf (we replace Ubuntu's stock file with its sample shares rather than edit it):

# /etc/samba/smb.conf
# Kubernetes volumes for csi-driver-smb: cluster nodes only, SMB3 with signing and encryption
[global]
   server role = standalone server
   server min protocol = SMB3
   server signing = mandatory
   server smb encrypt = required
   map to guest = never
   restrict anonymous = 2
   disable netbios = yes
   smb ports = 445
   load printers = no
   hosts allow = 192.168.10.17 192.168.10.20 192.168.10.21 127.0.0.1

[k8s]
   path = /srv/samba/k8s
   valid users = svc-k8s-smb
   read only = no
   create mask = 0660
   directory mask = 0770

server min protocol = SMB3 — no SMB1 and no SMB2. A subtlety: in Samba the value SMB3 means its latest variant, SMB3_11, so the server only admits clients with dialect 3.1.1. On the lab stand a mount with vers=3.0 got the same Dialect not supported by server refusal as vers=2.1 above. If some clients can do at most 3.0 or 3.0.2, write SMB3_00.
server signing = mandatory and server smb encrypt = required — signing and encryption are mandatory for all sessions; a client that cannot encrypt is refused.
map to guest = never and restrict anonymous = 2 — no guest and no anonymous access. never is the default anyway, but Ubuntu's own smb.conf sets bad user, so it is better to state the value explicitly.
disable netbios = yes and smb ports = 445 — plain TCP on 445 only, no NetBIOS: the nmbd name service does not start with this setting (systemd reports start condition unmet), and smbd stops listening on 139 — with Ubuntu's stock configuration it listened on both 139 and 445.
hosts allow — the cluster nodes' addresses and the local address for checks on the server itself; Samba refuses everyone else right after the TCP connection.
valid users — only the service account is let into the share. create mask and directory mask — the mode smbd creates files and directories with on the server's disk: 0660 and 0770, with no access for others.

Check and restart:

user@files:~$ testparm -s
Load smb config files from /etc/samba/smb.conf
Loaded services file OK.
Weak crypto is allowed by GnuTLS (e.g. NTLM as a compatibility fallback)

Server role: ROLE_STANDALONE

# Global parameters
[global]
	disable netbios = Yes
	load printers = No
	restrict anonymous = 2
	server min protocol = SMB3
	server role = standalone server
	server signing = required
	server smb transports = 445
	idmap config * : backend = tdb
	hosts allow = 192.168.10.17 192.168.10.20 192.168.10.21 127.0.0.1
	server smb encrypt = required


[k8s]
	create mask = 0660
	directory mask = 0770
	path = /srv/samba/k8s
	read only = No
	valid users = svc-k8s-smb
user@files:~$ sudo systemctl restart smbd

testparm shows only the differences from the defaults and the canonical parameter names: mandatory became required (synonyms), smb ports became server smb transports (that is what the parameter is called in Samba 4.23), and map to guest = never disappeared because it matches the default. The Weak crypto is allowed by GnuTLS line warns that the crypto library allows the algorithms NTLM needs; it has nothing to do with our configuration.

Windows Server

On Windows Server the same is done like this (commands follow Microsoft's documentation, they were not run on the lab stand). A service account in the domain — with no group membership besides Domain Users, with a long random password; logging on to computers is denied to it by Group Policy (Deny log on locally, Deny log on through Remote Desktop Services):

PS C:\> New-ADUser -Name 'svc-k8s-smb' -SamAccountName svc-k8s-smb -AccountPassword (Read-Host -AsSecureString) `
          -Enabled $true -CannotChangePassword $true -PasswordNeverExpires $true -Description 'Kubernetes SMB volumes'

-PasswordNeverExpires is deliberate: for a volume, a password expiring under the domain policy looks like a password change on the server, and, as shown below, the volumes stop working at the very first reconnect. The service account's password is changed by procedure, not by timer.

The directory, NTFS permissions and share — permissions only for this account and the administrators, without Everyone:

PS C:\> New-Item -ItemType Directory D:\Shares\k8s
PS C:\> icacls D:\Shares\k8s /inheritance:r /grant:r 'BUILTIN\Administrators:(OI)(CI)F' 'NT AUTHORITY\SYSTEM:(OI)(CI)F' 'EXAMPLE\svc-k8s-smb:(OI)(CI)M'
PS C:\> New-SmbShare -Name k8s -Path D:\Shares\k8s -ChangeAccess 'EXAMPLE\svc-k8s-smb' -FullAccess 'BUILTIN\Administrators' `
          -EncryptData $true -FolderEnumerationMode AccessBased

(OI)(CI)M is the Modify right (read, write, delete, but not change permissions) inherited by files and subdirectories. /inheritance:r removes permissions inherited from the drive: otherwise Users from drive D: would get access to the cluster's data. The share permission is Change, not Full: the service account does not need to change NTFS permissions.

Server settings — SMB1 off, encryption and signing mandatory:

PS C:\> Set-SmbServerConfiguration -EnableSMB1Protocol $false -EncryptData $true -RejectUnencryptedAccess $true `
          -RequireSecuritySignature $true -Force
PS C:\> Get-SmbServerConfiguration | Select-Object EnableSMB1Protocol, EncryptData, RejectUnencryptedAccess, RequireSecuritySignature

EncryptData at the server level encrypts all shares; if the server also serves old clients without SMB3, encryption is enabled only on the cluster's share (-EncryptData $true in New-SmbShare, as above). RejectUnencryptedAccess ($true by default anyway) denies access to an encrypted share to clients that cannot encrypt — without it they would get access in clear text. SMB3 encryption signs messages by itself; RequireSecuritySignature is for sessions without encryption.

Windows Firewall: if the server serves only the cluster, the rule for 445 is limited to the nodes' addresses; if it serves users too — to their networks. Ports 137–139 (NetBIOS) are not needed by the cluster:

PS C:\> New-NetFirewallRule -DisplayName 'SMB from k8s nodes' -Direction Inbound -Protocol TCP -LocalPort 445 `
          -RemoteAddress 192.168.10.17,192.168.10.20,192.168.10.21 -Action Allow

The built-in rules of the "File and Printer Sharing" group must then be narrowed or turned off, otherwise they allow 445 to everyone.

The server firewall and the policy for pods

files.example.com runs ufw with the same default policies and SSH rule for administrators as the NFS server in part 22. For SMB it is enough to open 445/tcp from the three nodes — UDP and the NetBIOS ports are not needed:

user@files:~$ for ip in 192.168.10.17 192.168.10.20 192.168.10.21; do
>   sudo ufw allow from $ip to any port 445 proto tcp comment 'SMB: k8s nodes'
> done
Rule added
Rule added
Rule added
user@files:~$ sudo ufw status | grep 445
445/tcp                    ALLOW       192.168.10.17              # SMB: k8s nodes
445/tcp                    ALLOW       192.168.10.20              # SMB: k8s nodes
445/tcp                    ALLOW       192.168.10.21              # SMB: k8s nodes

As with NFS, volumes are mounted by kubelet on the node, and pods do not need direct access to the server. It is enough to extend the GlobalNetworkPolicy from part 22 with a second address:

  egress:
    - action: Deny
      destination:
        nets:
          - 192.168.10.30/32
          - 192.168.10.31/32

For SMB this matters no less than for NFS: both hosts allow and the firewall let a pod through just like a node — the pod's packets leave with the node's address. Without the password the pod reads nothing, but then the password remains the only barrier, and an entry point into the authentication service open from any pod is extra attack surface. With the policy, a pod in demo cannot reach the server, while its volume, as we will see below, works:

$ kubectl -n demo exec deploy/writer -- nc -zv -w 3 192.168.10.31 445
nc: 192.168.10.31 (192.168.10.31:445): Connection timed out
command terminated with exit code 1

Installing the driver

The driver is developed in the kubernetes-csi organization; the current release is v1.20.4. Like csi-driver-nfs in the previous part, the project publishes manifests for every release in the deploy/v1.20.4/ directory; the install-driver.sh script applies them plus a DaemonSet for Windows nodes, which our cluster does not have, so we take the four Linux files:

$ V=v1.20.4
$ for f in rbac-csi-smb.yaml csi-smb-driver.yaml csi-smb-controller.yaml csi-smb-node.yaml; do curl -sLO https://raw.githubusercontent.com/kubernetes-csi/csi-driver-smb/$V/deploy/$V/$f; done
$ kubectl apply -f rbac-csi-smb.yaml -f csi-smb-driver.yaml -f csi-smb-controller.yaml -f csi-smb-node.yaml
serviceaccount/csi-smb-controller-sa created
serviceaccount/csi-smb-node-sa created
clusterrole.rbac.authorization.k8s.io/smb-external-provisioner-role created
clusterrolebinding.rbac.authorization.k8s.io/smb-csi-provisioner-binding created
clusterrole.rbac.authorization.k8s.io/smb-external-resizer-role created
clusterrolebinding.rbac.authorization.k8s.io/smb-csi-resizer-role created
clusterrole.rbac.authorization.k8s.io/csi-smb-node-secret-role created
clusterrolebinding.rbac.authorization.k8s.io/csi-smb-node-secret-binding created
csidriver.storage.k8s.io/smb.csi.k8s.io created
deployment.apps/csi-smb-controller created
daemonset.apps/csi-smb-node created
$ kubectl -n kube-system get pods -l 'app in (csi-smb-controller,csi-smb-node)' -o wide
NAME                                  READY   STATUS    RESTARTS      AGE    IP              NODE           NOMINATED NODE   READINESS GATES
csi-smb-controller-6f4f896698-x7r46   4/4     Running   1 (32s ago)   2m7s   192.168.10.20   k8s-node-1     <none>           <none>
csi-smb-node-929cp                    3/3     Running   1 (17s ago)   2m6s   192.168.10.20   k8s-node-1     <none>           <none>
csi-smb-node-pf88k                    3/3     Running   0             2m6s   192.168.10.17   k8s-master-1   <none>           <none>
csi-smb-node-t7nb8                    3/3     Running   0             2m6s   192.168.10.21   k8s-node-2     <none>           <none>
$ kubectl get csidriver smb.csi.k8s.io -o yaml | sed -n '/^spec:/,$p'
spec:
  attachRequired: false
  fsGroupPolicy: ReadWriteOnceWithFSType
  podInfoOnMount: true
  requiresRepublish: false
  seLinuxMount: false
  storageCapacity: false
  volumeLifecycleModes:
  - Persistent
  - Ephemeral

The same driver can be installed with the Helm chart from the repository (the chart version has no v, 1.20.4); Helm is part 27. The layout is the same as the NFS driver's:

csi-smb-controller — a Deployment in hostNetwork with four containers: csi-provisioner (v6.3.0), csi-resizer (v2.2.1), liveness-probe and the driver itself, smb (image smbplugin:v1.20.4, Debian 12 with cifs-utils 7.0). The controller needs to mount the share to create volume directories in it, so both the server and the firewall must admit whichever node it lands on. The driver has no snapshots; of the extra capabilities it has volume expansion and cloning.
csi-smb-node — a DaemonSet on all nodes, the master included, with node-driver-registrar, liveness-probe and smb. The restart at first start is the same as with the NFS driver: a sidecar container did not wait for the driver's socket while the driver image was being pulled.
CSIDriver: attachRequired: false — there is no attach-to-node step. podInfoOnMount: true — kubelet passes the pod's name, namespace and UID to the driver. Ephemeral — besides PVs, the driver supports inline ephemeral volumes right in the pod spec (for those the driver looks up the Secret only in the pod's namespace). fsGroupPolicy is not set in the manifest, and the API server filled in the default ReadWriteOnceWithFSType; in practice it decides nothing for this driver, because the driver takes over fsGroup itself — more on that below.

There is a NodeStage step. Unlike the NFS driver, this one mounts the share once per node for each PV — into a "global" directory /var/lib/kubelet/plugins/kubernetes.io/csi/smb.csi.k8s.io/<hash>/globalmount (the hash is the SHA-256 of volumeHandle) — and makes bind mounts from it into the pods' directories (NodePublish). All pods on a node using one volume share one cifs mount and one set of options. This matters when options change, as discussed below.

fsGroup through VOLUME_MOUNT_GROUP. The driver advertises the VOLUME_MOUNT_GROUP node capability, and kubelet, seeing it, does not change the files' group itself (on SMB that is impossible anyway — owners are emulated) but passes the pod's fsGroup to the driver in the volume_mount_group field. The driver appends gid=<fsGroup> to the mount options and gives the group full rights in the modes: rwx bits for the group are added to file_mode and dir_mode, and if the modes are not set, file_mode=0774,dir_mode=0775 are used. If gid is already in the options, the driver touches nothing. There is no recursive walk of the volume, so fsGroupChangePolicy speeds nothing up here.

Credentials

The service account's password is kept in a Secret:

apiVersion: v1
kind: Namespace
metadata:
  name: storage-credentials
---
apiVersion: v1
kind: Secret
metadata:
  name: smb-files-k8s
  namespace: storage-credentials
type: Opaque
stringData:
  username: svc-k8s-smb
  password: "<password from the secrets store>"

$ kubectl apply -f smb-secret.yaml
namespace/storage-credentials created
secret/smb-files-k8s created

For a Windows server in a domain, domain: EXAMPLE is added to the Secret, and the driver passes it as the domain= option. A Samba server does not need the domain.

Not in default and not in the application's namespace. In the official examples the Secret lives in default, and that is an anti-example: anyone who can read secrets in default gets access to it. A separate storage-credentials namespace, where only cluster administrators have rights to read secrets (RBAC, Kubernetes' role-based access control, is part 29), separates the password from the application teams. The manifest with the password itself is not put into Git; how to keep secrets in a repository is part 41.

Who else reads this password. For a PV, kubelet fetches the Secret and passes it to the driver in the NodeStageVolume call (in the driver's log the field looks exactly like this: "secrets":"***stripped***"), and external-provisioner passes it to the controller. But in the release manifests both of the driver's service accounts, the controller's and the nodes', are given a ClusterRole with get on all secrets in the cluster: the node DaemonSet reads them itself for inline ephemeral volumes. So whoever can kubectl exec into a driver pod in kube-system can read any Secret in the cluster — access to kube-system is equivalent to administrator rights anyway, but it is worth keeping in mind.

The driver then passes the password to mount.cifs on the command line mount -t cifs -o …,username=…,password=… (if the password contains a comma, a quote or a backtick — through a temporary credentials file). It does not end up in the driver's log — <masked> stands in its place there — but for the duration of the mount it is visible in the arguments of the mount process on the node. One more reason to keep only what the cluster needs on the nodes.

Password rotation. Changing the password in the Secret does not change it in volumes already mounted: the kernel remembers the password the volume was mounted with and uses it on every session reconnect. Let us check what happens if the password is changed on the server: working volumes keep writing while the session is alive, but after the very first disconnect (here — systemctl restart smbd on the server) the client tries to log in with the old password every 10 seconds, and the application gets Permission denied — even on a volume with hard:

user@files:~$ sudo smbpasswd svc-k8s-smb
New SMB password:
Retype new SMB password:
user@files:~$ sudo systemctl restart smbd
user@k8s-node-1:~$ sudo dmesg -T | grep -i cifs | tail -3
[Fri Oct 30 12:34:12 2026] CIFS: Status code returned 0xc000006d STATUS_LOGON_FAILURE
[Fri Oct 30 12:34:22 2026] CIFS: Status code returned 0xc000006d STATUS_LOGON_FAILURE
[Fri Oct 30 12:34:32 2026] CIFS: Status code returned 0xc000006d STATUS_LOGON_FAILURE
user@k8s-node-1:~$ sudo grep 'Session Status' /proc/fs/cifs/DebugData
	1) Address: 192.168.10.31 Uses: 1 Capability: 0x30006f	Session Status: 3 password no longer valid
$ kubectl -n demo logs writer-bc8b4f458-k6z47 | tail -2
sh: can't create /reports/visits.log: Permission denied
sh: can't create /reports/visits.log: Permission denied

This is misleading: "Permission denied" looks like a file permission error, while in fact the session cannot authenticate. When the old password was restored on the server, the client logged in on the next attempt, and writing resumed by itself, without a remount.

So the reliable way to change the password is two accounts with identical rights that take turns: create the second one, put it in the Secret, remount the volumes on all nodes (the order is below), make sure in /proc/fs/cifs/DebugData and in smbstatus on the server that the sessions use the new one, and only then change the password of the first or disable it. The kernel and mount.cifs also support an alternate password (password2, which the client tries if the main one fails), but csi-driver-smb takes only username, password and domain from the Secret.

StorageClass

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: smb
provisioner: smb.csi.k8s.io
parameters:
  source: //files.example.com/k8s
  subDir: ${pvc.metadata.namespace}-${pvc.metadata.name}-${pv.metadata.name}
  onDelete: archive
  csi.storage.k8s.io/provisioner-secret-name: smb-files-k8s
  csi.storage.k8s.io/provisioner-secret-namespace: storage-credentials
  csi.storage.k8s.io/node-stage-secret-name: smb-files-k8s
  csi.storage.k8s.io/node-stage-secret-namespace: storage-credentials
reclaimPolicy: Retain
volumeBindingMode: Immediate
allowVolumeExpansion: true
mountOptions:
  - vers=3
  - seal
  - dir_mode=0770
  - file_mode=0660
  - uid=10001
  - cache=strict
  - actimeo=1
  - nobrl
  - nolease
  - mfsymlinks
  - hard

source — the share as //server/share. The name is resolved by the driver's plugins through the cluster's CoreDNS (dnsPolicy: ClusterFirstWithHostNet, like the NFS driver).
subDir, onDelete — the same as with the NFS driver: name templates (with ${pv.metadata.name}, so that a new PVC with the same name does not get the old directory) and delete/retain/archive. And likewise, with reclaimPolicy: Retain, onDelete fires only after the administrator switches a released PV to Delete.
provisioner-secret-* — credentials for the controller: with them it mounts the share and creates the volume directory; without them no directory is created at all. The Secret's name is recorded in the PV's annotations, and DeleteVolume uses it to mount the share on deletion. node-stage-secret-* — for kubelet and the node plugin, when mounting on the node. Here it is one Secret; if you like, the controller can be given a separate account.
reclaimPolicy: Retain — for valuable data, as in the previous part.

mountOptions line by line

vers=3 — "SMB 3.0 or newer": the client offers 3.0, 3.0.2 and 3.1.1, and the server picks the highest common one. SMB2 is excluded, so SMB3 encryption and signing are always available. A strict vers=3.1.1 is stronger thanks to negotiation integrity, but if even one of the servers that volumes with these options mount from can do at most 3.0.2 (this happens with NAS devices and older Windows), its volumes simply will not mount. vers=3 is an explicit version without that trap; if all servers are definitely 3.1.1, you can specify it. With our Samba, naturally, 3.1.1 is negotiated.

seal — require SMB3 encryption from the client side, even if the server does not require it. A server that does require encryption is honoured by the cifs client even without seal, so the option is a guarantee against a server configured more weakly than expected. sign — require signing; the option exists in the kernel (/proc/fs/cifs/mount_params), but the mount.cifs manual has no separate entry for it. With seal, separate signing is not needed: encrypted messages are already protected against tampering.

dir_mode and file_mode — the mode the client shows for directories and files (0755 by default). The server does not pass POSIX permissions to the client, so these modes are the only thing the client checks when a process on the node accesses a file. The driver's official example sets dir_mode=0777,file_mode=0777, and that is an anti-example: any process of any pod the volume is mounted into, with any UID, can read, write and execute everything, and the server does not see it — for the server, everything is done by one service account. 0770 and 0660 limit access to the volume's owner and group.

uid and gid — who on the node "owns" all the volume's files (the kernel turns on forceuid and forcegid by itself). uid=10001 is the application's UID. gid is not set in the class: the driver substitutes it from the pod's fsGroup (VOLUME_MOUNT_GROUP). There is one mount per node and PV, so all pods using the volume must have the same fsGroup: the group is set by the pod that mounted the volume on the node first.

cache — the caching mode. strict (default) — cache data only while the server has granted a lease or an oplock, that is, while nobody else changes the file; none — do not cache at all, every read goes to the server; loose — cache with no consistency guarantees, unacceptable for RWX volumes on several nodes.

actimeo — how many seconds to cache file attributes and directory contents (1 by default). More means fewer requests to the server, but another node will not see a new file for longer. For RWX volumes it is left at the default or set to 0.

nobrl — do not send byte-range locks to the server, keep them on the client only. Windows enforces locks mandatorily: if one client has locked a range, a read of that range by another client gets an error instead of waiting. Some runtimes turn a file's share mode into a flock on the whole file, cifs sends it to the server, and a read from a neighbouring pod during a write ends with EACCES ("Permission denied", although permissions have nothing to do with it). The price of nobrl: locks stop working across nodes. It can be enabled only if the application does not rely on locks for data consistency between pods.

nolease — do not request leases or oplocks from the server. The server will not have to recall them from other clients, and the client does not cache data (with cache=strict and no lease, caching does not turn on anyway). For volumes that pods on different nodes read and write at the same time, this costs almost nothing; why it is needed is shown in the case study below.

mfsymlinks — store symbolic links as special text files (the Minshall–French format). Without POSIX extensions an SMB server does not keep symbolic links in a form Linux understands, while many applications and package managers rely on them; mfsymlinks files are understood by any server. In /proc/mounts of newer kernels the option shows up as symlink=mfsymlinks.

noperm is not on the list, and deliberately so: the option disables permission checks on the client, and then dir_mode, file_mode, uid and gid no longer restrict anything. It is advised when "permissions are not enough"; the right cure is correct uid, gid and fsGroup.

hard — as with NFS: on a disconnect, requests wait for recovery instead of failing. The cifs default is soft (that is what findmnt showed for the manual mount), and operations caught in the reconnect window fail immediately. hard hides short disconnects from the application but does not remove their cause; the price is the same as in the previous part — processes in the D state if the server is gone for long.

As with the NFS driver, DeleteVolume does not get the class options: on the lab stand the controller mounted the share for deletion with -o ,file_mode=0777,dir_mode=0777,<masked> — no vers and no seal. With our server that works (the client negotiates 3.1.1 by itself and encrypts because the server requires it), and for a server that needs special options they go into the same Secret under the mountOptions key — the driver takes them from there.

Kerberos instead of a password

If the file server is in a domain, mounting can use Kerberos: the sec=krb5 and cruid=<UID> options tell the client to take a ticket from that UID's credential cache. The driver supports this as follows: the ticket cache (a ccache file obtained with kinit for the service account, in base64) is put into the Secret under the key krb5cc_<UID> matching cruid, and on the node the driver writes it to /var/lib/kubelet/kerberos/ (the directory must exist — the driver does not create it). The ticket for the server is obtained by cifs.upcall, which the kernel runs on the node itself, so the nodes need cifs-utils, configured Kerberos and a default_ccache_name line pointing to that directory; the Helm chart can write that line itself (linux.krb5AutoConfig=true writes /etc/krb5.conf.d/90-csi-driver-smb.conf), the rest of the Kerberos setup is up to the node administrator. There is no password in the cluster, but a ticket has a limited lifetime, and the cache in the Secret must be refreshed regularly by an external process — otherwise after a few hours or days the volumes will stop reconnecting. For most clusters this is more work than benefit; it is justified where policy forbids NTLM. Kerberos was not tested on the lab stand.

A volume for an application, and permissions

A PVC and a two-replica Deployment in the demo namespace from part 11 — the same writer on busybox as in the previous part, only the volume is now reports on SMB:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: reports
  namespace: demo
spec:
  accessModes: ["ReadWriteMany"]
  storageClassName: smb
  resources:
    requests:
      storage: 5Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: writer
  namespace: demo
spec:
  replicas: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: writer
  template:
    metadata:
      labels:
        app.kubernetes.io/name: writer
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        runAsGroup: 10001
        fsGroup: 10001
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: writer
          image: busybox:1.37
          command: ["sh", "-c", "while true; do echo \"$(date -u +%FT%TZ) $HOSTNAME\" >> /reports/visits.log; sleep 10; done"]
          volumeMounts:
            - name: reports
              mountPath: /reports
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
      volumes:
        - name: reports
          persistentVolumeClaim:
            claimName: reports

$ kubectl apply -f sc.yaml -f writer.yaml
storageclass.storage.k8s.io/smb created
persistentvolumeclaim/reports created
deployment.apps/writer created
$ kubectl -n demo get pvc reports -w
NAME      STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
reports   Pending                                      smb            <unset>                 0s
reports   Pending   pvc-fc91f7d5-6db4-471b-a083-3b93c0bdae27   0                         smb            <unset>                 1s
reports   Bound     pvc-fc91f7d5-6db4-471b-a083-3b93c0bdae27   5Gi        RWX            smb            <unset>                 1s
$ kubectl -n demo get pods -o wide
NAME                      READY   STATUS    RESTARTS   AGE   IP              NODE         NOMINATED NODE   READINESS GATES
writer-554fccb9dc-8vb4g   1/1     Running   0          28s   172.20.172.10   k8s-node-2   <none>           <none>
writer-554fccb9dc-kvd28   1/1     Running   0          27s   172.20.211.60   k8s-node-1   <none>           <none>

The volume was created within a second — this is what it looks like in the controller's log (--v=5 in the manifest, lines shortened):

$ kubectl -n kube-system logs deploy/csi-smb-controller -c smb | grep -E 'CreateVolume|internally|Mounting cmd'
I1030 12:26:51.685032       1 utils.go:76] GRPC call: /csi.v1.Controller/CreateVolume
I1030 12:26:51.685043       1 utils.go:77] GRPC request: {"capacity_range":{"required_bytes":5368709120},"name":"pvc-fc91f7d5-6db4-471b-a083-3b93c0bdae27",…,"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{"mount_flags":["vers=3","seal","dir_mode=0770","file_mode=0660","uid=10001","cache=strict","actimeo=1","nobrl","nolease","mfsymlinks","hard"]}},"access_mode":{"mode":5}}]}
I1030 12:26:51.685200       1 controllerserver.go:350] internally mounting //files.example.com/k8s at /tmp/pvc-fc91f7d5-6db4-471b-a083-3b93c0bdae27
I1030 12:26:51.685419       1 mount_linux.go:270] Mounting cmd (mount) with arguments (-t cifs -o vers=3,seal,dir_mode=0770,file_mode=0660,uid=10001,cache=strict,actimeo=1,nobrl,nolease,mfsymlinks,hard,<masked> //files.example.com/k8s /tmp/pvc-fc91f7d5-6db4-471b-a083-3b93c0bdae27)
I1030 12:26:51.699383       1 controllerserver.go:368] internally unmounting /tmp/pvc-fc91f7d5-6db4-471b-a083-3b93c0bdae27

The share root is mounted with the class options, the directory is created, the share is unmounted — all in 17 ms. <masked> is the spot where the real command line has the user name and password.

And here is what the node plugin on k8s-node-1 received:

$ kubectl -n kube-system logs csi-smb-node-929cp -c smb | grep -E 'NodeStageVolume: targetPath|Mounting cmd' | head -2
I1030 12:26:53.298942       1 nodeserver.go:297] NodeStageVolume: targetPath(/var/lib/kubelet/plugins/kubernetes.io/csi/smb.csi.k8s.io/d94cf53eb14c47a64c6df39efd71480c44ad55bf9fc4712e6cb3e4619397435b/globalmount) volumeID(files.example.com/k8s#demo-reports-pvc-fc91f7d5-6db4-471b-a083-3b93c0bdae27#pvc-fc91f7d5-6db4-471b-a083-3b93c0bdae27#archive) context(…) mountflags([vers=3 seal dir_mode=0770 file_mode=0670 uid=10001 cache=strict actimeo=1 nobrl nolease mfsymlinks hard]) mountOptions([vers=3 seal dir_mode=0770 file_mode=0670 uid=10001 cache=strict actimeo=1 nobrl nolease mfsymlinks hard gid=10001])
I1030 12:26:53.299169       1 mount_linux.go:270] Mounting cmd (mount) with arguments (-t cifs -o vers=3,seal,dir_mode=0770,file_mode=0670,uid=10001,cache=strict,actimeo=1,nobrl,nolease,mfsymlinks,hard,gid=10001,<masked> //files.example.com/k8s/demo-reports-pvc-fc91f7d5-6db4-471b-a083-3b93c0bdae27 /var/lib/kubelet/plugins/kubernetes.io/csi/smb.csi.k8s.io/d94cf53eb14c47a64c6df39efd71480c44ad55bf9fc4712e6cb3e4619397435b/globalmount)

kubelet's request carried "volume_mount_group":"10001" — the pod's fsGroup. The driver appended gid=10001 and raised the group's rights: file_mode=0660 from the class became 0670. Inside the pod it looks like this:

$ kubectl -n demo exec deploy/writer -- sh -c 'id; ls -lan /reports; tail -2 /reports/visits.log; df -h /reports'
uid=10001 gid=10001 groups=10001
total 8
drwxrwx---    2 10001    10001            0 Oct 30 12:26 .
drwxr-xr-x    1 0        0             4096 Oct 30 12:26 ..
-rw-rwx---    1 10001    10001          450 Oct 30 12:27 visits.log
2026-10-30T12:27:37Z writer-554fccb9dc-kvd28
2026-10-30T12:27:37Z writer-554fccb9dc-8vb4g
Filesystem                Size      Used Available Use% Mounted on
//files.example.com/k8s/demo-reports-pvc-fc91f7d5-6db4-471b-a083-3b93c0bdae27
                         30.3G      7.2G     23.1G  24% /reports

The application writes without being root and without 0777, and both replicas append lines to the same file. But the file now has the execute bit for the group — a side effect of how the driver handles fsGroup. If that is undesirable, set gid in the class yourself (gid=10001 next to uid=10001): then the driver does not touch the modes, and the pod's fsGroup has no effect on this volume. On the lab stand a class with an explicit gid gave exactly file_mode=0660. Since uid is in the class anyway, the class effectively belongs to one application, and an explicit gid in it complicates nothing.

PVC size is not a quota, as with NFS: df shows the server's whole disk, not 5 GiB. Limits are enforced with quotas on the server (FSRM folder quotas on Windows, file system quotas with Samba) and by monitoring used space.

One connection and one session per node

In the previous part it turned out that there is one NFS client per "node — server" pair, and the connection options are set by the first mount. With cifs it is similar, but stricter. Let us add a second volume from the same server, from a class with different options (no seal, an explicit gid), and a pod on k8s-node-1 that mounts both volumes:

user@k8s-node-1:~$ sudo grep -E '^[0-9]\)|Sessions:|Session Status|\\\\' /proc/fs/cifs/DebugData
1) ConnectionId: 0x5 Hostname: files.example.com
	Sessions:
	1) Address: 192.168.10.31 Uses: 2 Capability: 0x30006f	Session Status: 1
	0) IPC: \\files.example.com\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
	1) \\files.example.com\k8s Mounts: 1 DevInfo: 0x20 Attributes: 0x1006f
	2) \\files.example.com\k8s Mounts: 1 DevInfo: 0x20 Attributes: 0x1006f
user@k8s-node-1:~$ sudo ss -tn -4 dst 192.168.10.31
State Recv-Q Send-Q Local Address:Port  Peer Address:Port
ESTAB 0      0      192.168.10.20:42516 192.168.10.31:445

All volumes of one server under one account on a node go through one TCP connection and one session (Uses: 2) — one logon, one set of keys. But there are two share connections: seal on one volume and its absence on the other mean different tree connects. A manual mount with the same options as the second class joined the existing tree connect (Mounts: 2). Unlike NFS, cifs does not silently attach a mount with incompatible options: according to the client's sources, a different dialect, echo_interval, retrans or security parameters give a separate connection, a different account gives a separate session, and a different seal value gives a separate tree connect.

There are two practical consequences. First: a dropped connection or a failed re-logon hits all volumes of this server on the node at once. Second: a password change, as we saw, breaks them all together too — one more reason to rotate accounts by procedure.

When the server is unavailable

Let us cut k8s-node-2 off from the server with a rule at the top of the INPUT chain on the server (as in the previous part, ufw deny does not work for this: it lets established connections through) and wait for almost four minutes:

user@files:~$ sudo iptables -I INPUT 1 -s 192.168.10.21 -p tcp --dport 445 -j DROP
user@k8s-node-2:~$ ps -eo pid,stat,wchan:25,cmd | grep 'while true' | grep -v grep     # after 30 seconds
 319125 Ds   -                         sh -c while true; do echo "$(date -u +%FT%TZ) $HOSTNAME" >> /reports/visits.log; sleep 10; done
user@k8s-node-2:~$ sudo dmesg -T | grep -i cifs | tail -1                               # after 3 minutes
[Fri Oct 30 12:40:23 2026] CIFS: VFS: \\files.example.com has not responded in 180 seconds. Reconnecting...
$ timeout 15 kubectl -n demo exec writer-bc8b4f458-k6z47 -- ls -l /reports; echo rc=$?
rc=124
user@files:~$ sudo iptables -D INPUT -s 192.168.10.21 -p tcp --dport 445 -j DROP

The pod's process hangs in the D state on a write, and kubectl exec with ls does not return — as with NFS. The client noticed the server was gone only after 180 seconds: it sends echo requests every echo_interval (60 s by default) and declares the server unresponsive when nothing has come from it for three intervals. After the rule was removed the client reconnected within seconds, and the write the process had been hanging on for almost four minutes completed:

user@files:~$ sudo grep -n -E 'T12:3[7-9]|T12:4[01]' /srv/samba/k8s/demo-reports-pvc-fc91f7d5-6db4-471b-a083-3b93c0bdae27/visits.log | sed -n '1,5p;26,31p'
130:2026-10-30T12:37:03Z writer-bc8b4f458-8wkrh
131:2026-10-30T12:37:06Z writer-bc8b4f458-k6z47
132:2026-10-30T12:37:13Z writer-bc8b4f458-8wkrh
133:2026-10-30T12:37:16Z writer-bc8b4f458-k6z47
134:2026-10-30T12:37:23Z writer-bc8b4f458-8wkrh
155:2026-10-30T12:40:53Z writer-bc8b4f458-8wkrh
156:2026-10-30T12:41:03Z writer-bc8b4f458-8wkrh
157:2026-10-30T12:37:26Z writer-bc8b4f458-k6z47
158:2026-10-30T12:41:13Z writer-bc8b4f458-8wkrh
159:2026-10-30T12:41:17Z writer-bc8b4f458-k6z47
160:2026-10-30T12:41:23Z writer-bc8b4f458-8wkrh

The line stamped 12:37:26 was written after 12:41:03: it is the very write that was waiting for the server — hard did not lose it. The replica on k8s-node-1 kept writing all along as if nothing had happened. With soft (the default), after the reconnect the application would have got an error on this write — EHOSTDOWN, "Host is down" — and we will meet it in the case study below. What to do when the server is down for long is the same as with NFS in the previous part: fix the server or the network first, not the pods.

Case study: "Host is down" on SMB volumes

A case from practice, in a cluster with a NAS from one vendor. Pods periodically got the Host is down error (EHOSTDOWN) when working with files on an SMB volume. The NAS itself was reachable; the vendor's support saw "ordinary client disconnects".

Kernel log. On the nodes — frequent cifs reconnects, dozens per minute on busy nodes, and before each one the same line:

user@k8s-node-1:~$ sudo dmesg -T | grep -i cifs | tail -4
[Thu Sep 24 07:52:44 2026] CIFS: VFS: \\<NAS address> RFC 1002 unknown response type 0xfe
[Thu Sep 24 07:52:44 2026] CIFS: VFS: \\<NAS address> RFC 1002 unknown response type 0xfe
[Thu Sep 24 07:52:45 2026] CIFS: VFS: \\<NAS address> RFC 1002 unknown response type 0xfe
[Thu Sep 24 07:52:45 2026] CIFS: VFS: \\<NAS address> RFC 1002 unknown response type 0xfe

The client expected a four-byte frame header with a zero type and read the byte 0xfe — the first byte of the SMB2 signature. That is, it had lost sync with the stream: it reads a frame header where there is actually the middle of a message. After that the client drops the connection and reconnects, and on a soft mount all operations caught in that window immediately get EHOSTDOWN. The times of errors in the application log matched these lines to the second.

Traffic capture. tcpdump on a node for a few dozen seconds and analysis in Wireshark showed: the framing on the wire is intact, the server does not corrupt the stream. The connection was always closed by the client, a fraction of a millisecond after one and the same frame from the NAS:

 ┌─────────────────┬──────────────────────┬──────────────────────┐
 │ 00 00 00 99     │ SMB2 CREATE          │ SMB2 CLOSE           │
 │ frame header    │ STATUS_PENDING       │ STATUS_PENDING       │
 │ length 153      │ NextCommand = 80     │ related command      │
 └─────────────────┴──────────────────────┴──────────────────────┘
   client reads      parses the first       FE 53 4D 42 taken
   frame length      interim response       for the header of a
                     and considers the      new frame: "type 0xfe"
                     frame finished

This is the reply to the client's compound CREATE + CLOSE request, which cifs uses for stat and similar operations. When the file being opened was held under a lease by another client — another pod on another node — the NAS sent the holder a lease break and, while waiting, answered the compound request with two interim STATUS_PENDING responses in one frame. The cifs client processed the first interim response and went to read the next frame header from the socket without reading the second part of the compound frame. It took the beginning of the second response for a header — hence 0xfe and the disconnect.

This explained everything else: the error occurred only with this NAS (other servers do not reply this way), did not depend on the dialect and grew with concurrent access to the same files — the rate jumped when one of the applications started running in a large number of replicas on all nodes. In the client's sources it is plain to see: in the cifs_demultiplex_thread receive loop an interim STATUS_PENDING response ends the processing of the frame, and the loop goes to read a new header, bypassing the branch that reads the rest of a compound frame. This code is the same in kernels 5.15, 6.8, 7.0 and in mainline as of October 2026: a kernel upgrade does not help.

The cure — mount options.

nolease — the cluster's clients stop requesting leases and oplocks. The NAS does not need to recall them from neighbouring pods before opening a file, and the very reason to answer STATUS_PENDING goes away. These volumes had cache=none, so the client was not caching anything anyway, and there was almost no loss.
nobrl — for an adjacent problem: conflicts of mandatory byte-range locks gave pods EACCES on reads during writes. The applications did not rely on cross-node locks, so locks were kept on the client.
vers=3 — at first they tried lowering the dialect to 3.0, but the disconnects remained: the cause was not the dialect. They returned to the highest common one (3.1.1 with the NAS), but not strictly: other servers the same volumes worked with had 3.0.2 as their maximum, and vers=3.1.1 would have broken those mounts.

After remounting there were two orders of magnitude fewer disconnects: from 50–80 per minute per node to 5–11 per node in 25 minutes. The remaining ones come from other NAS clients that keep taking oplocks (Windows workstations, for example); nolease in the cluster does not affect them. The next line of defence, had that not been enough, is hard: it does not remove the disconnects but hides them from the application, like the write that waited out the server outage above. In the case study it was kept as a fallback, because hard turns a real NAS outage into hung processes.

How to remount volumes

Options take effect only on a new mount, and this is easy to get wrong. In dynamic PVs mountOptions were copied from the class at creation time, so new options are added to every PV (kubectl patch pv … --type=json, as in the previous part; the same for static PVs). But even after that, kubectl rollout restart changes nothing: the NodeStage mount is one per node and PV, and in a rolling update the new pod comes up on the node before the old one leaves, so the mount is never released even for a moment. Let us check on the lab stand by changing actimeo=1 to actimeo=5 in the PV:

$ kubectl patch pv pvc-fc91f7d5-6db4-471b-a083-3b93c0bdae27 --type=json \
    -p '[{"op":"replace","path":"/spec/mountOptions","value":["vers=3","seal","dir_mode=0770","file_mode=0660","uid=10001","cache=strict","actimeo=5","nobrl","nolease","mfsymlinks","hard"]}]'
persistentvolume/pvc-fc91f7d5-6db4-471b-a083-3b93c0bdae27 patched
$ kubectl -n demo rollout restart deploy/writer
deployment.apps/writer restarted
$ kubectl -n demo get pods -o wide        # after the rollout
NAME                      READY   STATUS    RESTARTS   AGE   IP              NODE         NOMINATED NODE   READINESS GATES
writer-64474f4896-2g9cn   1/1     Running   0          74s   172.20.211.55   k8s-node-1   <none>           <none>
writer-64474f4896-8hvkq   1/1     Running   0          69s   172.20.172.7    k8s-node-2   <none>           <none>
user@k8s-node-1:~$ grep -o 'globalmount cifs [^ ]*' /proc/mounts | grep -o 'actimeo=[0-9]*'
actimeo=1

New pods on both nodes, and the old mount. The order is one node at a time:

kubectl cordon k8s-node-1 — new pods will not land on the node.
Delete the pods on the node that use SMB volumes (or kubectl drain if nothing on the node would mind moving); their controllers will bring up replacements on other nodes.
Wait until the node has no mounts from this server left: grep cifs /proc/mounts is empty (NodeUnstage unmounted the global mount).
kubectl uncordon k8s-node-1, wait for the deployments to become ready, move on to the next node.
At the end — kubectl rollout restart of the same deployments so the pods spread across the nodes again.

$ kubectl cordon k8s-node-1
node/k8s-node-1 cordoned
$ kubectl -n demo delete pod --field-selector spec.nodeName=k8s-node-1 -l app.kubernetes.io/name=writer
pod "writer-64474f4896-2g9cn" deleted from demo namespace
user@k8s-node-1:~$ grep cifs /proc/mounts || echo '(no cifs mounts)'
(no cifs mounts)
$ kubectl uncordon k8s-node-1
node/k8s-node-1 uncordoned

The same for k8s-node-2, then rollout restart, and on both nodes /proc/mounts shows actimeo=5. If the applications have no PodDisruptionBudget, you will have to watch the number of simultaneously unavailable replicas in step 2 yourself (PodDisruptionBudget is covered in part 13).

Hands-on

The PV — right after creation:

$ kubectl get pv pvc-fc91f7d5-6db4-471b-a083-3b93c0bdae27 -o yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: smb.csi.k8s.io
    volume.kubernetes.io/provisioner-deletion-secret-name: smb-files-k8s
    volume.kubernetes.io/provisioner-deletion-secret-namespace: storage-credentials
  creationTimestamp: "2026-10-30T12:26:51Z"
  finalizers:
  - kubernetes.io/pv-protection
  name: pvc-fc91f7d5-6db4-471b-a083-3b93c0bdae27
  resourceVersion: "1133559"
  uid: 9d0558a4-4e96-4939-9061-3c0711b2c83f
spec:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 5Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: reports
    namespace: demo
    resourceVersion: "1133549"
    uid: fc91f7d5-6db4-471b-a083-3b93c0bdae27
  csi:
    driver: smb.csi.k8s.io
    nodeStageSecretRef:
      name: smb-files-k8s
      namespace: storage-credentials
    volumeAttributes:
      csi.storage.k8s.io/pv/name: pvc-fc91f7d5-6db4-471b-a083-3b93c0bdae27
      csi.storage.k8s.io/pvc/name: reports
      csi.storage.k8s.io/pvc/namespace: demo
      onDelete: archive
      source: //files.example.com/k8s
      storage.kubernetes.io/csiProvisionerIdentity: 1793363183585-5388-smb.csi.k8s.io
      subDir: demo-reports-pvc-fc91f7d5-6db4-471b-a083-3b93c0bdae27
    volumeHandle: files.example.com/k8s#demo-reports-pvc-fc91f7d5-6db4-471b-a083-3b93c0bdae27#pvc-fc91f7d5-6db4-471b-a083-3b93c0bdae27#archive
  mountOptions:
  - vers=3
  - seal
  - dir_mode=0770
  - file_mode=0660
  - uid=10001
  - cache=strict
  - actimeo=1
  - nobrl
  - nolease
  - mfsymlinks
  - hard
  persistentVolumeReclaimPolicy: Retain
  storageClassName: smb
  volumeMode: Filesystem
status:
  lastPhaseTransitionTime: "2026-10-30T12:26:51Z"
  phase: Bound

The volumeHandle has four parts: server and share, subdirectory, PV name and deletion policy. The Secret is referenced in two places: nodeStageSecretRef — for mounting on the node, and the provisioner-deletion-secret-* annotations — for DeleteVolume. There is, of course, no password in the PV.

The mounts on the node — the global one and the bind mount into the pod (after the actimeo change):

user@k8s-node-1:~$ grep cifs /proc/mounts
//files.example.com/k8s/demo-reports-pvc-fc91f7d5-6db4-471b-a083-3b93c0bdae27 /var/lib/kubelet/plugins/kubernetes.io/csi/smb.csi.k8s.io/d94cf53eb14c47a64c6df39efd71480c44ad55bf9fc4712e6cb3e4619397435b/globalmount cifs rw,relatime,vers=3.1.1,cache=strict,upcall_target=app,nolease,username=svc-k8s-smb,uid=10001,forceuid,gid=10001,forcegid,addr=192.168.10.31,file_mode=0670,dir_mode=0770,seal,hard,nounix,serverino,mapposix,nobrl,mfsymlinks,reparse=nfs,nativesocket,symlink=mfsymlinks,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=5,closetimeo=1 0 0
//files.example.com/k8s/demo-reports-pvc-fc91f7d5-6db4-471b-a083-3b93c0bdae27 /var/lib/kubelet/pods/55ea45b3-af45-4971-804b-2f299042f8bc/volumes/kubernetes.io~csi/pvc-fc91f7d5-6db4-471b-a083-3b93c0bdae27/mount cifs rw,relatime,vers=3.1.1,cache=strict,upcall_target=app,nolease,username=svc-k8s-smb,uid=10001,forceuid,gid=10001,forcegid,addr=192.168.10.31,file_mode=0670,dir_mode=0770,seal,hard,nounix,serverino,mapposix,nobrl,mfsymlinks,reparse=nfs,nativesocket,symlink=mfsymlinks,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=5,closetimeo=1 0 0

Two lines with identical options. vers=3.1.1 is the negotiated dialect. gid=10001 and file_mode=0670 are the driver's handling of fsGroup. nounix — POSIX extensions were not requested, owners and modes are emulated. There is no password in the list: it was passed to the kernel separately. This is where options are checked, not in the manifest: only /proc/mounts shows what the volume is actually mounted with.

The client's internal state — connection, dialect, encryption, session, shares:

user@k8s-node-1:~$ sudo cat /proc/fs/cifs/DebugData
Display Internal CIFS Data Structures for Debugging
---------------------------------------------------
CIFS Version 2.59
Features: DFS,FSCACHE,STATS,DEBUG,ALLOW_INSECURE_LEGACY,CIFS_POSIX,UPCALL(SPNEGO),XATTR,ACL,WITNESS
CIFSMaxBufSize: 16384
Active VFS Requests: 0

Servers:
1) ConnectionId: 0x7 Hostname: files.example.com
ClientGUID: D96C6663-4CB8-E649-9719-6ABE45E30B22
Number of credits: 4149,1,1 Dialect 0x311 signed posix
Server capabilities: 0x30006f
TCP status: 1 Instance: 3
Local Users To Server: 1 SecMode: 0x3 Req On Wire: 0 Net namespace: 4026531833
In Send: 0 In MaxReq Wait: 0
Compression: no built-in support
Encryption: Negotiated cipher (AES128-GCM)

	Sessions:
	1) Address: 192.168.10.31 Uses: 1 Capability: 0x30006f	Session Status: 1
	Security type: RawNTLMSSP  SessionId: 0x80bf4ed6 encrypted
	User: 10001 Cred User: 0

	Shares:
	0) IPC: \\files.example.com\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
	PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 encrypted
	Share Capabilities: None	Share Flags: 0x8000
	tid: 0x50da3957	Maximal Access: 0x1f00a9

	1) \\files.example.com\k8s Mounts: 1 DevInfo: 0x20 Attributes: 0x1006f
	PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0x5c3565d6 encrypted
	Share Capabilities: None Aligned, Partition Aligned,	Share Flags: 0x8000
	tid: 0x51760654	Optimal sector size: 0x200	Maximal Access: 0x1f01ff
...

Dialect 0x311 — SMB 3.1.1; signed and SecMode: 0x3 — signing is enabled and required; posix — the server offers POSIX extensions (Samba 4.23 supports them by default), but the mount did not ask for them. Encryption: Negotiated cipher (AES128-GCM) is the negotiated cipher, and the session and shares are marked encrypted. Security type: RawNTLMSSP — logon via NTLM. Number of credits is how many requests the client can keep in flight; In MaxReq Wait above zero means requests are waiting for credits. Status: 1 on the session and share means connected; Instance: 3 is the TCP reconnect number: this node lived through the smbd restart and the password change from the experiments above. Further down the file is the list of the server's network interfaces for multichannel, which we omitted.

Operation and reconnect counters:

user@k8s-node-1:~$ sudo cat /proc/fs/cifs/Stats
Resources in use
CIFS Session: 1
Share (unique mount targets): 2
SMB Request/Response Buffer: 1 Pool size: 5
SMB Small Req/Resp Buffer: 1 Pool size: 30
Operations (MIDs): 0

2 session 2 share reconnects
Total vfs operations: 616 maximum at one time: 3

Max requests in flight: 3
1) \\files.example.com\k8s
SMBs: 553 since 2026-10-30 12:32:19 UTC
Bytes read: 0  Bytes written: 3403
Open files: 0 total (local), 0 open on server
TreeConnects: 2 total 0 failed
TreeDisconnects: 0 total 0 failed
Creates: 189 total 0 failed
Closes: 178 total 0 failed
Flushes: 1 total 0 failed
Reads: 0 total 0 failed
Writes: 77 total 0 failed
Locks: 0 total 0 failed
IOCTLs: 4 total 0 failed
QueryDirectories: 4 total 0 failed
ChangeNotifies: 0 total 0 failed
QueryInfos: 96 total 0 failed
SetInfos: 2 total 0 failed
OplockBreaks: 0 sent 0 failed

session reconnects and share reconnects are the main health figures: in a normal state they stay put. Here there are two of each — from the smbd restart and the failed logons after the password change. In the case study above, the session reconnect counter on busy nodes ran into the millions over a few weeks — that is where the problem should have been looked for, sooner than in application complaints. Locks: 0 — nobrl works, locks do not go to the server; OplockBreaks: 0 — with nolease there is nothing to break.

The kernel log:

user@k8s-node-1:~$ sudo dmesg -T | grep -i cifs | tail -2
[Fri Oct 30 12:43:57 2026] CIFS: enabling forceuid mount option implicitly because uid= option is specified
[Fri Oct 30 12:43:57 2026] CIFS: Attempting to mount //files.example.com/k8s

On mount — Attempting to mount and notices about implicit options, nothing else — good. Lines has not responded in N seconds. Reconnecting... mean the server did not respond (network or the server itself), unknown response type — the stream lost sync, as in the case study, STATUS_LOGON_FAILURE and failed to create a new SMB session — an authentication error (often a changed password). The time in dmesg -T is computed from the boot time and a kernel counter and may differ from the real time (by 17 seconds on the lab stand); for exact matching with application logs journalctl -k is better.

On the other side — the server. smbstatus shows who is connected, with which dialect, encryption and signing:

user@files:~$ sudo smbstatus

Samba version 4.23.6-Ubuntu-4.23.6+dfsg-1ubuntu2.2
PID     Username     Group        Machine                                   Protocol Version  Encryption           Signing
----------------------------------------------------------------------------------------------------------------------------------------
14130   svc-k8s-smb  svc-k8s-smb  192.168.10.20 (ipv4:192.168.10.20:57814)  SMB3_11           AES-128-GCM          AES-128-CMAC
14131   svc-k8s-smb  svc-k8s-smb  192.168.10.21 (ipv4:192.168.10.21:55986)  SMB3_11           AES-128-GCM          AES-128-CMAC

Service      pid     Machine       Connected at                     Encryption   Signing
---------------------------------------------------------------------------------------------
IPC$         14131   192.168.10.21 Fri Oct 30 12:41:07 PM 2026 UTC  AES-128-GCM  -
k8s          14131   192.168.10.21 Fri Oct 30 12:41:07 PM 2026 UTC  AES-128-GCM  AES-128-CMAC
IPC$         14130   192.168.10.20 Fri Oct 30 12:35:11 PM 2026 UTC  AES-128-GCM  AES-128-CMAC
k8s          14130   192.168.10.20 Fri Oct 30 12:35:04 PM 2026 UTC  AES-128-GCM  AES-128-CMAC

No locked files

One smbd process per node: one connection per node, just as DebugData showed. The connection time is the moment of the last reconnect: k8s-node-2 reconnected at 12:41:07 after the outage experiment. No locked files — there are no locks on the server.

Traffic for analysis in Wireshark — a graphical network protocol analyzer that can dissect SMB2/3 field by field:

user@k8s-node-1:~$ sudo timeout 12 tcpdump -i any -s 0 -w /tmp/smb.pcap host 192.168.10.31 and port 445
tcpdump: WARNING: any: That device doesn't support promiscuous mode
(Promiscuous mode not supported on the "any" device)
tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
19 packets captured
21 packets received by filter
0 packets dropped by kernel
user@k8s-node-1:~$ sudo tcpdump -r /tmp/smb.pcap -nn -X -c 1 'tcp[tcpflags] & tcp-push != 0' 2>/dev/null | sed -n '1p;5,6p'
12:36:43.914216 ens18 Out IP 192.168.10.20.57814 > 192.168.10.31.445: Flags [P.], seq 1371088206:1371088734, ack 1960316619, win 126, options [nop,nop,TS val 318912597 ecr 1880430020], length 528
	0x0030:  7015 15c4 0000 020c fd53 4d42 42cd d8c6  p........SMBB...
	0x0040:  ea98 d8db 62b6 ca70 6b4a 596b ab8d 18ce  ....b..pkJYk....

After the IP and TCP headers (52 bytes, the lines before 0x0030 are omitted) comes 00 00 02 0c — a frame header with a zero type and a length of 524 bytes — and after it fd 53 4d 42, the transform header: the message is encrypted. Commands and statuses inside are not visible — only frame lengths, timing and which side closed the TCP connection; dissecting by command requires the session key or a test share without encryption on which the problem reproduces. 0 packets dropped by kernel is a must: a capture with lost packets is useless for this kind of analysis. In Wireshark — the smb2 filter, the smb2.cmd, smb2.nt_status, smb2.flags.async columns; to find disconnects — tcp.flags.fin == 1 || tcp.flags.reset == 1, and look at which frame came right before.

Expansion and cloning. Expansion is instant: the driver just records the new size — it sets no quota on the share. A clone is a cp -a copy of files from the source volume's directory by the controller:

$ kubectl -n demo patch pvc reports -p '{"spec":{"resources":{"requests":{"storage":"10Gi"}}}}'
persistentvolumeclaim/reports patched
$ kubectl -n kube-system logs deploy/csi-smb-controller -c smb | grep 'ControllerExpandVolume(' | tail -1
I1030 12:44:10.097347       1 controllerserver.go:321] ControllerExpandVolume(files.example.com/k8s#demo-reports-pvc-fc91f7d5-6db4-471b-a083-3b93c0bdae27#pvc-fc91f7d5-6db4-471b-a083-3b93c0bdae27#archive) successfully, currentQuota: 10737418240 bytes
$ kubectl apply -f reports-copy.yaml        # a PVC with dataSource: kind PersistentVolumeClaim, name reports
persistentvolumeclaim/reports-copy created
$ kubectl -n kube-system logs deploy/csi-smb-controller -c smb | grep copied | tail -1
I1030 12:44:18.563142       1 controllerserver.go:415] copied /tmp/pvc-fc91f7d5-6db4-471b-a083-3b93c0bdae27/demo-reports-pvc-fc91f7d5-6db4-471b-a083-3b93c0bdae27/. -> /tmp/pvc-7026f4d6-053b-431d-8d0e-7361f1f3e8fb/demo-reports-copy-pvc-7026f4d6-053b-431d-8d0e-7361f1f3e8fb

A clone of a live volume is not a snapshot: files the application changed during the copy will end up in the copy in some intermediate state.

And the release. We delete the application and the claims — the PVs stay Released; the data has been checked and is no longer needed — we switch the PVs to Delete, and onDelete: archive fires:

$ kubectl -n demo delete deploy writer
deployment.apps "writer" deleted from demo namespace
$ kubectl -n demo delete pvc reports reports-copy
persistentvolumeclaim "reports" deleted from demo namespace
persistentvolumeclaim "reports-copy" deleted from demo namespace
$ kubectl patch pv pvc-fc91f7d5-6db4-471b-a083-3b93c0bdae27 -p '{"spec":{"persistentVolumeReclaimPolicy":"Delete"}}'
persistentvolume/pvc-fc91f7d5-6db4-471b-a083-3b93c0bdae27 patched
$ kubectl -n kube-system logs deploy/csi-smb-controller -c smb | grep -E 'Mounting cmd|archiving' | tail -2
I1030 12:45:31.137291       1 mount_linux.go:270] Mounting cmd (mount) with arguments (-t cifs -o ,file_mode=0777,dir_mode=0777,<masked> //files.example.com/k8s /tmp/pvc-fc91f7d5-6db4-471b-a083-3b93c0bdae27)
I1030 12:45:31.147817       1 controllerserver.go:224] archiving subdirectory /tmp/pvc-fc91f7d5-6db4-471b-a083-3b93c0bdae27/demo-reports-pvc-fc91f7d5-6db4-471b-a083-3b93c0bdae27 --> /tmp/pvc-fc91f7d5-6db4-471b-a083-3b93c0bdae27/archived-demo-reports-pvc-fc91f7d5-6db4-471b-a083-3b93c0bdae27
user@files:~$ sudo ls -la /srv/samba/k8s
total 16
drwxrwx--- 4 svc-k8s-smb svc-k8s-smb 4096 Oct 30 12:45 .
drwxr-xr-x 3 root        root        4096 Oct 30 12:22 ..
drwxrwx---+2 svc-k8s-smb svc-k8s-smb 4096 Oct 30 12:26 archived-demo-reports-copy-pvc-7026f4d6-053b-431d-8d0e-7361f1f3e8fb
drwxrwx---+2 svc-k8s-smb svc-k8s-smb 4096 Oct 30 12:26 archived-demo-reports-pvc-fc91f7d5-6db4-471b-a083-3b93c0bdae27

The second PV is switched the same way. The mount line without the class options is visible here too — -o ,file_mode=0777,dir_mode=0777. The archived-… directories are removed on the server by hand when they are definitely not needed.

Sources

csi-driver-smb: repository, release v1.20.4, release manifests, installing v1.20.4 and the Helm chart — driver components, images, RBAC, the chart version without v, krb5AutoConfig.
csi-driver-smb: driver parameters — source, subDir and its templates, onDelete, the Secret for the controller and the node, the volumeHandle format, mountOptions for DeleteVolume, Kerberos; StorageClass example — with the 0777 we do not repeat; node plugin source — gid from volume_mount_group, group rights, passing the password to mount.
Kubernetes CSI: CSIDriver Object and fsGroup Support — fsGroupPolicy, podInfoOnMount, delegating fsGroup to the driver; Secrets and Credentials — the provisioner-secret-* and node-stage-secret-* parameters; CSI Ephemeral Inline Volumes — the Ephemeral mode.
Container Storage Interface Specification — NodeStageVolume, VOLUME_MOUNT_GROUP and the volume_mount_group field, DeleteVolume without mount options.
Kubernetes: Storage Classes and Persistent Volumes — class fields, mount options, release and expansion; Configure a Security Context and delegating fsGroup to the CSI driver; Good practices for Kubernetes Secrets and RBAC — who reads a Secret; Safely Drain a Node — cordon and drain.
Linux client: mount.cifs(8) — vers, seal, uid/gid, file_mode/dir_mode, cache, actimeo, nobrl, nolease, mfsymlinks, hard/soft, echo_interval, password2, the PASSWD variable; cifs.upcall(8) — Kerberos; Linux kernel: CIFS usage — /proc/fs/cifs/DebugData, Stats, mount_params; fs/smb/client/connect.c — the cifs_demultiplex_thread receive loop, the "has not responded" check, the rules for sharing connections and sessions.
Samba: smb.conf(5) — server min protocol and the SMB3_11 variant, server signing, server smb encrypt, map to guest, restrict anonymous, disable netbios, smb ports, hosts allow, valid users, masks; smbpasswd(8), pdbedit(8), testparm(1) and smbstatus(1) — accounts, configuration checks, connections; Ubuntu Server: Samba as a file server; useradd(8) and ufw(8).
Windows Server: New-SmbShare and Set-SmbServerConfiguration — share and server parameters; SMB security enhancements — encryption, the RejectUnencryptedAccess default, AES-128-GCM, SMB1 since version 1709; SMB signing overview — signing, AES-CMAC and AES-GMAC; Detect, enable, and disable SMBv1, SMBv2, and SMBv3; New-ADUser, icacls and New-NetFirewallRule — the account, NTFS permissions, the firewall.
Protocol: MS-SMB2 — dialects, credits, compound requests, STATUS_PENDING, leases and breaks, signing and the transform header; MS-NLMP — NTLM; RFC 1002 — the NetBIOS session header; Overview of file sharing using SMB 3.
Diagnostics: Wireshark: SMB2 — SMB2/3 dissection and filters; tcpdump(1) — capture and -X.

Next — other storage: local volumes and local-path-provisioner, S3 through CSI and why it is not a file system, distributed Longhorn and Ceph, and where to keep databases in a cluster.