Vitaliy Leschenko’s blog
Виталий Лещенко

Kubernetes for beginners. Part 25: What other storage is there for the cluster?

All parts of the series “Kubernetes for beginners”

The three previous parts were about network file systems: an NFS server for the cluster (part 22), csi-driver-nfs (part 23) and csi-driver-smb (part 24). They are good for shared files, but we have already seen what they cannot do: databases are not put on them, the volume size limits nothing (df in the pod shows the whole export), and when the server is unreachable every pod with its volumes stalls at once. Today is an overview of the rest of what you meet in clusters without a cloud: local node disks (local-path-provisioner, the local volume, TopoLVM), S3 object storage mounted as a file system, and the distributed storage systems Longhorn, Ceph and OpenEBS. At the end comes the answer to the question all five parts of this block lead to: where to keep databases and which data goes where. The mechanics of PV, PVC, StorageClass and CSI are from part 21.

Local volumes and S3 were tested on the lab cluster: local-path-provisioner, a static local volume on a separate node disk, k8s-csi-s3 with the Garage S3 server. The distributed systems do not fit there even within their minimum requirements (more on that below), so their section follows the projects' documentation, without output.

Local node disks

The fastest disk for a pod is the one in its own node: no network, no server, no protocol, the fsync latency is the SSD's latency. The price is single but large: the data lives on one node. A pod bound to such a volume can only run there, and if the node dies for good, the data dies with it unless something higher up replicates it.

local-path-provisioner

local-path-provisioner is a small controller from the k3s developers that implements dynamic provisioning of volumes in local node directories. It is not a CSI driver but an old-style external provisioner: it watches PVCs of its class (rancher.io/local-path) itself and creates PVs, while kubelet mounts them with built-in means. It is installed with a single release manifest:

$ kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/v0.0.37/deploy/local-path-storage.yaml
namespace/local-path-storage created
serviceaccount/local-path-provisioner-service-account created
role.rbac.authorization.k8s.io/local-path-provisioner-role created
clusterrole.rbac.authorization.k8s.io/local-path-provisioner-role created
rolebinding.rbac.authorization.k8s.io/local-path-provisioner-bind created
clusterrolebinding.rbac.authorization.k8s.io/local-path-provisioner-bind created
deployment.apps/local-path-provisioner created
storageclass.storage.k8s.io/local-path created
configmap/local-path-config created
$ kubectl get storageclass local-path
NAME         PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
local-path   rancher.io/local-path   Delete          WaitForFirstConsumer   false                  25s

How it works. When a pod with a PVC of this class is assigned to a node, the controller starts a short-lived helper pod helper-pod-create-<PV name> on that node (in the local-path-storage namespace), which runs the setup script from the local-path-config ConfigMap: mkdir -m 0777 of the directory /opt/local-path-provisioner/<PV name>_<namespace>_<PVC name>. Then the controller creates a PV — of type hostPath by default, or of type local with the annotation volumeType: local on the PVC (or defaultVolumeType: local on the class) — with nodeAffinity to that node. kubelet mounts the directory into the pod with a bind mount. When the PVC is deleted (policy Delete), a similar helper-pod-delete-… pod runs teardown — rm -rf of the directory.

All of this is configured in the same ConfigMap: nodePathMap sets the directories for each node (they are worth moving to a separate disk rather than leaving on the root partition), helperPod.yaml is the helper pod template. The release template uses the image docker.io/library/busybox without a tag, that is latest from Docker Hub: in a cluster without internet access or with a "pinned versions only" rule it has to be replaced with an image from your own registry.

WaitForFirstConsumer. The mode from part 21: the class postpones creating the volume until the scheduler has chosen a node for the pod. For a local volume there is no other way: if the volume were created right away on k8s-node-1 while only k8s-node-2 had room for the pod, the pod would never start. Once created, the binding is permanent: the PV gets nodeAffinity, and every later start of the pod happens only on that node.

Limitations.

  • The PVC size is not enforced: the project documentation says plainly that the capacity limit is ignored. The volume is a directory on a shared file system, and it grows as long as the node disk has room. There is no expansion either (ALLOWVOLUMEEXPANSION false) — there is nothing to expand.
  • Access modes are only ReadWriteOnce and ReadWriteOncePod: a directory on one node. The controller rejects a PVC with ReadWriteMany (NodePath only supports ReadWriteOnce and ReadWriteOncePod (1.22+) access modes). The exception is sharedFileSystemPath: if a shared file system is mounted at the same directory on every node, the provisioner hands out volumes in it with any mode, but that is no longer local storage.
  • If the node is unreachable or taken out of service, the pod will not restart elsewhere; it stays Pending until the node returns. kubectl drain evicts the pod, but the new one waits for uncordon. If the node is lost for good, the PVC has to be deleted and created again, and the data restored from a backup or a replica.

When it fits. Lab and test clusters; caches and temporary data you would not miss; and — most importantly — applications that replicate data between their instances themselves: a PostgreSQL cluster with streaming replication, the RabbitMQ message broker with quorum queues (a queue stored on several broker nodes and agreed on by voting), the Elasticsearch search engine with its own shard copies. They do not need reliability from storage, they need the speed of a local disk, and they provide reliability themselves.

The local volume and static provisioning

The volume type local built into Kubernetes is the same thing without a controller: the administrator creates a PV for each disk or partition, usually mounted separately (for example, an NVMe disk for a database).

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-disk
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Retain
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: k8s-node-1-disk1
spec:
  capacity:
    storage: 200Mi
  accessModes: ["ReadWriteOncePod"]
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-disk
  local:
    path: /mnt/disks/disk1
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values: ["k8s-node-1"]

kubernetes.io/no-provisioner means the class has no provisioner: PVCs of this class bind only to PVs created in advance. WaitForFirstConsumer means something different here than with local-path: the scheduler takes into account which nodes have free matching PVs, picks a node and only then binds the PVC to a PV on that node. nodeAffinity is mandatory on a local PV — it is the binding to the node. Here the size is real: the volume is a separate file system, and writing past it ends with No space left on device. ReadWriteOncePod (the mode from part 21: exactly one pod in the whole cluster uses the volume) has to be declared in the static PV itself — a PVC with ReadWriteOncePod will not bind to a PV with only ReadWriteOnce.

The difference from hostPath that matters: local is a full PV that the scheduler understands, while hostPath in a pod is just a path on whatever node the pod happened to land on. Creating a PV for each disk by hand is automated by sig-storage-local-static-provisioner — a DaemonSet that finds mounted disks in a given node directory (for example, /mnt/disks/) and creates PVs for them, and after a volume is released wipes the disk and creates the PV again.

TopoLVM

TopoLVM is a CSI driver that carves volumes as LVM logical volumes (the Linux logical volume manager) out of a volume group on each node. Unlike local-path, the volume gets a real size (it is a separate block device with an ext4, XFS or Btrfs file system), supports expansion and block mode, and snapshots for thin logical volumes (thin provisioning). The scheduler learns each node's free space through storage capacity tracking (CSIStorageCapacity, part 21) or the TopoLVM scheduler extender and will not place a pod where there is no room. If a cluster has many local volumes, this is the next step after local-path.

S3 through CSI

S3 is an object storage HTTP protocol introduced by Amazon that became a standard: data lives in buckets as objects with a key name; they are put (PUT), read (GET, including ranges), deleted and listed by prefix (ListObjectsV2). Every request is signed with an access key: the client computes a request signature from the secret key (the AWS Signature Version 4 scheme, an HMAC over the method, path, headers and time), and the secret itself never crosses the network. S3 has no directories: reports/2026/10.pdf is a key with slashes, and a "directory" is a common prefix or an empty object whose name ends with /.

Compatible servers people run themselves: Ceph RGW (the S3 gateway of the distributed storage system Ceph, see below), Garage (a lightweight open-source distributed S3 server designed for small deployments on several machines) and others. MinIO, long the most popular of them, no longer maintains its open-source repository: it has been archived, and the company offers a separate product with its own license instead.

A bucket can be mounted as a volume through CSI drivers based on FUSE (Filesystem in Userspace) — a kernel mechanism that hands file system operations to a user-space process. The process on the node turns open, read, write into S3 requests.

  • k8s-csi-s3 is a driver that mounts buckets through GeeseFS, a fast FUSE file system for S3 (it can also use s3fs and rclone, but the authors strongly recommend GeeseFS). It supports dynamic provisioning: a new PVC is a new bucket or a pvc-<uid> prefix in the bucket named in the class; deleting the PVC removes the prefix. Any access mode works, including ReadWriteMany. It does not store file owners and modes on third-party S3 servers: everything appears to belong to one user with the same mode from the mount options.
  • Mountpoint for Amazon S3 CSI Driver is a driver based on Mountpoint, the AWS FUSE client optimized for large sequential reads. Static provisioning only: a PV points to an existing bucket. A compatible server's address can be set with --endpoint-url, but it is designed primarily for Amazon S3 and is installed with a Helm chart or an EKS add-on.

k8s-csi-s3 is installed with three release manifests; the server address and the keys live in a Secret, not in the class:

apiVersion: v1
kind: Secret
metadata:
  name: csi-s3-secret
  namespace: storage-credentials
stringData:
  accessKeyID: GK51446d5c5b4635172249f367
  secretAccessKey: <secret key>
  endpoint: http://s3.example.com:3900
  region: garage
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: s3
provisioner: ru.yandex.s3.csi
parameters:
  mounter: geesefs
  options: "--memory-limit 1000 --dir-mode 0770 --file-mode 0660 --uid 10001 --gid 10001"
  bucket: k8s-volumes
  csi.storage.k8s.io/provisioner-secret-name: csi-s3-secret
  csi.storage.k8s.io/provisioner-secret-namespace: storage-credentials
  csi.storage.k8s.io/controller-publish-secret-name: csi-s3-secret
  csi.storage.k8s.io/controller-publish-secret-namespace: storage-credentials
  csi.storage.k8s.io/node-stage-secret-name: csi-s3-secret
  csi.storage.k8s.io/node-stage-secret-namespace: storage-credentials
  csi.storage.k8s.io/node-publish-secret-name: csi-s3-secret
  csi.storage.k8s.io/node-publish-secret-namespace: storage-credentials

The Secret lives in the separate storage-credentials namespace — by the same logic as the SMB password in part 24: the key should have rights to this bucket only, not to the whole storage. --uid, --gid, --dir-mode, --file-mode set the owner and mode for every file of the volume at once; --memory-limit is GeeseFS's memory ceiling for the data cache in megabytes, and it applies to each volume on each node: ten S3 volumes on a node with the default 1000 means up to 10 GB of memory outside pod accounting.

Where GeeseFS runs. By default the node plugin starts GeeseFS not in its own container but on the node itself — as a transient systemd service geesefs-<volume name>.service (for that /run/systemd is mounted into the container, and the binary is copied to /var/lib/kubelet/plugins/ru.yandex.s3.csi/). This way an upgrade or restart of the driver does not break mounted volumes. With the --no-systemd option GeeseFS lives in the driver container, and when it restarts every volume on the node answers Transport endpoint is not connected — both cases are tested below.

Security. The driver controller gets a ClusterRole with get and list on all secrets in the cluster, and the node plugin runs privileged with access to the node's systemd. The service receives the S3 keys through environment variables, and the Environment property of a transient systemd service is served over D-Bus to any user on the node: systemctl show without sudo prints the secret key (shown below). So the volume key should have rights to one bucket, and the nodes should have no outside users. One more difference from NFS and SMB: the driver controller talks to the S3 server from the pod network (its pod is not hostNetwork), so closing pods' path to the server with an address-based network policy, as in part 22, is only possible with an exception for kube-system; the main boundary here is the key's rights.

It is not POSIX

A bucket mounted as a directory looks like a file system but behaves differently:

  • No atomic rename. S3 has no rename operation; the FUSE client copies the object under the new key and deletes the old one, and renaming a directory means copying every object with its prefix. Applications that write a temporary file and then atomically rename it into place (almost every database and many configuration systems do this) lose that guarantee on S3. Mountpoint supports renaming files only in the S3 Express One Zone storage class and cannot rename directories at all.
  • No locks between nodes. GeeseFS does not support locking: flock works only among processes of one node (the kernel handles it), while a pod on another node gets the same lock at the same time. Two writers of one object do not know about each other, and the last one wins.
  • No hard links, and GeeseFS keeps symbolic links, owners and modes only in Yandex Object Storage: on another S3 server a symbolic link looks like an empty file on the neighbouring node, and chmod and chown succeed and change nothing.
  • Partial overwrites are expensive. An S3 object is written whole. Changing 4 KiB in the middle of a 1 GiB file means assembling the object again (GeeseFS does it with a server-side copy, but the volume is the same). Mountpoint by default only does sequential writes of new files; overwriting needs --allow-overwrite and only with truncation, deletion needs --allow-delete.
  • Metadata consistency. The object list and attributes are cached by the client (GeeseFS --stat-cache-ttl defaults to one minute), and another node may not see a just-created file for a minute.
  • The volume size means nothing. df in the pod shows a petabyte, and the PVC capacity is just a number in the object.

When it fits. Large files written once and then read: media, archives, exports, datasets for processing; passing files between systems where one works with S3 through the API and the other only knows files. It does not fit: databases, append-only logs, anything that relies on rename and locks.

Better without a file system. If you write the application, it is better to talk to S3 through its API with an SDK (a library for your programming language): the application decides what to upload and when, gets explicit errors, can do multipart uploads and presigned URLs, and the FUSE layer with its non-obvious behaviour disappears. Then Kubernetes needs not a volume but a Secret with the keys and the storage address in the configuration.

Distributed storage

The next level is storage that lives in the cluster itself: it uses node disks and replicates data between them over the network. A pod gets a block volume that survives the loss of a node: a replica exists on another one.

      node 1                node 2                node 3
  ┌────────────┐        ┌────────────┐        ┌────────────┐
  │ pod        │        │            │        │            │
  │  │         │        │            │        │            │
  │  ▼         │        │            │        │            │
  │ volume ────┼───────►│ replica 2  │        │ replica 3  │
  │ replica 1  │◄───────┼────────────┼───────►│            │
  │ node disk  │        │ node disk  │        │ node disk  │
  └────────────┘        └────────────┘        └────────────┘
    a write is acknowledged once all healthy replicas have stored it
  • Longhorn is distributed block storage from the authors of Rancher (release 1.13). Each volume has its own engine process that synchronously writes data to several replicas on disks of different nodes. In the main data engine (V1) the engine exposes the volume to the node as a block device over iSCSI — a protocol that carries SCSI disk commands over TCP — so every node needs open-iscsi with a running iscsid; the newer V2 engine is built on SPDK (a library of user-space storage drivers) and exposes volumes over NVMe/TCP. It has a web UI, snapshots, backups to S3 or NFS, ReadWriteMany through a separate share-manager pod with an NFSv4 server for the volume (the nodes need an NFSv4 client). The documented minimum is three nodes with 4 vCPUs and 4 GiB of memory each, SSD or NVMe, and a 10 Gbit/s network between nodes. The simplest of the distributed systems to install.
  • Ceph is a distributed storage system that provides block devices (RBD, RADOS Block Device), a file system (CephFS) and S3-compatible object storage (RGW). Data is spread across OSD daemons (one per disk) by the CRUSH algorithm, which computes an object's location from the cluster map with no central directory; the cluster state is kept by monitors (MON), which need at least three for a quorum. In Kubernetes it is deployed by the Rook operator (release 1.21; an operator is a controller that manages a complex application through its own resources, covered in detail in part 28), and volumes are attached by the ceph-csi driver. An OSD needs a disk, partition or LVM logical volume without a file system (or a PVC in block mode), and the nodes need the rbd kernel module. The most powerful and most complex option: industrial storage that whole clouds run on.
  • OpenEBS (release 4.6) is a set of engines under one name: local volumes (on directories, LVM or ZFS — a file system with a built-in volume manager; the counterpart of local-path and TopoLVM) and replicated volumes, Replicated PV Mayastor, which creates an NVMe over TCP target for each volume — the NVMe-oF protocol that carries NVMe disk commands over the network — and requires dedicated CPU cores and huge pages. It lets you start with local volumes and add replication where needed.

All three are installed with Helm charts (Helm is covered in part 27) or their own manifests; on the lab cluster of two workers with 2 vCPUs and 4 GB each none of them meets even the minimum requirements, so here it is description only.

The operating cost is the same in kind and different in size:

  • Resources. At least three nodes with dedicated disks, memory and CPU for storage daemons on each node, and network: every write goes over the network to 2–3 replicas, and 1 Gbit/s quickly becomes the bottleneck — Longhorn names 10 Gbit/s as the basic requirement and recommends a dedicated storage network.
  • Latency. Synchronous replication adds a network round trip to every write. A distributed volume is slower than a local disk.
  • Failure complexity. Storage that lives in the cluster falls with the cluster: a failed Kubernetes upgrade, a node network outage or memory pressure on the nodes hit the data too. Recovering from a double failure, rebalancing, replacing disks is a separate skill set.
  • Upgrades. The storage has its own release cycle and compatibility with Kubernetes and kernel versions, and it has to be upgraded more carefully than applications.

If the organization has no people ready to own Ceph, distributed storage in a small cluster often brings more risk than it removes. A sensible path for three to five nodes: local volumes for what replicates itself, NFS or SMB for shared files, S3 for large objects, and Longhorn only if you truly need a block volume that survives node loss for an application that cannot replicate itself.

Databases in the cluster

At this point the storage block comes down to this: a database needs fast fsync, correct locking, POSIX semantics and protection from losing a node. Network file systems give the last one at the cost of the first three (as discussed in part 23: Prometheus explicitly does not support NFS, SQLite warns about broken locking on network file systems, every fsync gets a network round trip longer). Distributed block storage gives everything, but at the cost of complexity and one more copy of the data on top of what the database makes itself.

So the commonly accepted scheme is different: local disks and replication by the database itself — a shared-nothing architecture that the CloudNativePG documentation recommends for PostgreSQL.

  k8s-node-1                       k8s-node-2
 ┌─────────────────────────┐      ┌─────────────────────────┐
 │ postgres-1 (primary)    │ WAL  │ postgres-2 (replica)    │
 │ PVC local-disk ──► disk ├─────►│ PVC local-disk ──► disk │
 └─────────────────────────┘      └─────────────────────────┘
        operator: primary election, failover, backups to S3

Each instance is a pod with its own local volume; the primary sends its change log (WAL, the write-ahead log in PostgreSQL) to the replicas over the network; when a node is lost, the operator promotes one of the replicas to primary and creates a new replica from scratch on another node. Backups and the log go to S3 — outside the cluster and outside the nodes. All this is managed by operators: for PostgreSQL, for example, CloudNativePG (an open-source PostgreSQL operator; it backs up to S3 through the Barman Cloud plugin, the built-in upload has been deprecated since 1.26), for RabbitMQ its official cluster operator; how operators work is covered in part 28.

What you get: the speed of local NVMe, the database's own consistency guarantees, surviving node loss without distributed storage, and point-in-time recovery from S3. What it costs: at least two or three instances, each holding a full copy of the data.

Which data goes where

Data Storage Access mode Why
Replicated databases (PostgreSQL, MySQL, broker clusters) local PV, local-path, TopoLVM RWO (better RWOP) local disk speed; reliability comes from database replication
Non-replicated databases that must not be lost distributed block (Longhorn, Ceph RBD) RWO survives node loss
Prometheus metrics (a monitoring system), the local part of Loki logs (a log store, part 33) local disk RWO the TSDB needs POSIX; long-term storage goes to S3 through the application itself
etcd local SSD — (not a cluster PV) fsync latency; more in part 34
Files shared by several replicas: uploads, reports, static assets NFS (csi-driver-nfs) RWX a natural shared FS, permissions by UID
Files that Windows users need too SMB (csi-driver-smb) RWX the same share people use; SMB3 authentication and encryption
Ready-made reference data, read only static NFS or SMB PV ROX ro on the server and in the PV
Large objects, media, archives, backups S3 through the API; a CSI S3 driver if needed RWX (through FUSE) cheap and scalable; not POSIX
Caches, intermediate data emptyDir or local-path RWO fine to lose
Configuration and secrets ConfigMap, Secret — these are not data volumes (part 12)

RWOP (ReadWriteOncePod) is safer than RWO for databases: RWO limits the volume to one node, but two pods on that node will both mount it, while RWOP rules out the second pod entirely. local-path supports it; a static PV has to declare it in the PV itself.

Hands-on

The experiments run in the demo namespace from part 11, the pods use busybox.

local-path: a directory on the node

A 100 MiB PVC and a Deployment that writes its start time into it:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cache
  namespace: demo
spec:
  storageClassName: local-path
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 100Mi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cache
  namespace: demo
  labels:
    app.kubernetes.io/name: cache
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app.kubernetes.io/name: cache
  template:
    metadata:
      labels:
        app.kubernetes.io/name: cache
    spec:
      containers:
        - name: app
          image: busybox:1.37
          command: ["sh", "-c", "date >> /data/started.log; sleep infinity"]
          volumeMounts:
            - name: data
              mountPath: /data
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: cache

strategy: Recreate because the RWO volume is on one node: during an update the old pod has to leave before the new one comes. Until there is a pod, the PVC waits — as intended:

$ kubectl -n demo get pvc cache
NAME    STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
cache   Pending                                      local-path     <unset>                 5s
$ kubectl -n demo describe pvc cache | sed -n '/Events/,$p'
Events:
  Type    Reason                Age              From                         Message
  ----    ------                ----             ----                         -------
  Normal  WaitForFirstConsumer  2s (x2 over 6s)  persistentvolume-controller  waiting for first consumer to be created before binding

A pod appeared — the scheduler chose k8s-node-2, and a helper pod started there:

$ kubectl -n local-path-storage get pods -o wide
NAME                                                         READY   STATUS              RESTARTS   AGE   IP              NODE         NOMINATED NODE   READINESS GATES
helper-pod-create-pvc-4b0f1a3d-1c17-4468-b06c-dde209e63531   0/1     ContainerCreating   0          2s    <none>          k8s-node-2   <none>           <none>
local-path-provisioner-54745b98cf-rj4p5                      1/1     Running             0          50s   172.20.211.62   k8s-node-1   <none>           <none>
$ kubectl -n demo get pvc cache
NAME    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
cache   Bound    pvc-4b0f1a3d-1c17-4468-b06c-dde209e63531   100Mi      RWO            local-path     <unset>                 34s
$ kubectl -n demo describe pvc cache | sed -n '/Events/,$p'
Events:
  Type    Reason                 Age                From                                                                                                Message
  ----    ------                 ----               ----                                                                                                -------
  Normal  WaitForFirstConsumer   30s (x2 over 34s)  persistentvolume-controller                                                                         waiting for first consumer to be created before binding
  Normal  Provisioning           22s                rancher.io/local-path_local-path-provisioner-54745b98cf-rj4p5_27abeaf4-ef01-47f0-885a-be13148abbdc  External provisioner is provisioning volume for claim "demo/cache"
  Normal  ExternalProvisioning   15s (x2 over 22s)  persistentvolume-controller                                                                         Waiting for a volume to be created either by the external provisioner 'rancher.io/local-path' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  Normal  ProvisioningSucceeded  9s                 rancher.io/local-path_local-path-provisioner-54745b98cf-rj4p5_27abeaf4-ef01-47f0-885a-be13148abbdc  Successfully provisioned volume pvc-4b0f1a3d-1c17-4468-b06c-dde209e63531

Of the 13 seconds between Provisioning and ProvisioningSucceeded the helper pod spent almost all in ContainerCreating: the node was pulling busybox:latest. The provisioner log shows what happened:

$ kubectl -n local-path-storage logs deploy/local-path-provisioner | grep -v event.go
time="2026-10-30T13:19:15Z" level=info msg="Health server started on :8080"
time="2026-10-30T13:19:15Z" level=debug msg="Applied config: {\"nodePathMap\":[{\"node\":\"DEFAULT_PATH_FOR_NON_LISTED_NODES\",\"paths\":[\"/opt/local-path-provisioner\"]}],\"storageClassConfigs\":null}"
time="2026-10-30T13:19:15Z" level=debug msg="Provisioner started"
I1030 13:19:15.347581       1 controller.go:838] "Starting provisioner controller" component="rancher.io/local-path_local-path-provisioner-54745b98cf-rj4p5_27abeaf4-ef01-47f0-885a-be13148abbdc"
I1030 13:19:15.448643       1 controller.go:887] "Started provisioner controller" component="rancher.io/local-path_local-path-provisioner-54745b98cf-rj4p5_27abeaf4-ef01-47f0-885a-be13148abbdc"
time="2026-10-30T13:19:52Z" level=debug msg="config doesn't contain node k8s-node-2, use DEFAULT_PATH_FOR_NON_LISTED_NODES instead"
time="2026-10-30T13:19:52Z" level=info msg="Creating volume pvc-4b0f1a3d-1c17-4468-b06c-dde209e63531 at k8s-node-2:/opt/local-path-provisioner/pvc-4b0f1a3d-1c17-4468-b06c-dde209e63531_demo_cache"
time="2026-10-30T13:19:52Z" level=info msg="create the helper pod helper-pod-create-pvc-4b0f1a3d-1c17-4468-b06c-dde209e63531 into local-path-storage"
time="2026-10-30T13:20:05Z" level=info msg="Volume pvc-4b0f1a3d-1c17-4468-b06c-dde209e63531 has been created on k8s-node-2:/opt/local-path-provisioner/pvc-4b0f1a3d-1c17-4468-b06c-dde209e63531_demo_cache"
time="2026-10-30T13:20:05Z" level=info msg="Start of helper-pod-create-pvc-4b0f1a3d-1c17-4468-b06c-dde209e63531 logs"
time="2026-10-30T13:20:05Z" level=info msg="End of helper-pod-create-pvc-4b0f1a3d-1c17-4468-b06c-dde209e63531 logs"

The PV is a plain hostPath bound to the node:

$ kubectl get pv pvc-4b0f1a3d-1c17-4468-b06c-dde209e63531 -o yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    local.path.provisioner/selected-node: k8s-node-2
    pv.kubernetes.io/provisioned-by: rancher.io/local-path
...
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 100Mi
...
  hostPath:
    path: /opt/local-path-provisioner/pvc-4b0f1a3d-1c17-4468-b06c-dde209e63531_demo_cache
    type: DirectoryOrCreate
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - k8s-node-2
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-path
  volumeMode: Filesystem

With the annotation volumeType: local on the PVC the same directory is presented as a local PV (spec.local.path instead of spec.hostPath) — tested with a second PVC, nothing else changes.

We write 200 MiB into a 100 MiB volume:

$ kubectl -n demo exec deploy/cache -- sh -c 'dd if=/dev/zero of=/data/big bs=1M count=200 2>&1 | tail -1; ls -l /data; df -h /data'
209715200 bytes (200.0MB) copied, 0.036870 seconds, 5.3GB/s
total 204804
-rw-r--r--    1 root     root     209715200 Oct 30 13:20 big
-rw-r--r--    1 root     root            29 Oct 30 13:20 started.log
Filesystem                Size      Used Available Use% Mounted on
/dev/sda2                30.3G      5.9G     22.8G  21% /data

No error, no warning: the pod sees the node's whole root partition. On the node it is a plain directory with mode 0777:

user@k8s-node-2:~$ sudo ls -la /opt/local-path-provisioner/
total 12
drwxr-xr-x 3 root root 4096 Oct 30 13:20 .
drwxr-xr-x 5 root root 4096 Oct 30 13:19 ..
drwxrwxrwx 2 root root 4096 Oct 30 13:20 pvc-4b0f1a3d-1c17-4468-b06c-dde209e63531_demo_cache
user@k8s-node-2:~$ sudo du -sh /opt/local-path-provisioner/*
201M	/opt/local-path-provisioner/pvc-4b0f1a3d-1c17-4468-b06c-dde209e63531_demo_cache
user@k8s-node-2:~$ df -h /opt/local-path-provisioner
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2        31G  6.0G   23G  21% /

The directory is on the root partition: that is local-path's default, and exactly why its path is moved to a separate disk through nodePathMap. One pod without limits can fill the root partition and cause disk pressure for the whole node.

What happens when the node is taken out of service. We close k8s-node-2 to new pods and delete the pod — the same thing kubectl drain does:

$ kubectl cordon k8s-node-2
node/k8s-node-2 cordoned
$ kubectl -n demo delete pod -l app.kubernetes.io/name=cache
pod "cache-7dc5df85bc-ktqpg" deleted from demo namespace
$ kubectl -n demo get pods -l app.kubernetes.io/name=cache -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP       NODE     NOMINATED NODE   READINESS GATES
cache-7dc5df85bc-9lhkb   0/1     Pending   0          40s   <none>   <none>   <none>           <none>
$ kubectl -n demo describe pod cache-7dc5df85bc-9lhkb | sed -n '/Events/,$p'
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  39s   default-scheduler  0/3 nodes are available: 1 node(s) didn't match PersistentVolume's node affinity, 1 node(s) had untolerated taint(s), 1 node(s) were unschedulable. no new claims to deallocate, preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.

Three nodes, three reasons: k8s-node-1 does not match the volume's nodeAffinity, the master has the control-plane taint (part 13), k8s-node-2 is closed. The pod will wait forever; we bring the node back:

$ kubectl uncordon k8s-node-2
node/k8s-node-2 uncordoned
$ kubectl -n demo get pods -l app.kubernetes.io/name=cache -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP              NODE         NOMINATED NODE   READINESS GATES
cache-7dc5df85bc-9lhkb   1/1     Running   0          51s   172.20.172.63   k8s-node-2   <none>           <none>
$ kubectl -n demo exec deploy/cache -- cat /data/started.log
Fri Oct 30 13:20:09 UTC 2026
Fri Oct 30 13:22:41 UTC 2026

The data is in place — the second start appended a line. It is not a bug but a property of a local volume that has to be weighed when choosing: had the node not come back, the pod would have stayed Pending.

Deleting the PVC starts a second helper pod, and the directory disappears:

$ kubectl delete -f cache.yaml
persistentvolumeclaim "cache" deleted from demo namespace
deployment.apps "cache" deleted from demo namespace
$ kubectl -n local-path-storage logs deploy/local-path-provisioner --since=2m | grep -v event.go
time="2026-10-30T13:23:29Z" level=info msg="Deleting volume pvc-4b0f1a3d-1c17-4468-b06c-dde209e63531 at k8s-node-2:/opt/local-path-provisioner/pvc-4b0f1a3d-1c17-4468-b06c-dde209e63531_demo_cache"
time="2026-10-30T13:23:29Z" level=info msg="create the helper pod helper-pod-delete-pvc-4b0f1a3d-1c17-4468-b06c-dde209e63531 into local-path-storage"
time="2026-10-30T13:23:35Z" level=info msg="Volume pvc-4b0f1a3d-1c17-4468-b06c-dde209e63531 has been deleted on k8s-node-2:/opt/local-path-provisioner/pvc-4b0f1a3d-1c17-4468-b06c-dde209e63531_demo_cache"
...
user@k8s-node-2:~$ sudo ls -la /opt/local-path-provisioner/
total 8
drwxr-xr-x 2 root root 4096 Oct 30 13:23 .
drwxr-xr-x 5 root root 4096 Oct 30 13:19 ..

A local volume on a separate disk

For the experiment, instead of NVMe, a 256 MiB image file attached as a loop device (a block device on top of a file) and mounted on k8s-node-1:

user@k8s-node-1:~$ sudo fallocate -l 256M /var/tmp/disk1.img
user@k8s-node-1:~$ LOOP=$(sudo losetup --find --show /var/tmp/disk1.img); echo $LOOP
/dev/loop5
user@k8s-node-1:~$ sudo mkfs.ext4 -q -L disk1 $LOOP
user@k8s-node-1:~$ sudo mkdir -p /mnt/disks/disk1 && sudo mount $LOOP /mnt/disks/disk1
user@k8s-node-1:~$ df -h /mnt/disks/disk1
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop5      224M  152K  206M   1% /mnt/disks/disk1

The class and the PV are the manifest from the beginning of the part. The access mode trap: the first time the PV was declared with ReadWriteOnce, while the PVC and the pod asked for ReadWriteOncePod:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: db
  namespace: demo
spec:
  storageClassName: local-disk
  accessModes: ["ReadWriteOncePod"]
  resources:
    requests:
      storage: 200Mi
---
apiVersion: v1
kind: Pod
metadata:
  name: db
  namespace: demo
spec:
  containers:
    - name: app
      image: busybox:1.37
      command: ["sleep", "infinity"]
      volumeMounts:
        - name: data
          mountPath: /data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: db

$ kubectl -n demo describe pod db | sed -n '/Events/,$p'
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  68s   default-scheduler  0/3 nodes are available: 1 node(s) had untolerated taint(s), 2 node(s) didn't find available persistent volumes to bind. no new claims to deallocate, preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.

didn't find available persistent volumes to bind — the scheduler found no matching free PV on any worker; the PVC meanwhile is Pending with the same WaitForFirstConsumer event. The answer is the same if the PV is smaller than the request. We add the required mode to the PV, and binding goes through:

$ kubectl patch pv k8s-node-1-disk1 --type=merge -p '{"spec":{"accessModes":["ReadWriteOncePod"]}}'
persistentvolume/k8s-node-1-disk1 patched
$ kubectl -n demo get pvc db
NAME   STATUS   VOLUME             CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
db     Bound    k8s-node-1-disk1   200Mi      RWOP           local-disk     <unset>                 89s
$ kubectl -n demo get pod db -o wide
NAME   READY   STATUS    RESTARTS   AGE   IP             NODE         NOMINATED NODE   READINESS GATES
db     1/1     Running   0          89s   172.20.211.9   k8s-node-1   <none>           <none>

Here the size is real — the volume has its own file system:

$ kubectl -n demo exec db -- df -h /data
Filesystem                Size      Used Available Use% Mounted on
/dev/loop5              223.7M    152.0K    205.7M   0% /data
$ kubectl -n demo exec db -- dd if=/dev/zero of=/data/big bs=1M count=300
dd: error writing '/data/big': No space left on device
219+0 records in
218+0 records out
228589568 bytes (218.0MB) copied, 2.515043 seconds, 86.7MB/s
command terminated with exit code 1
user@k8s-node-1:~$ findmnt -n -o TARGET,SOURCE,FSTYPE -S /dev/loop5
/mnt/disks/disk1                                                                                               /dev/loop5 ext4
/var/lib/kubelet/pods/9f3f0845-d7d9-4b30-ac5a-0ba986ac6e7f/volumes/kubernetes.io~local-volume/k8s-node-1-disk1 /dev/loop5 ext4

kubelet bind-mounts the disk directory into the pod's volume directory (the kubernetes.io~local-volume plugin). After the pod and the PVC are deleted, Retain kicks in: the PV moves to Released with the old claimRef, the data stays on the disk, and a new PVC will not bind to it until the administrator wipes the disk and removes the claimRef (kubectl patch pv k8s-node-1-disk1 --type=json -p '[{"op":"remove","path":"/spec/claimRef"}]') — exactly the work sig-storage-local-static-provisioner takes over:

$ kubectl get pv
NAME               CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM     STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE
k8s-node-1-disk1   200Mi      RWOP           Retain           Released   demo/db   local-disk     <unset>                          2m21s
user@k8s-node-1:~$ sudo ls -la /mnt/disks/disk1
total 223260
drwxr-xr-x 3 root root      4096 Oct 30 13:26 .
drwxr-xr-x 3 root root      4096 Oct 30 13:24 ..
-rw-r--r-- 1 root root 228589568 Oct 30 13:26 big
drwx------ 2 root root     16384 Oct 30 13:24 lost+found

S3: the Garage server and k8s-csi-s3

The S3 server for the experiment is Garage 2.5 on a separate machine s3.example.com (192.168.10.33): one node, one copy of the data (replication_factor = 1 — the documentation says plainly it is for test deployments only), metadata in SQLite. Garage is a single static binary; the configuration:

# /etc/garage.toml
metadata_dir = "/var/lib/garage/meta"
data_dir = "/var/lib/garage/data"
db_engine = "sqlite"
replication_factor = 1

rpc_bind_addr = "127.0.0.1:3901"
rpc_public_addr = "127.0.0.1:3901"
rpc_secret = "<output of openssl rand -hex 32>"

[s3_api]
s3_region = "garage"
api_bind_addr = "0.0.0.0:3900"

[admin]
api_bind_addr = "127.0.0.1:3903"

Port 3900 is the S3 API, 3901 is communication between Garage nodes (local only for a single node), 3903 is the admin API. Garage does not do TLS itself: on the lab network it answers over HTTP; request signing does not reveal the secret, but the data travels in the clear. In a production setup a TLS reverse proxy goes in front of it, and endpoint gets an https:// address.

For the experiment a transient systemd-run service is enough (the Garage documentation has a unit file for a permanent setup); then the node gets a role in the cluster layout, and a bucket and a key with rights to it only are created:

user@s3:~$ sudo systemd-run --unit=garage /usr/local/bin/garage -c /etc/garage.toml server
Running as unit: garage.service; invocation ID: 5b947ec15de847bfb9ed5764c294f9c2
user@s3:~$ sudo garage -c /etc/garage.toml status
==== HEALTHY NODES ====
ID                Hostname     Address         Tags  Zone  Capacity          DataAvail  ResyncQueue  Version
42df63bf219b0dad  s3           127.0.0.1:3901              NO ROLE ASSIGNED             0            v2.5.0
user@s3:~$ sudo garage -c /etc/garage.toml layout assign -z dc1 -c 2G 42df63bf219b0dad
user@s3:~$ sudo garage -c /etc/garage.toml layout apply --version 1
...
New cluster layout with updated role assignment has been applied in cluster.
Data will now be moved around between nodes accordingly.
user@s3:~$ sudo garage -c /etc/garage.toml bucket create k8s-volumes
user@s3:~$ sudo garage -c /etc/garage.toml key create k8s-csi-s3
==== ACCESS KEY INFORMATION ====
Key ID:              GK51446d5c5b4635172249f367
Key name:            k8s-csi-s3
Created:             2026-10-30 13:28:25.940 +00:00
Validity:            valid
Expiration:          never

Can create buckets:  false
...
user@s3:~$ sudo garage -c /etc/garage.toml bucket allow --read --write --owner k8s-volumes --key k8s-csi-s3
==== KEYS FOR THIS BUCKET ====
Permissions  Access key                              Local aliases
RWO          GK51446d5c5b4635172249f367  k8s-csi-s3
user@s3:~$ for ip in 192.168.10.17 192.168.10.20 192.168.10.21; do
>   sudo ufw allow proto tcp from $ip to any port 3900 comment 'S3: k8s nodes'
> done
Rule added
Rule added
Rule added

The secret key is printed once at creation (and later with garage key info --show-secret); it does not appear in this text. The region is part of the SigV4 signature, and Garage only accepts requests addressed to the region from s3_region (to others it answers AuthorizationHeaderMalformed with a hint about the right region), hence region: garage in the driver Secret.

The k8s-csi-s3 v0.43.7 driver is three release manifests (the controller-provisioner, the CSIDriver object, the node plugin DaemonSet), all in kube-system:

$ U=https://raw.githubusercontent.com/yandex-cloud/k8s-csi-s3/v0.43.7/deploy/kubernetes
$ kubectl apply -f $U/provisioner.yaml -f $U/driver.yaml -f $U/csi-s3.yaml
serviceaccount/csi-s3-provisioner-sa created
clusterrole.rbac.authorization.k8s.io/csi-s3-external-provisioner-runner created
clusterrolebinding.rbac.authorization.k8s.io/csi-s3-provisioner-role created
service/csi-s3-provisioner created
statefulset.apps/csi-s3-provisioner created
csidriver.storage.k8s.io/ru.yandex.s3.csi created
serviceaccount/csi-s3 created
clusterrole.rbac.authorization.k8s.io/csi-s3 created
clusterrolebinding.rbac.authorization.k8s.io/csi-s3 created
daemonset.apps/csi-s3 created
$ kubectl -n kube-system get pods -l 'app in (csi-s3,csi-s3-provisioner)' -o wide
NAME                   READY   STATUS    RESTARTS      AGE   IP              NODE         NOMINATED NODE   READINESS GATES
csi-s3-nntbp           2/2     Running   0             97s   172.20.172.20   k8s-node-2   <none>           <none>
csi-s3-provisioner-0   2/2     Running   1 (23s ago)   99s   172.20.172.15   k8s-node-2   <none>           <none>
csi-s3-v7j98           2/2     Running   0             97s   172.20.211.7    k8s-node-1   <none>           <none>
$ kubectl get csidriver ru.yandex.s3.csi
NAME               ATTACHREQUIRED   PODINFOONMOUNT   STORAGECAPACITY   TOKENREQUESTS   REQUIRESREPUBLISH   MODES        AGE
ru.yandex.s3.csi   false            true             false             <unset>         false               Persistent   61s

The driver and sidecar images (provisioner v6.2.0, registrar v2.16.0) live in the cr.yandex registry. The node plugin landed only on the workers — the manifest has no toleration for the master's taint, so pods with S3 volumes cannot run on the master. The csi-provisioner restart on first start is the same as with csi-driver-nfs: the sidecar started before the driver opened its socket. The manifest has no fsGroupPolicy field, the API fills in ReadWriteOnceWithFSType, and with it kubelet applies fsGroup only to RWO volumes with a specified fsType — the pod's fsGroup does not change an S3 volume, the owner is set by the --uid/--gid options.

We create the Secret substituting the key from a file so it lands neither in the output nor in the shell history ($(…) also strips the trailing newline of the file — with it the signature would not match, so --from-file does not work here), then the class from the beginning of the part and a PVC with a Deployment across two nodes:

$ kubectl create namespace storage-credentials
namespace/storage-credentials created
$ kubectl -n storage-credentials create secret generic csi-s3-secret \
    --from-literal=accessKeyID=GK51446d5c5b4635172249f367 \
    --from-literal=secretAccessKey="$(cat ./garage-k8s-csi-s3.secret)" \
    --from-literal=endpoint=http://s3.example.com:3900 \
    --from-literal=region=garage
secret/csi-s3-secret created

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: media
  namespace: demo
spec:
  storageClassName: s3
  accessModes: ["ReadWriteMany"]
  resources:
    requests:
      storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: media
  namespace: demo
  labels:
    app.kubernetes.io/name: media
spec:
  replicas: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: media
  template:
    metadata:
      labels:
        app.kubernetes.io/name: media
    spec:
      securityContext:
        runAsUser: 10001
        runAsGroup: 10001
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: media
      containers:
        - name: app
          image: busybox:1.37
          command: ["sleep", "infinity"]
          volumeMounts:
            - name: media
              mountPath: /media
      volumes:
        - name: media
          persistentVolumeClaim:
            claimName: media

$ kubectl -n demo get pvc media
NAME    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
media   Bound    pvc-f2b79471-30fc-4c02-9008-49899e70f87e   1Gi        RWX            s3             <unset>                 26s
$ kubectl -n demo get pods -l app.kubernetes.io/name=media -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP              NODE         NOMINATED NODE   READINESS GATES
media-7f79bbf67d-4jr2n   1/1     Running   0          25s   172.20.172.18   k8s-node-2   <none>           <none>
media-7f79bbf67d-cghcr   1/1     Running   0          25s   172.20.211.8    k8s-node-1   <none>           <none>
$ kubectl get pv pvc-f2b79471-30fc-4c02-9008-49899e70f87e -o yaml
...
  csi:
...
    driver: ru.yandex.s3.csi
...
    volumeAttributes:
      bucket: k8s-volumes
      capacity: "1073741824"
      mounter: geesefs
      options: --memory-limit 1000 --dir-mode 0770 --file-mode 0660 --uid 10001 --gid
        10001
      storage.kubernetes.io/csiProvisionerIdentity: 1793367013849-1750-ru.yandex.s3.csi
    volumeHandle: k8s-volumes/pvc-f2b79471-30fc-4c02-9008-49899e70f87e
  persistentVolumeReclaimPolicy: Delete

volumeHandle is bucket/prefix. On the node there are two FUSE mounts (staging and the pod's volume directory), and behind them a systemd service:

user@k8s-node-1:~$ findmnt -t fuse.geesefs -o TARGET,SOURCE,FSTYPE,OPTIONS
TARGET                                                                                                                                   SOURCE      FSTYPE       OPTIONS
/var/lib/kubelet/plugins/kubernetes.io/csi/ru.yandex.s3.csi/a0978cd9ba03768c44659165dd0bc3f0a40595d4597aeb9a3c3484fdce698bbd/globalmount k8s-volumes fuse.geesefs rw,nosuid,nodev,relatime,user_id=65534,group_id=65534,default_permissions,allow_other
/var/lib/kubelet/pods/152f88ad-2f11-473d-b3ca-c0f0f0e87ab5/volumes/kubernetes.io~csi/pvc-f2b79471-30fc-4c02-9008-49899e70f87e/mount      k8s-volumes fuse.geesefs rw,nosuid,nodev,relatime,user_id=65534,group_id=65534,default_permissions,allow_other
user@k8s-node-1:~$ systemctl status 'geesefs-*' --no-pager -n 0
● geesefs-k8s_2dvolumes_2fpvc_2df2b79471_2d30fc_2d4c02_2d9008_2d49899e70f87e.service - GeeseFS mount for Kubernetes volume k8s-volumes/pvc-f2b79471-30fc-4c02-9008-49899e70f87e
     Loaded: loaded (/run/systemd/transient/geesefs-k8s_2dvolumes_2fpvc_2df2b79471_2d30fc_2d4c02_2d9008_2d49899e70f87e.service; transient)
  Transient: yes
    Drop-In: /run/systemd/system/geesefs-k8s_2dvolumes_2fpvc_2df2b79471_2d30fc_2d4c02_2d9008_2d49899e70f87e.service.d
             └─50-StopProps.conf
     Active: active (running) since Fri 2026-10-30 13:30:52 UTC; 44s ago
 Invocation: abd7fcbeb574439e985b9aeaa847c6a9
   Main PID: 957629 (geesefs)
      Tasks: 7 (limit: 1699)
     Memory: 6.9M (peak: 7.4M)
        CPU: 17ms
     CGroup: /system.slice/geesefs-k8s_2dvolumes_2fpvc_2df2b79471_2d30fc_2d4c02_2d9008_2d49899e70f87e.service
             └─957629 /var/lib/kubelet/plugins/ru.yandex.s3.csi/geesefs -f -o allow_other --endpoint http://s3.example.com:3900 --region garage --setuid 65534 --setgid 65534 --memory-limit 1000 --dir-mode 0770 --file-mode 0660 --uid 10001 --gid 10001 k8s-volumes:pvc-f2b79471-30fc-4c02-9008-49899e70f87e /var/lib/kubelet/plugins/kubernetes.io/csi/ru.yandex.s3.csi/a0978cd9ba03768c44659165dd0bc3f0a40595d4597aeb9a3c3484fdce698bbd/globalmount

GeeseFS runs as nobody (--setuid 65534), and the staging directory is the same SHA-256 of volumeHandle as in part 21. And here is what an ordinary node user sees without sudo (I replaced the 64-character secret key):

user@k8s-node-1:~$ systemctl show 'geesefs-k8s_2dvolumes_2fpvc_2df2b79471_2d30fc_2d4c02_2d9008_2d49899e70f87e.service' -p Environment
Environment=AWS_ACCESS_KEY_ID=GK51446d5c5b4635172249f367 AWS_SECRET_ACCESS_KEY=<secret key>

Now the behaviour. The pod on k8s-node-1 creates files and directories:

$ kubectl -n demo exec media-7f79bbf67d-cghcr -- sh -c 'id; df -h /media; echo hello > /media/a.txt; mkdir -p /media/reports/2026; echo r > /media/reports/2026/10.txt; ls -la /media'
uid=10001 gid=10001 groups=10001
Filesystem                Size      Used Available Use% Mounted on
k8s-volumes               1.0P         0      1.0P   0% /media
total 9
drwxrwx---    2 10001    10001         4096 Oct 30 13:31 .
drwxr-xr-x    1 root     root          4096 Oct 30 13:30 ..
-rw-rw----    1 10001    10001            6 Oct 30 13:31 a.txt
drwxrwx---    2 10001    10001            0 Oct 30 13:31 reports
$ kubectl -n demo exec media-7f79bbf67d-cghcr -- sh -c 'chmod 0600 /media/a.txt; echo chmod=$?; ls -l /media/a.txt; ln /media/a.txt /media/hard.txt; echo ln=$?; ln -s a.txt /media/sym.txt; echo symlink=$?'
chmod=0
-rw-rw----    1 10001    10001            6 Oct 30 13:31 /media/a.txt
ln: /media/hard.txt: Operation not supported
ln=1
symlink=0
$ kubectl -n demo exec media-7f79bbf67d-cghcr -- sh -c 'mv /media/a.txt /media/b.txt; mv /media/reports /media/archive; ls -R /media'
/media:
archive
b.txt
sym.txt

/media/archive:
2026

/media/archive/2026:
10.txt

A petabyte in df, chmod "succeeds" but the mode stays, hard links are not supported. On the neighbouring node the symbolic link turns out to be an empty file — on Garage GeeseFS does not restore it:

$ kubectl -n demo exec media-7f79bbf67d-4jr2n -- ls -l /media
total 1
drwxrwx---    2 10001    10001            0 Oct 30 13:32 archive
-rw-rw----    1 10001    10001            6 Oct 30 13:32 b.txt
-rw-rw----    1 10001    10001            0 Oct 30 13:32 sym.txt

Consistency: the pod on k8s-node-2 has already read the directory, the pod on k8s-node-1 creates c.txt at 13:33:06, and the second node lists the directory every five seconds:

13:33:06 archive b.txt sym.txt
13:33:11 archive b.txt sym.txt
...
13:34:01 archive b.txt sym.txt
13:34:06 archive b.txt c.txt sym.txt

Exactly one minute — --stat-cache-ttl. Locks: the pod on k8s-node-1 takes a flock for 20 seconds, and meanwhile a pod on the other node and a second process on the same node try:

$ kubectl -n demo exec media-7f79bbf67d-cghcr -- flock -n /media/lock -c 'echo A got lock; sleep 20' &
A got lock
$ kubectl -n demo exec media-7f79bbf67d-4jr2n -- sh -c 'flock -n /media/lock -c "echo B got lock too"; echo rc=$?'
B got lock too
rc=0
$ kubectl -n demo exec media-7f79bbf67d-cghcr -- sh -c 'flock -n /media/lock -c "echo A2 got lock"; echo rc=$?'
rc=1

On one node the kernel holds the lock, and the second process is refused; the pod on the other node knows nothing about it.

Driver restart. We delete the node plugin pod on k8s-node-1 — the DaemonSet creates a new one, and GeeseFS keeps running in its own service:

$ kubectl -n kube-system delete pod csi-s3-v7j98
pod "csi-s3-v7j98" deleted from kube-system namespace
$ kubectl -n demo exec media-7f79bbf67d-cghcr -- sh -c 'echo after-restart > /media/d.txt; cat /media/d.txt'
after-restart
user@k8s-node-1:~$ ps -o pid,etime,cmd -C geesefs | cut -c1-80
    PID     ELAPSED CMD
 957629       04:37 /var/lib/kubelet/plugins/ru.yandex.s3.csi/geesefs -f -o allo

The same experiment with a class where --no-systemd is added to options, and a pod nosd on k8s-node-1:

$ kubectl -n kube-system delete pod csi-s3-nlqqr
pod "csi-s3-nlqqr" deleted from kube-system namespace
$ kubectl -n demo exec nosd -- ls /data
ls: /data: Transport endpoint is not connected
command terminated with exit code 1
$ kubectl -n demo exec nosd -- sh -c 'echo y > /data/y.txt'
sh: can't create /data/y.txt: Transport endpoint is not connected
command terminated with exit code 1
$ kubectl -n demo get pod nosd
NAME   READY   STATUS    RESTARTS   AGE
nosd   1/1     Running   0          38s

The GeeseFS process died with the driver container, the volume is dead, and the pod stays Running — it has no probe that would notice. Moreover, after such a pod is deleted, kubelet logs GetMountRefs found corrupted mount … treating as unmounted path, the driver does not remove the dead mount on NodeUnstageVolume, and staging has to be unmounted on the node by hand (sudo umount …/globalmount). So --no-systemd only if the nodes have no systemd.

What is in the bucket — a ListObjectsV2 request signed with SigV4 straight from curl (the key is substituted from a file so it stays out of the history and the process list):

user@s3:~$ printf 'user = "GK51446d5c5b4635172249f367:%s"\n' "$(sudo cat /root/garage-k8s-csi-s3.secret)" \
    | curl -sS -K - --aws-sigv4 aws:amz:garage:s3 'http://127.0.0.1:3900/k8s-volumes?list-type=2' \
    | grep -oE '<Key>[^<]*</Key>|<Size>[^<]*</Size>' | paste - -
<Key>pvc-f2b79471-30fc-4c02-9008-49899e70f87e/</Key>	<Size>0</Size>
<Key>pvc-f2b79471-30fc-4c02-9008-49899e70f87e/archive/</Key>	<Size>0</Size>
<Key>pvc-f2b79471-30fc-4c02-9008-49899e70f87e/archive/2026/</Key>	<Size>0</Size>
<Key>pvc-f2b79471-30fc-4c02-9008-49899e70f87e/archive/2026/10.txt</Key>	<Size>2</Size>
<Key>pvc-f2b79471-30fc-4c02-9008-49899e70f87e/b.txt</Key>	<Size>6</Size>
<Key>pvc-f2b79471-30fc-4c02-9008-49899e70f87e/c.txt</Key>	<Size>4</Size>
<Key>pvc-f2b79471-30fc-4c02-9008-49899e70f87e/d.txt</Key>	<Size>14</Size>
<Key>pvc-f2b79471-30fc-4c02-9008-49899e70f87e/lock</Key>	<Size>0</Size>
<Key>pvc-f2b79471-30fc-4c02-9008-49899e70f87e/sym.txt</Key>	<Size>0</Size>

Directories are empty objects with a trailing /, the renamed directory reports became a set of new keys archive/…, the symbolic link is an empty object (GeeseFS keeps its target in object metadata, which Garage does not return in listings). d.txt is the write after the driver restart; the nosd volume's prefix is already gone: its PVC was deleted, and the driver removed the prefix.

Stopping a volume. When the Deployment is scaled to zero, the driver stops the service on NodeUnstageVolume; GeeseFS running as nobody cannot unmount itself, after 20 seconds (TimeoutStopSec from 50-StopProps.conf) systemd kills it, and ExecStopPost runs umount -f -l:

user@k8s-node-2:~$ sudo journalctl -u 'geesefs-*' -o cat --since '-5min'
...
2026/10/30 13:38:30.580733 main.INFO Received terminated, attempting to unmount...
Stopping geesefs-k8s_2dvolumes_2fpvc_2df2b79471_2d30fc_2d4c02_2d9008_2d49899e70f87e.service - GeeseFS mount for Kubernetes volume k8s-volumes/pvc-f2b79471-30fc-4c02-9008-49899e70f87e...
2026/10/30 13:38:50.617082 main.ERROR Failed to unmount in response to terminated: exit status 1: /usr/bin/fusermount3: failed to unmount /var/lib/kubelet/plugins/kubernetes.io/csi/ru.yandex.s3.csi/a0978cd9ba03768c44659165dd0bc3f0a40595d4597aeb9a3c3484fdce698bbd/globalmount: Permission denied
geesefs-k8s_2dvolumes_2fpvc_2df2b79471_2d30fc_2d4c02_2d9008_2d49899e70f87e.service: State 'stop-sigterm' timed out. Killing.
geesefs-k8s_2dvolumes_2fpvc_2df2b79471_2d30fc_2d4c02_2d9008_2d49899e70f87e.service: Killing process 329074 (geesefs) with signal SIGKILL.
geesefs-k8s_2dvolumes_2fpvc_2df2b79471_2d30fc_2d4c02_2d9008_2d49899e70f87e.service: Main process exited, code=killed, status=9/KILL
geesefs-k8s_2dvolumes_2fpvc_2df2b79471_2d30fc_2d4c02_2d9008_2d49899e70f87e.service: Failed with result 'timeout'.
Stopped geesefs-k8s_2dvolumes_2fpvc_2df2b79471_2d30fc_2d4c02_2d9008_2d49899e70f87e.service - GeeseFS mount for Kubernetes volume k8s-volumes/pvc-f2b79471-30fc-4c02-9008-49899e70f87e.

The mount is gone and the service has disappeared, but the drop-in directory /run/systemd/system/geesefs-….service.d stays until the node reboots. Here the files were written long before the stop, but GeeseFS uploads data asynchronously, and the SIGKILL after 20 seconds is dangerous for a volume whose writes have not reached S3 yet: the application must close its files (fsync, close) before the pod stops. Deleting the PVC removes the prefix in the bucket:

$ kubectl -n kube-system logs csi-s3-provisioner-0 -c csi-s3 | grep -E 'DeleteVolume|removed'
I1030 13:39:22.351104       1 driver.go:119] GRPC call: /csi.v1.Controller/DeleteVolume
I1030 13:39:22.359356       1 controllerserver.go:130] Prefix pvc-f2b79471-30fc-4c02-9008-49899e70f87e removed
user@s3:~$ sudo garage -c /etc/garage.toml bucket info k8s-volumes | sed -n '/Size/,/Objects/p'
Size:            0 B (0 B)
Objects:         0

Cleanup

The drivers are removed with the same manifests (kubectl delete -f), taking their StorageClasses and the local-path-storage and storage-credentials namespaces with them; the Retain PV (k8s-node-1-disk1) is deleted by hand. On the nodes k8s-csi-s3 leaves the directory /var/lib/kubelet/plugins/ru.yandex.s3.csi with a copy of the GeeseFS binary and the service drop-ins in /run/systemd/system, local-path leaves an empty /opt/local-path-provisioner, plus the driver images and busybox:latest — all removed by hand (crictl rmi with exact names). The loop disk: umount, losetup -d, delete the image file.

Sources

Next — Kustomize: how to stop copying manifests between environments, base and overlays, patches, ConfigMap and Secret generators, and kubectl apply -k.

Only registered users can leave a comment.

Sign in or register to leave a comment.

© 2006-2026 Vitaliy Leschenko