7 October 2026, 16:37Виталий Лещенко

PostgreSQL backups from Kubernetes to Object Storage: no CSI, and a key that can't delete

In the series on highly available Kubernetes on BGP I didn't say a word about backups. Time to fix that. This article covers how a nightly pg_dump from the cluster ends up in a private Yandex Object Storage bucket, why I didn't mount the bucket via CSI, what permissions the key needs, and the pitfalls I hit with the bucket policy.

I did everything below for two services, each with its own cloud and its own bucket. So from the very start the scheme assumes there's no shared "one key for everything".

How it was and what was wrong with it

The original scheme was simple and familiar: a CronJob ran pg_dump --format=custom once a night and put the file on a PersistentVolume, then deleted dumps older than 14 days with find -mtime +14 -delete.

The volume was on an SMB share. The problem was that the dumps sat at the same site as the database itself, and the file server had no backups of its own. Lose the site, and you lose both the data and its copies. A backup stored next to the original protects you from DROP TABLE, but not from a fire.

Object Storage in the cloud is a different site, cheap storage and lifecycle rules out of the box. All that was left was to decide how to write to it.

The CSI option and why I dropped it

The first thought was to mount the bucket as a volume and change nothing in the CronJob. Yandex has an official driver, k8s-csi-s3 (a fork of ctrox/csi-s3 that mounts through GeeseFS), and it does work. But for backups it's more trouble than it's worth:

FUSE on the node. The bucket is mounted on the host, and the driver needs privileged pods and mountPropagation. One more component with node-level permissions that has to be kept up to date.
Rotation over the network. find -delete on a mounted bucket means listing and deleting objects through FUSE: slow and full of surprises.
Node dependency. A hung mount, and the CronJob hangs along with it. I'd already been through that with SMB.
One StorageClass, one secret. For two clouds I'd have had to set up static PVs with their own secrets, and on top of that the driver's chart by default creates buckets itself with reclaimPolicy: Delete: delete the PVC, and you've deleted the bucket with your backups.

For the task "put one file somewhere once a day" you don't need to mount a file system. Uploading an object is enough.

The scheme

A Job with two steps in one pod:

The dump initContainer (the postgres image) runs pg_dump into a temporary emptyDir and checks that the archive is readable.
The upload container (the rclone/rclone image) uploads the file to the bucket.

The init container gives you ordering and a short circuit: if the dump fails, the upload doesn't start, the Job fails, and the failed Job stays in the history. There's no need to build your own image with pg_dump and rclone inside.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: blog-backup
spec:
  schedule: "30 3 * * *"
  timeZone: "Europe/Minsk"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 7
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 2
      activeDeadlineSeconds: 1800
      template:
        spec:
          restartPolicy: Never
          securityContext:
            runAsNonRoot: true
            runAsUser: 70      # postgres in the postgres:*-alpine image
            runAsGroup: 70
            fsGroup: 70
          initContainers:
            - name: dump
              image: postgres:18-alpine
              env:
                - { name: PGHOST, value: pg.example.com }
                - { name: PGPORT, value: "5432" }
                - { name: PGDATABASE, value: blog }
                - { name: PGUSER, value: blog }
                - name: PGPASSWORD
                  valueFrom:
                    secretKeyRef: { name: blog-backup, key: PGPASSWORD }
              command:
                - sh
                - -ec
                - |
                  ts="$(date +%Y%m%d-%H%M%S)"
                  out="/backups/${PGDATABASE}-${ts}.dump"
                  pg_dump --format=custom --compress=6 --file="${out}"
                  [ -s "${out}" ]
                  pg_restore --list "${out}" > /dev/null   # the archive must be readable before it leaves the pod
                  ls -lh /backups
              volumeMounts:
                - { name: backups, mountPath: /backups }
          containers:
            - name: upload
              image: rclone/rclone:1.75.1
              env:
                - { name: RCLONE_CONFIG_YC_TYPE, value: s3 }
                - { name: RCLONE_CONFIG_YC_PROVIDER, value: Other }
                - { name: RCLONE_CONFIG_YC_ENDPOINT, value: https://storage.yandexcloud.net }
                - { name: RCLONE_CONFIG_YC_REGION, value: ru-central1 }
                - name: RCLONE_CONFIG_YC_ACCESS_KEY_ID
                  valueFrom:
                    secretKeyRef: { name: blog-backup, key: S3_ACCESS_KEY }
                - name: RCLONE_CONFIG_YC_SECRET_ACCESS_KEY
                  valueFrom:
                    secretKeyRef: { name: blog-backup, key: S3_SECRET_KEY }
                - { name: S3_DEST, value: "yc:blog-db-backups/postgres" }
              args:
                - copy
                - /backups
                - $(S3_DEST)
                - --s3-no-check-bucket
                - --s3-no-head
                - --no-check-dest
                - -v
              volumeMounts:
                - { name: backups, mountPath: /backups, readOnly: true }
          volumes:
            - name: backups
              emptyDir:
                sizeLimit: 512Mi

A couple of notes on the manifest.

rclone without a config file. The yc remote is described entirely by the RCLONE_CONFIG_YC_* variables: type, provider, endpoint, region and keys. Neither a ConfigMap nor rclone config is needed, and the keys come from a Secret, just like the database password.

Three flags on copy. --s3-no-check-bucket: don't try to create the bucket (the key has no permission for that, and rightly so). --s3-no-head and --no-check-dest: don't check whether the object exists at the destination; the name contains a timestamp, and the key may not even be allowed to list. Without these flags rclone fails on a restricted key before it even uploads.

emptyDir with sizeLimit. The dump lives in the pod exactly as long as it takes to upload. Set the limit to twice the dump size: if it overflows, the pod gets evicted.

Checking the archive. pg_restore --list reads the table of contents and fails on a corrupt file. A cheap check that catches a truncated dump before it turns out to be the only copy.

In the Helm chart all of this is parameterized: the bucket name, the prefix, the images and the keys come from values, and the keys from a sops-encrypted secrets file.

The bucket

Each service gets a separate bucket for backups in its own cloud, not the application's working bucket. That way retention and delete permissions won't affect the application's files.

yc storage bucket create --name blog-db-backups \
  --default-storage-class standard --max-size $((10 * 1024**3)) \
  --public-read=false --public-list=false --public-config-read=false

Retention is a bucket rule, not a script

Rotation used to be done by find in the CronJob. Now it's a bucket lifecycle rule: objects under the postgres/ prefix are deleted after 14 days, incomplete multipart uploads after one day.

{"lifecycleRules": [{
  "id": "expire-dumps", "enabled": true,
  "filter": {"prefix": "postgres/"},
  "expiration": {"days": "14"},
  "abortIncompleteMultipartUpload": {"daysAfterExpiration": "1"}
}]}

yc storage bucket update --name blog-db-backups --lifecycle-rules-from-file lifecycle.json

The rule works even if the CronJob doesn't run. And, more importantly, the key in the pod no longer needs permission to delete.

One oddity: yc storage bucket get --full shows expiration.date: 0001-01-01T00:00:00Z for the rule next to days: 14. That's the zero value of an unset field, not a date in the past; the rule works by days.

Permissions: a role on the bucket

A service account for uploads, one per bucket:

yc iam service-account create --name db-backup-uploader
yc iam access-key create --service-account-name db-backup-uploader

It needs a role on this bucket only, not on the folder: storage.uploader. That role allows uploading, listing and reading objects, but not deleting them or creating buckets. A key leaked from the cluster can't wipe the backups, and such a key can read the dumps anyway: the database password sits in the same Secret.

In the console, a role on a bucket is assigned on the "Security" tab. I didn't find a yc command for it, but there is a REST API, with two quirks: the path needs the bucket's resource_id, not its name (with the name it replies "The specified bucket does not exist"), and the method is PATCH (POST gets Method Not Allowed):

RID=$(yc storage bucket get --name blog-db-backups --full --format json | jq -r .resource_id)
SA=$(yc iam service-account get --name db-backup-uploader --format json | jq -r .id)

curl -s -X PATCH "https://storage.api.cloud.yandex.net/storage/v1/buckets/$RID:updateAccessBindings" \
  -H "Authorization: Bearer $(yc iam create-token)" -H "Content-Type: application/json" \
  -d "{\"accessBindingDeltas\":[{\"action\":\"ADD\",\"accessBinding\":{\"roleId\":\"storage.uploader\",\"subject\":{\"id\":\"$SA\",\"type\":\"serviceAccount\"}}}]}"

For restores there's a second account, db-backup-reader, with the storage.viewer role on the same bucket. It reads and lists, but doesn't write. The upload key isn't used in the restore pod.

A check worth doing before you rely on the scheme: run all four operations with both keys. Here's what I got:

Operation	uploader	reader
upload	yes	403
list	yes	yes
download	yes	yes
delete	403	403

Pitfall: a bucket policy narrows permissions and locks out the owner

At first I granted access not with a role but with a bucket policy: Principal: {CanonicalUser: <account id>}, Action: s3:PutObject. The policy was saved and read back fine, yet for twelve minutes straight uploads answered AccessDenied, both through rclone and through boto3. A policy alone wasn't enough for an account with no role on the bucket.

I added the storage.uploader role, uploads started working, and I stopped there. An hour later it turned out that the cloud owner (the resource-manager.clouds.owner role) could neither list, nor read, nor delete objects in the new buckets: AccessDenied on everything. In the old buckets, which had no policy, everything worked.

An experiment on a test bucket made it all clear:

a bucket without a policy: the owner sees the objects;
add a policy with Allow for another subject: the owner gets 403;
add Allow s3:* for the owner to the same policy: access comes back.

In other words, when a policy exists, access is the intersection of the role and the policy: the role grants the permission, the policy must not deny it, and anyone not in the policy doesn't get through, even the cloud owner. I didn't find this stated explicitly in the Yandex documentation I read, so it's a conclusion from observation; but that's exactly how it behaves.

Two practical rules follow from this:

If you don't need a policy, don't create one. A role on the bucket covers the task completely, while a policy adds a second layer that's easy to forget about.
If you do need a policy, include yourself in it. Otherwise only those listed in it will have access to the bucket.

One more thing: I couldn't remove the policy with yc. --policy '{}' and a policy with an empty Statement list are saved as {"Version": "2012-10-17"} and don't restore access. It only went away from the console.

As for the "write-only key" I believed in for a while, that was the policy too. It cut the permissions of storage.uploader down to a single PutObject. Without the policy the role works as in the table above.

Restoring

While everything lived on a volume, restores used a Deployment with zero replicas: the postgres image, the same volume at /backups, the PG* variables already set. Scale up a replica, exec inside, run pg_restore.

With a bucket, the volume is replaced by an emptyDir, and next to it there's a second container, fetch: the same rclone, but with the read key:

kubectl scale deployment/blog-restore --replicas=1

kubectl exec deploy/blog-restore -c fetch -- sh -c 'rclone ls "$S3_DEST"'
kubectl exec deploy/blog-restore -c fetch -- sh -c 'rclone copyto "$S3_DEST/blog-20261006-185006.dump" /backups/blog-20261006-185006.dump'

kubectl exec -it deploy/blog-restore -c restore -- sh
# inside:
pg_restore --list /backups/blog-20261006-185006.dump | head
pg_restore --clean --if-exists --no-owner -d "$PGDATABASE" /backups/blog-20261006-185006.dump

kubectl scale deployment/blog-restore --replicas=0

Before a real restore, stop everything that writes to the database, or --clean will fight with live connections.

Which user to connect as depends on who owns the objects. If the application's role owns the database and all the tables (as with the blog), restore as that role. If the objects belong to different roles or the database has extensions, you need administrator rights, and that's a separate Secret the backup pod must not have.

It's worth testing the restore procedure right away rather than waiting for a disaster: bring up the pod, download yesterday's dump, read pg_restore --list, connect with psql. You don't have to touch the database itself.

What I chose not to do

Bucket versioning. File names carry a timestamp, nothing gets overwritten, and the pod's key can't delete. Versioning would only double the storage.
Client-side encryption. The bucket is private, traffic goes over TLS, and the read keys are separate. That's enough for my data; for other people's personal data I'd encrypt the dump before uploading.
Alerts. Kubernetes won't tell you a dump wasn't uploaded unless the Job fails, and a failed Job just lands in the history. I have an alert on failed Jobs in Prometheus; if you don't, now's the time.

Summary

A CronJob made of a pg_dump init container and an rclone container instead of a CSI driver: nothing is mounted, no privileged pods, and the secret lives in the service's namespace.
A separate private bucket per service, in its own cloud. Different clouds come naturally: each has its own Secret.
The storage.uploader role on the bucket: the pod's key can't delete. Retention is a bucket rule.
A bucket policy, if there is one, narrows permissions on top of the roles and locks out everyone not listed in it, including the cloud owner. Don't create one unless you need it.
A restore Deployment with a fetch container and a read key, tested before it's needed.