Kubernetes for beginners. Part 5: How do you build an image and where do you keep it?

Until now the cluster has been running other people's images: nginx:1.29 and busybox:1.37 from Docker Hub. In the first part an image was described in a single phrase — a packaged file system plus startup parameters; in the second we saw that the kubelet asks containerd over CRI to "pull the image" and "start the container"; and in the previous part we built the cluster we'll run them on. Today we'll go through the whole chain: what an image physically consists of, how containerd stores and unpacks it on a node, how to build our own hello application into a small and secure image, how to put it into the private GitHub Container Registry, and how to teach the cluster nodes to pull images from there.

hello is the application that runs through the whole series: in the following parts it will carry pods, Deployments, services, ingress, metrics and logs. That is why the source code is given in full and the image is built by the book.

---

What an image is made of

The image format is described by the OCI Image specification (the Open Container Initiative is an organization under the Linux Foundation that in 2015 standardized the image format and the container runtime environment based on what Docker had built). Alongside it are OCI Distribution, the protocol for distributing images over HTTP, and OCI Runtime, the description of how a low-level runtime such as runc starts a container. The Docker format ("Image Manifest V2, Schema 2") almost matches OCI — only the mediaType strings differ — and containerd understands both.

An image is not a single file but a small tree of JSON documents and archives that refer to each other by the hash of their content:

 tag ghcr.io/example/hello:1.0
   │
   ▼
 Index  (application/vnd.oci.image.index.v1+json)        sha256:a5066cd4…
   ├─ manifest linux/amd64  ──────────────────────────────┐
   ├─ manifest linux/arm64                                │
   ├─ attestation for amd64 (platform unknown/unknown)    │
   └─ attestation for arm64                               │
                                                          ▼
 Manifest (application/vnd.oci.image.manifest.v1+json)   sha256:bc289980…
   ├─ config ─► Config (…image.config.v1+json)           sha256:f768da29…
   │              architecture, os, Entrypoint, Env, User,
   │              rootfs.diff_ids, history
   └─ layers
        ├─ layer 1 (…image.layer.v1.tar+gzip)             sha256:2cc7ee28…
        ├─ …
        └─ layer 14: our binary /hello                     sha256:28a9c83d…

A layer is a tar archive of file system changes, usually compressed with gzip, less often with zstd. Each layer contains the files added or changed in one build step. A deleted file is recorded in the layer as an empty "whiteout" file .wh.<name>, and a directory whose contents are replaced entirely is marked with .wh..wh..opq. That is why RUN rm in a Dockerfile does not make the image smaller: the file stays in the previous layer, and the new layer only hides it.

Config is a JSON document that says how to run the image: architecture and OS, Entrypoint and Cmd, environment variables, working directory, user, exposed ports, labels, stop signal. It also holds rootfs.diff_ids — the hashes of the layers in uncompressed form — and history, the log of build steps.

Manifest is the table of contents of the image for one platform: a reference to the config and the list of layers in order. Each reference is a descriptor: the content type (mediaType), the hash (digest) and the size in bytes (size).

Index (a manifest list in Docker terms) is a table of contents over tables of contents: a list of manifests, each with its platform. A single tag hello:1.0 serves both x86-64 and ARM nodes: containerd on a node reads the index and takes the manifest for its own architecture. Entries with the platform unknown/unknown are attestations added by BuildKit: information on how and from what the image was built (provenance) and, on request, a list of the packages inside (SBOM). The runtime skips them.

Content addressing

The key property of the whole construction: every object is named by its digest — sha256: followed by the SHA-256 of its bytes. A manifest contains the digests of the config and the layers, an index contains the digests of the manifests. The result is a Merkle tree: the digest of the index uniquely determines every byte of every layer for every platform. It is impossible to change something inside and keep the digest.

Three consequences follow:

• Integrity. Having downloaded a blob, containerd computes its hash and compares it with the digest from the manifest. A layer that was substituted or damaged in transit will be rejected even if the registry is compromised — provided that the manifest digest itself was obtained from a trusted place.
• Deduplication. A base image layer shared by a dozen applications is stored in the registry and on the node once and downloaded once.
• A tag is a pointer, a digest is a name. The tag 1.0 is a record in the registry saying "this tag points to such-and-such digest", and it can be overwritten. A digest cannot be overwritten. We'll come back to this in the section on tags.

A layer has two hashes. The manifest holds the digest of the compressed archive: that is what gets downloaded and verified. The config (diff_ids) holds the digest of the uncompressed tar: it does not depend on which compressor and compression level were used to build the layer. containerd uses a third value, the chainID: a hash of the layer's diff_id and the chainID of all the layers beneath it. The same layer on top of different bases gives different chainIDs, and that is correct: the unpacked file system "this layer on top of that base" is a different one.

---

How containerd stores and unpacks an image

When the kubelet asks containerd over CRI to pull ghcr.io/example/hello:1.0, the following happens:

1. Resolve. containerd asks the registry for the manifest by tag and gets a digest — for a multi-architecture image, the digest of the index.
2. Platform selection. The linux/amd64 manifest is taken from the index (all the lab nodes are x86-64).
3. Fetch. The config and the layers that are not yet present locally are downloaded into the content store — the directory /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/, where each file is named by its digest. The metadata — which images exist and which blobs belong to them — lives in a bbolt database (an embedded key-value store written in Go), /var/lib/containerd/io.containerd.metadata.v1.bolt/meta.db.
4. Unpack. Each layer is unpacked by a snapshotter into a separate file system snapshot.

containerd separates everything into namespaces: the kubelet's images and containers live in k8s.io, Docker Engine, if it is installed on the same machine, uses moby, and ctr looks at default by default. Hence a common confusion: ctr images ls on a cluster node shows an empty list, while ctr -n k8s.io images ls shows all the images.

The overlayfs snapshotter

The default snapshotter is overlayfs, a kernel file system that stacks several directories into one: the lower ones (lowerdir) are read-only, the upper one (upperdir) is writable, and on read a file is taken from the topmost layer that has it.

 /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/
   95/fs    ◄── layer 1 of the base image, unpacked (committed, read-only)
   96/fs    ◄── layer 2 …
   …
   108/fs   ◄── layer 14: /hello               (committed, read-only)
   109/fs   ◄── container upperdir             (active, writable)
   109/work ◄── overlayfs work directory

 container root = overlay:
   lowerdir = 108/fs:107/fs:…:95/fs   (top to bottom)
   upperdir = 109/fs
   workdir  = 109/work

Each layer is unpacked once into its own directory snapshots/N/fs and marked as a committed snapshot keyed by its chainID. When a container starts, the snapshotter creates an active snapshot on top of the image's chain of layers and mounts the overlay at /run/containerd/io.containerd.runtime.v2.task/k8s.io/<id>/rootfs — that is the container's root. Everything the container writes goes into its upperdir; when the container is removed, the active snapshot is deleted together with what was written. Ten containers from the same image share the same lower directories and spend space only on their own changes.

There are other snapshotters too: native (full copies of directories, for file systems without overlay), btrfs and zfs (snapshots by means of those file systems), stargz and similar ones (lazy loading: the container starts before the whole image is downloaded, and files are fetched as they are accessed). The lab uses overlayfs: it is the default and the best option for most cases.

After unpacking, the compressed blobs stay in the content store by default: they are needed so that a layer does not have to be downloaded again and so that the image can be passed on (ctr export). Unneeded blobs and snapshots — those referenced by no image and no container — are removed by containerd's garbage collector.

---

The hello application

A small HTTP server in Go with no external dependencies — only the standard library:

• GET / responds with hello from <hostname>; in a pod the hostname is the pod name, so the response shows which replica answered;
• GET /healthz — "the process is alive", for the liveness probe;
• GET /readyz — "ready to accept requests": 200 while the server is running and 503 after SIGTERM is received; for the readiness probe (both probes are covered in the next part);
• GET /metrics — the request counter and the build version in the Prometheus text format;
• the port comes from the PORT variable, 8080 by default;
• on SIGTERM it stops accepting new connections and waits for the current requests to finish.

go.mod:

module example.com/hello

go 1.27

main.go:

package main

import (
	"context"
	"errors"
	"fmt"
	"log"
	"net"
	"net/http"
	"os"
	"os/signal"
	"sync/atomic"
	"syscall"
	"time"
)

// version is set at build time: -ldflags "-X main.version=1.0".
var version = "dev"

var (
	requests atomic.Uint64
	ready    atomic.Bool
)

func main() {
	port := os.Getenv("PORT")
	if port == "" {
		port = "8080"
	}
	hostname, err := os.Hostname()
	if err != nil {
		log.Fatalf("hostname: %v", err)
	}

	mux := http.NewServeMux()
	mux.HandleFunc("GET /{$}", func(w http.ResponseWriter, r *http.Request) {
		requests.Add(1)
		log.Printf("%s %s %s", r.RemoteAddr, r.Method, r.URL.Path)
		w.Header().Set("X-App-Version", version)
		fmt.Fprintf(w, "hello from %s\n", hostname)
	})
	mux.HandleFunc("GET /healthz", func(w http.ResponseWriter, r *http.Request) {
		fmt.Fprintln(w, "ok")
	})
	mux.HandleFunc("GET /readyz", func(w http.ResponseWriter, r *http.Request) {
		if !ready.Load() {
			http.Error(w, "not ready", http.StatusServiceUnavailable)
			return
		}
		fmt.Fprintln(w, "ok")
	})
	mux.HandleFunc("GET /metrics", func(w http.ResponseWriter, r *http.Request) {
		w.Header().Set("Content-Type", "text/plain; version=0.0.4; charset=utf-8")
		fmt.Fprintln(w, "# HELP hello_requests_total Requests served by the / handler.")
		fmt.Fprintln(w, "# TYPE hello_requests_total counter")
		fmt.Fprintf(w, "hello_requests_total %d\n", requests.Load())
		fmt.Fprintln(w, "# HELP hello_build_info Build information.")
		fmt.Fprintln(w, "# TYPE hello_build_info gauge")
		fmt.Fprintf(w, "hello_build_info{version=%q} 1\n", version)
	})

	srv := &http.Server{
		Handler:           mux,
		ReadHeaderTimeout: 5 * time.Second,
	}

	ln, err := net.Listen("tcp", ":"+port)
	if err != nil {
		log.Fatalf("listen: %v", err)
	}
	ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM, syscall.SIGINT)
	defer stop()

	go func() {
		if err := srv.Serve(ln); err != nil && !errors.Is(err, http.ErrServerClosed) {
			log.Fatalf("serve: %v", err)
		}
	}()
	ready.Store(true)
	log.Printf("hello %s listening on %s", version, ln.Addr())

	<-ctx.Done()
	ready.Store(false)
	log.Printf("signal received, shutting down")
	shutdownCtx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
	defer cancel()
	if err := srv.Shutdown(shutdownCtx); err != nil {
		log.Printf("shutdown: %v", err)
	}
	log.Printf("bye")
}

A few places that matter for Kubernetes.

Logs go to stdout and stderr. The log package writes to stderr; the application keeps no files. containerd collects the container's streams into files on the node, the kubelet serves them through kubectl logs, and the log collector (part 33) ships them to storage. An application that writes to a file inside the container loses its logs together with the container.

A configuration error means an immediate exit. If PORT is taken or set incorrectly, net.Listen returns an error and the process exits with code 1. That is better than running without an open port: Kubernetes will see the failure, restart the container and show the reason in the status and the logs.

SIGTERM. When a pod is stopped, the kubelet sends SIGTERM to the container's main process and waits, 30 seconds by default, before sending SIGKILL. srv.Shutdown closes the listening socket, waits for the requests in progress to finish and closes keep-alive connections. A process that ignores SIGTERM is killed every time after 30 seconds, with requests cut off. How this fits together with removing the pod from load balancing is covered in the next part.

Metrics format. /metrics serves the text format of Prometheus, a monitoring system that itself periodically polls (scrapes) the HTTP addresses of applications and stores the numbers it gets as time series; details are in part 32. The format is simple: # HELP lines with a description and # TYPE lines with the type (counter is a counter that only grows, gauge is an arbitrary value), followed by lines of the form "name{labels} value". A hello_build_info metric with the value 1 and the version in a label is the common way to expose the build version. In a real application metrics are written with the prometheus/client_golang library; here we did without it to keep the source self-contained.

The version is baked in at build time with the linker flag -X main.version=1.0 and is visible in the X-App-Version response header, in the metric and in the first log line. The response body does not depend on the version.

---

Dockerfile

# syntax=docker/dockerfile:1

# --- build ---
FROM --platform=$BUILDPLATFORM golang:1.27 AS build
ARG TARGETOS TARGETARCH
ARG VERSION=dev
WORKDIR /src
COPY go.mod ./
RUN --mount=type=cache,target=/go/pkg/mod go mod download
COPY *.go ./
RUN --mount=type=cache,target=/go/pkg/mod \
    --mount=type=cache,target=/root/.cache/go-build \
    CGO_ENABLED=0 GOOS=$TARGETOS GOARCH=$TARGETARCH \
    go build -trimpath -ldflags="-s -w -X main.version=${VERSION}" -o /out/hello .

# --- final image ---
FROM gcr.io/distroless/static-debian13:nonroot
ARG VERSION=dev
LABEL org.opencontainers.image.title="hello" \
      org.opencontainers.image.version="${VERSION}" \
      org.opencontainers.image.source="https://git.example.com/demo/hello"
COPY --from=build /out/hello /hello
USER 65532:65532
EXPOSE 8080
ENTRYPOINT ["/hello"]

And .dockerignore, so that unnecessary files — and with them secrets and .git — do not end up in the build context:

*
!go.mod
!go.sum
!*.go

Let's go through it line by line.

# syntax=docker/dockerfile:1 is a directive for BuildKit: use the current stable version of the Dockerfile syntax. It is needed for RUN --mount.

Multi-stage. Two FROMs mean two stages. The first (build) is a full image with Go, the compiler and the sources, almost a gigabyte. The second is the final one, and exactly one file is copied into it from the first. Nothing that was in the build stage — the compiler, the module cache, the sources — gets into the final image.

--platform=$BUILDPLATFORM, TARGETOS, TARGETARCH. BuildKit sets these arguments itself: BUILDPLATFORM is the platform of the machine the build runs on, TARGETOS/TARGETARCH is the platform we are building for. The build stage always runs on the "native" architecture, and Go cross-compiles the binary for the target one. Without this, a build for arm64 on an x86-64 machine would run in the QEMU emulator and be many times slower.

Cache mounts. --mount=type=cache attaches to a step a directory that BuildKit keeps between builds but does not put into a layer: the Go module cache and compiler cache survive a rebuild, and the image does not grow because of them.

Order of steps. First go.mod and downloading the dependencies, then the sources. BuildKit caches every step and rebuilds starting from the first one that changed: an edit to main.go will not force the modules to be downloaded again. The general rule: what changes rarely goes higher.

CGO_ENABLED=0 gives a fully static binary that does not need libc. -trimpath removes the build machine's paths from the binary (the build becomes reproducible), -s -w removes the symbol table and debug information (a third smaller).

distroless/static. A base image from Google's distroless project: it has no shell, no package manager and no utilities — only root CA certificates (/etc/ssl/certs), time zones, /etc/passwd with the users root and nonroot (UID 65532), and /tmp. Less than a megabyte compressed. The nonroot tag makes nonroot the default user. The -debian13 suffix is the Debian version whose packages the image is built from; it is worth specifying explicitly so that a change of the base distribution does not happen unnoticed.

Why this is better than FROM ubuntu or FROM alpine. Fewer files mean fewer vulnerabilities that a scanner will find in the image and that will have to be closed with updates. No shell means that an attacker who gains code execution in the process has none of the usual tools. A smaller size means a faster download to a new node when scaling. The flip side is that kubectl exec ... sh into such a container does not work; for debugging there is kubectl debug with an ephemeral container, covered in part 31.

An alternative is FROM scratch, an empty image. It also works for a Go binary, but you will have to copy the CA certificates, /etc/passwd and the time zones yourself; distroless/static is exactly "scratch plus these three things", built and kept up to date for you.

USER 65532:65532. The process in the container does not run as root. The nonroot tag already does this, but a numeric UID in the Dockerfile is useful to Kubernetes: with runAsNonRoot: true (part 30) the kubelet checks that the image user is not 0, and it can do that only by number, not by name. Port 8080 is above 1024, so no privileges are needed for it.

LABEL — labels in the image config following the OCI convention (org.opencontainers.image.*): where the sources are, which version. Registries and scanners display them. GitHub Container Registry uses the org.opencontainers.image.source label with the address of a GitHub repository to link the image to that repository and its access permissions.

Pinning base images. golang:1.27 and static-debian13:nonroot are tags, and their owners rebuild the images under them with security updates. For a reproducible build the base is pinned by digest: FROM gcr.io/distroless/static-debian13:nonroot@sha256:<digest>. BuildKit itself records in the provenance which digest it took the base by: the build log shows that nonroot turned into nonroot@sha256:18f4cc26…. Updating such lines by hand is inconvenient, so bots like Renovate or Dependabot take care of it: they watch for new digests and open a pull request with the update, which goes through the usual review.

---

Building: docker buildx

Behind docker build there has long been BuildKit, a build engine that turns a Dockerfile into a graph of steps, runs independent steps in parallel, caches them and can build for several platforms. buildx is a Docker CLI plugin for working with BuildKit directly. We build on the workstation in Docker Desktop.

For a multi-architecture build the most convenient option is a builder with the docker-container driver — a separate container with BuildKit that does not need multi-platform support in Docker's own storage. First a build without pushing, to check that everything compiles:

$ docker buildx create --name multi --driver docker-container --use
multi
$ docker buildx build --platform linux/amd64,linux/arm64 \
    --build-arg VERSION=1.0 --progress=plain \
    -t ghcr.io/example/hello:1.0 .
#1 [internal] booting buildkit
#1 DONE 14.1s
...
#5 [linux/amd64 internal] load metadata for docker.io/library/golang:1.27
#6 [linux/amd64 internal] load metadata for gcr.io/distroless/static-debian13:nonroot
#12 [linux/amd64 stage-1 1/2] FROM gcr.io/distroless/static-debian13:nonroot@sha256:18f4cc26c9ffa7f20a359298be7deadac5c607c8e03a02b8119f1a5fc2f82c99
...
#19 [linux/amd64 build 6/6] RUN --mount=type=cache,target=/go/pkg/mod     --mount=type=cache,target=/root/.cache/go-build     CGO_ENABLED=0 GOOS=linux GOARCH=amd64     go build -trimpath -ldflags="-s -w -X main.version=1.0" -o /out/hello .
#19 DONE 6.3s
#20 [linux/amd64->arm64 build 6/6] RUN --mount=type=cache,target=/go/pkg/mod     --mount=type=cache,target=/root/.cache/go-build     CGO_ENABLED=0 GOOS=linux GOARCH=arm64     go build -trimpath -ldflags="-s -w -X main.version=1.0" -o /out/hello .
#20 DONE 6.3s
...
WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load

The linux/amd64->arm64 line is that very cross-compilation: the arm64 binary was built on amd64. If the Dockerfile had no --platform=$BUILDPLATFORM, BuildKit would run the steps of the arm64 stage under emulation through QEMU and binfmt_misc (a kernel mechanism that hands the execution of a foreign executable format to a specified interpreter) — it works, but slowly. The warning at the end is honest: for now the result lives only in the builder's cache. --push will send it to the registry, --load into Docker's local storage; Docker Desktop stores images in containerd and therefore accepts multi-architecture images too.

You can check the image without leaving the workstation:

$ docker buildx build --platform linux/amd64 --build-arg VERSION=1.0 -t hello-test:1.0 --load .
$ docker run -d --rm --name hello-test -p 18080:8080 hello-test:1.0
$ curl -si http://localhost:18080/ | grep -E 'HTTP|X-App|hello'
HTTP/1.1 200 OK
X-App-Version: 1.0
hello from 149774fac689
$ curl -s http://localhost:18080/metrics | grep -v '^#'
hello_requests_total 1
hello_build_info{version="1.0"} 1
$ docker stop hello-test

In Docker the container's hostname is its short ID; in Kubernetes it will be the pod name.

Building without Docker

Docker is not required on the build machine, and in CI inside Kubernetes it is better not to have it. The popular recipes "mount /var/run/docker.sock into the build pod" or "run Docker-in-Docker with privileged: true" give anyone who can change the build script root on the node: through the Docker socket you can start a privileged container with the host's root file system. Alternatives:

• BuildKit on its own: the buildkitd daemon and the buildctl client. It is the same engine as inside docker buildx, but without Docker; it can run without root (rootless) and run as a pod in the cluster.
• buildah and podman are daemonless tools from Red Hat. buildah build (and podman build, which calls buildah) builds from the same Dockerfile; buildah can also build an image with shell script commands, without a Dockerfile. They run without root through user namespaces.
• kaniko is a builder that executes Dockerfile steps inside an ordinary unprivileged container, right in its file system, and packs the layers itself. The original Google project was archived in 2025; development continues in community forks.
• ko is an image builder for Go without a Dockerfile: it compiles the application and puts the binary on a distroless base. For Go services like hello it is the shortest path.

All of them produce an image in OCI format, which any registry and any runtime understands.

---

The registry: GitHub Container Registry

A registry is an HTTP service that stores blobs and manifests and serves them over the OCI Distribution protocol: all requests go to addresses of the form /v2/<name>/manifests/<tag or digest> and /v2/<name>/blobs/<digest>. There are many registries: Docker Hub, cloud registries, your own server on the network. In this series application images live in GitHub Container Registry (ghcr.io): it has a free storage and traffic quota for private images, it has the same accounts and permissions as GitHub repositories, and its certificate is issued by a public certificate authority, so the nodes need no configuration to trust it.

An image name in ghcr.io is ghcr.io/<owner>/<name>, where the owner is a GitHub user or organization. In this article the owner is example and the image is ghcr.io/example/hello; on your own lab substitute your own. A new image pushed to ghcr.io is private by default: only someone with permissions on the package can pull it.

Tokens

ghcr.io does not accept the GitHub account password — only tokens. Two are needed, with different permissions:

Who	Token scope	Why
build machine	write:packages	push images (includes reading)
cluster nodes	read:packages	pull only

These are classic personal access tokens (Settings → Developer settings → Personal access tokens → Tokens (classic)): fine-grained tokens do not work with the GitHub package registry. The nodes' token must not be able to do anything except pull packages: it will be stored in the cluster in a Secret, and everything it allows is allowed to everyone who can read that Secret. Both have a limited lifetime, for example 90 days, with a reminder to replace them.

On a workstation where the GitHub CLI is already set up, you don't have to create the push token by hand: gh will add the required scope to its session, and the token itself will go to Docker through a pipe, appearing neither on screen nor in the shell history:

$ gh auth refresh -h github.com -s write:packages
$ gh auth token | docker login ghcr.io -u example --password-stdin
Login Succeeded

Docker keeps credentials in the store specified in ~/.docker/config.json: in Docker Desktop that is "credsStore": "desktop", the system password store of Windows or macOS. If no store is configured, the password ends up in config.json itself in base64, that is, in plain text; on a shared build machine that is unacceptable.

Pushing

The same build command with --push:

$ docker buildx build --platform linux/amd64,linux/arm64 \
    --build-arg VERSION=1.0 --progress=plain \
    -t ghcr.io/example/hello:1.0 --push .
...
#20 [linux/amd64 build 6/6] RUN --mount=type=cache,target=/go/pkg/mod     --mount=type=cache,target=/root/.cache/go-build     CGO_ENABLED=0 GOOS=linux GOARCH=amd64     go build -trimpath -ldflags="-s -w -X main.version=1.0" -o /out/hello .
#20 CACHED
...
#22 exporting to image
#22 exporting layers 0.3s done
#22 exporting manifest sha256:bc289980e0a6cbd930a319094f25249a36634818823ab3bfc2d6a9382cd0ec6d 0.0s done
#22 exporting config sha256:f768da2949737845e4a79d5141c37186a611d44d79f0ecef3ddce921d3b5d00e 0.0s done
#22 exporting attestation manifest sha256:1a82f2e36a46e5d60c34b29cba26ae51fc181cf92b91cd83fa8ed31e52ae81c0 0.1s done
#22 exporting manifest sha256:e909d5afc7fecc910dce9195b1b3784252bfedc47ba3e1154266c7600866adc7 0.0s done
#22 exporting config sha256:bcbc7faf90e2af1a8fa61cb3730b35cc77bba1c4d0f11ec95e788ad527f17bd3 0.0s done
#22 exporting attestation manifest sha256:e9fc21e46675ff5ac5907435fc376e217b31c3ca94a3c11e98b0c6a92e4bf71a 0.0s done
#22 exporting manifest list sha256:a5066cd40ef37b9b889ddfbe9c1860b3afe231b96b975000e0edca3fdd6f4a4a 0.0s done
#23 [auth] example/hello:pull,push token for ghcr.io
#23 DONE 0.0s
#22 pushing layers 6.0s done
#22 pushing manifest for ghcr.io/example/hello:1.0@sha256:a5066cd40ef37b9b889ddfbe9c1860b3afe231b96b975000e0edca3fdd6f4a4a 2.9s done
#22 DONE 9.5s

All the build steps came from the cache (CACHED): the sources had not changed. The registry received the index (manifest list), a manifest and a config for each platform, and two provenance attestations that BuildKit adds by default; --sbom=true adds the package list as well. The [auth] line means BuildKit obtained from ghcr.io a token for reading and writing the example/hello repository; we'll look at how this exchange works at the end of the part.

Your own registry: Distribution and Harbor

A cloud registry is not always suitable: the cluster may sit in a network without internet access, the organization's rules may require keeping images in-house, and pulling from the cloud at every scale-up means external traffic and an external dependency. In that case the registry is run in-house.

Distribution is the reference implementation of the OCI Distribution protocol, formerly Docker Registry, now a project of the CNCF (the Cloud Native Computing Foundation, which is also home to Kubernetes). It is a single Go program in the official registry image (the 3.x branch) that stores blobs and manifests on disk or in S3 object storage. To work on a network it needs three things:

• TLS. A certificate for the registry name, usually from the organization's internal certificate authority (how certificates and chains work is covered in part 23 of the Networking for beginners series). A registry without TLS is possible, but then passwords and images travel over the network in the clear, and every client has to be explicitly switched to "insecure registry" mode.
• Passwords. An htpasswd file with bcrypt hashes (htpasswd -B from the apache2-utils package); Distribution ignores entries with other hashes. This is authentication only, without authorization: anyone who logs in can pull, push and delete images in any repository. Permissions such as "this one only reads" require a separate token server — or Harbor.
• Garbage collection. Deleting a tag through the API deletes only the reference; the blobs stay on disk until registry garbage-collect is run, and for the duration of garbage collection the registry is switched to read-only mode.

Harbor is an organization-grade registry, also a CNCF project. Inside it has the same Distribution, and around it: projects with user roles, robot accounts with pull-only rights for clusters, login through LDAP or OIDC (a single sign-on protocol on top of OAuth 2.0), a web interface, vulnerability scanning of images with the open source scanner Trivy (it matches the packages and libraries in the layers against databases of known vulnerabilities), immutable tags, retention policies, replication between registries and a caching proxy for Docker Hub. It is installed with Docker Compose or into the cluster with a Helm chart (Helm is a package manager for Kubernetes, covered in part 27).

The certificate of your own registry is issued by an internal CA that is not in the nodes' system store, and without configuration a pull fails with x509: certificate signed by unknown authority. The right way is to tell containerd which CA to trust for this registry. In containerd 2.x registry settings live in the directory set by the config_path parameter — a subdirectory per host with a hosts.toml file. In the default configuration of containerd 2.2 this parameter is empty, so it is set explicitly in /etc/containerd/config.toml from part 4:

[plugins.'io.containerd.cri.v1.images'.registry]
  config_path = '/etc/containerd/certs.d'

and on every node the CA and hosts.toml are placed in the directory /etc/containerd/certs.d/<registry name>/:

server = "https://registry.example.com"

[host."https://registry.example.com"]
  capabilities = ["pull", "resolve"]
  ca = "/etc/containerd/certs.d/registry.example.com/ca.crt"

capabilities is what the node is allowed to do with this host: pull means downloading blobs, resolve means turning a tag into a digest; nodes don't need push. The same mechanism is used to configure a Docker Hub mirror: a docker.io/ directory with a host section for your caching proxy. Two bad options that are often recommended: skip_verify = true disables certificate verification entirely, and the node will hand the password to, and take images from, anyone who answers to that name; adding the CA to the system store works, but makes every program on the node trust the internal CA, and for all names.

---

imagePullSecrets

The nodes trust the ghcr.io certificate, but ghcr.io will not hand over a private image without credentials. The first attempt to run a pod with ghcr.io/example/hello:1.0:

$ kubectl get pod nosecret
NAME       READY   STATUS         RESTARTS   AGE
nosecret   0/1     ErrImagePull   0          13s
$ kubectl get events --field-selector involvedObject.name=nosecret -o custom-columns=REASON:.reason,MSG:.message
REASON    MSG
Pulling   Pulling image "ghcr.io/example/hello:1.0"
Failed    Failed to pull image "ghcr.io/example/hello:1.0": failed to pull and unpack image "ghcr.io/example/hello:1.0": failed to resolve image: failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://ghcr.io/token?scope=repository%3Aexample%2Fhello%3Apull&service=ghcr.io: 401 Unauthorized
Failed    Error: ErrImagePull
BackOff   Back-off pulling image "ghcr.io/example/hello:1.0"
Failed    Error: ImagePullBackOff

containerd asked ghcr.io for an anonymous token to read example/hello and was refused: the image is private.

Kubernetes keeps registry credentials in a Secret of type kubernetes.io/dockerconfigjson, which holds the contents of a ~/.docker/config.json file under the .dockerconfigjson key. We create it with a read:packages token, without leaving the token in the shell history:

$ read -rsp 'ghcr read token: ' T; echo
$ kubectl create secret docker-registry ghcr-pull \
    --docker-server=ghcr.io \
    --docker-username=example \
    --docker-password="$T"
secret/ghcr-pull created
$ unset T

The same as a manifest:

apiVersion: v1
kind: Secret
metadata:
  name: ghcr-pull
  namespace: default
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: <base64 of the JSON below>

Inside, after base64 decoding, is JSON of the form {"auths":{"ghcr.io":{"username":"example","password":"…","auth":"…"}}}. Base64 is not encryption: anyone who can read Secrets in this namespace knows the token. How to restrict reading and enable encryption of secrets in etcd is covered in parts 12 and 29. Such a manifest is not put into Git in the clear.

The Secret is attached to a pod with the imagePullSecrets field:

apiVersion: v1
kind: Pod
metadata:
  name: hello
spec:
  imagePullSecrets:
  - name: ghcr-pull
  containers:
  - name: hello
    image: ghcr.io/example/hello:1.0
    ports:
    - containerPort: 8080

The secret must be in the same namespace as the pod. When pulling the image, the kubelet goes through the secrets from imagePullSecrets, finds the entry for the host ghcr.io and passes the credentials to containerd in the CRI PullImage request. The token is not written to disk on the node.

Writing imagePullSecrets into every pod is inconvenient and easy to forget. It is more convenient to attach the secret to a ServiceAccount, the account under which the pods of a namespace run (details in part 29). Every namespace has a default ServiceAccount, which a pod gets if no other one is specified:

$ kubectl patch serviceaccount default \
    -p '{"imagePullSecrets":[{"name":"ghcr-pull"}]}'
serviceaccount/default patched

When a pod is created, the ServiceAccount admission plugin — a part of the API server that supplements objects before they are saved — copies the service account's imagePullSecrets into the pod spec. Only at creation: pods that are already running do not change. A pod from a manifest without the imagePullSecrets field gets it after creation:

$ kubectl get pod hello -o jsonpath='{.spec.imagePullSecrets}{"\n"}'
[{"name":"ghcr-pull"}]

From part 11 on, applications will live in the demo namespace, and this binding will have to be done there again: both the Secret and the ServiceAccount belong to a namespace.

One more important detail is what happens to an image that is already on the node. It used to work like this: pod A with the right secret pulled a private image onto k8s-node-1, and after that pod B in another namespace, without any secrets, could run the same image with imagePullPolicy: IfNotPresent — the kubelet did not go to the registry and did not check permissions. Since Kubernetes 1.35 the KubeletEnsureSecretPulledImages check (beta) is enabled by default: the kubelet remembers which credentials an image was pulled with and does not let a pod without suitable credentials use it until it has confirmed with the registry that access is allowed. Let's check: the image is already on k8s-node-1, and a pod in the other namespace, which has neither a Secret nor a binding, is started on the same node:

$ kubectl -n other get pod hello
NAME    READY   STATUS         RESTARTS   AGE
hello   0/1     ErrImagePull   0          12s
$ kubectl -n other get events -o custom-columns=REASON:.reason,MSG:.message
REASON    MSG
Pulling   Pulling image "ghcr.io/example/hello:1.0"
Failed    Failed to pull image "ghcr.io/example/hello:1.0": failed to pull and unpack image "ghcr.io/example/hello:1.0": failed to resolve image: failed to authorize: failed to fetch anonymous token: ...401 Unauthorized
Failed    Error: ErrImagePull
BackOff   Back-off pulling image "ghcr.io/example/hello:1.0"
Failed    Error: ImagePullBackOff

The kubelet did not let the pod use someone else's pull and went to the registry, where the pod has no permissions. The behavior is set by the imagePullCredentialsVerificationPolicy field in the kubelet configuration; the default is NeverVerifyPreloadedImages: all images are verified except those loaded onto the node bypassing the kubelet.

For cloud registries, instead of static tokens, kubelet credential providers are used: executable plugins on the node that the kubelet calls before a pull and that issue a short-lived token (this is how the nodes of cloud Kubernetes services access their cloud's registries). For ghcr.io from your own cluster this is not needed.

---

imagePullPolicy

When the kubelet goes to the registry is decided by the container's imagePullPolicy field:

Value	Behavior
IfNotPresent	pull only if there is no image with this name on the node
Always	on every container start, ask the registry which digest the tag points to and pull the missing layers
Never	never go to the registry; if the image is not on the node, the container does not start

The default value depends on the image reference. You can check what the API server will fill in with a server-side dry run:

$ for i in ghcr.io/example/hello ghcr.io/example/hello:latest ghcr.io/example/hello:1.0 \
    ghcr.io/example/hello@sha256:a5066cd40ef37b9b889ddfbe9c1860b3afe231b96b975000e0edca3fdd6f4a4a; do
    echo -n "$i -> "; kubectl run x --image=$i --dry-run=server -o jsonpath='{.spec.containers[0].imagePullPolicy}{"\n"}'
  done
ghcr.io/example/hello -> Always
ghcr.io/example/hello:latest -> Always
ghcr.io/example/hello:1.0 -> IfNotPresent
ghcr.io/example/hello@sha256:a5066cd40ef37b9b889ddfbe9c1860b3afe231b96b975000e0edca3fdd6f4a4a -> IfNotPresent

Image	Default
hello (no tag)	Always
hello:latest	Always
hello:1.0	IfNotPresent
hello@sha256:… or hello:1.0@sha256:…	IfNotPresent

The default is filled in once — when the object is created. If you later change the tag in a Deployment from latest to 1.0, the template keeps the Always that was set at creation.

Always is cheaper than it sounds: if the digest matches, only the manifest is downloaded and the layers come from the cache. But it makes starting a container dependent on the registry being available: if the registry is down, even crashed containers whose image is already on the node are not restarted. For a cloud registry it is also a dependency on the internet. IfNotPresent with immutable tags or digests is more reliable.

If pulling the image fails, the pod shows ErrImagePull and then ImagePullBackOff: the kubelet retries with a growing delay up to a ceiling of 300 seconds. By default the kubelet pulls images one at a time (serializeImagePulls: true in its configuration); on nodes where many pods start at once, parallel pulls can be allowed (serializeImagePulls: false and maxParallelImagePulls).

---

Tags versus digests

A tag is a mutable pointer. hello:1.0 today and hello:1.0 after someone has rebuilt and re-pushed it are different images, and the cluster will not notice: k8s-node-1 already has the old one, and with IfNotPresent that is the one it will start, while k8s-node-2 will pull the new one. Two replicas of the same Deployment will run different code.

latest is doubly bad. It is just the default tag and guarantees no "latest version": it points to whatever was last pushed without a tag. From image: hello:latest in a manifest it is impossible to tell which version is running; rolling back "to the previous latest" is impossible, since it no longer exists; and the default imagePullPolicy: Always adds a dependency on the registry. latest is not used in the cluster's manifests at all.

The right way:

• Immutable version tags. Every build gets a new tag: a semantic version 1.0.3, a commit hash, a build number. A tag that has been pushed once is never overwritten — ghcr.io and Distribution do not prevent it, so the discipline is on the CI side; Harbor can enforce it.
• Pinning by digest — for a full guarantee:

image: ghcr.io/example/hello:1.0@sha256:a5066cd40ef37b9b889ddfbe9c1860b3afe231b96b975000e0edca3fdd6f4a4a

If both a tag and a digest are specified, the image is pulled by digest only, and the tag remains for humans. The index digest works for nodes of any architecture. You can find it like this:

$ docker buildx imagetools inspect ghcr.io/example/hello:1.0 | head -3
Name:      ghcr.io/example/hello:1.0
MediaType: application/vnd.oci.image.index.v1+json
Digest:    sha256:a5066cd40ef37b9b889ddfbe9c1860b3afe231b96b975000e0edca3fdd6f4a4a

Nobody is going to write digests in by hand: this is done by the build pipeline or a GitOps tool (part 41), which updates the manifest after the image is pushed.

Whatever the manifest says, the pod reports what is actually running: status.containerStatuses[].imageID holds the digest by which containerd found the image.

---

Image garbage collection on the node

Images accumulate on a node: every new version of an application leaves the previous one behind. They are cleaned up by the kubelet, not by containerd on its own: every five minutes it checks how full the image file system is and, if usage is above the threshold, removes unused images, starting with those that have gone unused the longest. The parameters are in KubeletConfiguration, and the kubelet exposes the effective values through the API server:

$ kubectl get --raw /api/v1/nodes/k8s-node-2/proxy/configz | jq '.kubeletconfig | {imageGCHighThresholdPercent, imageGCLowThresholdPercent, imageMinimumGCAge, imageMaximumGCAge, serializeImagePulls, imagePullCredentialsVerificationPolicy}'
{
  "imageGCHighThresholdPercent": 85,
  "imageGCLowThresholdPercent": 80,
  "imageMinimumGCAge": "2m0s",
  "imageMaximumGCAge": "0s",
  "serializeImagePulls": true,
  "imagePullCredentialsVerificationPolicy": "NeverVerifyPreloadedImages"
}

Field	Default	Meaning
imageGCHighThresholdPercent	85	disk usage at which cleanup starts
imageGCLowThresholdPercent	80	disk usage to clean down to
imageMinimumGCAge	2m	an image younger than this is not removed, even if unused
imageMaximumGCAge	0s (disabled)	remove an unused image older than this without waiting for the disk to fill up

imageMaximumGCAge has been stable since 1.35; on nodes where application versions change often, it makes sense to set it, for example, to 168h (a week), so that old images do not sit around until the disk fills up. Images of running containers are never removed.

The Kubernetes documentation advises against third-party "cleaners" such as crictl rmi --prune run from cron: the kubelet keeps its own accounting and may be surprised when an image disappears. Once, by hand, when space is short, it is acceptable. If the disk fills up faster than the kubelet can clean it, pod eviction on disk pressure (DiskPressure) kicks in; it is covered in part 10.

---

Hands-on

The registry over the HTTP API. Without credentials ghcr.io responds with 401 and tells you where to get a token:

$ curl -si https://ghcr.io/v2/ | head -5
HTTP/1.1 401 Unauthorized
Content-Type: application/json
docker-distribution-api-version: registry/2.0
Strict-Transport-Security: max-age=63072000; includeSubDomains; preload
www-authenticate: Bearer realm="https://ghcr.io/token",service="ghcr.io",scope="repository:user/image:pull"

This is the token authentication of the Distribution protocol: the registry itself does not check passwords but sends the client to a token server (realm). The client comes there with its credentials and an access scope (scope: which repository and what to do with it), receives a short-lived token, and then accesses the registry with an Authorization: Bearer header. containerd does exactly this on every pull, and it was at this step that the pod without a secret got 401. Let's repeat it by hand on the master with the same read:packages token that is stored in the Secret: curl will ask for it as a password, so it will not stay in the shell history.

user@k8s-master-1:~$ TOKEN=$(curl -s -u example \
    'https://ghcr.io/token?scope=repository:example/hello:pull&service=ghcr.io' | jq -r .token)
Enter host password for user 'example':
user@k8s-master-1:~$ curl -s -H "Authorization: Bearer $TOKEN" https://ghcr.io/v2/example/hello/tags/list
{"name":"example/hello","tags":["1.0"]}

The index by tag. The response type is selected with the Accept header, and the digest comes in the Docker-Content-Digest header:

user@k8s-master-1:~$ curl -s -D - -H "Authorization: Bearer $TOKEN" \
    -H 'Accept: application/vnd.oci.image.index.v1+json' \
    https://ghcr.io/v2/example/hello/manifests/1.0
HTTP/1.1 200 OK
Content-Length: 1609
Content-Type: application/vnd.oci.image.index.v1+json
docker-content-digest: sha256:a5066cd40ef37b9b889ddfbe9c1860b3afe231b96b975000e0edca3fdd6f4a4a
docker-distribution-api-version: registry/2.0
...

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:bc289980e0a6cbd930a319094f25249a36634818823ab3bfc2d6a9382cd0ec6d",
      "size": 2939,
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:e909d5afc7fecc910dce9195b1b3784252bfedc47ba3e1154266c7600866adc7",
      "size": 2939,
      "platform": {
        "architecture": "arm64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:1a82f2e36a46e5d60c34b29cba26ae51fc181cf92b91cd83fa8ed31e52ae81c0",
      "size": 837,
      "annotations": {
        "vnd.docker.reference.digest": "sha256:bc289980e0a6cbd930a319094f25249a36634818823ab3bfc2d6a9382cd0ec6d",
        "vnd.docker.reference.type": "attestation-manifest"
      },
      "platform": {
        "architecture": "unknown",
        "os": "unknown"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:e9fc21e46675ff5ac5907435fc376e217b31c3ca94a3c11e98b0c6a92e4bf71a",
      "size": 837,
      "annotations": {
        "vnd.docker.reference.digest": "sha256:e909d5afc7fecc910dce9195b1b3784252bfedc47ba3e1154266c7600866adc7",
        "vnd.docker.reference.type": "attestation-manifest"
      },
      "platform": {
        "architecture": "unknown",
        "os": "unknown"
      }
    }
  ]
}

The amd64 manifest, this time by digest, and its last layer is our binary:

user@k8s-master-1:~$ curl -s -H "Authorization: Bearer $TOKEN" \
    -H 'Accept: application/vnd.oci.image.manifest.v1+json' \
    https://ghcr.io/v2/example/hello/manifests/sha256:bc289980e0a6cbd930a319094f25249a36634818823ab3bfc2d6a9382cd0ec6d \
    | jq '{config: .config.digest, layers: (.layers | length), last: .layers[-1]}'
{
  "config": "sha256:f768da2949737845e4a79d5141c37186a611d44d79f0ecef3ddce921d3b5d00e",
  "layers": 14,
  "last": {
    "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
    "digest": "sha256:28a9c83dfc2d1139a8a6d4610867e774208003f2e85e9ccfc8207ec6cd35f14a",
    "size": 2529007
  }
}

First come the thirteen layers of the distroless base image — certificates, /etc/passwd, time zones and so on — and ours is last: 2.5 MB of compressed binary. The config is an ordinary blob downloaded via /v2/<name>/blobs/<digest>; ghcr.io answers this request with a 307 redirect to its blob storage, hence the -L flag for curl:

user@k8s-master-1:~$ curl -sL -H "Authorization: Bearer $TOKEN" \
    https://ghcr.io/v2/example/hello/blobs/sha256:f768da2949737845e4a79d5141c37186a611d44d79f0ecef3ddce921d3b5d00e \
    | jq '{architecture, os, config: .config | {User, Entrypoint, Env, ExposedPorts, Labels}}'
{
  "architecture": "amd64",
  "os": "linux",
  "config": {
    "User": "65532:65532",
    "Entrypoint": [
      "/hello"
    ],
    "Env": [
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
      "SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt"
    ],
    "ExposedPorts": {
      "8080/tcp": {}
    },
    "Labels": {
      "org.opencontainers.image.source": "https://git.example.com/demo/hello",
      "org.opencontainers.image.title": "hello",
      "org.opencontainers.image.version": "1.0"
    }
  }
}
user@k8s-master-1:~$ unset TOKEN

Pulling on the node. Let's run the pod from the manifest above and look at the events:

$ kubectl apply -f hello-pod.yaml
pod/hello created
$ kubectl get events --field-selector involvedObject.name=hello
LAST SEEN   TYPE     REASON      OBJECT      MESSAGE
22s         Normal   Scheduled   pod/hello   Successfully assigned default/hello to k8s-node-2
19s         Normal   Pulling     pod/hello   Pulling image "ghcr.io/example/hello:1.0"
1s          Normal   Pulled      pod/hello   Successfully pulled image "ghcr.io/example/hello:1.0" in 18.723s (18.723s including waiting). Image size: 3395899 bytes.
0s          Normal   Created     pod/hello   Container created
0s          Normal   Started     pod/hello   Container started

$ kubectl get pod hello -o jsonpath='{.status.containerStatuses[0].imageID}{"\n"}'
ghcr.io/example/hello@sha256:a5066cd40ef37b9b889ddfbe9c1860b3afe231b96b975000e0edca3fdd6f4a4a

Three and a half megabytes for the whole image, yet the pull took almost twenty seconds: the registry is far away, and besides the bytes themselves, the token, the index, the manifest, the config and fourteen layers have to be fetched in separate requests, and for every blob ghcr.io redirects the client to its storage (this can be seen in the walk-through of the HTTP API above). Your own registry on the same network serves such an image noticeably faster; for a cloud one it is worth keeping IfNotPresent so that a container restart does not wait for the registry.

What is left on the node. crictl sees images through the eyes of CRI, ctr through the eyes of containerd itself:

user@k8s-node-2:~$ sudo crictl images --digests | grep -E 'IMAGE|hello'
IMAGE                                     TAG                 DIGEST              IMAGE ID            SIZE
ghcr.io/example/hello                     1.0                 a5066cd40ef37       f768da2949737       3.4MB

DIGEST is the digest of the index by which the image was obtained, IMAGE ID is the digest of the config for the node's platform. It is the config that identifies an image on the node: two references to the same image (a tag and a digest) give a single row.

user@k8s-node-2:~$ sudo ctr images ls
REF TYPE DIGEST SIZE PLATFORMS LABELS
user@k8s-node-2:~$ sudo ctr -n k8s.io images ls | grep hello
ghcr.io/example/hello:1.0                                                              application/vnd.oci.image.index.v1+json sha256:a5066cd40ef37b9b889ddfbe9c1860b3afe231b96b975000e0edca3fdd6f4a4a 3.2 MiB linux/amd64,linux/arm64 io.cri-containerd.image=managed
ghcr.io/example/hello@sha256:a5066cd40ef37b9b889ddfbe9c1860b3afe231b96b975000e0edca3fdd6f4a4a application/vnd.oci.image.index.v1+json sha256:a5066cd40ef37b9b889ddfbe9c1860b3afe231b96b975000e0edca3fdd6f4a4a 3.2 MiB linux/amd64,linux/arm64 io.cri-containerd.image=managed

user@k8s-node-2:~$ sudo ls /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/ | grep -E '^(a5066c|bc2899|e909d5|f768da|28a9c8)'
28a9c83dfc2d1139a8a6d4610867e774208003f2e85e9ccfc8207ec6cd35f14a
a5066cd40ef37b9b889ddfbe9c1860b3afe231b96b975000e0edca3fdd6f4a4a
bc289980e0a6cbd930a319094f25249a36634818823ab3bfc2d6a9382cd0ec6d
f768da2949737845e4a79d5141c37186a611d44d79f0ecef3ddce921d3b5d00e

The default namespace is empty; the kubelet's images are in k8s.io. The content store holds the index, the amd64 manifest, the config and the layers; the arm64 manifest (e909d5…) and the attestations are not there: the node does not need them. The files are named by their digests, and the sha256sum of any of them matches its name.

The snapshots and the overlay mount of the running container:

user@k8s-node-2:~$ CID=$(sudo crictl ps --name hello -q)
user@k8s-node-2:~$ sudo ctr -n k8s.io snapshots ls | grep -E "^KEY|^$CID|^sha256:b3b8d3"
KEY                                                                     PARENT                                                                  KIND
cde4a6d2b3914a3dfe6311031441443677a65a768ae3767529e06060b1c3de87        sha256:b3b8d3ba9a241bc70d3c5483b50354bf657b524012762da671f0c7602b72b051 Active
sha256:b3b8d3ba9a241bc70d3c5483b50354bf657b524012762da671f0c7602b72b051 sha256:d90b4cf528e38e9d1df9b742b6d6aef28307c8c7cd897b08959860b2556bb106 Committed

user@k8s-node-2:~$ mount | grep "$CID"
overlay on /run/containerd/io.containerd.runtime.v2.task/k8s.io/cde4a6d2b391…/rootfs type overlay (rw,relatime,
lowerdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/108/fs:…/snapshots/107/fs:…:…/snapshots/95/fs,
upperdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/109/fs,
workdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/109/work,uuid=on,nouserxattr)

user@k8s-node-2:~$ sudo ls -l /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/108/fs
total 5732
-rwxr-xr-x 1 root root 5865632 Oct 24 14:40 hello
user@k8s-node-2:~$ sudo find /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/109/fs
/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/109/fs
/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/109/fs/var
/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/109/fs/var/run
/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/109/fs/var/run/secrets
/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/109/fs/var/run/secrets/kubernetes.io
/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/109/fs/var/run/secrets/kubernetes.io/serviceaccount
/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/109/fs/etc
/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/109/fs/etc/hostname
/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/109/fs/etc/resolv.conf
/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/109/fs/etc/hosts

The mount line is wrapped for readability and the paths are shortened. The key of a committed snapshot is the layer's chainID, the key of the active one is the container ID. lowerdir has fourteen layer directories, with our binary on top. The container has not written anything into upperdir itself yet, but it already contains empty mount points: the files prepared by the kubelet are mounted on them, on top of the image — /etc/hosts, /etc/hostname, /etc/resolv.conf and the directory with the ServiceAccount token.

Finally, how the kubelet sees the image disk for garbage collection:

user@k8s-node-2:~$ sudo crictl imagefsinfo | jq '.status.imageFilesystems[0] | {mountpoint: .fsId.mountpoint, usedBytes: .usedBytes.value}'
{
  "mountpoint": "/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs",
  "usedBytes": "1251205120"
}

This is the file system whose usage the kubelet compares with the 85 and 80 percent thresholds.

---

Our own image is in a private registry, and the nodes know the read token. The hello pod we started for the check is so far just a way to pull the image; we have not yet looked at what a pod really is.

Sources

• OCI Image Specification and OCI Distribution Specification — the image format and the registry protocol.
• containerd: the path of content from the registry to a snapshot, snapshotters, configuring registries with hosts.toml.
• Docker: Dockerfile reference, multi-stage, multi-platform builds, cache mounts, docker login; BuildKit, buildah, ko.
• distroless — base images without a shell or a package manager.
• GitHub: working with the Container Registry and personal access tokens.
• Kubernetes: images (imagePullPolicy, credential verification for already pulled images), pulling an image from a private registry, kubelet credential providers, image garbage collection, kubelet configuration.
• Distribution and Harbor — your own registry.

---

Next — Pod: why Kubernetes runs pods rather than containers, what the pause container does, init and sidecar containers, liveness, readiness and startup probes, restarts and CrashLoopBackOff, graceful shutdown on SIGTERM.