Kubernetes for beginners. Part 42: How to run a cluster on a laptop: kind, k3s and minikube
All parts of the series “Kubernetes for beginners”We built the series' training cluster in part 4: three virtual machines, containerd, kubeadm init and join, Calico, a kubeconfig on the laptop. That is what a real cluster looks like, and we went through the whole series on it — up to GitOps with Argo CD in the previous part. But for many tasks three machines are too much: checking a manifest before rolling it out, running tests in CI, showing a colleague how a Deployment works, playing with a new version of Kubernetes without touching the lab. For that there are clusters that come up on a laptop in a minute and are deleted with a single command: kind, k3s and minikube. This is a bonus part at the end of the series and the only one about them: the main examples in the series used kubeadm clusters. So here we will look at each of them seriously — how it is built inside, how its cluster differs from the one we assembled by hand, and where it will let you down.
All three run Kubernetes: their API is the same one described in part 3, and ordinary Deployment and Service manifests work without changes. Examples with Windows pods, CRDs or external systems need the corresponding nodes and components. What differs is what surrounds the API: where the nodes live, what stores the state, which CNI is used, where LoadBalancers and volumes come from.
kind: the nodes are containers
kind (Kubernetes IN Docker) appeared as a tool for testing Kubernetes itself: the project needed to bring up a cluster from freshly built code in CI, quickly and many times a day. The idea is to make each "node" of the cluster a container in Docker or Podman. Docker is a container engine with the dockerd daemon; Podman is an engine compatible with it at the command level but without a daemon: containers run as child processes of the user. kind works with both.
Inside a node container there is almost a complete machine: systemd as the first process, with containerd and kubelet under it — the same ones we installed as packages in part 4. The kindest/node node image is built for a specific version of Kubernetes and already contains kubeadm, kubelet, kubectl and the images of the control plane components, so almost nothing is downloaded from the internet when a cluster is created. kind starts the node containers and then runs kubeadm init inside the first one and kubeadm join inside the others. In other words, kind is kubeadm with containers instead of virtual machines.
Laptop (Linux, or the Docker Desktop VM on macOS/Windows)
┌────────────────────────────────────────────────────────────────────┐
│ dockerd / podman │
│ │
│ docker network "kind" 172.18.0.0/16 │
│ ┌──────────────────────┐ ┌──────────────────────┐ ┌─────────────┐ │
│ │ lab-control-plane │ │ lab-worker │ │ lab-worker2 │ │
│ │ 172.18.0.2 │ │ 172.18.0.3 │ │ 172.18.0.4 │ │
│ │ systemd │ │ systemd │ │ systemd │ │
│ │ ├ containerd │ │ ├ containerd │ │ ├ ... │ │
│ │ │ ├ kube-apiserver │ │ │ ├ kindnet │ │ │ │
│ │ │ ├ etcd │ │ │ ├ kube-proxy │ │ │ │
│ │ │ └ ... │ │ │ └ pods │ │ │ │
│ │ └ kubelet │ │ └ kubelet │ │ │ │
│ └──────────▲───────────┘ └──────────────────────┘ └─────────────┘ │
│ │ port 6443 published on 127.0.0.1:<random> │
└─────────────┼──────────────────────────────────────────────────────┘
│ kubectl on the laptop
The node container runs privileged: kubelet inside it has to create cgroups and network namespaces and mount volumes, and containerd has to run nested containers. The pods of a kind cluster are containers inside a container, sharing the laptop's kernel.
Around kubeadm, kind installs its own minimal set:
- kindnet — kind's own CNI. Each node gets a subnet from the pod network (
10.244.0.0/16by default), and thekindnetddaemon on each node adds routes to the neighbours' subnets via their addresses in the Docker network — without encapsulation, because all the nodes are in one L2 segment. Since kind 0.24 the bundle also includes a minimal NetworkPolicy implementation — the kube-network-policies controller from the authors of kind; it runs inside the samekindnetdand writes nftables rules (in the log of thekindnetpod these are the linesStarting controller name="kube-network-policies"andSyncing nftables rules). - local-path-provisioner — a volume provisioner from Rancher: on a volume request it creates a directory on the node's disk and gives it to the pod. The cluster has a StorageClass
standardmarked as the default class. - kube-proxy in iptables mode (the default) or nftables mode — it is switched on in the cluster configuration.
- CoreDNS — the cluster's DNS server, the same one kubeadm installs.
Installation and the first cluster
kind is a single static Go binary. On Linux:
$ curl -Lo kind https://kind.sigs.k8s.io/dl/v0.33.0/kind-linux-amd64
$ curl -Lo kind.sha256 https://github.com/kubernetes-sigs/kind/releases/download/v0.33.0/kind-linux-amd64.sha256sum
$ sed 's/kind-linux-amd64/kind/' kind.sha256 | sha256sum -c
kind: OK
$ chmod +x kind && sudo mv kind /usr/local/bin/
$ kind version
kind v0.33.0 go1.26.7 linux/amd64
On macOS it is brew install kind, on Windows winget install Kubernetes.kind or the binary from the same page. Docker or Podman must already be installed.
Version 0.33 brings up Kubernetes 1.37.0 by default — the same minor version as the lab after part 35 (1.37.2; the kind 0.33.0 release ships a 1.37.0 image for the 1.37 branch). For another version the node image is specified explicitly. The list of ready-made images with digests is in the description of each kind release; the authors recommend taking the image with its @sha256:… digest, because an image under the same tag is rebuilt with new kind releases and may not work with an older version of kind. Let's pin the default version the same way — then a kind upgrade will not change the cluster unnoticed:
$ kind create cluster --name lab \
--image kindest/node:v1.37.0@sha256:a1ed56cfb0e7b93589bdf97c8cd566405a265939e3620fc4f5de89adff580ae5
Creating cluster "lab" ...
✓ Ensuring node image (kindest/node:v1.37.0) 🖼
✓ Preparing nodes 📦
✓ Writing configuration 📜
✓ Starting control-plane 🕹️
✓ Installing CNI 🔌
✓ Installing StorageClass 💾
Set kubectl context to "kind-lab"
You can now use your cluster with:
kubectl cluster-info --context kind-lab
Have a nice day! 👋
In the lab the command ran for 33 seconds, most of which was downloading the node image.
kind appended a cluster, a user and the context kind-lab to ~/.kube/config and made it current. A context is the "cluster, user, namespace" triple from the kubeconfig that we looked at in part 4; switching between the lab and kind is kubectl config use-context. The API server address in the kubeconfig is https://127.0.0.1:<port>: port 6443 of the container is published only on the laptop's loopback, the cluster is not reachable from outside, and that is how it should stay.
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
lab-control-plane NotReady control-plane 2s v1.37.0
NotReady right after creation is normal: kind returns control without waiting for kindnet to set up the node's network, and a few seconds later the node becomes Ready. The single node is both the control plane and a worker: in a cluster without separate worker nodes kind removes from it the node-role.kubernetes.io/control-plane:NoSchedule taint that kubeadm puts on the master, so that user pods can be scheduled there. In the multi-node cluster below the taint on the master stays. Taints are covered in detail in part 13.
A multi-node cluster
The set of nodes, the network and kubeadm patches are defined in a configuration file:
# kind-lab.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: lab
networking:
podSubnet: 172.20.0.0/16 # same as in the lab, to avoid confusion
serviceSubnet: 10.96.0.0/12
kubeProxyMode: nftables # same as in the lab
nodes:
- role: control-plane
image: kindest/node:v1.37.0@sha256:a1ed56cfb0e7b93589bdf97c8cd566405a265939e3620fc4f5de89adff580ae5
extraPortMappings: # node port 30080 → laptop 127.0.0.1:8080
- containerPort: 30080
hostPort: 8080
listenAddress: 127.0.0.1
- role: worker
image: kindest/node:v1.37.0@sha256:a1ed56cfb0e7b93589bdf97c8cd566405a265939e3620fc4f5de89adff580ae5
- role: worker
image: kindest/node:v1.37.0@sha256:a1ed56cfb0e7b93589bdf97c8cd566405a265939e3620fc4f5de89adff580ae5
labels:
disk: ssd
$ kind delete cluster --name lab
Deleting cluster "lab" ...
Deleted nodes: ["lab-control-plane"]
$ kind create cluster --config kind-lab.yaml
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
lab-control-plane Ready control-plane 29s v1.37.0
lab-worker Ready <none> 20s v1.37.0
lab-worker2 Ready <none> 20s v1.37.0
$ kubectl -n kube-system get cm kube-proxy -o jsonpath='{.data.config\.conf}' | grep '^mode'
mode: nftables
The image is already in the Docker cache, and the three nodes are created in 20 seconds. The pods got addresses from 172.20.0.0/16, and the nodes themselves from the Docker network 172.18.0.0/16.
If the configuration has more than one control-plane, kind adds one more container — a load balancer in front of the API servers (in recent versions this is Envoy, an L4 and L7 proxy server), and the kubeconfig will point at it. This way you can get a feel for a highly available control plane, but it adds no reliability: all the "masters" are containers on one laptop. How such a control plane is built on real machines is covered in the series about Kubernetes on BGP.
The extraPortMappings field is an ordinary container port publication (docker run -p). A NodePort service with port 30080 will be reachable at 127.0.0.1:8080 on the laptop. The publication is set only when the cluster is created: a port cannot be added to a Docker container on the fly.
The kubeadmConfigPatches field lets you pass to kubeadm pieces of ClusterConfiguration, KubeletConfiguration and other types that we wrote by hand in part 4 — for example, to enable API server auditing or change kubelet parameters.
A local image: kind load
The main joy of kind in development is that an image built on the laptop can be handed to the cluster without a registry. We built our own hello application in part 5, and here we will take a two-line Dockerfile: the nginx base image and our own page. How a Dockerfile works and what image layers are is also in part 5.
$ cat Dockerfile
FROM nginx:1.29
COPY index.html /usr/share/nginx/html/
$ docker build -t demo/web:dev .
$ kind load docker-image demo/web:dev --name lab
Image: "demo/web:dev" with ID "sha256:902178ca06f85d0e937b22447962845b19804f53e648bc93d6f24a355eb5c04b" not yet present on node "lab-worker", loading...
Image: "demo/web:dev" with ID "sha256:902178ca06f85d0e937b22447962845b19804f53e648bc93d6f24a355eb5c04b" not yet present on node "lab-worker2", loading...
Image: "demo/web:dev" with ID "sha256:902178ca06f85d0e937b22447962845b19804f53e648bc93d6f24a355eb5c04b" not yet present on node "lab-control-plane", loading...
kind load docker-image does a docker save — exports the image into a tar archive — and imports it into the containerd of each node (ctr images import in the k8s.io namespace, where containerd keeps kubelet's images). For Podman and for ready-made archives there is kind load image-archive file.tar.
A subtlety almost everyone trips over: a container in a pod has an imagePullPolicy field — when kubelet should pull the image from a registry. If the image tag is latest or not specified, the default is Always: kubelet will go to the registry, will not find demo/web:latest there, and the pod will get stuck in ErrImagePull, even though the image is on the node. With the dev tag the default is IfNotPresent — the image on the node will be used. That is why images for kind are given a meaningful tag rather than latest; the policy is covered in detail in part 5.
$ kubectl create deployment web --image=demo/web:dev --replicas=2
deployment.apps/web created
$ kubectl create service nodeport web --tcp=80:80 --node-port=30080
service/web created
$ kubectl get deployment web -o jsonpath='{.spec.template.spec.containers[0].imagePullPolicy}{"\n"}'
IfNotPresent
$ curl -s http://127.0.0.1:8080/
<h1>hello from kind</h1>
The request arrived at the published port of the lab-control-plane container, and kube-proxy inside it forwarded it to one of the pods — both pods being on the worker nodes. If curl right after the rollout returned an empty response, kube-proxy had not yet written the rules for the new service: a second or two later the request goes through. The Deployment and Service here are just scaffolding; they are covered in part 7 and part 15.
LoadBalancer in kind
By default a LoadBalancer service in kind stays with EXTERNAL-IP <pending> forever: in Kubernetes, load balancer addresses are assigned by the cloud controller (cloud-controller-manager), and kind has no cloud. The authors' solution is cloud-provider-kind: a separate program that runs on the laptop (it needs access to Docker and the right to open ports), watches LoadBalancer services in all kind clusters and creates a load balancer container in the kind network for each one. Its address from the Docker network becomes the EXTERNAL-IP.
$ curl -LO https://github.com/kubernetes-sigs/cloud-provider-kind/releases/download/v0.12.0/cloud-provider-kind_0.12.0_linux_amd64.tar.gz
$ curl -LO https://github.com/kubernetes-sigs/cloud-provider-kind/releases/download/v0.12.0/cloud-provider-kind_0.12.0_checksums.txt
$ grep linux_amd64 cloud-provider-kind_0.12.0_checksums.txt | sha256sum -c
cloud-provider-kind_0.12.0_linux_amd64.tar.gz: OK
$ tar xzf cloud-provider-kind_0.12.0_linux_amd64.tar.gz cloud-provider-kind
$ sudo install -m 0755 cloud-provider-kind /usr/local/bin/
$ sudo cloud-provider-kind # leave it running in a separate terminal
$ kubectl expose deployment web --name=web-lb --type=LoadBalancer --port=80
service/web-lb exposed
$ kubectl get svc web-lb
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
web-lb LoadBalancer 10.97.254.5 172.18.0.5 80:30933/TCP 20s
$ docker ps --format 'table {{.Names}}\t{{.Image}}\t{{.Ports}}' | grep kindccm
kindccm-e1b23e1f7834 envoyproxy/envoy:v1.33.2 0.0.0.0:32768->10000/tcp, [::]:32768->10000/tcp
$ curl -s http://172.18.0.5/
<h1>hello from kind</h1>
The load balancer is a container with the Envoy proxy server, and the address 172.18.0.5 was assigned to it in the kind network. Note the PORTS column: Docker publishes the Envoy port on all of the laptop's addresses, so the service becomes reachable from the local network on a random port — worth keeping in mind on a public network. In version 0.12, cloud-provider-kind also implements Ingress and Gateway API (the --enable-default-ingress and --gateway-channel flags at the start of its log): without a separate ingress controller, kind understands the objects from part 18.
On Linux the addresses of the kind network are reachable from the laptop directly. On macOS and Windows Docker lives inside a virtual machine, and the 172.18.0.x addresses cannot be reached from the host without additional forwarding — this is kind's main networking pitfall on these systems, and it also applies to NodePort without extraPortMappings. In the lab, LoadBalancer is provided by MetalLB from part 17.
k3s: one binary instead of a cluster of packages
k3s was made by Rancher (now SUSE) for nodes with few resources: edge devices, Raspberry Pi, CI, small servers. Today it is a project of the CNCF — the Cloud Native Computing Foundation, which is also home to Kubernetes itself. This is not "Kubernetes in containers" but a real distribution for Linux machines — physical or virtual — only assembled differently from kubeadm.
One binary. The API server, scheduler, controller-manager, kubelet, kube-proxy, containerd, runc, the CNI plugins, kubectl, crictl, ctr — all inside a single executable /usr/local/bin/k3s of about 80 MB. The control plane components run not as pods, as with kubeadm, but as goroutines of a single k3s server process; on worker nodes k3s agent runs. One systemd service per node: k3s.service on the server, k3s-agent.service on the agent.
State storage. By default k3s stores API objects not in etcd but in SQLite — an embedded database in a single file, /var/lib/rancher/k3s/server/db/state.db. Between the API server and SQLite sits the kine layer ("kine is not etcd"): it implements the part of the etcd protocol that kube-apiserver uses (get, list, watch, transactions) on top of an SQL database. Through the same kine, k3s can store its state in an external PostgreSQL or MySQL. The third option is embedded etcd: the --cluster-init flag on the first server, and k3s will start etcd inside its own process; more servers can then be joined to it to get a highly available control plane. The number of servers is then odd — three, five: etcd holds quorum by majority, and a fourth server adds no resilience. A cluster on SQLite can be migrated to embedded etcd by restarting the server with --cluster-init.
Built-in components. What we install ourselves in a kubeadm cluster, k3s brings along:
| Component | What it is | How to disable |
|---|---|---|
| Flannel | a CNI from CoreOS: a subnet per node, VXLAN between nodes | --flannel-backend=none (and install your own CNI) |
| CoreDNS | the cluster's DNS | --disable=coredns |
| Traefik 3 | an ingress controller: a reverse proxy that accepts HTTP(S) from outside and distributes it to services | --disable=traefik |
| ServiceLB (klipper-lb) | an implementation of LoadBalancer services without a cloud | --disable=servicelb |
| local-path-provisioner | volumes in a directory on the node's disk, StorageClass local-path |
--disable=local-storage |
| metrics-server | collects pod CPU and memory for kubectl top and autoscaling |
--disable=metrics-server |
| network policy controller | a piece of kube-router (a router for Kubernetes) that turns NetworkPolicy into iptables rules | --disable-network-policy |
| helm-controller | the controller of the HelmChart resource: installs Helm charts (application packages for Kubernetes, covered in part 27) from manifests |
--disable-helm-controller |
| Gateway API CRDs | the Gateway, HTTPRoute and other types from part 18, understood by the built-in Traefik |
--disable=gateway-api-crd |
Flannel is one of the first CNI plugins: it gives each node a subnet from 10.42.0.0/16 (the k3s default) and wraps packets between nodes in VXLAN — L2 frames inside UDP datagrams to port 8472. VXLAN itself is covered in detail in part 18 of the networking series. The k3s service network is 10.43.0.0/16, and the DNS address is 10.43.0.10.
ServiceLB (formerly klipper-lb) is simple. For each LoadBalancer service it creates a DaemonSet in kube-system — a pod on each eligible node with the port available. The pod declares a hostPort with the service port, that is, it takes this port on the node's address, and forwards incoming traffic to the service's ClusterIP with iptables rules. The node addresses end up in the service's EXTERNAL-IP. Hence the limitations: one node cannot have two LoadBalancers with the same port. The second Service stays Pending if the port is taken on every eligible node, and the "external address" is a node's address, which disappears together with the node.
Installation
The official way is the script from get.k3s.io. Before running a script from the internet under sudo, it is worth downloading and reading it; environment variables set the version and the role:
user@k3s-1:~$ curl -sfL https://get.k3s.io -o install-k3s.sh
user@k3s-1:~$ less install-k3s.sh
user@k3s-1:~$ sudo INSTALL_K3S_CHANNEL=v1.37 sh install-k3s.sh
[INFO] Finding release for channel v1.37
[INFO] Using v1.37.2+k3s1 as release
[INFO] Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.37.2%2Bk3s1/sha256sum-amd64.txt
[INFO] Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.37.2%2Bk3s1/k3s
[INFO] Verifying binary download
[INFO] Installing k3s to /usr/local/bin/k3s
[INFO] Skipping installation of SELinux RPM
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Creating /usr/local/bin/ctr symlink to k3s
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s.service
[INFO] systemd: Enabling k3s unit
Created symlink '/etc/systemd/system/multi-user.target.wants/k3s.service' → '/etc/systemd/system/k3s.service'.
[INFO] systemd: Starting k3s
Here k3s-1 (192.168.122.11) and k3s-2 (192.168.122.12) are two training virtual machines with Ubuntu Server 26.04 on the laptop, in the default libvirt network, separate from the lab. The v1.37 channel selects the latest release in the 1.37 branch. In the lab it selected v1.37.2+k3s1. To reproduce the example with the same release, set INSTALL_K3S_VERSION=v1.37.2+k3s1 on both nodes. The script creates the kubectl, crictl and ctr links only if such commands do not exist yet: on a machine with the kubectl package installed it will print Skipping /usr/local/bin/kubectl symlink to k3s, command exists in PATH, and kubectl will not point at k3s.
Half a minute later:
user@k3s-1:~$ sudo k3s kubectl get nodes
NAME STATUS ROLES AGE VERSION
k3s-1 Ready control-plane 29s v1.37.2+k3s1
user@k3s-1:~$ sudo k3s kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-577d995dff-wnd5r 1/1 Running 0 65s
kube-system helm-install-gateway-api-crd-brl9s 0/1 Completed 0 66s
kube-system helm-install-traefik-6pmh2 0/1 Completed 1 (47s ago) 66s
kube-system helm-install-traefik-crd-nfj49 0/1 Completed 0 66s
kube-system local-path-provisioner-56c6584955-g4qvf 1/1 Running 0 65s
kube-system metrics-server-778c4649b4-zj7fw 1/1 Running 0 65s
kube-system svclb-traefik-59103bc3-cp4bx 2/2 Running 0 45s
kube-system traefik-68df944ff7-jlbh2 1/1 Running 0 45s
Not a single control plane pod: the API server, scheduler and controller-manager live inside the k3s server process, so they are neither in kube-system nor in /etc/kubernetes/manifests — the directory from which kubelet in a kubeadm cluster runs the control plane's static pods (pods described by files on the node rather than by objects in the API). The helm-install-* pods are the helm-controller, which installed the Traefik and Gateway API CRD charts from the files in /var/lib/rancher/k3s/server/manifests/; the first attempt of helm-install-traefik fails and restarts until the Traefik CRDs are registered — this is normal. The svclb-traefik-… pod is ServiceLB, which took ports 80 and 443 of the node for the Traefik service. The server has a single role, control-plane: the old master label is no longer set in 1.37.
k3s writes the administrator kubeconfig to /etc/rancher/k3s/k3s.yaml, a file readable only by root. To work from the laptop, the file is copied and server: https://127.0.0.1:6443 in it is changed to the node's address:
$ ssh user@k3s-1 sudo cat /etc/rancher/k3s/k3s.yaml | sed 's/127.0.0.1/192.168.122.11/; s/: default/: k3s-lab/' > ~/.kube/k3s-lab.yaml
$ chmod 600 ~/.kube/k3s-lab.yaml
$ KUBECONFIG=~/.kube/k3s-lab.yaml kubectl get nodes
After sed the cluster, user and context in the file are named k3s-lab, and kubectl on the laptop sees the same nodes as k3s kubectl on the server.
This file contains a client certificate with cluster-admin rights, the same level as admin.conf with kubeadm; it should be handled like the root password.
The second node
The agent joins with a token that the server put into /var/lib/rancher/k3s/server/node-token:
user@k3s-1:~$ sudo cat /var/lib/rancher/k3s/server/node-token
K10276633…::server:63e0…
user@k3s-2:~$ curl -sfL https://get.k3s.io -o install-k3s.sh
user@k3s-2:~$ less install-k3s.sh
user@k3s-2:~$ sudo INSTALL_K3S_CHANNEL=v1.37 K3S_URL=https://192.168.122.11:6443 \
K3S_TOKEN='K10276633…::server:63e0…' sh install-k3s.sh
...
[INFO] Creating uninstall script /usr/local/bin/k3s-agent-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s-agent.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s-agent.service
[INFO] systemd: Enabling k3s-agent unit
Created symlink '/etc/systemd/system/multi-user.target.wants/k3s-agent.service' → '/etc/systemd/system/k3s-agent.service'.
[INFO] systemd: Starting k3s-agent
user@k3s-1:~$ sudo k3s kubectl get nodes
NAME STATUS ROLES AGE VERSION
k3s-1 Ready control-plane 71s v1.37.2+k3s1
k3s-2 Ready <none> 18s v1.37.2+k3s1
The presence of K3S_URL turns the installation into an agent. In a full token, the cluster CA hash follows the K10 prefix. On its first connection, the agent checks the CA against this hash and validates the server certificate before sending credentials, just as kubeadm join checks --discovery-token-ca-cert-hash. The agent brings up a local load balancer on 127.0.0.1:6444, through which kubelet and kube-proxy reach the servers: if there are several servers, the agent switches between them by itself.
user@k3s-2:~$ sudo ss -lntp | grep 6444
LISTEN 0 4096 127.0.0.1:6444 0.0.0.0:* users:(("k3s-agent",pid=2129,fd=4))
LISTEN 0 4096 [::1]:6444 [::]:* users:(("k3s-agent",pid=2129,fd=7))
user@k3s-2:~$ sudo grep server: /var/lib/rancher/k3s/agent/kubelet.kubeconfig
server: https://127.0.0.1:6444
How k3s differs from a kubeadm cluster
- The control plane components are not pods and are not visible through
kubectl; the logs are injournalctl -u k3s, the parameters arek3s serverflags or/etc/rancher/k3s/config.yaml. - containerd is built in, with its own socket
/run/k3s/containerd/containerd.sockand directory/var/lib/rancher/k3s/agent/containerd; registry settings are in/etc/rancher/k3s/registries.yaml, not in/etc/containerd/certs.d. Thecrictlandctrbundled with k3s already point at the right socket. - The state is in SQLite, not in etcd:
etcdctland the etcd snapshots from part 34 do not apply to such a cluster; for a backup, save the entire/var/lib/rancher/k3s/server/db/directory and the/var/lib/rancher/k3s/server/tokenfile. - The pod and service networks are
10.42.0.0/16and10.43.0.0/16, not our172.20.0.0/16and10.96.0.0/12; they are changed with the--cluster-cidrand--service-cidrflags. - Upgrading is done by replacing the binary (or with the same script and a new channel), not with
kubeadm upgrade.
k3s pitfalls
- Traefik and ServiceLB take 80 and 443. If the node already has a web server, or you want MetalLB and your own ingress controller, they have to be disabled at installation:
INSTALL_K3S_EXEC="--disable=traefik --disable=servicelb". On an existing cluster,--disable=traefikmakes k3s uninstall the packaged Traefik addon. To disable ServiceLB, set--disable=servicelbon every server in the cluster. - The built-in Traefik gets overwritten. Editing its manifest in
/var/lib/rancher/k3s/server/manifests/traefik.yamlis pointless: k3s will rewrite the file on startup. Settings go into a separateHelmChartConfigresource. - Ports between nodes. Flannel needs UDP 8472, kubelet TCP 10250, the API TCP 6443. A host firewall that blocks 8472 gives you a cluster where pods on different nodes cannot see each other, while pods on the same node can.
- local-path ties the data to a node. A volume is a
/var/lib/rancher/k3s/storage/…directory on a specific node: a pod with such a volume will not move to another node, and the data will be lost together with the disk. For tests this is fine, for data it is not; storage is covered in detail in part 21. - A single server on SQLite is a single point of failure. For a training cluster this is fine; for anything bigger — embedded etcd on three servers.
There is also k3d — a wrapper that runs k3s in Docker containers in the style of kind. If you need k3s specifically on a laptop without virtual machines, that is the one.
minikube: a cluster in a virtual machine or a container
minikube is the oldest of the three, a project of Kubernetes itself, conceived as "a cluster for a developer on a workstation". What sets it apart is drivers: minikube can create a node as a virtual machine in a hypervisor or as a container.
- Virtual machines: KVM (the Linux kernel hypervisor, the
kvm2driver), QEMU (an emulator and hypervisor that works on all OSes), Hyper-V (the Windows hypervisor), VFkit (a wrapper around the macOS Virtualization.framework), VirtualBox, Parallels, VMware. The VM boots minikube's own image, the minikube ISO — a minimal Linux with containerd and kubelet. - Containers:
dockerandpodman— the node, as with kind, is a container with systemd inside (thekicbaseimage). noneandssh— install the cluster directly on the current or a remote Linux machine; this is for special cases.
The preferred drivers: on Linux docker or kvm2, on macOS docker or vfkit, on Windows hyperv or docker. A VM gives a real kernel separate from the laptop — important when you need kernel modules, your own sysctl parameters or a check of how an application behaves on a "clean" machine. A container is faster and lighter.
Inside the minikube node kubeadm runs too. Since v1.39 the default runtime is containerd, as in our cluster (previously it was Docker Engine with the cri-dockerd adapter); it can be chosen with the --container-runtime flag.
Installation and the first cluster
$ curl -LO https://github.com/kubernetes/minikube/releases/download/v1.39.0/minikube-linux-amd64
$ curl -LO https://github.com/kubernetes/minikube/releases/download/v1.39.0/minikube-linux-amd64.sha256
$ echo "$(cat minikube-linux-amd64.sha256) minikube-linux-amd64" | sha256sum -c
minikube-linux-amd64: OK
$ sudo install minikube-linux-amd64 /usr/local/bin/minikube
$ minikube start --driver=docker --kubernetes-version=v1.37.2 \
--container-runtime=containerd --cpus=2 --memory=3g --nodes=2 --cni=calico
* minikube v1.39.0 on Ubuntu 26.04
! Specified Kubernetes version 1.37.2 is newer than the newest supported version: v1.37.0. Use `minikube config defaults kubernetes-version` for details.
! Specified Kubernetes version 1.37.2 not found in Kubernetes version list
* Searching the internet for Kubernetes version...
* Kubernetes version 1.37.2 found in GitHub version list
* Using the docker driver based on user configuration
* Using Docker driver with root privileges
* Starting "minikube" primary control-plane node in "minikube" cluster
* Pulling base image v0.0.51 ...
* Downloading Kubernetes v1.37.2 preload ...
> gcr.io/k8s-minikube/kicbase: 507.60 MiB / 507.61 MiB 100.00% 21.92 MiB
* Preparing Kubernetes v1.37.2 on containerd 2.3.4 ...
* Configuring Calico (Container Networking Interface) ...
* Verifying Kubernetes components...
- Using image gcr.io/k8s-minikube/storage-provisioner:v5
* Enabled addons: default-storageclass, storage-provisioner
* Starting "minikube-m02" worker node in "minikube" cluster
* Pulling base image v0.0.51 ...
* Found network options:
- NO_PROXY=192.168.49.2
* Preparing Kubernetes v1.37.2 on containerd 2.3.4 ...
- env NO_PROXY=192.168.49.2
* Verifying Kubernetes components...
* Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
The series' lab runs 1.37.2, while minikube 1.39 knows versions only up to 1.37.0. It warned about this, found 1.37.2 in the Kubernetes releases on GitHub and installed it — with a version newer than the minikube authors tested, this works, but without guarantees. Together with downloading the kicbase node image (half a gigabyte), the start took almost three minutes. The nodes are containers with Debian 12 inside, on the laptop's kernel:
$ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
minikube Ready control-plane 27s v1.37.2 192.168.49.2 <none> Debian GNU/Linux 12 (bookworm) 7.0.0-42-generic (amd64) containerd://2.3.4
minikube-m02 NotReady <none> 8s v1.37.2 192.168.49.3 <none> Debian GNU/Linux 12 (bookworm) 7.0.0-42-generic (amd64) containerd://2.3.4
--cni=calico installs the same CNI as in the lab (by default minikube picks the CNI by itself: bridge for a single node, kindnet for several). --cpus and --memory are set per node: on a machine with 4 cores and 8 GB, two nodes of 3 GB each are a reasonable limit. Profiles (-p name) let you keep several clusters at once; the context in the kubeconfig is named after the profile.
Addons
minikube's strong point is addons: ready-made components that are enabled with a single command.
$ minikube addons list | grep -E 'dashboard |ingress |metrics|registry |metallb'
│ dashboard │ minikube │ disabled │ Kubernetes │
│ ingress │ minikube │ disabled │ Kubernetes │
│ metallb │ minikube │ disabled │ 3rd party (MetalLB) │
│ metrics-server │ minikube │ disabled │ Kubernetes │
│ registry │ minikube │ disabled │ minikube │
$ minikube addons enable metrics-server
...
- Using image registry.k8s.io/metrics-server/metrics-server:v0.9.0
* The 'metrics-server' addon is enabled
$ minikube addons enable ingress
...
- Using image registry.k8s.io/ingress-nginx/controller:v1.15.1
- Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.6.9
* Verifying ingress addon...
* The 'ingress' addon is enabled
Minikube 1.39 also has a traefik addon. Enable it with minikube addons enable traefik.
ingress is the ingress-nginx controller based on the nginx web server (the ingress-nginx project itself has not been maintained since March 2026; for new setups Traefik or Gateway API are better, covered in part 18), metrics-server is for kubectl top, registry is an image registry inside the cluster, metallb is the same MetalLB as in the lab, dashboard is the Kubernetes Dashboard web interface (minikube dashboard opens it in a browser). storage-provisioner is minikube's provisioner that creates volumes as directories on the node (the standard class, an analogue of local-path).
The addons list table is not accurate in everything: after minikube start --nodes=2 the default-storageclass and storage-provisioner rows in it show disabled, even though minikube start printed Enabled addons: default-storageclass, storage-provisioner, and the StorageClass standard and the storage-provisioner pod exist in the cluster and work. It is more reliable to check what is actually enabled in the cluster itself.
Images and access to services
Your own image — without a registry, as with kind:
$ minikube image load demo/web:dev
$ minikube image ls | grep demo
docker.io/demo/web:dev
You can also build it right inside the node: minikube image build -t demo/web:dev ..
A NodePort service is opened with minikube service web --url: on Linux with the docker driver it simply prints the node's address, while on macOS and Windows, where the node's address is not reachable from the host, it brings up a tunnel and prints http://127.0.0.1:<port>, which works as long as the command is not stopped.
$ kubectl create deployment web --image=demo/web:dev --replicas=2
$ kubectl expose deployment web --type=NodePort --port=80
$ minikube service web --url
http://192.168.49.2:30180
For LoadBalancer there is minikube tunnel in a separate terminal. It adds a route on the laptop to the service network via the minikube node's address (which is why it needs sudo) and sets the EXTERNAL-IP of LoadBalancer services equal to their ClusterIP:
$ kubectl expose deployment web --name=web-lb --type=LoadBalancer --port=80
$ minikube tunnel
Status:
machine: minikube
pid: 21867
route: 10.96.0.0/12 -> 192.168.49.2
minikube: Running
services: [web-lb]
errors:
minikube: no errors
router: no errors
loadbalancer emulator: no errors
$ kubectl get svc web-lb
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
web-lb LoadBalancer 10.107.252.59 10.107.252.59 80:32623/TCP 16s
$ curl -s http://10.107.252.59/
<h1>hello from kind</h1>
The page is still the same — it is served by the demo/web:dev image built for kind. The tunnel works as long as it is running. If its process is killed, the 10.96.0.0/12 route on the laptop stays, and minikube delete does not remove it either — after the cluster is deleted it hangs around with the linkdown status. minikube tunnel --cleanup removes the records of previous tunnels and immediately starts a new one, so it is not a one-off cleanup command. The most reliable approach is to check ip route after experimenting and delete the leftovers by hand: sudo ip route del 10.96.0.0/12.
minikube pitfalls
- The default resources are modest: 2 CPUs and memory calculated from the host's RAM — usually a few gigabytes. For anything heavier than a couple of pods, set
--cpusand--memoryat the first start: they cannot be changed on an existing cluster, only by recreating it. - The driver is chosen once. The driver of an existing profile cannot be changed —
minikube deleteand start over. - The network to the node. With the docker driver on macOS and Windows the node's address is not reachable from the host; everything goes through
minikube serviceandminikube tunnel. - Addons determine which manifests work. Enable Traefik for Ingresses that use its annotations. The
standardclass uses minikube's local provisioner. Install the appropriate CSI driver to try the series' storage manifests.
How all three differ from a kubeadm cluster
Everything concerning the API and objects is the same. What diverges is what lies below the API:
LoadBalancer. In a cluster on bare metal, an address for a LoadBalancer service is assigned by a separate component: in our case it is MetalLB from part 17, which answers ARP requests for the addresses of its pool. kind has cloud-provider-kind with load balancer containers, k3s has ServiceLB with hostPort on node addresses, minikube has a tunnel with a route on the laptop. The manifest is the same, but the packet path and the behaviour on failure are not. Testing on a laptop how a service survives a node failure is pointless.
Storage. All three provide directory volumes on the node by default. The NFS or SMB volumes the lab works with behave differently: the volume is accessible from any node, there are permissions on the server side, locks, network latency. Errors with fsGroup, root_squash and directory permissions are not reproduced on a laptop.
Network. kindnet and Flannel do not implement NetworkPolicy the way Calico does (in k3s policies are handled by a separate kube-router controller, in kind by its own minimal one); Calico-specific resources such as GlobalNetworkPolicy are absent in them. In kind, kube-proxy works through iptables by default, while the lab uses nftables. The kind node network is a Docker bridge, and MTU, routing between subnets and VXLAN over a real network do not occur in it.
Kernel and runtime. The pods of kind and minikube (with the docker driver) run on the laptop's kernel, nested in the node container. The cgroups, sysctl, seccomp and AppArmor restrictions are those that Docker allowed. On macOS and Windows it is the kernel of the Docker Desktop service VM altogether.
Resilience. A kind cluster does not always survive a Docker restart: the container addresses in the kind network may change, and certificates issued for the old addresses will stop matching. These are disposable clusters — they are not repaired but recreated, and that is the right way to work with them: the cluster configuration in a file, creation in a minute.
Comparison
| kind | k3s | minikube | |
|---|---|---|---|
| Node | a Docker/Podman container | a Linux machine (VM, bare metal) | a VM or a container, depending on the driver |
| Kubernetes installation inside | kubeadm | its own binary, without kubeadm | kubeadm |
| State storage | etcd | SQLite (kine), embedded etcd, PostgreSQL/MySQL | etcd |
| Control plane | static pods | goroutines of the k3s server process |
static pods |
| Default CNI | kindnet | Flannel (VXLAN) | bridge / kindnet, chosen with --cni |
| LoadBalancer | cloud-provider-kind (separate) | ServiceLB, built in | minikube tunnel |
| Ingress | through cloud-provider-kind | Traefik, built in | traefik or ingress addon |
| Volumes | local-path, class standard |
local-path, class local-path |
hostpath, class standard |
| Multiple nodes | yes, with a configuration file | yes, with agents on other machines | yes, --nodes |
| Your own image without a registry | kind load |
import into containerd (k3s ctr images import) |
minikube image load |
| Creation time | ~1 min | ~30 s per node | 1–3 min |
| Suitable for permanent use | no | yes | no |
Which one when
- kind — CI and checking manifests. The cluster comes up in any runner that has Docker, the configuration is in a single file, the Kubernetes version is chosen by the image. The tests of Kubernetes itself and of most operators are run on kind. A multi-node cluster on a single laptop is also a job for it.
- k3s — when you need a real, long-lived cluster on modest hardware: a home server, a branch office, an edge device, a training cluster on a single VM. It is a full-fledged distribution, used in production as well. For a laptop — through k3d or in a VM.
- minikube — development on a workstation, especially on macOS and Windows: VM drivers, addons and tunnels solve most "how do I reach the service" problems. If you need a separate kernel, it is also the one.
For completeness: there is MicroK8s — a Canonical distribution installed as a snap package which, like minikube, enables components with the microk8s enable command; k0s — another "single binary" distribution, from Mirantis; and the built-in Kubernetes in Docker Desktop and Rancher Desktop — a checkbox in the settings that brings up a single-node cluster in a service VM. They are built along the same lines as the ones covered above.
Hands-on
kind nodes are containers:
$ docker ps --format 'table {{.Names}}\t{{.Image}}\t{{.Ports}}'
NAMES IMAGE PORTS
lab-worker kindest/node:v1.37.0
lab-worker2 kindest/node:v1.37.0
lab-control-plane kindest/node:v1.37.0 127.0.0.1:37483->6443/tcp, 127.0.0.1:8080->30080/tcp
$ docker inspect lab-control-plane --format '{{.HostConfig.Privileged}}'
true
$ docker exec lab-worker ps -e -o pid,comm | head -12
PID COMMAND
1 systemd
95 systemd-journal
139 containerd
239 kubelet
290 containerd-shim
321 containerd-shim
333 pause
352 pause
409 kube-proxy
584 kindnetd
788 containerd-shim
PID 1 is systemd, and under it are the same processes as on k8s-node-1. You can go inside the node and use the same tools as in the lab:
$ docker exec -it lab-worker bash
root@lab-worker:/# crictl ps
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD NAMESPACE
f560e14023694 50bfe947daf23 59 seconds ago Running web 0 317fd06885853 web-85c764c6f7-fkfpz default
2be56b80e9552 4626fe10df5b9 About a minute ago Running kindnet-cni 0 0b114641a368f kindnet-skbsr kube-system
39934b0c1394f d6a28daf3e6b0 About a minute ago Running kube-proxy 0 fb7cf17ed540e kube-proxy-lznlh kube-system
root@lab-worker:/# ip route
default via 172.18.0.1 dev eth0
172.18.0.0/16 dev eth0 proto kernel scope link src 172.18.0.3
172.20.0.0/24 via 172.18.0.2 dev eth0
172.20.1.0/24 via 172.18.0.4 dev eth0
172.20.2.2 dev veth25d6c26d scope host
This is how kindnet works: the subnets of neighbouring nodes are ordinary routes via their addresses in the Docker network, and a pod has a veth with a /32 route on the node. No encapsulation at all.
The image loaded with kind load, in the node's containerd:
root@lab-worker:/# ctr -n k8s.io images ls -q | grep demo
docker.io/demo/web:dev
The name was expanded to docker.io/demo/web:dev: containerd treats a short name without a registry as an image from Docker Hub. If imagePullPolicy were Always, that is exactly where kubelet would go.
k3s — one process instead of static pods:
user@k3s-1:~$ ps -e -o pid,rss,args | grep -E 'k3s (server|agent)|containerd ' | grep -v grep
3650 970152 /usr/local/bin/k3s server
3666 192072 containerd
user@k3s-1:~$ ls /etc/kubernetes/manifests
ls: cannot access '/etc/kubernetes/manifests': No such file or directory
user@k3s-1:~$ sudo ls -la /var/lib/rancher/k3s/server/db/
total 23800
drwx------ 2 root root 4096 Nov 7 15:13 .
drwx------ 8 root root 4096 Nov 7 15:13 ..
-rw-r--r-- 1 root root 12087296 Nov 7 15:13 state.db
-rw-r--r-- 1 root root 32768 Nov 7 15:14 state.db-shm
-rw-r--r-- 1 root root 12236432 Nov 7 15:14 state.db-wal
The API server, scheduler, controller-manager and kubelet are in a single k3s server process (about 950 MB of memory for everything a minute after startup), containerd is a separate child process, and its arguments are not visible in ps. On the agent, k3s agent takes about 190 MB. The cluster state is an SQLite file with a WAL journal. You can look inside with ordinary sqlite3: kine stores objects in the kine table with keys like /registry/pods/kube-system/coredns-… — the same key scheme as etcd in a kubeadm cluster.
user@k3s-1:~$ sudo sqlite3 /var/lib/rancher/k3s/server/db/state.db \
"select name from kine where name like '/registry/services/specs/%' group by name;"
/registry/services/specs/default/kubernetes
/registry/services/specs/kube-system/kube-dns
/registry/services/specs/kube-system/metrics-server
/registry/services/specs/kube-system/traefik
ServiceLB on the node:
user@k3s-1:~$ sudo k3s kubectl get svc -n kube-system traefik
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
traefik LoadBalancer 10.43.65.154 192.168.122.12,192.168.122.11 80:31293/TCP,443:30503/TCP 58s
user@k3s-1:~$ sudo iptables -t nat -S | grep 'CNI-DN.*dport 80 .*DNAT'
-A CNI-DN-11cd7dd21d93ed04fed3d -p tcp -m tcp --dport 80 -j DNAT --to-destination 10.42.0.8:80
EXTERNAL-IP holds the addresses of the nodes themselves. The CNI-DN-… chain is the work of the portmap CNI plugin, which implements the hostPort of the svclb-traefik pod: a packet to port 80 of the node goes into the ServiceLB pod, and from there to the ClusterIP of Traefik.
minikube — the node and its route:
$ minikube ssh -- sudo crictl pods --name storage
POD ID CREATED STATE NAME NAMESPACE ATTEMPT RUNTIME
76dbfd8e2ed2f About a minute ago Ready storage-provisioner kube-system 0 (default)
$ ip route | grep -E '10.96|192.168.49'
10.96.0.0/12 via 192.168.49.2 dev br-a52b9036e7e3
192.168.49.0/24 dev br-a52b9036e7e3 proto kernel scope link src 192.168.49.1
minikube tunnel added a route to the service network via the minikube node's address on the Docker bridge — this is exactly how packets to the "external" LoadBalancer address get into the cluster.
Deletion is a single command for each: kind delete cluster --name lab, minikube delete, /usr/local/bin/k3s-uninstall.sh on the server and k3s-agent-uninstall.sh on the agents.
Single-machine clusters will come in handy even after the whole series: manifests, charts and GitOps configuration are convenient to check first on kind, k3s or minikube — create a cluster in a minute and throw it away without regret. But how LoadBalancers, network volumes and the network between nodes will behave, only a real cluster like our lab will show.
Sources
- kind: quick start and loading images, cluster configuration, v0.33.0 and node images, LoadBalancer, cloud-provider-kind.
- k3s: architecture, installation, v1.37.2+k3s1 release, installer variables, networking services, packaged components, tokens, backup and restore.
- minikube: quick start, v1.39.0 release, drivers, addons, Traefik addon, local images, accessing services.
- Kubernetes: images and pull policy, ingress-nginx retirement.
Only registered users can leave a comment.
Sign in or register to leave a comment.