Kubernetes for beginners. Part 28: What are CRDs and operators?
All parts of the series “Kubernetes for beginners”In part 1 we defined Kubernetes as a set of controllers, each bringing the actual state to the desired one, and in part 3 its API as a uniform set of resources with spec and status that can be read, changed and watched for changes. Both ideas are open to extension: you can add your own resource type to the API and run your own controller next to it to serve it. Together this is called an operator, and we have already used two: the Tigera operator installed Calico in part 4, cert-manager issued certificates in part 19. Today — how it works: CustomResourceDefinition, schema and validation, the controller and its loop, then two real operators for databases and queues, and how to assess an operator before installing it. Kustomize from part 26 and Helm from part 27 come in handy here too: operators are usually installed with release manifests or charts.
Everything in this part was tested on the lab cluster: the CRD and a controller on controller-runtime, CloudNativePG 1.30.1 and RabbitMQ Cluster Operator 2.23.2. Left untested are PostgreSQL backups to S3, the kubectl cnpg plugin, CRD version conversion and Messaging Topology Operator — they follow the documentation.
A resource without code in the apiserver
A CustomResourceDefinition (CRD) is a built-in resource of the apiextensions.k8s.io group that describes a new type: its group, name, versions and schema. Create a CRD — and a second later kube-apiserver serves a new REST path, stores the objects in etcd, validates them against the schema, serves watch, supports kubectl get, apply, explain, RBAC and everything else built-in resources have. Not a line of code is written in the apiserver, and nothing is restarted.
What a CRD does not do is give the object behaviour. A Greeting object will sit in etcd and nothing will happen until some controller starts watching it. In this it differs from built-in resources only in quantity: a Deployment is also "just data" processed by a controller inside kube-controller-manager.
There is a second, older extension mechanism — API aggregation: a separate API server registers itself with an APIService object, and kube-apiserver proxies requests for its group to it. This is how metrics-server works (part 32). It is needed when data is not stored in etcd or custom storage logic is required; for everything else CRDs are used.
The example for the whole part is a Greeting resource: "I want the hello application of such a version, with such a greeting, in so many replicas, on such a host". The controller will turn it into a Deployment, a Service and an Ingress.
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: greetings.demo.example.com # <plural>.<group>, must be exactly this
spec:
group: demo.example.com
scope: Namespaced # or Cluster
names:
kind: Greeting
listKind: GreetingList
plural: greetings
singular: greeting
shortNames: [gr]
categories: [demo] # kubectl get demo shows all resources of the category
versions:
- name: v1
served: true
storage: true
subresources:
status: {}
scale:
specReplicasPath: .spec.replicas
statusReplicasPath: .status.readyReplicas
labelSelectorPath: .status.selector
additionalPrinterColumns:
- name: Version
type: string
jsonPath: .spec.version
- name: Replicas
type: integer
jsonPath: .spec.replicas
- name: Ready
type: integer
jsonPath: .status.readyReplicas
- name: URL
type: string
jsonPath: .status.url
priority: 1 # only in -o wide
- name: Age
type: date
jsonPath: .metadata.creationTimestamp
selectableFields:
- jsonPath: .spec.version
schema:
openAPIV3Schema:
type: object
required: [spec]
properties:
spec:
type: object
required: [version]
properties:
version:
type: string
pattern: '^[0-9]+\.[0-9]+(\.[0-9]+)?$'
description: Tag of the ghcr.io/example/hello image
message:
type: string
maxLength: 200
default: hello
replicas:
type: integer
minimum: 1
maximum: 10
default: 2
host:
type: string
maxLength: 253
x-kubernetes-validations:
- rule: self == oldSelf
message: host cannot be changed, create a new Greeting
x-kubernetes-validations:
- rule: "!has(self.host) || self.host.endsWith('.apps.example.com')"
message: host must be in the apps.example.com zone
fieldPath: .host
- rule: "!has(self.host) || self.replicas >= 2"
messageExpression: "'a public application needs at least 2 replicas, got %d'.format([self.replicas])"
reason: FieldValueForbidden
status:
type: object
properties:
readyReplicas:
type: integer
selector:
type: string
url:
type: string
observedGeneration:
type: integer
format: int64
conditions:
type: array
x-kubernetes-list-type: map
x-kubernetes-list-map-keys: [type]
items:
type: object
required: [type, status]
properties:
type: {type: string}
status: {type: string, enum: ["True", "False", "Unknown"]}
reason: {type: string}
message: {type: string}
lastTransitionTime: {type: string, format: date-time}
observedGeneration: {type: integer, format: int64}
Let us go through it piece by piece.
Group, versions, served and storage
The group is a domain name you own (demo.example.com), so as not to collide with other people's CRDs. The *.k8s.io and *.kubernetes.io groups are reserved for the Kubernetes project and require an annotation with a link to an approved proposal.
Versions — one resource can have several, like built-in ones (v1alpha1, v1beta1, v1). Each has two flags:
served— whether this version is served through the API. By turningservedoff a version is "taken out of service": requests to it get 404, but objects in etcd are not touched.storage— in which version objects are written to etcd. Exactly one version hasstorage: true.
If there are several versions and the schemas differ, objects have to be converted: the client writes v1 while etcd stores v2, or the other way round. spec.conversion.strategy: None fits only when the schemas are the same (only apiVersion changes). Otherwise it is Webhook: the apiserver sends the object to an HTTPS service (usually in the same pod as the controller) in a ConversionReview request and gets back the object in the required version. Conversion must be lossless in both directions, otherwise reading and writing through different versions will corrupt data.
Objects written in an old storage version stay in it until they are rewritten. The list of versions actually present in etcd is shown in the CRD's status.storedVersions, and an old version can be removed from the CRD only when it is not there: for that all objects are rewritten (storage migration is done by the operator itself on upgrade or by a tool like kube-storage-version-migrator), and then storedVersions is edited.
Schema: structural schema, defaults, validation
Since apiextensions.k8s.io/v1 the schema is mandatory and must be structural: the root, every object field and every array item has a non-empty type, and everything described inside allOf/anyOf/oneOf is also described outside. This restriction is not a whim: knowing the type of every node, the apiserver can do three things.
Pruning — fields not in the schema are silently dropped on write. A typo replica: 3 instead of replicas: 3 will not be saved, and kubectl apply with --validate=strict (the default behaviour) returns an error: Greeting in version "v1" cannot be handled as a Greeting: strict decoding error: unknown field "spec.replica". If some subtree needs arbitrary YAML (for example, a piece of a program's configuration), it is marked x-kubernetes-preserve-unknown-fields: true.
Defaults — default: 2 on replicas is applied when reading from etcd and on write, so the controller always sees the field filled in.
Validation — by OpenAPI keywords (type, pattern, minimum, maxLength, enum, required) and by CEL rules in x-kubernetes-validations. CEL (Common Expression Language) is an expression language from Google without loops or side effects that is guaranteed to finish in bounded time; the apiserver evaluates it itself, without an external call. CEL validation has been stable since Kubernetes 1.29.
Rule fields:
| Field | Purpose |
|---|---|
rule |
the expression; self is the value of the field where the rule is, oldSelf is its previous value on update |
message |
error text |
messageExpression |
error text computed by a CEL expression (takes precedence over message) |
reason |
machine-readable reason: FieldValueInvalid (default), FieldValueForbidden, FieldValueRequired, FieldValueDuplicate |
fieldPath |
which field to point to in the error |
optionalOldSelf |
evaluate a rule with oldSelf also on object creation |
The apiserver estimates CEL expressions in advance, when the CRD is created: every rule gets a worst-case "cost" estimate (string lengths, number of list items), and a schema that does not fit the budget is rejected as a whole. The first version of this CRD built the message by concatenation, '… got ' + string(self.replicas), and the apiserver did not accept it:
The CustomResourceDefinition "greetings.demo.example.com" is invalid:
* spec.validation.openAPIV3Schema.properties[spec].x-kubernetes-validations[1].messageExpression: Forbidden: estimated messageExpression cost exceeds budget by factor of more than 100x (try simplifying the rule, or adding maxItems, maxProperties, and maxLength where arrays, maps, and strings are declared)
…
The estimator does not know the length of a string produced from a number and treats it as unbounded, and so is a concatenation with it. string(self.replicas) alone passes, and so does formatting, '… got %d'.format([self.replicas]) — that is what we kept. In general the error itself suggests the cure: maxLength, maxItems, maxProperties on the fields the rules work with.
A rule with oldSelf is a transition rule: self == oldSelf on host makes the field immutable after creation, just like the built-in spec.selector. Another useful detail is ratcheting (stable since 1.33): if you tightened the schema while the cluster already holds objects that do not satisfy the new rules, they can still be updated without fixing old violations — only changed fields are checked. Without it, tightening a schema would break all existing objects.
selectableFields are fields usable in kubectl get greetings --field-selector spec.version=1.0 (stable since 1.32); before that the field selector for CRDs worked only on metadata.name and metadata.namespace. The apiserver does the filtering itself: --field-selector spec.version=1.1 on the lab returned No resources found in demo namespace., and with 1.0 — the hello object.
Printer columns are the kubectl get columns: what to show instead of a single NAME AGE. Columns with priority above zero are visible only in -o wide.
The status and scale subresources
subresources.status: {} enables the /status subresource, the same as for built-in types. Consequences:
- a write to the main resource ignores changes to
status, and a write to/statusignores everything exceptstatus. The user writes the desired state, the controller the observed one, and they do not overwrite each other; - rights are separated: the user gets
updateongreetings, the controller also ongreetings/status(RBAC and subresources — in part 29); metadata.generationincreases only when fields outsidemetadataandstatuschange, that is, in effect when the desired state changes. The controller writes intostatus.observedGenerationwhich generation it has already processed, and any client can tell by comparing two numbers whether the controller has caught up with the latest changes. This is exactly howkubectl rollout statuswatches a Deployment.
subresources.scale plugs the resource into the common scaling mechanism: kubectl scale greeting/hello --replicas=4 and HorizontalPodAutoscaler (scaling by metrics, part 32) work with Greeting the same way as with a Deployment. You specify where in the object the desired number of replicas, the actual number and the pod selector in string form are.
Hands-on: the CRD
$ kubectl apply -f greeting-crd.yaml
customresourcedefinition.apiextensions.k8s.io/greetings.demo.example.com created
$ kubectl get crd greetings.demo.example.com -o jsonpath='{range .status.conditions[*]}{.type}={.status}{"\n"}{end}'
NamesAccepted=True
Established=True
$ kubectl api-resources --api-group=demo.example.com
NAME SHORTNAMES APIVERSION NAMESPACED KIND
greetings gr demo.example.com/v1 true Greeting
$ kubectl explain greeting.spec.version
GROUP: demo.example.com
KIND: Greeting
VERSION: v1
FIELD: version <string>
DESCRIPTION:
Tag of the ghcr.io/example/hello image
Established=True means the apiserver has started serving the new path. Let us look at the path itself and at validation:
$ kubectl get --raw /apis/demo.example.com/v1/namespaces/demo/greetings
{"apiVersion":"demo.example.com/v1","items":[],"kind":"GreetingList","metadata":{"continue":"","resourceVersion":"1261791"}}
$ cat <<'EOF' | kubectl apply -f -
apiVersion: demo.example.com/v1
kind: Greeting
metadata:
name: hello
namespace: demo
spec:
version: "1.0"
replicas: 1
host: hello.example.org
EOF
The Greeting "hello" is invalid:
* spec.host: Invalid value: host must be in the apps.example.com zone
* spec: Forbidden: a public application needs at least 2 replicas, got 1
Both errors came from the apiserver, there is no controller yet. The corrected object:
$ cat <<'EOF' | kubectl apply -f -
apiVersion: demo.example.com/v1
kind: Greeting
metadata:
name: hello
namespace: demo
spec:
version: "1.0"
message: hi
host: hello.apps.example.com
EOF
greeting.demo.example.com/hello created
$ kubectl -n demo get gr
NAME VERSION REPLICAS READY AGE
hello 1.0 2 1s
REPLICAS 2 is the default from the schema, READY is empty: there is nobody to fill in the status. The transition rule on host is also checked by the apiserver:
$ kubectl -n demo patch gr hello --type=merge -p '{"spec":{"host":"hello2.apps.example.com"}}'
The Greeting "hello" is invalid: spec.host: Invalid value: "hello2.apps.example.com": host cannot be changed, create a new Greeting
Where the object really lives is etcd, under a key by group and resource; unlike built-in resources, which the apiserver stores in Protobuf, custom ones are stored as JSON. The etcdctl utility is in the etcd image, so the easiest way is to run it inside the container with crictl exec, as in part 12; jq keeps the essentials of the object:
user@k8s-master-1:~$ sudo crictl exec "$(sudo crictl ps --name etcd -q)" etcdctl \
--endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key \
get /registry/demo.example.com/greetings/demo/hello --print-value-only \
| jq -c '{apiVersion, kind, metadata: {name: .metadata.name, namespace: .metadata.namespace, generation: .metadata.generation, uid: .metadata.uid}, spec}'
{"apiVersion":"demo.example.com/v1","kind":"Greeting","metadata":{"name":"hello","namespace":"demo","generation":1,"uid":"1cccbbb4-fcb6-4617-a2a5-2a91cbc31c37"},"spec":{"host":"hello.apps.example.com","message":"hi","replicas":2,"version":"1.0"}}
replicas: 2 is written in etcd: the default was applied already on write. For comparison — the beginning of the value of a built-in object, the demo namespace: instead of JSON there is binary Protobuf with the k8s\0 prefix:
user@k8s-master-1:~$ sudo crictl exec "$(sudo crictl ps --name etcd -q)" etcdctl … \
get /registry/namespaces/demo --print-value-only | head -c 12 | od -c | head -1
0000000 k 8 s \0 \n 017 \n 002 v 1 022 \t
The etcdctl utility and how etcd works are the subject of part 34; here it is enough to see that a CRD is no different from the rest of the cluster's data.
The controller
A controller is a program that watches objects through the API and brings the world in line with them. It is always built the same way:
kube-apiserver
▲ │ list + watch (Greeting, Deployment, Service, Ingress)
create/update│ ▼
patch status │ ┌──────────────┐
│ │ informer │ local object cache,
│ │ (cache) │ add / update / delete events
│ └──────┬───────┘
│ │ key "demo/hello"
│ ▼
│ ┌──────────────┐
│ │ work queue │ queue of keys: duplicates collapse,
│ │ │ errors — retry with growing delay
│ └──────┬───────┘
│ ▼
│ ┌──────────────┐
└─────┤ Reconcile │ read the desired state (from cache), look
│ (key) │ at what exists, create / fix / delete,
└──────────────┘ write status
The informer keeps a watch on the needed types and a local copy of the objects in memory. The controller reads only from this cache, not from the apiserver directly: dozens of controllers re-reading all objects every second would bring the apiserver down. At start the informer does a list, then receives changes through watch with resourceVersion (part 3) and periodically re-syncs (resync).
The work queue receives not events but keys namespace/name. If an object changed five times in a second, the queue holds one key, and Reconcile runs once — with the latest version. If Reconcile returned an error, the key goes back into the queue with an exponential delay. One key is never processed by two workers at the same time.
Reconcile is the function "bring the object with this key to the desired state". Its main property is that it does not know what changed. It is not told "the replicas field changed from 2 to 3", it is given the name of the object, and every time it compares desired with actual from scratch. This approach is called level-triggered, as opposed to edge-triggered (by event), and it makes the controller robust: a missed event, a controller restart, a manual edit of the Deployment — the next Reconcile call fixes everything. Hence the requirement of idempotency: calling Reconcile ten times in a row must give the same result as once.
The controller watches not only its own objects but also child ones: if someone deletes the Deployment created for a Greeting, the Deployment's delete event must enqueue the key of its owner. The link is stored in the Deployment itself.
ownerReferences and garbage collection
Every object created by the controller gets its owner recorded in metadata.ownerReferences:
metadata:
name: hello
namespace: demo
ownerReferences:
- apiVersion: demo.example.com/v1
kind: Greeting
name: hello
uid: 1cccbbb4-fcb6-4617-a2a5-2a91cbc31c37
controller: true
blockOwnerDeletion: true
Two parties use this. The controller — to find the owner from a child object's event (controller: true means the "managing" owner, there can be only one). And the garbage collector in kube-controller-manager: when the owner is deleted, it deletes all objects that list this owner. This is how deleting a Deployment deletes its ReplicaSets, and they delete pods; likewise deleting a Greeting will delete the Deployment, Service and Ingress without a single line of deletion code in our controller.
The deletion mode is chosen when the owner is deleted (kubectl delete --cascade=…): background (default: the owner is deleted at once, the children follow), foreground (the owner stays with deletionTimestamp until children with blockOwnerDeletion: true are deleted), orphan (children stay, the owner references are removed from them). A restriction: owner and child must be in the same namespace, or the owner must be cluster-scoped. A namespaced object cannot own a cluster-scoped one.
Finalizers
The garbage collector cleans up objects inside the cluster. But a controller can also create external resources: a DNS record, an S3 bucket, a database account. The collector knows nothing about them. That is what finalizers are for — a list of strings in metadata.finalizers.
While the list is not empty, the object is not deleted: kubectl delete only sets metadata.deletionTimestamp, the object stays visible, and this change is an event for the controller. The controller sees deletionTimestamp, cleans up external resources and removes its string from finalizers. When the list is empty, the apiserver deletes the object for good.
kubectl delete greeting hello
│
▼
apiserver: finalizers not empty → sets deletionTimestamp, object stays
│
▼
controller: Reconcile sees deletionTimestamp
│ → delete the DNS record (external resource)
│ → PATCH: remove "demo.example.com/cleanup" from finalizers
▼
apiserver: finalizers empty → object deleted
│
▼
garbage collector: delete Deployment, Service, Ingress by ownerReferences
The downside: if the controller is removed or broken, objects with its finalizer hang in Terminating forever, and the namespace with them will not be deleted either. A typical picture after removing an operator before its resources. Removing the finalizer by hand (kubectl patch … --type=merge -p '{"metadata":{"finalizers":null}}') is possible, but then external resources stay uncleaned — it is a conscious decision, not a way to "fix" things. The right order to remove an operator: first its resources, then the operator itself, then the CRDs.
status and conditions
status is what the controller observes: the number of ready replicas, the address, errors. A common format for reporting state is conditions — a list of conditions, each with:
| Field | Meaning |
|---|---|
type |
condition name in PascalCase: Ready, Available, Progressing, Degraded |
status |
True, False or Unknown |
reason |
machine-readable reason in PascalCase: DeploymentNotReady, ImagePullFailed |
message |
text for a human |
lastTransitionTime |
when status last changed (not when the condition was rewritten) |
observedGeneration |
which generation of the object it refers to |
The Ready condition is what kubectl wait --for=condition=Ready, kstatus in Helm 4 (part 27) and GitOps tools check. Positive polarity (Ready=True is good) reads easier than Failed=False. Status must be fully reconstructible from observing the world: the controller keeps nothing in it that it cannot compute again — otherwise losing the status (and it is lost when etcd is backed up through kubectl get -o yaml and restored) would lose data too.
A minimal controller on controller-runtime
There is no need to write informers and queues by hand. client-go is the official Go library for working with the API, and it has informers and a work queue. controller-runtime is a Kubernetes SIGs library on top of client-go: a manager with a shared cache, controller registration, webhooks, leader election with several replicas. Most operators are written with it. Kubebuilder is a project generator for controller-runtime: from Go structs with marker annotations it generates a CRD with a schema and CEL rules, RBAC, a controller skeleton and tests. Operator SDK is a Red Hat project on top of Kubebuilder: the same Go operators plus operators in Ansible and Helm and integration with OLM (Operator Lifecycle Manager — the operator installation and upgrade manager built into OpenShift, Red Hat's Kubernetes distribution).
The essence of the Greeting controller without the generated scaffolding:
func (r *GreetingReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
var g demov1.Greeting
if err := r.Get(ctx, req.NamespacedName, &g); err != nil {
// object deleted — the garbage collector removes the children
return ctrl.Result{}, client.IgnoreNotFound(err)
}
labels := map[string]string{"app.kubernetes.io/name": "hello", "app.kubernetes.io/instance": g.Name}
// The desired Deployment. CreateOrUpdate reads the current one, calls the function
// and writes only if something changed.
dep := &appsv1.Deployment{ObjectMeta: metav1.ObjectMeta{Name: g.Name, Namespace: g.Namespace}}
_, err := controllerutil.CreateOrUpdate(ctx, r.Client, dep, func() error {
dep.Labels = labels
dep.Spec.Replicas = g.Spec.Replicas
dep.Spec.Selector = &metav1.LabelSelector{MatchLabels: labels}
dep.Spec.Template.Labels = labels
dep.Spec.Template.Spec.Containers = []corev1.Container{{
Name: "hello",
Image: "ghcr.io/example/hello:" + g.Spec.Version,
Env: []corev1.EnvVar{{Name: "GREETING", Value: g.Spec.Message}},
Ports: []corev1.ContainerPort{{Name: "http", ContainerPort: 8080}},
}}
// ownerReference: Greeting is the managing owner of the Deployment
return controllerutil.SetControllerReference(&g, dep, r.Scheme)
})
if err != nil {
return ctrl.Result{}, err // the key goes back to the queue with a delay
}
// ... Service and Ingress the same way ...
// Status — a separate request to the /status subresource.
g.Status.ReadyReplicas = dep.Status.ReadyReplicas
g.Status.Selector = "app.kubernetes.io/instance=" + g.Name
g.Status.URL = "https://" + g.Spec.Host
g.Status.ObservedGeneration = g.Generation
ready := metav1.ConditionFalse
if dep.Status.ReadyReplicas == *g.Spec.Replicas {
ready = metav1.ConditionTrue
}
meta.SetStatusCondition(&g.Status.Conditions, metav1.Condition{
Type: "Ready", Status: ready, Reason: "DeploymentStatus",
Message: fmt.Sprintf("%d/%d replicas ready", dep.Status.ReadyReplicas, *g.Spec.Replicas),
ObservedGeneration: g.Generation,
})
return ctrl.Result{}, r.Status().Update(ctx, &g)
}
func (r *GreetingReconciler) SetupWithManager(mgr ctrl.Manager) error {
return ctrl.NewControllerManagedBy(mgr).
For(&demov1.Greeting{}). // own objects
Owns(&appsv1.Deployment{}). // child events → owner's key
Owns(&corev1.Service{}).
Owns(&networkingv1.Ingress{}).
Complete(r)
}
This code was built on the lab with controller-runtime v0.25.2 (the k8s.io/* libraries v0.37) and Go 1.27.3. Instead of Kubebuilder, the Greeting and GreetingList types were written by hand — Go structs with spec and status fields and the deep-copy functions DeepCopyObject required by the runtime.Object interface; the main function creates a manager (ctrl.NewManager), registers the built-in types and Greeting in its scheme and starts the controller. The Service and Ingress are created the same way as the Deployment, with the same labels.
All the ideas of the previous sections are visible here: Reconcile gets only a name, reads the object from the cache, brings the Deployment to the desired form idempotently, sets the owner, writes status through the subresource. Owns subscribes the controller to child Deployments, and their changes enqueue the owner's key. Deletion is not handled at all — the garbage collector does it. A finalizer would be needed only if the controller created something outside the cluster.
In the cluster a controller is a Deployment with a ServiceAccount that a ClusterRole (or a Role if it watches one namespace) allows get, list, watch on greetings, update on greetings/status and the full set on Deployments, Services and Ingresses. Kubebuilder generates these rights from // +kubebuilder:rbac:… markers above the Reconcile function. If the controller has more than one replica, only one is active: they compete for a Lease object in coordination.k8s.io, like kube-scheduler and kube-controller-manager.
Hands-on: the controller at work
To try the controller you do not have to build an image and install it as a Deployment: controller-runtime finds the kubeconfig the same way kubectl does, and the program can run right on the administrator's machine (go run . or a built binary) — this is how controllers are debugged. On the lab the binary ran on the master. It is better not to build it there: compiling the client-go dependencies took both master cores for several minutes, kube-apiserver failed its liveness probe and restarted, and kube-controller-manager and kube-scheduler spent some time in CreateContainerError — a workstation is a better place to build. The hello object from the previous section already exists, and a few seconds after start:
$ kubectl -n demo get deploy,svc,ingress -l app.kubernetes.io/instance=hello
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/hello 1/2 2 1 64s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/hello ClusterIP 10.102.187.61 <none> 80/TCP 64s
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress.networking.k8s.io/hello traefik hello.apps.example.com 80 64s
$ kubectl -n demo get gr -o wide
NAME VERSION REPLICAS READY URL AGE
hello 1.0 2 2 https://hello.apps.example.com 9m49s
$ kubectl -n demo get gr hello -o jsonpath='{.status.conditions[0].message}{"\n"}'
2/2 replicas ready
$ kubectl -n demo get deploy hello -o jsonpath='{.metadata.ownerReferences[0].kind}/{.metadata.ownerReferences[0].name} controller={.metadata.ownerReferences[0].controller}{"\n"}'
Greeting/hello controller=true
The Ingress ADDRESS column is empty and PORTS is only 80: the lab had no Ingress controller, and our controller does not create a tls section; in a cluster with Traefik from part 18 the address appears. The controller log also had an error among the reconcile lines:
{"level":"error","msg":"Reconciler error","controller":"greeting",…,"error":"Operation cannot be fulfilled on deployments.apps \"hello\": the object has been modified; please apply your changes to the latest version and try again"}
This is the optimistic locking from part 3: between reading the Deployment from the cache and writing it, kube-controller-manager changed it, and the write with a stale resourceVersion was rejected. Reconcile returned an error, the key went back to the queue with a delay, the next pass went through cleanly — exactly as designed. Resilience to manual edits — level-triggered in action:
$ kubectl -n demo scale deploy hello --replicas=5
deployment.apps/hello scaled
$ kubectl -n demo get deploy hello -w
NAME READY UP-TO-DATE AVAILABLE AGE
hello 2/5 2 2 79s
hello 2/2 2 2 80s
hello 2/2 2 2 80s
hello 2/2 2 2 81s
hello 2/2 5 2 82s
hello 2/2 2 2 83s
The controller got the Deployment change event, enqueued demo/hello, and Reconcile restored replicas: 2 from the Greeting — within a second, the extra pods did not even have time to become ready. Scaling through the resource itself, thanks to the scale subresource:
$ kubectl -n demo scale greeting hello --replicas=3
greeting.demo.example.com/hello scaled
$ kubectl -n demo wait greeting/hello --for=condition=Ready --timeout=60s
greeting.demo.example.com/hello condition met
$ kubectl -n demo get greeting hello -o jsonpath='{.metadata.generation} {.status.observedGeneration}{"\n"}'
2 2
$ kubectl get --raw /apis/demo.example.com/v1/namespaces/demo/greetings/hello/scale
{"kind":"Scale","apiVersion":"autoscaling/v1","metadata":{"name":"hello","namespace":"demo",…},"spec":{"replicas":3},"status":{"replicas":3,"selector":"app.kubernetes.io/instance=hello"}}
Generation 2, the controller processed generation 2 — the state has caught up with the desired one. The Scale object at …/greetings/hello/scale is assembled by the apiserver itself from the paths in subresources.scale: this is what kubectl scale and HorizontalPodAutoscaler work with.
A finalizer can be seen in action without code too — by setting it by hand. The controller does not handle it, so it has to be removed by hand as well:
$ kubectl -n demo patch greeting hello --type=merge -p '{"metadata":{"finalizers":["demo.example.com/cleanup"]}}'
greeting.demo.example.com/hello patched
$ kubectl -n demo delete greeting hello --wait=false
greeting.demo.example.com "hello" deleted from demo namespace
$ kubectl -n demo get greeting hello -o jsonpath='{.metadata.deletionTimestamp} {.metadata.finalizers}{"\n"}'
2026-10-31T05:06:33Z ["demo.example.com/cleanup"]
$ kubectl -n demo get deploy -l app.kubernetes.io/instance=hello
NAME READY UP-TO-DATE AVAILABLE AGE
hello 3/3 3 3 2m4s
kubectl says deleted, but the object is still there: it only got a deletionTimestamp, and the child Deployment keeps running. Remove the finalizer — and the apiserver deletes the object, and the garbage collector then removes the Deployment, Service and Ingress:
$ kubectl -n demo patch greeting hello --type=merge -p '{"metadata":{"finalizers":null}}'
greeting.demo.example.com/hello patched
$ kubectl -n demo get greeting
No resources found in demo namespace.
$ kubectl -n demo get deploy,svc,ingress -l app.kubernetes.io/instance=hello
No resources found in demo namespace.
The operator
An operator is a controller with CRDs that embed the knowledge of an administrator of a specific system. The term was coined in 2016 at CoreOS, the company that created etcd: an "operator" does what a human operator used to do — deploys a database cluster, replaces a failed node, upgrades the version in the right order, takes backups. Our Greeting is a controller but not an operator: it has no knowledge beyond "create a Deployment". The difference is visible on real examples.
CloudNativePG
CloudNativePG (CNPG) is a PostgreSQL operator, a CNCF sandbox project. It is installed with the official release manifest or a Helm chart into the cnpg-system namespace; the current version is 1.30.1:
$ kubectl apply --server-side -f https://github.com/cloudnative-pg/cloudnative-pg/releases/download/v1.30.1/cnpg-1.30.1.yaml
…
deployment.apps/cnpg-controller-manager serverside-applied
mutatingwebhookconfiguration.admissionregistration.k8s.io/cnpg-mutating-webhook-configuration serverside-applied
validatingwebhookconfiguration.admissionregistration.k8s.io/cnpg-validating-webhook-configuration serverside-applied
--server-side is not a whim here: the clusters.postgresql.cnpg.io CRD takes about 460 KiB in the manifest, poolers about 660 KiB, while a client-side apply would write a copy of each object into the last-applied-configuration annotation, and all annotations of an object together are limited to 256 KiB. The main resource is Cluster:
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: pg
namespace: demo
spec:
instances: 3
imageName: ghcr.io/cloudnative-pg/postgresql:18.6-standard-trixie
storage:
storageClass: local-path
size: 20Gi
bootstrap:
initdb:
database: app
owner: app
affinity:
topologyKey: kubernetes.io/hostname
plugins:
- name: barman-cloud.cloudnative-pg.io
isWALArchiver: true
parameters:
barmanObjectName: s3-store
The PostgreSQL image comes from the project's official set: minimal (PostgreSQL only), standard (plus locales and popular extensions) and the deprecated system with built-in Barman Cloud for the old backup method. Without imageName CNPG 1.30.1 takes 18.6-system-trixie; with the backup plugin (see below) standard is enough. The local-path storage class is from part 25.
What the operator does with this object — and what an administrator would have done before:
- Deploys a PostgreSQL cluster of three instances: a primary and two replicas with streaming replication (PostgreSQL sends its write-ahead log, WAL, to the replicas over the network). It does not use a StatefulSet: it creates the pods
pg-1,pg-2,pg-3and their PVCs itself, because it needs to manage each instance separately — which is the primary, which to recreate, which volume to attach. - Creates services
pg-rw(always points to the primary),pg-ro(replicas only) andpg-r(any instance). The application connects topg-rw.demo.svcand does not need to know who is the primary after a switch. The password of theappuser is in Secretpg-app(typekubernetes.io/basic-auth, with ready-made connection strings); next to it the operator creates its own certificate authoritypg-ca, the server certificatepg-server, the replication certificatepg-replicationand two PodDisruptionBudgets —pgfor replicas andpg-primaryfor the primary. - Fails over: if the primary is unavailable, the operator picks the replica with the most recent WAL, promotes it and redirects
pg-rw. A planned switchover — for example, before node maintenance — is done with the plugin,kubectl cnpg promote pg pg-2. - Upgrades PostgreSQL in the right order: on an
imageNamechange it first recreates the replicas one at a time, waiting for each to become ready, and only then turns to the primary. How exactly is set byprimaryUpdateMethod: by defaultrestart— the primary is restarted in place with the new image (on the lab writes were unavailable for about 20 seconds), whileswitchoverfirst moves the role to an already updated replica, and the pause shrinks to the switchover time.primaryUpdateStrategy: supervisedinstead of the defaultunsupervisedleaves this last step to the administrator. - Takes backups: a continuous WAL archive and base backups to S3 object storage (Amazon's object storage protocol, also implemented by many other systems), which gives point-in-time recovery (PITR). Since version 1.26 the built-in
barmanObjectStoresetting is deprecated, and backups moved to the Barman Cloud plugin (current version 0.15.1) with its ownObjectStoreresource. The plugin is installed with a separate manifest into the samecnpg-systemand requires cert-manager from part 19: the operator and the plugin talk over gRPC (a remote procedure call protocol on top of HTTP/2) with mutual TLS certificate verification, and the plugin manifest requests the certificates withIssuerandCertificateobjects. On the lab the schemas of both resources below were checked withkubectl apply --dry-run=serverwith the CRDs installed, but the backups themselves were not:
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: s3-store
namespace: demo
spec:
retentionPolicy: "30d"
configuration:
destinationPath: s3://pg-backups/
endpointURL: https://s3.example.com
s3Credentials:
accessKeyId:
name: pg-backup-s3
key: ACCESS_KEY_ID
secretAccessKey:
name: pg-backup-s3
key: ACCESS_SECRET_KEY
wal:
compression: gzip
---
apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata:
name: pg-nightly
namespace: demo
spec:
schedule: "0 0 2 * * *" # cron format with seconds: every day at 02:00:00
backupOwnerReference: self
cluster:
name: pg
method: plugin
pluginConfiguration:
name: barman-cloud.cloudnative-pg.io
Recovery is a new Cluster with bootstrap.recovery and a source from the same store; a copy for testing backups is brought up the same way, and that is worth doing regularly. Storage for PostgreSQL data is local node disks or block volumes, not NFS: why was covered in part 23 and part 25. The operator already provides replication at the PostgreSQL level, so local volumes without their own replication are a normal choice here.
On the lab the Cluster was created without the plugins section (the plugin was not installed) and at first without imageName:
$ kubectl -n demo get cluster pg
NAME AGE INSTANCES READY STATUS PRIMARY
pg 4m13s 3 3 Cluster in healthy state pg-1
$ kubectl -n demo get pods -l cnpg.io/cluster=pg -L cnpg.io/instanceRole -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES INSTANCEROLE
pg-1 1/1 Running 0 2m36s 172.20.172.33 k8s-node-2 <none> <none> primary
pg-2 1/1 Running 0 69s 172.20.211.41 k8s-node-1 <none> <none> replica
pg-3 1/1 Running 0 28s 172.20.211.42 k8s-node-1 <none> <none> replica
$ kubectl -n demo get svc,sts
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/pg-r ClusterIP 10.100.227.180 <none> 5432/TCP 4m13s
service/pg-ro ClusterIP 10.106.229.62 <none> 5432/TCP 4m12s
service/pg-rw ClusterIP 10.101.30.144 <none> 5432/TCP 4m12s
There is no StatefulSet, the instances are created one by one: first a pg-1-initdb Job initializes the database on the first instance's volume, then the replicas clone it. pg-2 and pg-3 ended up on the same node: the lab has two workers, and CNPG's anti-affinity is preferred by default — with podAntiAffinityType: required the third instance would wait in Pending for a third node. Delete the primary pod and watch the cluster every two seconds:
$ kubectl -n demo delete pod pg-1
pod "pg-1" deleted from demo namespace
05:12:16 pg 4m29s 3 2 Failing over pg-1
05:12:21 pg 4m33s 3 2 Failing over pg-2
05:12:23 pg 4m36s 3 2 Waiting for the instances to become active pg-2
05:12:45 pg 4m57s 3 3 Cluster in healthy state pg-2
$ kubectl -n demo get pods -l cnpg.io/cluster=pg -L cnpg.io/instanceRole
NAME READY STATUS RESTARTS AGE INSTANCEROLE
pg-1 1/1 Running 0 24s replica
pg-2 1/1 Running 0 113s primary
pg-3 1/1 Running 0 72s replica
Five seconds later pg-2 became the primary, and the recreated pg-1 came back as a replica; the cluster is fully healthy after half a minute. Then an image change — imageName: …:18.6-standard-trixie with a patch (a selection of the same lines where the status changed):
05:13:35 Waiting for the instances to become active pg-3 is recreated (replica)
05:14:03 Waiting for the instances to become active pg-3 ready, then pg-1 (replica)
05:14:33 Primary instance is being restarted without a switchover pg-2 (primary)
05:14:54 Cluster in healthy state the primary is still pg-2
Replicas were updated one at a time, the primary was restarted in place — the default primaryUpdateMethod: restart. Deleting the Cluster also deletes its PVCs: seconds after kubectl delete cluster pg there were neither pods nor volumes left in the namespace.
RabbitMQ Cluster Operator
RabbitMQ Cluster Operator is the official operator of the RabbitMQ message broker from its developers. It is installed with a single release manifest (the current one is 2.23.2) and creates the rabbitmq-system namespace, but cert-manager from part 19 must already be in the cluster: the manifest requests the certificate for its webhooks with Issuer and Certificate objects. Without it kubectl apply creates everything else and reports no matches for kind "Certificate" in version "cert-manager.io/v1", the operator pod waits for a non-existent Secret with the certificate, and any RabbitmqCluster is rejected — failed calling webhook "mrabbitmqcluster-v1beta1.kb.io" … connect: connection refused. The resource is RabbitmqCluster:
apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
name: mq
namespace: demo
spec:
replicas: 3
persistence:
storageClassName: local-path
storage: 10Gi
resources:
requests:
cpu: 500m
memory: 1Gi
limits:
memory: 1Gi
rabbitmq:
additionalConfig: |
default_queue_type = quorum
From it the operator creates the StatefulSet mq-server (here stable pod names are exactly what is needed: RabbitMQ nodes know each other by name), the mq service for clients and the headless mq-nodes for communication between nodes, Secret mq-default-user with the administrator credentials, a ConfigMap with the configuration and Secret mq-erlang-cookie — the shared Erlang secret without which the nodes do not join into a cluster:
$ kubectl -n demo get rabbitmqcluster,sts,svc,secret,cm -o wide | grep mq
rabbitmqcluster.rabbitmq.com/mq True True 2m27s
statefulset.apps/mq-server 3/3 2m26s rabbitmq rabbitmq:4.3.4-management
service/mq ClusterIP 10.111.195.78 <none> 5672/TCP,15672/TCP,15692/TCP 2m27s app.kubernetes.io/name=mq
service/mq-nodes ClusterIP None <none> 4369/TCP,25672/TCP 2m27s app.kubernetes.io/name=mq
secret/mq-default-user Opaque 8 2m26s
secret/mq-erlang-cookie Opaque 1 2m27s
configmap/mq-plugins-conf 1 2m26s
configmap/mq-server-conf 2 2m26s
The mq ports: 5672 — AMQP (the binary protocol over which clients publish messages to the broker and receive them from queues), 15672 — the management web UI, 15692 — Prometheus metrics; mq-nodes has 4369 — the Erlang name service (epmd; RabbitMQ is written in Erlang, and its nodes find each other through it) and 25672 — inter-node communication. Then comes what an administrator would have done by hand following instructions: on a version or configuration update the pods restart one at a time, and the preStop hook of each pod does not let it stop until the quorum queues (RabbitMQ's replicated queues based on Raft — consensus replication with leader election, where a write is confirmed by a majority of copies) still have enough live copies. The hook itself is a few rabbitmq-upgrade commands:
$ kubectl -n demo get sts mq-server -o jsonpath='{.spec.template.spec.containers[0].lifecycle.preStop.exec.command[2]}'
if [ ! -z "$(cat /etc/pod-info/skipPreStopChecks)" ]; then exit 0; fi; rabbitmq-upgrade await_online_quorum_plus_one -t 604800 && (rabbitmq-upgrade await_online_synchronized_mirror -t 604800 || true) && rabbitmq-upgrade drain -t 604800
await_online_quorum_plus_one waits until every quorum queue keeps its quorum even after this node stops, and drain puts the node into maintenance mode — clients and queue leaders move off it before it stops. After a cluster change the operator runs rabbitmq-queues rebalance all — redistributes queue leaders across the nodes (disabled by the skipPostDeploySteps field). The operator computes memory for the pod limit itself: the configuration gets total_memory_available_override_value = 858993460 — 80% of the 1 GiB limit, and from that value RabbitMQ computes its vm_memory_high_watermark threshold (0.6 by default), at which it stops accepting messages. The remaining 20% is headroom for memory RabbitMQ does not account for, so that the container does not hit the limit and get OOM-killed before the broker starts throttling publishers.
Queues, exchanges, users and permissions are handled by a separate operator, Messaging Topology Operator, with the resources Queue, Exchange, Binding, User, Permission, Policy: then the whole broker topology is described in YAML in Git next to the application.
How to assess an operator before installing it
An operator is code with broad rights that will run in the cluster unattended for months and make decisions for you, including deleting data. Before installing, it is worth checking:
Maturity. How old the project is, how often releases come out, whether there is a version support policy, who is behind the project (a foundation, a vendor, a single enthusiast). A useful scale is the operator capability levels from Operator SDK: basic install → seamless upgrades → full lifecycle (backups, recovery, failure handling) → deep insights (metrics and analysis) → autopilot (self-tuning). An operator that can only install is essentially a chart with an extra process.
Permissions. Look at the operator's ClusterRole before installing (helm template or the manifest) and after — what it can actually do:
$ kubectl auth can-i --list --as=system:serviceaccount:cnpg-system:cnpg-manager \
| grep -E '^(Resources|configmaps|secrets|services|pods|pods/exec|persistentvolumeclaims|jobs.batch|nodes) '
Resources Non-Resource URLs Resource Names Verbs
configmaps [] [] [create delete get list patch update watch]
secrets [] [] [create delete get list patch update watch]
services [] [] [create delete get list patch update watch]
persistentvolumeclaims [] [] [create delete get list patch watch]
pods/exec [] [] [create delete get list patch watch]
pods [] [] [create delete get list patch watch]
jobs.batch [] [] [create delete get list patch watch]
nodes [] [] [get list watch]
$ kubectl auth can-i get secrets -n kube-system --as=system:serviceaccount:cnpg-system:cnpg-manager
yes
A database operator reading Secrets in all namespaces is common, but it means that compromising it compromises all secrets of the cluster. Many operators can run in single- or multi-namespace mode with a Role instead of a ClusterRole — if that fits, prefer it. How to read these rights and why secrets, pods and create are dangerous — in part 29.
Webhooks. Operators often install a ValidatingWebhookConfiguration and a MutatingWebhookConfiguration — registration of their own HTTPS service that the apiserver calls when objects are created. With failurePolicy: Fail and a broad objectSelector, a crashed operator blocks object creation far beyond its CRDs. Check what exactly the webhook is subscribed to — kubectl get validatingwebhookconfigurations -o yaml. CNPG, for example, has all nine webhooks (five validating and four mutating) with failurePolicy: Fail, but they are subscribed only to its own resources — clusters, backups, scheduledbackups, databases, poolers: a crashed operator will not let you create a Cluster, but will not affect the pods and Deployments of other applications.
CRD upgrades. How the CRDs are upgraded with a new version: a separate manifest, a chart template, the operator itself at start? Is there a conversion webhook, and are storedVersions migrated? What of this has to be done by hand is usually in the release notes, and skipping a step leads to a new operator that does not understand old objects.
Removal. What happens to data when a Cluster or the operator itself is deleted, are there finalizers, are PVCs deleted. For CNPG, for example, PVCs are deleted together with the Cluster by default (seconds on the lab), and the backup is the only thing left.
Compatibility. Which Kubernetes versions the operator is tested with, and whether it will get in the way of a cluster upgrade (part 35).
Familiar operators
Now we can take a fresh look at what is already on the lab:
- The Tigera operator from part 4 watches the
Installationresource (operator.tigera.io/v1) and from it deploys calico-node, calico-typha, calico-kube-controllers and their configuration. Upgrading Calico means changing the operator version, and then it upgrades the components on the nodes in the right order itself. - cert-manager from part 19 is a set of controllers:
Certificate→CertificateRequest→Order→Challenge(for ACME) → a Secret with the certificate.CertificatehasReadyinstatus.conditions, and the whole owner reference mechanism applies to it: delete aCertificate, and the collector deletes the intermediate objects. - MetalLB from part 17 is configured with the
IPAddressPoolandL2Advertisementresources — CRDs by which its controller and the speakers on the nodes hand out addresses.
On the lab, at the moment when Calico, cert-manager, CNPG with the plugin's CRD, the RabbitMQ operator and our Greeting were installed, the CRDs by group looked like this:
$ kubectl get crd -o custom-columns=GROUP:.spec.group --no-headers | sort | uniq -c | sort -rn
22 crd.projectcalico.org
11 postgresql.cnpg.io
9 operator.tigera.io
4 cert-manager.io
2 acme.cert-manager.io
1 rabbitmq.com
1 policy.networking.k8s.io
1 demo.example.com
1 barmancloud.cnpg.io
52 types, of which one is ours. In a cluster with a dozen components there are often more than a hundred CRDs, and almost all the "Kubernetes" an administrator works with is no longer built-in resources but other people's API extensions.
Cleanup
Remove in reverse order, as said above: first the Cluster and the RabbitmqCluster (the operator deletes the pods and PVCs), then the operators with the same manifests (kubectl delete -f …; the CRDs go with them, and with the CRDs all objects of those types), then our CRD and cert-manager. The cert-manager leader election Lease objects in kube-system (cert-manager-controller and cert-manager-cainjector-leader-election) are not removed by the manifest — they are deleted by hand.
Sources
- Kubernetes: Custom Resources and Extend the Kubernetes API with CustomResourceDefinitions — CRDs, structural schema, pruning, defaults, CEL rules and their cost, ratcheting,
selectableFields, subresources, columns; Versions in CustomResourceDefinitions —served,storage, conversion andstoredVersions. - Kubernetes: Common Expression Language in Kubernetes and cel.dev — the CEL language, string functions and
format, cost estimation. - Kubernetes: Controllers and Operator pattern — the reconcile loop and operators.
- Kubernetes: Owners and Dependents, Garbage Collection and Finalizers —
ownerReferences, cascading deletion modes, finalizers. - Kubernetes API Conventions —
specandstatus,observedGeneration, the conditions format; Kubernetes API Concepts —resourceVersion, watch and optimistic locking. - controller-runtime (package reference), The Kubebuilder Book and Operator SDK: Operator Capability Levels — the library, the generator and operator maturity levels.
- CloudNativePG (release v1.30.1): Installation and upgrades, Rolling Updates —
primaryUpdateMethodandprimaryUpdateStrategy, Failover; Barman Cloud Plugin —ObjectStore, cert-manager requirements; PostgreSQL images —minimal,standard,system. - RabbitMQ Cluster Operator: installation and usage, release v2.23.2; Quorum Queues and Memory Threshold and Limit — quorum queues,
total_memory_available_override_valueandvm_memory_high_watermark.
Next — authentication and RBAC: how kube-apiserver learns who came to it, why Kubernetes has no "user" object, how to give an administrator access with a certificate and an application with a ServiceAccount token, and which rights are really dangerous.
Only registered users can leave a comment.
Sign in or register to leave a comment.