Kubernetes for beginners. Part 11: Why are namespaces, labels and annotations needed?

In the previous part we created the quota-test namespace in passing, put a quota on it and deleted it with one command together with everything inside. And since part 7 a Deployment finds its pods by labels, the Service from part 15 will find them the same way, and the DaemonSet from part 8 selects nodes by labels. Time to look at these mechanisms on their own. Namespaces, labels and annotations live in the metadata of every object, which we met in part 3, and they are exactly how objects find each other. At the end of the part the training namespace demo appears, where all further examples will live.

---

Namespace: a scope for names

A namespace is first of all a scope of name uniqueness. Two Deployments named hello cannot exist in one namespace; in different ones they can. The namespace itself is an ordinary API object, Namespace, and most other objects record in metadata.namespace which namespace they belong to. An object's key in etcd (the store where the API server keeps all objects) includes the namespace: /registry/deployments/demo/hello.

Right after installation a cluster has four namespaces:

Namespace	What is in it
default	everything created without specifying a namespace; better not used in production
kube-system	cluster components: kube-proxy, CoreDNS (the cluster DNS server), static control plane pods
kube-public	objects anyone can read, even without authentication; kubeadm keeps the cluster-info ConfigMap there for joining nodes
kube-node-lease	Lease objects, one per node; the kubelet renews its own every 10 seconds with a 40-second lease duration, and that is how the control plane knows the node is alive

The components we installed created their own: calico-system and tigera-operator for Calico. The kube- prefix is reserved for the system, so don't name your namespaces like that. A namespace name must be a DNS label per RFC 1123 — up to 63 characters of lowercase Latin letters, digits and hyphens — because it is part of service DNS names: hello.demo.svc.cluster.local. The API server will not accept a name with a capital letter: kubectl create namespace Demo2 is refused with metadata.name: Invalid value: "Demo2": a lowercase RFC 1123 label must consist of lower case alphanumeric characters or '-', and must start and end with an alphanumeric character.

The same fact leads to a non-obvious danger the documentation warns about. If you create a namespace named like a public top-level domain, for example com or dev, with a service example in it, then pods of any namespace looking up example.com without a trailing dot will reach that service through the DNS search list instead of the internet. That is why the right to create namespaces is given only to trusted users. How the search list works is in part 16.

---

What a namespace isolates and what it does not

A namespace is often perceived as a "separate environment". That is only partly true.

Isolates or restricts per namespace:

• object names;
• permissions: Role and RoleBinding in RBAC (the mechanism controlling access to the API, part 29) act within a single namespace — a team can get full rights in team-a and none in team-b;
• quotas and defaults: ResourceQuota and LimitRange from the previous part;
• network policies: a NetworkPolicy (part 20) describes rules for the pods of its namespace and can select peers by namespace labels;
• pod security level: Pod Security Admission (the built-in check of pod specs for dangerous settings, part 30) is switched on by labels on the namespace;
• access to Secrets, ConfigMaps and ServiceAccounts: a pod can mount a Secret and use a ServiceAccount only from its own namespace;
• owner references: an object can be owned only by an object of its own namespace or by a cluster-scoped object.

Does not isolate:

• the network. By default any pod can open a connection to any other pod in the cluster, whatever namespaces they are in. Without NetworkPolicy namespaces are invisible on the network;
• nodes and the kernel. Pods of different namespaces run on the same nodes, side by side in one Linux kernel; the isolation between them is container isolation, not Kubernetes namespaces;
• node resources — without quotas one namespace can take the whole cluster;
• DNS — a service name from any namespace resolves from any other by its full name;
• cluster-scoped objects — more on them below.

In short: a namespace is a boundary for the API (names, permissions, policies bound to it), but not a security boundary by itself. Multi-tenancy — different teams or customers in one cluster — is built from a namespace plus RBAC, NetworkPolicy, ResourceQuota and Pod Security, and only all of them together give isolation.

A small note on terms: Kubernetes namespaces and Linux namespaces (the kernel's net, pid, mnt and others that container isolation is built on) are different things with similar names. The former separate objects in the API, the latter separate processes on a node.

Objects outside namespaces

Some resources exist at the level of the whole cluster: they describe the cluster itself or things shared by everyone.

Resource	Why it is not namespaced
Namespace	a namespace cannot sit in a namespace
Node	a node is shared by everyone
PersistentVolume, StorageClass, CSIDriver	cluster storage; the PersistentVolumeClaim, however, is namespaced (part 21)
ClusterRole, ClusterRoleBinding	permissions for the whole cluster (part 29)
CustomResourceDefinition	the definition of a new API type (part 28)
PriorityClass, IngressClass, RuntimeClass	shared classes that namespaced objects refer to
ValidatingWebhookConfiguration, MutatingWebhookConfiguration	settings of the API server's request handlers

The full list comes from kubectl api-resources --namespaced=false, and --namespaced=true lists the types that live in namespaces. On such objects metadata.namespace is empty, and kubectl ignores the -n flag.

---

Deleting a namespace and finalizers

kubectl delete namespace demo deletes the namespace with everything in it, but not instantly. It is a good example of how deletion works in Kubernetes in general.

A finalizer is a string in an object's list of finalizers meaning "before the object disappears, someone has to finish some work". While the list is not empty, deletion only marks the object: the API server sets metadata.deletionTimestamp, and the object stays visible. The controller that set the finalizer sees the mark, does its work and removes its string. When no strings are left, the API server removes the object from etcd. We saw this in earlier parts: batch.kubernetes.io/job-tracking on Job pods; volume claims have kubernetes.io/pvc-protection, which prevents deleting a PVC while a pod uses it.

A namespace has a special finalizer — kubernetes in the spec.finalizers field. Deletion goes like this:

 kubectl delete namespace demo
        │
        ▼
 API server: deletionTimestamp, status.phase = Terminating
 no new objects can be created in the namespace
        │
        ▼
 namespace controller (kube-controller-manager):
   1. asks the API which namespaced object types exist at all
   2. deletes all pods and waits until they are gone
   3. deletes all other objects of all types
   4. waits until they are gone too — with their finalizers
   5. removes "kubernetes" from spec.finalizers
        │
        ▼
 API server deletes the Namespace

Step 2 is a 1.34 change (ordered namespace deletion became stable): pods go first, then everything else. Before that the order was arbitrary, and a pod could keep running for a while after its NetworkPolicy had already been deleted — that is, unprotected.

The progress of deletion is reflected in the namespace's status.conditions, and if a namespace is "stuck" in Terminating, the answer is almost always there:

Condition	What it means when True
NamespaceDeletionDiscoveryFailure	the controller could not get the list of API types — usually because of an unavailable aggregated API (one the API server does not serve itself but proxies to a third-party server), for example a crashed metrics-server, the component that provides pod usage metrics (part 32)
NamespaceDeletionGroupVersionParsingFailure	some API group could not be parsed
NamespaceDeletionContentFailure	an error while deleting objects
NamespaceContentRemaining	objects are still left — listed by type
NamespaceFinalizersRemaining	the remaining objects have finalizers — listed by name

The most common cause of a stuck namespace is an object with a finalizer nobody is left to remove: an operator (a controller for custom object types, part 28) was deleted before its objects, and its finalizers stayed on them forever. The right fix is to bring the controller back, let it clean up after itself and only then delete. If the controller is never coming back, the finalizer is removed by hand from the specific object, understanding what work it was supposed to do (for example, delete an external resource — a disk or a load balancer in the cloud — which you will now have to delete yourself).

The internet often advises "curing" a stuck namespace by clearing its spec.finalizers through the /finalize subresource. This removes the namespace from the API but leaves in etcd every object the controller did not manage to delete — without a namespace, invisible to kubectl and useless. If a namespace with the same name is created again later, they "come back to life" in it. This trick is a last resort, not a way to delete.

---

Labels

A label is a key-value pair in metadata.labels that describes an object so that it can be selected: "all pods of the hello application", "all nodes with SSDs", "everything belonging to the canary release".

Key syntax: an optional prefix and a name separated by /. The name is up to 63 characters of letters, digits, -, _, ., starting and ending with a letter or digit. The prefix is a DNS subdomain of up to 253 characters: app.kubernetes.io/name, example.com/team. The kubernetes.io/ and k8s.io/ prefixes are reserved for Kubernetes components; the API server does not check this (kubectl label pod … kubernetes.io/foo=bar goes through), so it is a convention, not a prohibition. A key without a prefix is considered private to the user. The value is up to 63 characters by the same rules and may be empty.

The length and character limits are not accidental: labels are meant for selection, not for storing data, and the server has to compare them quickly with a selector on every request. Filtering happens on the API server side: kubectl get pods -l app=web receives only matching pods from the server, and controllers subscribe to changes the same way, with a selector, without pulling the whole cluster.

Selectors

A label selector is a condition on labels. There are two kinds of requirements:

• equality-based: app=web (or app==web), tier!=frontend;
• set-based: env in (prod,staging), env notin (dev), canary (the key exists), !canary (the key does not exist).

Comma-separated requirements are combined with a logical AND; there is no OR between requirements — in replaces it. A subtlety: tier!=frontend and env notin (dev) also select objects that do not have that label at all.

In manifests a selector is written in two ways:

selector:
  matchLabels:                   # equality, pair by pair
    app.kubernetes.io/name: hello
  matchExpressions:              # sets
  - key: environment
    operator: In                 # In, NotIn, Exists, DoesNotExist
    values: [prod, staging]

All the conditions of matchLabels and matchExpressions are combined with AND. This form is understood by Deployment, ReplicaSet, StatefulSet, DaemonSet, Job, NetworkPolicy, PodDisruptionBudget. Service and the old ReplicationController have a plain map as a selector, equality only.

The link between controllers and pods rests on selectors, and there are two traps here:

• the selector of a Deployment, ReplicaSet, StatefulSet and DaemonSet is immutable after creation (in apps/v1): an attempt to change it ends with spec.selector: Invalid value: …: field is immutable. To change the selector, the object is recreated. So labels that will change — the version, for example — must not go into a selector;
• selectors must not overlap. If two Deployments select the same pods, their ReplicaSets will consider each other's pods their own and fight: one deletes the "extra" ones, the other creates the "missing" ones. Kubernetes does not forbid this. A ReplicaSet is protected from some of these cases by the pod-template-hash label, which the Deployment adds both to the template and to the selector, but only care protects against overlap between different applications.

Recommended labels

The Kubernetes project proposes a common set of labels with the app.kubernetes.io/ prefix so that tools — kubectl, Helm, monitoring dashboards — understand objects the same way:

Label	Meaning	Example
app.kubernetes.io/name	application name	hello
app.kubernetes.io/instance	a specific installation instance	hello-demo
app.kubernetes.io/version	application version	1.0
app.kubernetes.io/component	role in the architecture	web, database
app.kubernetes.io/part-of	the larger application this is part of	shop
app.kubernetes.io/managed-by	what manages the object	kubectl, Helm

Put them on all objects of an application — Deployment, Service, ConfigMap, pod template — but take only the unchanging ones into a selector: name and instance. version in a selector is that very trap: when version 1.1 is released, the Deployment selector would have to change, and it is immutable. The Helm package manager (part 27) sets these labels itself, with managed-by: Helm.

Kubernetes also sets well-known labels itself, and the scheduler and controllers rely on them: on nodes kubernetes.io/hostname, kubernetes.io/os, kubernetes.io/arch, node-role.kubernetes.io/control-plane, in clouds topology.kubernetes.io/zone; on namespaces kubernetes.io/metadata.name with the namespace name (the API server sets it and silently restores it if the label is changed or removed, which makes it convenient for selecting a namespace in a NetworkPolicy); on controller pods pod-template-hash, controller-revision-hash, batch.kubernetes.io/job-name, which we saw in earlier parts.

---

Annotations

An annotation is also a key-value pair, in metadata.annotations, with the same key syntax but a different purpose: to hold data about an object that nobody selects by. The value is an arbitrary string, even multi-line JSON; the limit is 256 KiB for all annotations of an object together, keys and values. The API server rejects a request that exceeds it: The Deployment "hello" is invalid: metadata.annotations: Too long: may not be more than 262144 bytes.

Annotations are read by tools and controllers:

• kubectl.kubernetes.io/last-applied-configuration — a copy of the manifest that client-side kubectl apply keeps to work out at the next apply which fields you removed (server-side apply from part 3 keeps this differently, in managedFields);
• deployment.kubernetes.io/revision — the revision number on a ReplicaSet;
• kubernetes.io/change-cause — the reason for a change shown by kubectl rollout history;
• batch.kubernetes.io/cronjob-scheduled-timestamp — the scheduled time of a Job from part 9;
• annotations of external controllers: Reloader (part 12) restarts a Deployment when its configuration changes, driven by an annotation on it, and cert-manager (part 19) issues a certificate driven by an annotation on an Ingress.

How to choose between a label and an annotation:

Question	Label	Annotation
Do you need to select objects by this value?	yes	no
Value length and format	up to 63 characters, restricted set	any string, up to 256 KiB in total
Who usually reads it	controller selectors, kubectl -l	tools, controllers, people
Examples	application, component, environment, team	repository link, commit hash, on-call contact, controller settings

The commit hash an image was built from is an annotation: there is no need to select by it, but it is useful to see while investigating. The owning team is a label: reports and quotas are built on it.

---

Field selectors

Labels are what you have hung on an object. Sometimes you need to select by a field of the object itself: all pods on node k8s-node-2, all pods that are not running, all events of a particular object. That is what a field selector is for:

$ kubectl get pods -A --field-selector spec.nodeName=k8s-node-2
$ kubectl get pods -A --field-selector status.phase!=Running,status.phase!=Succeeded
$ kubectl get events --field-selector involvedObject.name=web-0
$ kubectl get secrets -A --field-selector type=kubernetes.io/tls

Only =, ==, != are supported, and only certain fields: for all types metadata.name and metadata.namespace, then per type. For pods — spec.nodeName, spec.restartPolicy, spec.schedulerName, spec.serviceAccountName, spec.hostNetwork, status.phase, status.podIP, status.podIPs, status.nominatedNodeName; for events — the involvedObject.* fields, reason, reportingComponent, source, type; for Secrets — type; for Services — spec.type and spec.clusterIP; for nodes — spec.unschedulable; for namespaces — status.phase; for ReplicaSets — status.replicas, for Jobs — status.successful, for CertificateSigningRequests — spec.signerName. A field selector combines with a label selector: --field-selector status.phase=Running -l app.kubernetes.io/instance=debug. The API server rejects an unsupported field, and the error text depends on the type: for ConfigMaps and other types that have only the common fields it is "data.a" is not a known field selector: only "metadata.name", "metadata.namespace", for pods field label not supported: spec.containers. There are no in and notin operators here: status.phase in (Running) gets invalid selector. It is with the selector spec.nodeName=<its node> that the kubelet subscribes to the pods of its node without receiving the whole cluster.

---

kubectl and labels

$ kubectl label pod web-0 tier=backend               # add
$ kubectl label pod web-0 tier=api --overwrite       # change
$ kubectl label pod web-0 tier-                      # remove
$ kubectl annotate deployment hello example.com/owner='team-a@example.com'
$ kubectl get pods --show-labels                     # all labels in a column
$ kubectl get pods -L app.kubernetes.io/version      # a label's value in its own column
$ kubectl get pods -l 'app.kubernetes.io/name=hello,environment in (prod,staging)'
$ kubectl get all -A -l app.kubernetes.io/part-of=shop

Changing a label on a pod owned by a ReplicaSet is a way to "take" the pod out of its control for investigation: remove the label the ReplicaSet selects it by, and it creates a replacement, while the old pod keeps living on its own, without traffic from the Service (if the Service selects by the same label), and can be examined calmly.

---

Context and the default namespace

Every kubectl command without -n works in the default namespace, and that is stored not in the cluster but in your kubeconfig — the ~/.kube/config file we got in part 4. It has three lists and a pointer:

apiVersion: v1
kind: Config
clusters:                 # where to connect: API server address and CA
- name: kubernetes
  cluster:
    server: https://k8s-api.example.com:6443
    certificate-authority-data: LS0t...
users:                    # what to connect with: certificate, token
- name: kubernetes-admin
  user:
    client-certificate-data: LS0t...
    client-key-data: LS0t...
contexts:                 # context = cluster + user + namespace
- name: kubernetes-admin@kubernetes
  context:
    cluster: kubernetes
    user: kubernetes-admin
    namespace: demo
current-context: kubernetes-admin@kubernetes

A context ties together a cluster, credentials and a default namespace; current-context says which one is active. Commands:

$ kubectl config get-contexts
$ kubectl config set-context --current --namespace=demo
$ kubectl config view --minify            # the current context only
$ kubectl config use-context <name>

For those who switch often there are the small utilities kubectx and kubens: the first switches contexts, the second the namespace of the current context.

The namespace in a manifest and the namespace in the context interact like this: if metadata.namespace is set in the file, the object is created there whatever the context says; if not, in the namespace from -n or from the context. If the file names one namespace and the -n flag another, kubectl refuses with the namespace from the provided object "demo" does not match the namespace "default". You must pass '--namespace=demo' to perform this operation. The practice is to choose one style and stick to it: either the namespace is written in every manifest (more reliable, especially when deploying from Git), or it is written nowhere and given at apply time (that is what Kustomize does — the manifest assembly tool built into kubectl, part 26). Relying on the "current context" in scripts is dangerous: one day it will be the wrong one.

As for the credentials in this file: kubernetes-admin from kubeadm's admin.conf is a full cluster administrator, and keeping it on a laptop as the only way of access is bad practice. How to give people separate accounts with restricted rights is in part 29.

---

The training namespace demo

From this part on, all examples live in the demo namespace. Let's create it with a manifest — with labels and right away with the resource defaults from the previous part:

apiVersion: v1
kind: Namespace
metadata:
  name: demo
  labels:
    app.kubernetes.io/part-of: k8s-for-beginners
    environment: lab
---
apiVersion: v1
kind: LimitRange
metadata:
  name: defaults
  namespace: demo
spec:
  limits:
  - type: Container
    defaultRequest:
      cpu: 100m
      memory: 128Mi
    default:
      memory: 128Mi

And move hello from part 7 there, now with the recommended labels:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello
  namespace: demo
  labels: &labels
    app.kubernetes.io/name: hello
    app.kubernetes.io/instance: hello
    app.kubernetes.io/version: "1.0"
    app.kubernetes.io/component: web
    app.kubernetes.io/part-of: k8s-for-beginners
    app.kubernetes.io/managed-by: kubectl
  annotations:
    example.com/source: https://git.example.com/demo/hello
spec:
  replicas: 3
  selector:
    matchLabels:                       # unchanging labels only
      app.kubernetes.io/name: hello
      app.kubernetes.io/instance: hello
  template:
    metadata:
      labels: *labels
    spec:
      containers:
      - name: hello
        image: ghcr.io/example/hello:1.0
        ports:
        - name: http
          containerPort: 8080
        resources:
          requests:
            cpu: 100m
            memory: 64Mi
          limits:
            memory: 64Mi

&labels and *labels are a YAML anchor and alias: the same set of labels is inserted both into the object and into the pod template without repetition. kubectl expands them before sending to the server. The ghcr-pull Secret for the registry from part 5 has to be created in the new namespace too (with the same kubectl create secret docker-registry, but with -n demo) and attached to the default ServiceAccount of the demo namespace: a Secret from default is not available to a pod in demo — that is one of the namespace boundaries.

---

Hands-on

Namespaces and the label the cluster itself puts on them:

$ kubectl get namespaces --show-labels
NAME              STATUS   AGE    LABELS
calico-system     Active   4d3h   app.kubernetes.io/component=Installation.operator.tigera.io,app.kubernetes.io/instance=default,app.kubernetes.io/managed-by=tigera-operator,app.kubernetes.io/name=calico-system,app.kubernetes.io/part-of=Calico,k8s-app=calico-system,kubernetes.io/metadata.name=calico-system,name=calico-system,pod-security.kubernetes.io/enforce-version=latest,pod-security.kubernetes.io/enforce=privileged
default           Active   4d3h   kubernetes.io/metadata.name=default
kube-node-lease   Active   4d3h   kubernetes.io/metadata.name=kube-node-lease
kube-public       Active   4d3h   kubernetes.io/metadata.name=kube-public
kube-system       Active   4d3h   kubernetes.io/metadata.name=kube-system
tigera-operator   Active   4d3h   kubernetes.io/metadata.name=tigera-operator,name=tigera-operator,pod-security.kubernetes.io/enforce=privileged

$ kubectl apply -f demo-namespace.yaml
namespace/demo created
limitrange/defaults created

$ kubectl config set-context --current --namespace=demo
Context "kubernetes-admin@kubernetes" modified.

$ kubectl config get-contexts
CURRENT   NAME                          CLUSTER      AUTHINFO           NAMESPACE
*         kubernetes-admin@kubernetes   kubernetes   kubernetes-admin   demo

calico-system and tigera-operator carry Pod Security labels: the pods of Calico and its operator run privileged in these namespaces, and that is declared explicitly. And the Tigera operator (the controller that installs Calico) has also put the recommended app.kubernetes.io/* labels on its calico-system — with part-of=Calico and managed-by=tigera-operator. The namespace object itself:

$ kubectl get namespace demo -o yaml
apiVersion: v1
kind: Namespace
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Namespace","metadata":{"annotations":{},"labels":{"app.kubernetes.io/part-of":"k8s-for-beginners","environment":"lab"},"name":"demo"}}
  creationTimestamp: "2026-10-28T13:36:08Z"
  labels:
    app.kubernetes.io/part-of: k8s-for-beginners
    environment: lab
    kubernetes.io/metadata.name: demo
  name: demo
  resourceVersion: "762975"
  uid: 55406057-7dd7-4ac9-a424-55bdf02d445f
spec:
  finalizers:
  - kubernetes
status:
  phase: Active

The kubernetes finalizer is in place from the moment of creation, the kubernetes.io/metadata.name label was added by the API server, and the last-applied-configuration annotation by kubectl apply. The API server always writes creationTimestamp in UTC. The namespace name label cannot be changed — the API server silently restores it:

$ kubectl label ns demo kubernetes.io/metadata.name=other --overwrite
namespace/demo labeled

$ kubectl get ns demo -L kubernetes.io/metadata.name
NAME   STATUS   AGE     METADATA.NAME
demo   Active   4m21s   demo

Which types live outside namespaces:

$ kubectl api-resources --namespaced=false -o name | head -16
componentstatuses
namespaces
nodes
persistentvolumes
mutatingadmissionpolicies.admissionregistration.k8s.io
mutatingadmissionpolicybindings.admissionregistration.k8s.io
mutatingwebhookconfigurations.admissionregistration.k8s.io
validatingadmissionpolicies.admissionregistration.k8s.io
validatingadmissionpolicybindings.admissionregistration.k8s.io
validatingwebhookconfigurations.admissionregistration.k8s.io
customresourcedefinitions.apiextensions.k8s.io
apiservices.apiregistration.k8s.io
selfsubjectreviews.authentication.k8s.io
tokenreviews.authentication.k8s.io
selfsubjectaccessreviews.authorization.k8s.io
selfsubjectrulesreviews.authorization.k8s.io

Further down this list are certificatesigningrequests, storageclasses, priorityclasses, clusterroles and Calico's own types — bgppeers, ippools and others: network settings are shared by the whole cluster. Calico's types appear twice: as CRDs in the crd.projectcalico.org group and in the projectcalico.org group served by calico-apiserver — the very kind of aggregated API that can make namespace deletion hang. In total our cluster has 79 types outside namespaces and 41 in namespaces.

A pod from the private registry will not start in the new namespace until it has its own ghcr-pull. Having created the Secret with -n demo, we attach it to the default ServiceAccount — the context already points at demo:

$ kubectl patch serviceaccount default -p '{"imagePullSecrets":[{"name":"ghcr-pull"}]}'
serviceaccount/default patched

Now hello in demo and selections by labels:

$ kubectl apply -f hello.yaml
deployment.apps/hello created

$ kubectl get pods -L app.kubernetes.io/version,app.kubernetes.io/component
NAME                     READY   STATUS    RESTARTS   AGE   VERSION   COMPONENT
hello-6d5c6db6b4-b5kn2   1/1     Running   0          10s   1.0       web
hello-6d5c6db6b4-wc775   1/1     Running   0          10s   1.0       web
hello-6d5c6db6b4-zhgzw   1/1     Running   0          10s   1.0       web

$ kubectl get pods -l 'app.kubernetes.io/name=hello,app.kubernetes.io/version notin (0.9)' -o name
pod/hello-6d5c6db6b4-b5kn2
pod/hello-6d5c6db6b4-wc775
pod/hello-6d5c6db6b4-zhgzw

$ kubectl get pods --field-selector spec.nodeName=k8s-node-2 -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP              NODE         NOMINATED NODE   READINESS GATES
hello-6d5c6db6b4-b5kn2   1/1     Running   0          18s   172.20.172.20   k8s-node-2   <none>           <none>
hello-6d5c6db6b4-wc775   1/1     Running   0          18s   172.20.172.19   k8s-node-2   <none>           <none>

The scheduler put two of the three pods on k8s-node-2 and the third, zhgzw, on k8s-node-1.

Let's take one pod out of the ReplicaSet's control by changing a label from the selector:

$ kubectl label pod hello-6d5c6db6b4-zhgzw app.kubernetes.io/instance=debug --overwrite
pod/hello-6d5c6db6b4-zhgzw labeled

$ kubectl get pods -L app.kubernetes.io/instance
NAME                     READY   STATUS    RESTARTS   AGE   INSTANCE
hello-6d5c6db6b4-b5kn2   1/1     Running   0          31s   hello
hello-6d5c6db6b4-fnnj2   1/1     Running   0          3s    hello
hello-6d5c6db6b4-wc775   1/1     Running   0          31s   hello
hello-6d5c6db6b4-zhgzw   1/1     Running   0          31s   debug

$ kubectl get pod hello-6d5c6db6b4-zhgzw -o jsonpath='{.metadata.ownerReferences}{"\n"}'

The ReplicaSet stopped seeing the pod, created the replacement fnnj2 and removed the owner reference from the detached pod — it is now a standalone pod that nobody will restart or delete. After the investigation it is deleted by hand: kubectl delete pod hello-6d5c6db6b4-zhgzw.

Without --overwrite, kubectl does not change an existing label or annotation, and invalid keys and values are rejected before anything is saved: the key is checked by the API server, the value by kubectl itself, before the request is even sent:

$ kubectl label pod hello-6d5c6db6b4-zhgzw app.kubernetes.io/instance=debug2
error: 'app.kubernetes.io/instance' already has a value (debug), and --overwrite is false

$ kubectl annotate deployment hello example.com/owner='team-a@example.com'
deployment.apps/hello annotated
$ kubectl annotate deployment hello example.com/owner=x
error: --overwrite is false but found the following declared annotation(s): 'example.com/owner' already has a value (team-a@example.com)
$ kubectl annotate deployment hello example.com/owner-
deployment.apps/hello annotated

$ kubectl label pod hello-6d5c6db6b4-zhgzw 'Bad Key=1'
The Pod "hello-6d5c6db6b4-zhgzw" is invalid: metadata.labels: Invalid value: "Bad Key": name part must consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyName',  or 'my.name',  or '123-abc', regex used for validation is '([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9]')

$ kubectl label pod hello-6d5c6db6b4-zhgzw example_com/team=a
The Pod "hello-6d5c6db6b4-zhgzw" is invalid: metadata.labels: Invalid value: "example_com/team": prefix part a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')

$ kubectl label pod hello-6d5c6db6b4-zhgzw tier=-bad
error: invalid label value: "tier=-bad": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')

$ kubectl label pod hello-6d5c6db6b4-zhgzw tier=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
error: invalid label value: "tier=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa": must be no more than 63 bytes

Note that kubectl reports removing an annotation with the same word annotated, while removing a label is unlabeled. The length limit is counted in bytes, not characters. And the Deployment selector really cannot be changed — not even by adding a label that is already on all its pods:

$ kubectl patch deployment hello --type=merge -p '{"spec":{"selector":{"matchLabels":{"app.kubernetes.io/version":"1.0"}}}}'
The Deployment "hello" is invalid: spec.selector: Invalid value: {"matchLabels":{"app.kubernetes.io/instance":"hello","app.kubernetes.io/name":"hello","app.kubernetes.io/version":"1.0"}}: field is immutable

A namespace that won't go away. Let's create trash with a ConfigMap carrying a finalizer nobody will come for:

$ kubectl create namespace trash
namespace/trash created
$ kubectl -n trash create configmap stuck --from-literal=a=b
configmap/stuck created
$ kubectl -n trash patch configmap stuck --type=merge -p '{"metadata":{"finalizers":["example.com/hold"]}}'
configmap/stuck patched

$ kubectl delete namespace trash --wait=false
namespace "trash" deleted

$ kubectl get namespace trash
NAME    STATUS        AGE
trash   Terminating   32s

$ kubectl get namespace trash -o jsonpath='{range .status.conditions[*]}{.type}={.status}: {.message}{"\n"}{end}'
NamespaceDeletionDiscoveryFailure=False: All resources successfully discovered
NamespaceDeletionGroupVersionParsingFailure=False: All legacy kube types successfully parsed
NamespaceDeletionContentFailure=False: All content successfully deleted, may be waiting on finalization
NamespaceContentRemaining=True: Some resources are remaining: configmaps. has 1 resource instances
NamespaceFinalizersRemaining=True: Some content in the namespace has finalizers remaining: example.com/hold in 1 resource instances

$ kubectl -n trash get configmap stuck -o jsonpath='{.metadata.deletionTimestamp}{"\n"}'
2026-10-28T13:38:18Z

$ kubectl -n trash create configmap new --from-literal=a=b
error: failed to create configmap: configmaps "new" is forbidden: unable to create new content in namespace trash because it is being terminated

Everything follows the scheme: the controller deleted what it could (including the kube-root-ca.crt ConfigMap present in every namespace), the stuck ConfigMap is marked for deletion but holds on to its finalizer, the namespace conditions name the culprit directly, and nothing new can be created in a namespace being deleted. Remove the finalizer, and the namespace disappears by itself, though not in the same second: the controller re-checks a stuck namespace after a pause, and on the lab it was gone within a few seconds:

$ kubectl -n trash patch configmap stuck --type=merge -p '{"metadata":{"finalizers":null}}'
configmap/stuck patched

$ kubectl get namespace trash
NAME    STATUS        AGE
trash   Terminating   47s

$ kubectl get namespace trash
Error from server (NotFound): namespaces "trash" not found

---

Sources

• Namespaces — the initial namespaces, DNS names and the danger of names that match top-level domains, objects outside namespaces.
• Share a Cluster with Namespaces — the Active and Terminating phases, deleting a namespace and the finalizers field.
• KEP-5080: Ordered Namespace Deletion — why pods are deleted before the other objects.
• Finalizers — how finalizers delay the deletion of an object.
• Object Names and IDs — naming rules: RFC 1123 DNS labels and subdomains.
• Labels and Selectors — label syntax, equality-based and set-based selectors, matchLabels and matchExpressions.
• Recommended Labels — the app.kubernetes.io/* labels.
• Annotations — key syntax and the 256 KiB limit.
• Field Selectors — supported fields per type and the error text.
• Well-Known Labels, Annotations and Taints — a reference of the labels and annotations Kubernetes sets itself.
• Organizing Cluster Access Using kubeconfig Files — clusters, users and contexts in kubeconfig.

---

Labels, annotations and namespaces are metadata. But an application also has its own configuration: the database address, parameters, passwords. So far we have passed them as variables right in the manifest, and in part 9 we took the database password from a Secret without explaining what it is.

Next — ConfigMap and Secret: how to pass configuration into a pod as variables and files, what happens when it changes and why variables are not updated, how to restart pods on a change with Reloader, why base64 is not encryption, who can read Secrets and how to encrypt secrets in etcd.