Kubernetes for beginners. Part 32: How do metrics and autoscaling work?

In the previous part we examined failures through events and logs — through whatever was left in the cluster at the moment of asking. Today is about numbers collected in advance: how much CPU and memory pods really consume, how Kubernetes itself changes the number of replicas based on those numbers, and where to keep the metrics history so that a week later you can answer what happened at night. We start with the built-in pipeline — the kubelet, metrics-server, kubectl top and the HorizontalPodAutoscaler — then install Prometheus with Grafana and Alertmanager using one Helm chart (part 27), teach it to scrape metrics from hello and write the first alert rules. Requests and limits from part 10 are needed constantly here: autoscaling calculates its percentages from them.

The demo namespace stays as part 30 left it: under the restricted level, with hello running as UID 10001 without capabilities. This will affect where we generate load from.

---

Two metrics pipelines

Metrics travel along two independent paths in a cluster, and mixing them up is a common mistake:

 Resource pipeline (built in)                    Full pipeline (monitoring)
 ────────────────────────────                    ──────────────────────────

 kubelet on every node                           exporters: kubelet, node-exporter,
   /metrics/resource                             kube-state-metrics, application /metrics
        │  every 15 s                                    │  every 30 s (pull)
        ▼                                                ▼
 metrics-server                                  Prometheus
   last two points in memory                       TSDB on disk, weeks of history
        │                                                │
        ▼                                                ├──► Grafana: graphs
 kube-apiserver                                          ├──► rules → Alertmanager
   /apis/metrics.k8s.io (aggregation)                    └──► adapter → custom/external
        │                                                     metrics API (optional)
        ├──► kubectl top
        ├──► HorizontalPodAutoscaler
        └──► VerticalPodAutoscaler

The resource pipeline is minimal: only CPU and memory of pods and nodes, only "now", no history. Kubernetes itself needs it for autoscaling, and a person needs it for kubectl top. The full pipeline is a monitoring system: thousands of metrics, on-disk storage, queries, graphs and alerts. Neither replaces the other: the HPA does not need Prometheus to scale on CPU, and metrics-server is useless for the question "what happened yesterday".

---

Where the kubelet gets its numbers

The source of both pipelines is the kubelet. It gets container statistics from containerd over CRI and from the cgroups it created for the pods itself (part 10): cpu.stat (usage_usec — accumulated CPU time) and memory.current, memory.stat (the working set is computed from them — used memory minus the inactive file cache the kernel can evict). Built into the kubelet is cAdvisor, a Google library that walks the cgroups and collects detailed statistics from them.

The kubelet exposes this on its port 10250 at several paths:

Path	What is there	Who reads it
/metrics/resource	a minimal set in Prometheus format: CPU and working set per node, pod and container	metrics-server
/stats/summary	the Summary API — the same and more (network, file systems, volumes) in JSON	old metrics-server versions, helper utilities
/metrics/cadvisor	everything cAdvisor collected: hundreds of per-container metrics	Prometheus
/metrics/probes	results of liveness, readiness and startup probes	Prometheus
/metrics	metrics of the kubelet itself: operation timings, PLEG, volumes (kubelet_volume_stats_*)	Prometheus

You can look through the API server proxy without logging in to the node:

$ kubectl get --raw /api/v1/nodes/k8s-node-2/proxy/metrics/resource | grep -E 'hello-74d65597f7-5lfft|^node_'
container_cpu_usage_seconds_total{container="hello",namespace="demo",pod="hello-74d65597f7-5lfft"} 0.396541 1793735197827
container_memory_working_set_bytes{container="hello",namespace="demo",pod="hello-74d65597f7-5lfft"} 6.623232e+06 1793735197827
container_start_time_seconds{container="hello",namespace="demo",pod="hello-74d65597f7-5lfft"} 1.7937347742225347e+09
container_swap_limit_bytes{container="hello",namespace="demo",pod="hello-74d65597f7-5lfft"} 0 1793735197827
container_swap_usage_bytes{container="hello",namespace="demo",pod="hello-74d65597f7-5lfft"} 0 1793735197827
node_cpu_usage_seconds_total 13472.803 1793735195256
node_memory_working_set_bytes 1.55904e+09 1793735195256
node_swap_usage_bytes 0 1793735195256
pod_cpu_usage_seconds_total{namespace="demo",pod="hello-74d65597f7-5lfft"} 0.394523 1793735193027
pod_memory_working_set_bytes{namespace="demo",pod="hello-74d65597f7-5lfft"} 6.606848e+06 1793735193027
pod_swap_usage_bytes{namespace="demo",pod="hello-74d65597f7-5lfft"} 0 1793735193027

The last number on a line is the measurement timestamp in milliseconds. CPU here is a counter: seconds of processor time since the container started. Consumption "in cores" is only obtained as the difference between two measurements divided by the interval between them. The next request showed 0.444832 with the timestamp 1793735209621, that is, 11.8 seconds later: (0.444832 − 0.396541) / 11.8 ≈ 0.004 cores, or 4m. Memory is an instantaneous value in bytes: 6.6 MB here.

---

metrics-server

metrics-server is a small component from kubernetes-sigs. Every 15 seconds it scrapes /metrics/resource from all kubelets, keeps the last two points per object in memory, computes CPU from them and serves the result as the metrics.k8s.io/v1beta1 API. A separate APIService object registers this API in the kube-apiserver aggregation layer: the API server does not handle requests to /apis/metrics.k8s.io/... itself but proxies them to the metrics-server service.

What metrics-server does not do, in the words of its own documentation:

• it does not keep history — only the latest value; restarting the metrics-server pod loses nothing, because there is nothing to lose;
• it is not for monitoring — do not connect Prometheus or graphs to it, the kubelet's /metrics/resource and /metrics/cadvisor exist for that;
• it is not accurate in detail — the CPU value is averaged over a window and short spikes are smoothed out;
• it knows no other metrics besides CPU and memory.

Versions 0.9.x support Kubernetes 1.34 and newer. It is installed with the official manifest at an explicit version:

$ kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.9.1/components.yaml
serviceaccount/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
service/metrics-server created
deployment.apps/metrics-server created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created

The Deployment in the manifest has the arguments --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname (reach the kubelet by node IP), --kubelet-use-node-status-port (take the port from the node status) and --metric-resolution=15s. The --kubelet-insecure-tls flag is not there: metrics-server verifies the certificate the kubelet presents on port 10250 against the cluster CA. On our cluster the check passes right away: back in part 4 we enabled serverTLSBootstrap: true and approved the kubelet-serving CSRs by hand, so every kubelet answers with a certificate signed by the cluster CA. Half a minute later metrics-server is ready:

$ kubectl get apiservice v1beta1.metrics.k8s.io
NAME                     SERVICE                      AVAILABLE   AGE
v1beta1.metrics.k8s.io   kube-system/metrics-server   True        32s

Why not --kubelet-insecure-tls

On a kubeadm cluster installed without serverTLSBootstrap, the kubelet generates a self-signed certificate for port 10250 by itself: the issuer is the kubelet itself (CN=k8s-node-2-ca@1793733646), and the SAN holds only the node name, with no IP address. Such a certificate does not pass verification against the cluster CA. You can see this on one node by temporarily removing the field from /var/lib/kubelet/config.yaml: an error appears in the metrics-server log, and the node loses its numbers in kubectl top.

$ kubectl logs -n kube-system deploy/metrics-server | grep -m1 'Failed to scrape'
E1103 19:20:50.298744       1 scraper.go:149] "Failed to scrape node" err="Get \"https://192.168.10.21:10250/metrics/resource\": tls: failed to verify certificate: x509: cannot validate certificate for 192.168.10.21 because it doesn't contain any IP SANs" node="k8s-node-2"

$ kubectl top nodes
NAME           CPU(cores)   CPU(%)      MEMORY(bytes)   MEMORY(%)
k8s-master-1   104m         5%          1927Mi          58%
k8s-node-1     40m          2%          735Mi           22%
k8s-node-2     <unknown>    <unknown>   <unknown>       <unknown>

The APIService stays True meanwhile: the other nodes respond. The first answer in a search is to add the --kubelet-insecure-tls flag to metrics-server. It turns off verification of the kubelet certificate completely. Why that is bad: on every scrape metrics-server presents the token of its ServiceAccount to the kubelet, and without certificate verification it will present it to anyone who manages to intercept traffic in the node network. The metrics-server documentation explicitly calls this flag an option for testing. The right way is to give the kubelets certificates signed by the cluster CA, as is done on our cluster.

Signed kubelet certificates

With serverTLSBootstrap: true the kubelet requests its serving certificate through the certificates.k8s.io API, the same way it requested its client certificate when joining the cluster: it creates a CertificateSigningRequest (CSR) object with the kubernetes.io/kubelet-serving signer and lists its names and addresses in it. kube-controller-manager approves client CSRs of nodes automatically, but not serving ones: Kubernetes cannot verify on its own that a node is entitled to a certificate for the requested names and addresses. That is why in part 4 we looked at what exactly each node was asking for before approving. After approval kube-controller-manager signs the certificate with the cluster CA, and the kubelet picks it up into /var/lib/kubelet/pki/kubelet-server-current.pem and answers with it.

If a cluster is already running without this field, it is enabled in two places. The kubelet-config ConfigMap is the source kubeadm takes configuration from for new nodes and during upgrades:

$ kubectl edit cm kubelet-config -n kube-system
data:
  kubelet: |
    apiVersion: kubelet.config.k8s.io/v1beta1
    kind: KubeletConfiguration
    serverTLSBootstrap: true
    ...

On every existing node the same field is appended to /var/lib/kubelet/config.yaml, the kubelet is restarted and the resulting CSRs are approved the same way as during cluster installation:

user@k8s-node-1:~$ echo 'serverTLSBootstrap: true' | sudo tee -a /var/lib/kubelet/config.yaml
user@k8s-node-1:~$ sudo systemctl restart kubelet

The certificate lives for a year and is renewed along the same path: the kubelet creates a new CSR in advance, and it has to be approved again — otherwise, when the old certificate expires, metrics-server stops scraping that node. So as not to forget, approval is automated with a controller such as kubelet-csr-approver, which approves only requests whose names and IPs match the node. A controller that approves everything is worse than manual approval: anyone holding a node client certificate gets a serving certificate for any name.

The same signed certificates let you close a second hole of the same kind: by default kube-apiserver in kubeadm does not verify the kubelet certificate when it proxies kubectl logs and exec. The experiment above shows this too: kubectl get --raw /api/v1/nodes/k8s-node-2/proxy/... keeps working with a self-signed certificate. Since the kubelets answer with certificates from the cluster CA, the API server can be given --kubelet-certificate-authority with that CA. The reverse direction, from the API server to metrics-server, is marked insecureSkipTLSVerify: true on the APIService in the manifest; a tidy replacement is to issue a certificate for metrics-server with cert-manager from part 19 and set caBundle.

kubectl top

$ kubectl top nodes
NAME           CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)
k8s-master-1   145m         7%       1853Mi          56%
k8s-node-1     146m         7%       728Mi           22%
k8s-node-2     65m          3%       1492Mi          45%

$ kubectl top pods -n demo --containers
POD                      NAME    CPU(cores)   MEMORY(bytes)
hello-74d65597f7-5lfft   hello   5m           6Mi
hello-74d65597f7-8vkl6   hello   4m           6Mi
hello-74d65597f7-qqdxp   hello   4m           6Mi

Node percentages are calculated from allocatable — what the kubelet is ready to give to pods, not all the memory of the machine. Memory is the working set, so the numbers will not match free on the node: free reads /proc/meminfo, while the working set comes from the cgroup and includes the active part of the file cache. It is the working set that is compared with the limit on OOM and with the kubelet eviction thresholds, so that is what to look at when deciding on requests and limits.

---

HorizontalPodAutoscaler

HorizontalPodAutoscaler (HPA) is an autoscaling/v2 object that changes replicas on a Deployment, a StatefulSet or any resource with a scale subresource. Its controller lives in kube-controller-manager and every 15 seconds (--horizontal-pod-autoscaler-sync-period) does the same thing for each HPA: gets the metrics, computes the desired number of replicas and writes it to the target's spec.replicas. From there the ordinary Deployment controller from part 7 takes over.

The algorithm

desiredReplicas = ceil( currentReplicas × currentMetricValue / desiredMetricValue )

Three pods with a 70% CPU target and an actual 140% give ceil(3 × 140 / 70) = 6. At 75% it is ceil(3 × 75 / 70) = ceil(3.21) = 4, but there will be no scaling: if the ratio of current to target differs from 1 by less than the tolerance (10% by default, --horizontal-pod-autoscaler-tolerance), the HPA does nothing. The tolerance can also be set in the HPA itself, with the tolerance field in behavior.scaleUp and behavior.scaleDown: the feature appeared in 1.33, has been beta and on by default since 1.35, and became stable in 1.37.

Details that explain "strange" behaviour:

• Percentages are of requests. A utilization of 70% means 70% of resources.requests.cpu, not of a core and not of the limit. If even one container of a pod has no request for that resource, the HPA cannot calculate utilization and stays idle, reporting this in its conditions. One more reason to always set requests.
• Unready pods count as consuming 0% when scaling up on CPU, and pods without metrics count as 0% when growing and 100% when shrinking. During the first 5 minutes after a pod starts (--horizontal-pod-autoscaler-cpu-initialization-period) its CPU is taken into account only if the sample was taken when the pod was already ready: the CPU spike at the start of a Go or Java application must not trigger further growth.
• Several metrics — the HPA computes the desired number for each and takes the maximum.
• replicas in the manifest. If an HPA manages a Deployment, the replicas field is removed from the Deployment manifest: otherwise every kubectl apply returns the replica count to the one in the file, fighting the HPA.

Metric types

Type	From	Example
Resource	metrics.k8s.io (metrics-server)	CPU or memory averaged across pods: Utilization as a percentage of requests or AverageValue in absolute units
ContainerResource	metrics.k8s.io	the same, but for one container of a pod, so that a sidecar does not dilute the average
Pods	custom.metrics.k8s.io	a metric with a value per pod: requests per second per pod
Object	custom.metrics.k8s.io	a metric of one cluster object: requests per second on an Ingress
External	external.metrics.k8s.io	a metric outside the cluster: the queue length in a message broker

The custom.metrics.k8s.io and external.metrics.k8s.io APIs do not exist in a cluster by themselves: like metrics-server, they are registered in the aggregation layer by a separate adapter, a component that translates HPA requests into requests to the monitoring system. For Prometheus that is prometheus-adapter (slowly developed) or KEDA, covered below.

behavior

The behavior field sets how fast the HPA may change the number of replicas, separately up and down. The defaults:

behavior:
  scaleDown:
    stabilizationWindowSeconds: 300
    policies:
    - type: Percent
      value: 100
      periodSeconds: 15
  scaleUp:
    stabilizationWindowSeconds: 0
    policies:
    - type: Percent
      value: 100
      periodSeconds: 15
    - type: Pods
      value: 4
      periodSeconds: 15
    selectPolicy: Max

The stabilization window: the HPA remembers its recommendations for the last N seconds and, when shrinking, takes the highest of them. Load dropped for a minute and came back — the replicas stay untouched. By default the window is 5 minutes down and zero up: growth can start immediately. Policies limit the step: up, no more than doubling or +4 pods in 15 seconds (the larger of the two is taken, selectPolicy: Max; from three replicas, straight to seven), down, everything extra at once. For a service that warms up slowly or holds long connections, sharp shrinking hurts, and the downward policy is made gentler — for example, no more than one pod per minute.

An example with hello

The hello manifest from part 30 has no resources, and without requests.cpu the HPA has nothing to calculate from. Let's set them and start with three replicas:

$ kubectl set resources deployment hello -n demo --requests=cpu=100m,memory=32Mi --limits=memory=64Mi
deployment.apps/hello resource requirements updated
$ kubectl scale deployment hello -n demo --replicas=3
deployment.apps/hello scaled

The HPA keeps average CPU usage at 70% of the request, with three to ten replicas, and removes pods no faster than one per minute:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: hello
  namespace: demo
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: hello
  minReplicas: 3
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Pods
        value: 1
        periodSeconds: 60

The load comes from pods with an endless loop of requests to the service. They cannot run in demo: busybox without a securityContext is rejected by the restricted level (pods "..." is forbidden: violates PodSecurity "restricted:latest"). So the generator lives in the default namespace and reaches the service by the name hello.demo (part 16). Two pods first:

$ kubectl apply -f hello-hpa.yaml
horizontalpodautoscaler.autoscaling/hello created
$ kubectl create deployment load -n default --image=busybox:1.37 --replicas=2 -- \
    /bin/sh -c 'while true; do wget -q -O /dev/null http://hello.demo; done'
deployment.apps/load created

$ kubectl get hpa hello -n demo --watch
NAME    REFERENCE          TARGETS       MINPODS   MAXPODS   REPLICAS   AGE
hello   Deployment/hello   cpu: 0%/70%   3         10        3          75s
hello   Deployment/hello   cpu: 15%/70%   3         10        3          105s
hello   Deployment/hello   cpu: 59%/70%   3         10        3          2m1s
hello   Deployment/hello   cpu: 74%/70%   3         10        3          2m16s
hello   Deployment/hello   cpu: 75%/70%   3         10        3          2m31s
hello   Deployment/hello   cpu: 74%/70%   3         10        3          3m1s
hello   Deployment/hello   cpu: 76%/70%   3         10        3          6m16s

For a full seven minutes the target is exceeded, and there are still three replicas. That is the tolerance at work: 75 / 70 = 1.07, the difference from one is under 10%, so the HPA does nothing even though the formula would give four. Let's apply more load — six pods:

$ kubectl delete deployment load -n default
$ kubectl create deployment load -n default --image=busybox:1.37 --replicas=6 -- \
    /bin/sh -c 'while true; do wget -q -O /dev/null http://hello.demo; done'

$ kubectl get hpa hello -n demo --watch
NAME    REFERENCE          TARGETS       MINPODS   MAXPODS   REPLICAS   AGE
hello   Deployment/hello   cpu: 0%/70%   3         10        3          22m
hello   Deployment/hello   cpu: 22%/70%   3         10        3          22m
hello   Deployment/hello   cpu: 134%/70%   3         10        3          22m
hello   Deployment/hello   cpu: 136%/70%   3         10        6          23m
hello   Deployment/hello   cpu: 126%/70%   3         10        6          23m
hello   Deployment/hello   cpu: 75%/70%    3         10        6          23m
hello   Deployment/hello   cpu: 48%/70%    3         10        6          23m
hello   Deployment/hello   cpu: 28%/70%    3         10        6          24m

ceil(3 × 136 / 70) = ceil(5.83) = 6: within the default policies, so in one step. The 126% line with six replicas is a lag: the new pods have not entered the measurement yet. After that the average moves between 20 and 60%, below the target, yet the replicas do not go down: the stabilization window holds the highest recommendation of the last five minutes, and the load is still running. Let's remove it:

$ kubectl delete deployment load -n default
$ kubectl events -n demo --for hpa/hello
LAST SEEN   TYPE     REASON              OBJECT                          MESSAGE
16m         Normal   SuccessfulRescale   HorizontalPodAutoscaler/hello   New size: 6; reason: cpu resource utilization (percentage of request) above target
6m13s       Normal   SuccessfulRescale   HorizontalPodAutoscaler/hello   New size: 5; reason: All metrics below target
5m13s       Normal   SuccessfulRescale   HorizontalPodAutoscaler/hello   New size: 4; reason: All metrics below target
4m12s       Normal   SuccessfulRescale   HorizontalPodAutoscaler/hello   New size: 3; reason: All metrics below target

The first decrease comes 5 minutes 47 seconds after the load is removed (the stabilization window plus the last high sample), then one pod per minute, as the policy says. Until the window has passed, conditions shows AbleToScale with the reason ScaleDownStabilized and the message recent recommendations were higher than current one, applying the highest recent recommendation. The full state of the HPA is in status:

$ kubectl get hpa hello -n demo -o yaml | sed -n '/^status:/,$p'
status:
  conditions:
  - lastTransitionTime: "2026-11-03T18:14:13Z"
    message: recommended size matches current size
    reason: ReadyForNewScale
    status: "True"
    type: AbleToScale
  - lastTransitionTime: "2026-11-03T18:14:13Z"
    message: the HPA was able to successfully calculate a replica count from cpu resource
      utilization (percentage of request)
    reason: ValidMetricFound
    status: "True"
    type: ScalingActive
  - lastTransitionTime: "2026-11-03T18:46:50Z"
    message: the desired replica count is less than the minimum replica count
    reason: TooFewReplicas
    status: "True"
    type: ScalingLimited
  currentMetrics:
  - resource:
      current:
        averageUtilization: 1
        averageValue: 1m
      name: cpu
    type: Resource
  currentReplicas: 3
  desiredReplicas: 3
  lastScaleTime: "2026-11-03T18:48:51Z"

ScalingActive=False means the HPA cannot get metrics: no metrics-server, no request on a container, the adapter is not responding. ScalingLimited=True means it has hit minReplicas or maxReplicas.

Other autoscalers

VerticalPodAutoscaler (VPA) is a separate project from the kubernetes/autoscaler repository: it watches the actual consumption of pods and recommends or sets their requests and limits itself. Since in-place pod resize became stable in Kubernetes 1.35, the VPA can change resources without recreating the pod where possible, in the InPlaceOrRecreate mode. It is not used together with an HPA on CPU: they would fight. A useful safe mode is updateMode: "Off", where the VPA only writes recommendations and a person decides.

KEDA (Kubernetes Event-Driven Autoscaling) is an operator that scales on external events: the length of a RabbitMQ or Kafka queue, the result of a Prometheus query, a schedule. A ScaledObject describes the source, KEDA implements external.metrics.k8s.io for it and creates an ordinary HPA; on top of the HPA it can scale to zero replicas and back.

Cluster Autoscaler changes not the number of pods but the number of nodes: it adds a machine when pods hang in Pending for lack of resources, and removes an underloaded one. It works through the API of a cloud provider or a virtualization platform that can create machines on request. In a cluster on your own servers, like ours, it has nothing to call, so it is not here; capacity is planned from metrics in advance.

---

Prometheus

Prometheus is a monitoring system and time series database, a CNCF project. Its design in four points.

Data model. A time series is a metric name plus a set of labels (key–value pairs), and a sequence of "time — number" points for that combination. container_cpu_usage_seconds_total{namespace="demo", pod="hello-…-5lfft", container="hello"} is one series; another pod is another series. Every unique combination of labels is a new series in the database, so labels never hold anything with an unbounded number of values (user IDs, URLs with parameters): this is called cardinality, and it is the main enemy of Prometheus memory.

Pull. Prometheus itself goes to targets over HTTP for /metrics at a set interval and takes the current values in a text format. It builds the target list with service discovery: for Kubernetes, through the cluster API, watching pods, services, endpoints and nodes. If a target did not respond, its up{...} series becomes 0 — this is how "the application is silent" is told apart from "the application says all is bad".

Exporters. Not every program can serve /metrics. An exporter is a separate process that takes state from something else and translates it into the Prometheus format: node-exporter reads /proc and /sys of the node (CPU, memory, disks, network, file systems), kube-state-metrics watches API objects and turns their state into metrics (how many replicas are desired and ready, the reason a container last terminated, the pod phase, PVC capacity). Applications such as hello serve their metrics themselves.

Storage. A local TSDB: fresh points live in memory and in a write-ahead log (WAL) on disk, every two hours they are flushed into an immutable block (a directory with chunks, an index and a tombstones file), and blocks are later compacted into larger ones. A point takes 1–2 bytes on disk thanks to compression.

PromQL in five minutes

There are four metric types: counter (only grows, resets on restart: requests, CPU seconds), gauge (any current value: memory, queue length), histogram (a distribution over _bucket{le="..."} buckets plus _sum and _count: response time) and summary (quantiles calculated in the application itself).

container_memory_working_set_bytes{namespace="demo", container="hello"}

This is an instant vector — one latest value per matching series. Counters are useless in this form; they are viewed through the rate of change over a window, a range vector [5m]:

rate(container_cpu_usage_seconds_total{namespace="demo", container="hello"}[5m])

rate is the average per-second growth over 5 minutes, corrected for counter resets; for CPU seconds that is exactly cores. increase(...[1h]) is the growth over an hour. Aggregation removes extra labels:

sum by (pod) (rate(container_cpu_usage_seconds_total{namespace="demo"}[5m]))

histogram_quantile(0.95, sum by (le) (rate(apiserver_request_duration_seconds_bucket{verb="GET"}[5m])))

The first is CPU per pod. The second is the 95th percentile of kube-apiserver response time for GET requests, from its histogram; on our cluster it is about 0.07 seconds. The window in rate should fit at least four scrape intervals: with 30 seconds, from two minutes.

---

kube-prometheus-stack

You can assemble all this by hand, but it takes long. kube-prometheus-stack is a Helm chart from the Prometheus community that installs a ready set:

• Prometheus Operator — an operator (part 28) with a set of CRDs: Prometheus and Alertmanager describe the servers themselves (the operator turns them into StatefulSets), ServiceMonitor and PodMonitor say what to scrape, PrometheusRule holds recording and alerting rules, AlertmanagerConfig holds notification routes. The operator watches these objects and generates the Prometheus configuration from them, so targets and rules are added with manifests next to the application;
• Prometheus with ready ServiceMonitors for the kubelet and cAdvisor, the API server, CoreDNS, kube-controller-manager, kube-scheduler, etcd and kube-proxy;
• kube-state-metrics and node-exporter (a DaemonSet on every node);
• Alertmanager — a service that receives firing alerts from Prometheus, groups them, suppresses dependent ones (a node is down, no need for a separate alert on each of its pods), observes repeat intervals and sends them out over channels: mail, messengers, webhook;
• Grafana — a web interface for graphs and dashboards on top of Prometheus and other sources, with ready cluster dashboards (29 in this chart version);
• 155 alert rules in 35 PrometheusRule objects from the kubernetes-mixin project and others: KubePodCrashLooping, KubePersistentVolumeFillingUp, CPUThrottlingHigh, KubeNodeNotReady and so on.

Storage: a local volume, not NFS

Prometheus writes to the TSDB constantly, keeps files open through mmap and relies on POSIX file system semantics. Its documentation says plainly: non-POSIX-compliant file systems are not supported, unrecoverable corruption may happen, and NFS is not supported. So volumes from csi-driver-nfs (part 23) are not suitable for Prometheus, for exactly the reason databases are not placed on NFS. It needs a local disk of the node: the local-path StorageClass from part 25 (local-path-provisioner is a controller that creates a volume as a directory on the node where the pod started) or a static local PV on a separate disk. If you removed it after that part, one manifest installs it again:

$ kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/v0.0.37/deploy/local-path-storage.yaml

A local volume ties the pod to the node: if k8s-node-2 goes down, Prometheus waits for it to come back. For monitoring a small cluster that is acceptable; where it is not, two replicas are run (replicas: 2 in the Prometheus resource), each with its own local volume on its own node and independent scraping of the same targets.

How much space. The formula from the documentation: size = retention in seconds × points per second × bytes per point. On our three-node cluster this chart gives 76 thousand series and 2,100 points per second after an hour of work. Over 15 days: 1,296,000 × 2,100 × 2 ≈ 5.4 GB, plus the WAL and headroom for block compaction and cluster growth — a 20 GB volume is enough. Your cluster will show its real numbers: prometheus_tsdb_head_series and sum(rate(prometheus_tsdb_head_samples_appended_total[5m])). Retention is set by two limits at once: retention (time) and retentionSize (size, some 10–15% below the volume), whichever comes first. Prometheus took about 370 MiB of memory on our cluster; that is the reference point for requests.

Installation

node-exporter mounts node directories and runs in the node's network namespace, so we label the monitoring namespace with the privileged level right away (part 30). The Grafana admin password goes into a Secret created in advance, not into values (a secret in values ends up in the Helm release history and in Git):

$ kubectl create namespace monitoring
$ kubectl label namespace monitoring pod-security.kubernetes.io/enforce=privileged
$ kubectl create secret generic grafana-admin -n monitoring \
    --from-literal=admin-user=admin --from-literal=admin-password="$(openssl rand -base64 24)"

Minimal values:

# kps-values.yaml
fullnameOverride: kps

prometheus:
  prometheusSpec:
    retention: 15d
    retentionSize: 17GB
    # take ServiceMonitor, PodMonitor and PrometheusRule from all namespaces,
    # not only those with the label release: kps
    serviceMonitorSelectorNilUsesHelmValues: false
    podMonitorSelectorNilUsesHelmValues: false
    ruleSelectorNilUsesHelmValues: false
    # find targets through EndpointSlice, not through the deprecated Endpoints API
    serviceDiscoveryRole: EndpointSlice
    resources:
      requests: { cpu: 200m, memory: 1Gi }
      limits: { memory: 2Gi }
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: local-path
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 20Gi

alertmanager:
  alertmanagerSpec:
    storage:
      volumeClaimTemplate:
        spec:
          storageClassName: local-path
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 1Gi

grafana:
  admin:
    existingSecret: grafana-admin

# our kubelets answer with certificates from the cluster CA, so verification can be on
kubelet:
  serviceMonitor:
    tlsConfig:
      insecureSkipVerify: false

The last block is the same subject as with metrics-server. By default the chart scrapes kubelets with insecureSkipVerify: true, because on an ordinary kubeadm cluster their certificates are self-signed. Ours are signed by the cluster CA, and with verification on all nine kubelet targets (three paths per node) stay up: the chart takes the CA from the kube-root-ca.crt ConfigMap, which exists in every namespace.

The chart is installed from an OCI registry with one command at an explicit version (Helm in detail is in part 27):

$ helm install kps oci://ghcr.io/prometheus-community/charts/kube-prometheus-stack \
    --version 92.17.1 -n monitoring -f kps-values.yaml
...
STATUS: deployed
REVISION: 1
DESCRIPTION: Install complete
$ kubectl get pods -n monitoring
NAME                                      READY   STATUS    RESTARTS        AGE
alertmanager-kps-alertmanager-0           2/2     Running   0               8m34s
kps-grafana-6d697c9dd6-x947c              3/3     Running   2 (2m14s ago)   10m
kps-kube-state-metrics-67c8fc75bb-bn5cl   1/1     Running   0               10m
kps-operator-5f77976978-76cft             1/1     Running   0               10m
kps-prometheus-node-exporter-6b98q        1/1     Running   0               10m
kps-prometheus-node-exporter-7wzp5        1/1     Running   0               10m
kps-prometheus-node-exporter-t5rkg        1/1     Running   0               10m
prometheus-kps-prometheus-0               2/2     Running   0               8m32s

This is Prometheus 3.15, Alertmanager 0.34, operator 0.94 and Grafana 13.3. The two Grafana restarts are not a failure: on nodes with two vCPUs the first start takes several minutes, and the container misses its liveness probe twice (Container grafana failed liveness probe, will be restarted). The Prometheus pod has two containers: Prometheus itself and config-reloader, which watches the configuration generated by the operator and asks Prometheus to reread it without a restart.

A kubeadm peculiarity. In kubeadm, kube-controller-manager, kube-scheduler and etcd metrics listen only on 127.0.0.1 by default, and kube-proxy serves metrics on 127.0.0.1:10249. Prometheus in a pod cannot reach them: right after installation the targets of these four components are down, and the alerts include etcdInsufficientMembers (critical, firing at once), TargetDown, KubeControllerManagerInstanceUnreachable, KubeSchedulerInstanceUnreachable and KubeProxyInstanceUnreachable. There are two options.

The first is to move them to the node address: bind-address: 0.0.0.0 in kubeadm extraArgs for the controller manager and the scheduler, the node address in etcd's listen-metrics-urls, and metricsBindAddress: 0.0.0.0:10249 in the kube-proxy ConfigMap followed by kubectl rollout restart ds/kube-proxy -n kube-system. After that all four become up on our cluster. The protection differs: ports 10257 and 10259 of the controller manager and the scheduler require authentication (403 without a token), while etcd and kube-proxy serve metrics to anyone (200), so those ports are closed with a firewall from everyone except the pod subnet.

The second is to honestly turn these targets off in values if you do not need them: kubeControllerManager.enabled: false, kubeScheduler.enabled: false, kubeEtcd.enabled: false, kubeProxy.enabled: false. Along with the targets the chart removes the related alerts.

Access to the interfaces. To begin with, kubectl port-forward -n monitoring svc/kps-grafana 3000:80 and svc/kps-prometheus 9090. Permanent access to Grafana goes through an Ingress with a certificate from part 18 and part 19; Prometheus and Alertmanager have no authentication of their own and are not published without an authenticating proxy.

A ServiceMonitor for hello

hello from part 5 already serves /metrics: a request counter and the build version. You can look through the API server proxy without starting a separate pod:

$ kubectl get --raw /api/v1/namespaces/demo/services/hello:http/proxy/metrics
# HELP hello_requests_total Requests served by the / handler.
# TYPE hello_requests_total counter
hello_requests_total 2122
# HELP hello_build_info Build information.
# TYPE hello_build_info gauge
hello_build_info{version="1.0"} 1

For Prometheus to start collecting them you need a ServiceMonitor, an object that tells the operator: "scrape the endpoints of services with these labels, on this port, at this path". We put the label on the Service back in part 15, and the port is given by its name from the Service (for hello it is http):

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: hello
  namespace: demo
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: hello
  endpoints:
  - port: http
    path: /metrics
    interval: 30s

The operator adds a job with Kubernetes service discovery to the Prometheus configuration: Prometheus itself watches the EndpointSlices of services with these labels (that is what serviceDiscoveryRole: EndpointSlice in values sets; by default the operator still uses the Endpoints API, deprecated in Kubernetes 1.33) and scrapes the addresses of all pods behind them; to every series Prometheus adds the labels namespace, pod, service, container, endpoint, job and instance. A PodMonitor does the same without a service, by pod labels; it is handy for DaemonSets and pods with no service.

If a "deny all ingress" policy from part 20 is in effect in the demo namespace, everything depends on timing. Prometheus keeps persistent connections to its targets, and Calico does not break connections that are already open: on our cluster the targets stayed up for almost four minutes after default-deny, until we removed the policy ourselves. But once the pods change, a new connection cannot be established:

$ kubectl rollout restart deploy/hello -n demo
$ curl -s 'localhost:9090/api/v1/targets?state=active' | jq -r '.data.activeTargets[] | select(.labels.job=="hello") | "\(.labels.pod) \(.health) \(.lastError)"'
hello-74d65597f7-5lfft down Get "http://172.20.172.25:8080/metrics": context deadline exceeded
hello-74d65597f7-qqdxp down Get "http://172.20.211.63:8080/metrics": context deadline exceeded
hello-74d65597f7-8vkl6 down Get "http://172.20.211.61:8080/metrics": context deadline exceeded

So monitoring may break not when the policy is applied but at the next rollout. An allow policy is needed, only for the monitoring namespace and only on the metrics port; with it the targets return to up at the next scrape:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-prometheus
  namespace: demo
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: hello
  policyTypes: ["Ingress"]
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: monitoring
      podSelector:
        matchLabels:
          app.kubernetes.io/name: prometheus
    ports:
    - port: 8080
      protocol: TCP

Useful queries

CPU throttling — the share of CFS scheduler periods in which a container hit its CPU limit (part 10). Above 25%, the application is being slowed down regularly, even if average usage is low. The series exist only for containers with a CPU limit: hello will not appear in this query, and a test pod with limits.cpu: 50m and an endless loop shows exactly 1 on our cluster, that is, 100%:

sum by (namespace, pod, container) (rate(container_cpu_cfs_throttled_periods_total{container!=""}[5m]))
  /
sum by (namespace, pod, container) (rate(container_cpu_cfs_periods_total{container!=""}[5m]))

Restarts over the last hour, from kube-state-metrics:

increase(kube_pod_container_status_restarts_total[1h]) > 0

OOM — containers last terminated by OOM, and the OOM events themselves from cAdvisor:

kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} == 1
increase(container_oom_events_total{container!=""}[1h]) > 0

Memory relative to the limit — who is close to OOM:

max by (namespace, pod, container) (container_memory_working_set_bytes{container!=""})
  /
max by (namespace, pod, container) (kube_pod_container_resource_limits{resource="memory"})
  > 0.9

Requests versus consumption — for choosing requests: the ratio of real CPU to requested CPU over a day. Values around 0.1 mean the pod orders ten times more than it spends and takes up room on the node for nothing:

sum by (namespace, pod) (rate(container_cpu_usage_seconds_total{container!=""}[1d]))
  /
sum by (namespace, pod) (kube_pod_container_resource_requests{resource="cpu"})

Free space on a PVC — kubelet metrics for mounted volumes, and a forecast that the volume fills up within four hours:

kubelet_volume_stats_available_bytes / kubelet_volume_stats_capacity_bytes < 0.1

predict_linear(kubelet_volume_stats_available_bytes[6h], 4 * 3600) < 0

predict_linear fits a linear regression over the points of 6 hours and extrapolates 4 hours ahead. These metrics exist only for volumes whose driver implements statistics. csi-driver-nfs and csi-driver-smb do, but for a shared NFS export with subDir "free" shows the space of the whole export, not a volume quota: on our cluster a 1 GiB PVC reports a capacity of 17 GB. local-path volumes, on the other hand, are plain node directories and have no kubelet_volume_stats_* series at all, so these queries cannot see the volume of Prometheus itself, and the KubePersistentVolumeFillingUp alert does not watch it. For that one, look at the node file system:

node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} < 0.1

An alert rule

Rules are written as a PrometheusRule object. Every alert rule is a PromQL expression: as long as it returns at least one series, the alert is active. for is how long the expression must stay true in a row before the alert goes from pending to firing and on to Alertmanager. hello has two indicators, target availability and the request counter, so that is what we write the rules on:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: hello
  namespace: demo
spec:
  groups:
  - name: hello
    rules:
    - alert: HelloTargetDown
      expr: up{namespace="demo", service="hello"} == 0
      for: 2m
      labels:
        severity: warning
      annotations:
        summary: Prometheus cannot scrape metrics from a hello pod
        description: 'Pod {{ $labels.pod }} has not answered on /metrics for 2 minutes.'
    - alert: HelloHighRequestRate
      expr: sum(rate(hello_requests_total{namespace="demo"}[5m])) > 50
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: hello receives more than 50 requests per second
        description: 'Currently {{ $value | humanize }} requests per second, the HPA may not keep up.'

Alertmanager needs the severity label for routing: critical wakes the on-call engineer, warning goes to the work channel. Besides alerts, a PrometheusRule holds recording rules — precomputed expressions saved as new series: a heavy dashboard query is calculated once per interval, not every time the page opens.

The chart's rule set has a Watchdog alert that fires always. It is not an error but a check of the whole chain: it is wired to an external "dead man's switch" service that raises the alarm if Watchdog stops arriving — that is, if monitoring itself is broken.

---

A hands-on look

The metrics API directly. kubectl top is just a client of the aggregated API:

$ kubectl get --raw /apis/metrics.k8s.io/v1beta1/namespaces/demo/pods/hello-74d65597f7-5lfft | jq '{window, containers}'
{
  "window": "27.174s",
  "containers": [
    {
      "name": "hello",
      "usage": {
        "cpu": "4203724n",
        "memory": "6228Ki"
      }
    }
  ]
}

window is the interval between the two points CPU was calculated from: usually about 15 seconds, but the timestamps are set by the kubelet and the window can be longer; 4203724n is nanocores, about 4.2m.

The kubelet certificate — after serverTLSBootstrap it is signed by the cluster CA, not by the kubelet itself:

user@k8s-master-1:~$ sudo openssl x509 -in /var/lib/kubelet/pki/kubelet-server-current.pem -noout -issuer -subject -ext subjectAltName -enddate
issuer=CN=kubernetes
subject=O=system:nodes, CN=system:node:k8s-master-1
X509v3 Subject Alternative Name:
    DNS:k8s-master-1, IP Address:192.168.10.17
notAfter=Oct 24 10:32:48 2027 GMT

What the operator made of the ServiceMonitor. The generated configuration lies in a Secret, compressed with gzip:

$ kubectl get secret prometheus-kps-prometheus -n monitoring -o jsonpath='{.data.prometheus\.yaml\.gz}' \
    | base64 -d | gunzip | grep -A16 'job_name: serviceMonitor/demo/hello'
- job_name: serviceMonitor/demo/hello/0
  honor_labels: false
  kubernetes_sd_configs:
  - role: endpointslice
    namespaces:
      names:
      - demo
  scrape_interval: 30s
  metrics_path: /metrics
  relabel_configs:
  - source_labels:
    - job
    target_label: __tmp_prometheus_job_name
  - action: keep
    source_labels:
    - __meta_kubernetes_service_label_app_kubernetes_io_name
    - __meta_kubernetes_service_labelpresent_app_kubernetes_io_name

Targets and their state — through the Prometheus HTTP API:

$ kubectl port-forward -n monitoring svc/kps-prometheus 9090 &
$ curl -s 'localhost:9090/api/v1/targets?state=active' | jq -r '.data.activeTargets[] | select(.labels.namespace=="demo") | "\(.labels.pod) \(.health) \(.lastScrapeDuration)"'
hello-74d65597f7-5lfft up 0.000396057
hello-74d65597f7-qqdxp up 0.000679508
hello-74d65597f7-8vkl6 up 0.000627355

$ curl -s localhost:9090/api/v1/query --data-urlencode 'query=prometheus_tsdb_head_series' | jq -r '.data.result[0].value[1]'
76398

The TSDB on disk — in the local volume, on the node where Prometheus runs. You cannot look from inside the container: the image is distroless and has no ls (exec: "ls": executable file not found in $PATH), and the PV tells you the directory path:

$ kubectl get pv $(kubectl get pvc -n monitoring prometheus-kps-prometheus-db-prometheus-kps-prometheus-0 -o jsonpath='{.spec.volumeName}') -o jsonpath='{.spec.hostPath.path}'
/opt/local-path-provisioner/pvc-3aa8da09-dea5-4d57-baaa-abeb390d8832_monitoring_prometheus-kps-prometheus-db-prometheus-kps-prometheus-0
user@k8s-node-2:~$ sudo ls -1 /opt/local-path-provisioner/pvc-3aa8da09-…/prometheus-db
01M6HPRQAS1NC6XZHWA1Y7GYB4
chunks_head
lock
queries.active
wal

The directory with a ULID-style name is a finished block: two hours of data by default, and blocks get compacted into larger ones over time. wal is the write-ahead log, chunks_head holds memory-mapped chunks of fresh data, lock protects the directory from a second process, and queries.active keeps the queries in progress. These are the files Prometheus works with through mmap and locks, and that is why they need a local disk.

An alert at work. Under load from six pods hello receives over a thousand requests per second. 25 seconds after the start the expression is true and the alert is pending; five minutes later it is firing, and at that moment it appears in Alertmanager:

$ curl -s localhost:9090/api/v1/alerts | jq -r '.data.alerts[] | select(.labels.alertname | startswith("Hello")) | "\(.labels.alertname) \(.state) \(.activeAt)"'
HelloHighRequestRate firing 2026-11-03T19:07:55.540015347Z

$ kubectl port-forward -n monitoring svc/kps-alertmanager 9093 &
$ curl -s 'localhost:9093/api/v2/alerts?filter=alertname=~"Hello.*"' | jq -r '.[] | "\(.labels.alertname) \(.status.state) \(.startsAt) \(.annotations.description)"'
HelloHighRequestRate active 2026-11-03T19:12:55.540Z Currently 1.258k requests per second, the HPA may not keep up.

activeAt is when the expression became true; startsAt in Alertmanager is when for passed. The default-deny experiment from the ServiceMonitor section also exercises the second rule: two minutes after the targets went down, HelloTargetDown has one firing alert per pod, each with its name in the description (Pod hello-74d65597f7-8vkl6 has not answered on /metrics for 2 minutes.). We have not configured any receivers in Alertmanager, so for now the alert goes no further than its API.

Cleanup: kubectl delete -f for the default-deny and allow-prometheus policy files, and kubectl delete deployment load -n default if the generator is still running. We keep metrics-server, the HPA and Prometheus: the next part continues on them.

---

Metrics answer "how much" and "when": how many requests, when the growth began. They do not answer "which request exactly and why" — that takes logs and traces, tied to metrics by shared labels and identifiers.

---

Sources

• Metrics Server — purpose and limitations, version compatibility, the --kubelet-insecure-tls flag.
• Resource metrics pipeline and Node metrics data — where the kubectl top numbers come from, kubelet paths.
• Certificate Management with kubeadm — serverTLSBootstrap and kubelet-serving CSRs; kubelet-csr-approver — automatic approval.
• Horizontal Pod Autoscaling and HorizontalPodAutoscaler Walkthrough — the algorithm, tolerance, behavior, metric types.
• Vertical Pod Autoscaler, KEDA, Cluster Autoscaler, prometheus-adapter — other autoscalers and the metrics adapter.
• Prometheus: Overview, Data model, Metric types — design and data model.
• Prometheus: Storage — the TSDB, the size formula, the NFS exclusion.
• Querying basics and Query functions — PromQL, rate, histogram_quantile, predict_linear.
• Alerting rules and Alertmanager — alert rules, grouping and inhibition.
• kube-prometheus-stack — the chart and its values; kubernetes-mixin — alert rules and dashboards.
• Prometheus Operator: Getting Started and API reference — ServiceMonitor, PrometheusRule, serviceDiscoveryRole.
• kube-state-metrics, node_exporter, Grafana — components of the set.

---

Next — part 33: logs and traces. How container logs are laid out on the node, collecting them with an agent into Loki, LogQL, API server auditing, tracing with OpenTelemetry and Tempo, and moving from a graph to a log and from a log to a trace in Grafana.