5 November 2026, 10:36Виталий Лещенко

Kubernetes for beginners. Part 33: How to collect logs and traces

All parts of the series “Kubernetes for beginners”

In part 31 a container's logs disappeared along with the pod, and events after an hour. In the previous part Prometheus learned to keep metrics for weeks and to report what went wrong and when. Today is about the two remaining pillars of observability: logs collected from all nodes into one searchable place, and traces that show the path of an individual request through services. We will look at how container logs are laid out on a node, put a collection agent on every node and Loki for storage, turn on API server auditing, add OpenTelemetry tracing to hello with storage in Tempo, and link it all in Grafana so that you can go from a point on a graph to a log, and from a log line to the trace of a request.

Container logs on the node

An application writes to stdout and stderr, file descriptors 1 and 2. In a container they are not connected to a terminal: when creating the container, containerd connects them to named pipes (FIFOs) through its intermediary process (the shim), and the CRI plugin of containerd reads the pipes and writes everything it receives, line by line, to a file on the node. The kubelet sets the path to the file when it creates the container:

/var/log/pods/<namespace>_<pod>_<pod uid>/<container>/<run number>.log

The run number is restartCount: after a restart 1.log appears, then 2.log, and the kubelet keeps the files of the current and the previous run, which is what kubectl logs --previous shows. The line format is the CRI log format:

user@k8s-node-1:~$ sudo tail -3 /var/log/pods/demo_hello-9ffdcbdf5-92v6k_ab2f90cc-01f9-40c7-ac10-0102698ef2cd/hello/0.log
2026-11-04T05:31:35.903831685Z stderr F 2026/11/04 05:31:35 172.20.211.2:41660 GET /
2026-11-04T05:31:35.914578454Z stderr F 2026/11/04 05:31:35 172.20.211.2:41718 GET /
2026-11-04T05:31:35.915862288Z stderr F 2026/11/04 05:31:35 172.20.211.2:41722 GET /

Four space-separated fields: the time containerd received the line (RFC 3339 with nanoseconds), the stream (stdout or stderr), a tag and the line itself. For now hello writes through Go's standard log package, which prints to stderr by default, hence stderr here. The tag F means a full line; P means part of a long line: containerd cuts lines longer than 16 KiB (max_container_log_line_size in its configuration), and a log collector has to glue the sequence P … P F back together. A pod that printed a line of 40,000 characters followed by the word short leaves four records in the file:

user@k8s-node-1:~$ sudo cut -c1-60 /var/log/pods/default_longline_a313dbbb-a488-4179-ad33-9eb550d113eb/longline/0.log
2026-11-04T05:31:53.41651155Z stdout P xxxxxxxxxxxxxxxxxxxxx
2026-11-04T05:31:53.4165661Z stdout P xxxxxxxxxxxxxxxxxxxxxx
2026-11-04T05:31:53.416670001Z stdout F xxxxxxxxxxxxxxxxxxxx
2026-11-04T05:31:53.41667465Z stdout F short

kubectl logs glues them itself and shows the whole line. crictl logs reads the same file.

Rotation is the kubelet's job, not the application's. Every 10 seconds (containerLogMonitorInterval) it checks the size of the files and, if the current one is larger than containerLogMaxSize (10Mi by default), asks containerd to reopen the file: the old one is renamed to 0.log.<time>, older ones are compressed with gzip, and no more than containerLogMaxFiles (5 by default) files per container are kept. Both parameters are set in KubeletConfiguration. So a node holds at most about 50 MiB of logs per container, and kubectl logs shows only the current file: whatever has gone into rotation can no longer be seen through kubectl.

For the convenience of collectors the kubelet also maintains the /var/log/containers/ directory, a flat list of symbolic links of the form <pod>_<namespace>_<container>-<container id>.log pointing to the current files in /var/log/pods/.

Why stdout and not a file

An application that writes logs to a file inside the container breaks this whole chain:

the file grows in the container's writable layer (the upper overlayfs layer) or in an emptyDir, which uses up the pod's ephemeral-storage, and the kubelet will eventually evict the pod with Evicted, as in part 31;
there is no rotation unless the application does it itself, and doing it inside every application is a bad idea;
the file disappears with the container on restart;
it is invisible both to kubectl logs and to collection agents, which read /var/log/pods/.

So the rule is simple: an application in Kubernetes writes logs to stdout and stderr, one record per line, and the platform takes care of storage, rotation and delivery. If a program can only write to a file, a sidecar container (part 6) is placed next to it, which reads the file from a shared emptyDir and prints it to its own stdout (tail -F). This is a workaround, not the norm.

Structured logs

A person can read the line request served method=GET path=/ status=200, but a machine has to parse it with a regular expression, which breaks at the first format change. A structured log is a record in a machine-readable format, most often JSON, one object per line: fields with names and types that can be filtered and aggregated. A bonus: multi-line messages (stack traces) are packed into a single JSON field and stop falling apart into separate records. Go has the standard log/slog package for this, and version 1.2 of hello switches to it:

slog.SetDefault(slog.New(slog.NewJSONHandler(os.Stdout, nil)))

slog.InfoContext(ctx, "request served",
    "method", r.Method, "path", r.URL.Path, "status", http.StatusOK, "duration_ms", ms)

The record now goes to stdout, and instead of an address and a path in a line there are named fields. From here on the logs of hello look like this. The build, tracing and other changes of version 1.2 are in the section on traces.

What must not get into a log in any format: passwords, tokens, Authorization headers, cookies, personal data. More people read logs than secrets, and logs are kept longer.

Collecting logs: an agent on every node

The files live on the nodes, but you need to search across all of them at once, and after a pod is deleted. The standard scheme is an agent on every node (a DaemonSet, part 8): it mounts the node's /var/log/pods through hostPath, reads new lines (remembering its position in each file), parses the CRI format, adds labels from the Kubernetes API (namespace, pod, container, application labels) and sends everything to storage.

 k8s-node-1                                  k8s-node-2
 ┌────────────────────────────────┐          ┌────────────────────────────────┐
 │ hello ──stdout──► containerd   │          │ hello ──stdout──► containerd   │
 │                       │        │          │                       │        │
 │                       ▼        │          │                       ▼        │
 │ /var/log/pods/.../0.log        │          │ /var/log/pods/.../0.log        │
 │          │                     │          │          │                     │
 │          ▼                     │          │          ▼                     │
 │ agent (DaemonSet): CRI, labels │          │ agent (DaemonSet): CRI, labels │
 └──────────┼─────────────────────┘          └──────────┼─────────────────────┘
            │ HTTP push                                 │ HTTP push
            │                                           │
            └─────────────────────┬─────────────────────┘
                                  ▼
                       Loki ──► S3: chunks and index
                                  ▲
                                  │ LogQL
                               Grafana

Which agent to take:

Grafana Alloy is the Grafana Labs agent that collects logs, metrics and traces; its configuration is a graph of components in Alloy's own configuration language. This is the recommended agent for Loki.
Promtail is the former Loki agent, which many will still come across in articles and charts. It reached end of life on March 2, 2026: there will be no more updates, and the command alloy convert --source-format=promtail translates a Promtail configuration into an Alloy one. It is not used in new installations.
Fluent Bit is a lightweight log collector and processor written in C, from the CNCF Fluent project; it is also installed as a DaemonSet and can send to Loki, Elasticsearch, S3 and dozens of other places. A good choice if logs need to go to several systems.

We take Alloy.

Alloy configuration

The configuration is a set of components connected by outputs and inputs. Pod logs need five of them:

// only the pods of its own node: each DaemonSet instance reads its own files
discovery.kubernetes "pods" {
  role = "pod"
  selectors {
    role  = "pod"
    field = "spec.nodeName=" + sys.env("HOSTNAME")
  }
}

// Loki labels from pod metadata and the path to the log file
discovery.relabel "pods" {
  targets = discovery.kubernetes.pods.targets
  rule {
    source_labels = ["__meta_kubernetes_namespace"]
    target_label  = "namespace"
  }
  rule {
    source_labels = ["__meta_kubernetes_pod_name"]
    target_label  = "pod"
  }
  rule {
    source_labels = ["__meta_kubernetes_pod_container_name"]
    target_label  = "container"
  }
  rule {
    source_labels = ["__meta_kubernetes_pod_label_app_kubernetes_io_name"]
    target_label  = "app"
  }
  rule {
    source_labels = ["__meta_kubernetes_pod_node_name"]
    target_label  = "node"
  }
  rule {
    source_labels = ["__meta_kubernetes_pod_uid", "__meta_kubernetes_pod_container_name"]
    separator     = "/"
    target_label  = "__path__"
    replacement   = "/var/log/pods/*$1/*.log"
  }
}

local.file_match "pods" {
  path_targets = discovery.relabel.pods.output
}

loki.source.file "pods" {
  targets    = local.file_match.pods.targets
  forward_to = [loki.process.pods.receiver]
}

loki.process "pods" {
  // parse the CRI format and glue lines tagged P
  stage.cri {}
  // from JSON lines move trace_id into structured metadata
  stage.json {
    expressions = { trace_id = "trace_id" }
  }
  stage.structured_metadata {
    values = { trace_id = "" }
  }
  forward_to = [loki.write.default.receiver]
}

loki.write "default" {
  endpoint {
    url = "http://loki-gateway.monitoring.svc/loki/api/v1/push"
  }
}

discovery.kubernetes watches pods through the API, discovery.relabel turns metadata into labels and computes __path__, a pattern for the container's files; local.file_match expands the pattern into real files, loki.source.file reads them from the remembered position, loki.process parses the lines, and loki.write sends them to Loki in batches. The restriction by spec.nodeName matters: without it every Alloy instance would watch all pods in the cluster. The HOSTNAME variable here is the name of the node, not the pod: the chart sets it through a fieldRef to spec.nodeName. The app label comes from app.kubernetes.io/name, the very label hello has had in previous parts: dots and slashes in a label name turn into underscores. Alloy adds two labels itself: filename with the file path and stream with the stream name; service_name is set by Loki.

Alloy is installed with a Helm chart in one command (Helm is covered in detail in part 27); the configuration above goes into the values:

# alloy-values.yaml
controller:
  type: daemonset
  tolerations:                # so that the agent also lands on the master, where it reads the audit log below
  - key: node-role.kubernetes.io/control-plane
    effect: NoSchedule
alloy:
  mounts:
    varlog: true          # hostPath /var/log of the node, read-only
  configMap:
    content: |
      // the configuration above

$ helm repo add grafana https://grafana.github.io/helm-charts
$ helm install alloy grafana/alloy --version 1.15.0 -n monitoring -f alloy-values.yaml

The Alloy pod needs a hostPath on /var/log, and hostPath volumes are already forbidden at the baseline level of Pod Security Admission from part 30. The monitoring namespace runs at the privileged level anyway: node-exporter from the previous part also mounts node directories and lives in the node's network namespace. That is why nothing but observability components goes into such a namespace, and the right to create pods in it is given to a narrow circle of administrators.

Cluster events live for an hour (part 31), and Alloy can save them to Loki: the loki.source.kubernetes_events component reads events through the API and turns them into log lines. In a DaemonSet it would collect the same events on every node, so it is run in a separate Alloy instance with one replica, or Alloy clustering is enabled, which divides the work between instances. A second release of the same chart is enough for us:

# alloy-events-values.yaml
controller:
  type: deployment
  replicas: 1
alloy:
  configMap:
    content: |
      loki.source.kubernetes_events "cluster" {
        job_name   = "kubernetes-events"
        log_format = "json"
        forward_to = [loki.write.default.receiver]
      }

      loki.write "default" {
        endpoint {
          url = "http://loki-gateway.monitoring.svc/loki/api/v1/push"
        }
      }

Loki

Loki is a log storage system from Grafana Labs, built "like Prometheus, but for logs". The main difference from systems like Elasticsearch is what gets indexed.

Elasticsearch and similar systems build a full-text index: every word of every line goes into an inverted index, a search for any substring is instant, but the index is comparable in size to the data itself and needs a lot of memory and disk. Loki indexes only labels, a small set of pairs like {namespace="demo", container="hello"}. Lines with the same set of labels form a stream; the lines of a stream are put into chunks, compressed and written to object storage. A query first selects streams by the index and then goes through their chunks by "brute force", parallelizing the reads. The index ends up tiny, storage cheap, and search speed depends on how precisely the labels narrow down the selection.

This leads to the main rule of Loki: low label cardinality. Every new combination of label values is a new stream with its own chunks. Labels are what you select a source by: cluster, namespace, application, container, node. Not labels: anything with many values, such as trace id, user id, request path, client IP, response code. Such fields stay in the text of the line and are extracted at query time, and what you often need to search for exactly (like a trace id) goes, since Loki 3, into structured metadata: key-value pairs attached to each line that do not create new streams. This is exactly what stage.structured_metadata does in the Alloy configuration.

How Loki stores data

Lines arrive at the distributor, which checks limits and spreads streams across ingesters. An ingester accumulates chunks in memory (with a write-ahead log on a local disk in case of a crash) and flushes them to object storage when a chunk is full or a stream goes quiet. Both chunks and the index go to object storage, the index in the TSDB format (the same approach as in Prometheus), one file per day. The compactor merges small index files every few minutes and deletes data older than the retention period. The querier runs queries, reading fresh data from ingesters and older data from storage.

The object storage is S3 or something compatible (part 25): it is cheap, scales by itself and relieves Loki of data replication. A local file system as the main storage is acceptable only for experiments.

Deployment modes

All Loki components are built into one binary, and the -target flag selects which of them to run:

Mode	How it works	When
Monolithic	all components in one process, `-target=all`	up to ~20 GB of logs per day; with several replicas on shared S3, with fault tolerance
Simple Scalable	three groups: `write`, `read`, `backend`	declared deprecated, will be removed in Loki 4.0; not to be used in new installations
Microservices	each component is a separate Deployment or StatefulSet	very large volumes, separate scaling of each component

For our cluster, the monolithic mode with one replica. The Helm charts of Loki, Tempo and Grafana were handed over to the community in 2026 and live in the grafana-community repository; they are installed from there too, from an OCI registry.

Installation

The S3 server is Garage on s3.example.com from part 25, with port 3900 open to the nodes. We create buckets and a separate key with rights only to them:

user@s3:~$ sudo garage -c /etc/garage.toml bucket create loki-chunks
user@s3:~$ sudo garage -c /etc/garage.toml bucket create loki-ruler
user@s3:~$ sudo garage -c /etc/garage.toml key create loki
user@s3:~$ sudo garage -c /etc/garage.toml bucket allow --read --write --owner loki-chunks --key loki
user@s3:~$ sudo garage -c /etc/garage.toml bucket allow --read --write --owner loki-ruler --key loki

The access keys go into a Secret and reach the configuration through environment variables (-config.expand-env=true allows Loki to substitute ${...} in its config):

$ kubectl create secret generic loki-s3 -n monitoring \
    --from-literal=S3_ACCESS_KEY_ID=... --from-literal=S3_SECRET_ACCESS_KEY=...

# loki-values.yaml
deploymentMode: Monolithic

loki:
  auth_enabled: false            # a single tenant; without this every request needs X-Scope-OrgID
  commonConfig:
    replication_factor: 1
  schemaConfig:
    configs:
    - from: "2026-10-01"
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: loki_index_
        period: 24h
  storage:
    type: s3
    bucketNames:
      chunks: loki-chunks
      ruler: loki-ruler
    s3:
      endpoint: s3.example.com:3900
      region: garage             # the region is part of the request signature, Garage accepts only its own
      accessKeyId: ${S3_ACCESS_KEY_ID}
      secretAccessKey: ${S3_SECRET_ACCESS_KEY}
      s3ForcePathStyle: true
      insecure: true             # in the lab network Garage answers over HTTP
  limits_config:
    retention_period: 30d
    allow_structured_metadata: true
  compactor:
    retention_enabled: true
    delete_request_store: s3

singleBinary:
  replicas: 1
  extraArgs: ["-config.expand-env=true"]
  extraEnvFrom:
  - secretRef:
      name: loki-s3
  persistence:
    storageClass: local-path     # the ingester's WAL goes on a local disk
    size: 10Gi

# memcached caches and test components are not needed on a small cluster
chunksCache: { enabled: false }
resultsCache: { enabled: false }
lokiCanary: { enabled: false }
test: { enabled: false }

# replicas of the other modes are zero
backend: { replicas: 0 }
read: { replicas: 0 }
write: { replicas: 0 }
ingester: { replicas: 0 }
querier: { replicas: 0 }
queryFrontend: { replicas: 0 }
queryScheduler: { replicas: 0 }
distributor: { replicas: 0 }
compactor: { replicas: 0 }
indexGateway: { replicas: 0 }
bloomPlanner: { replicas: 0 }
bloomBuilder: { replicas: 0 }
bloomGateway: { replicas: 0 }

$ helm install loki oci://ghcr.io/grafana-community/helm-charts/loki \
    --version 20.4.1 -n monitoring -f loki-values.yaml
$ kubectl get pods -n monitoring -l app.kubernetes.io/instance=loki
NAME                            READY   STATUS    RESTARTS   AGE
loki-0                          2/2     Running   0          112s
loki-gateway-7ccfd99688-5p9kl   2/2     Running   0          113s

schemaConfig is the version of the storage format with a start date; when the format changes, a new entry with a future date is added, and old data is read by the old schema. loki-gateway is an nginx in front of Loki, a single entry point for writing and reading. Loki has no authentication of its own: with auth_enabled: false anyone who can reach the gateway reads all the logs, so access to loki-gateway is restricted with NetworkPolicy: Alloy may write, Grafana may read.

LogQL

The Loki query language resembles PromQL. A query starts with a stream selector in curly braces, which works through the index and should be as precise as possible:

{namespace="demo", container="hello"}

Then comes a pipeline of stages separated by |. Line filters are the cheapest and work on the raw text:

{namespace="demo", container="hello"} |= "error" != "healthz"
{namespace="demo", container="hello"} |~ "\"status\":5[0-9]{2}"

Parsers turn a line into fields: | json for JSON, | logfmt for key=value, | pattern "<ip> - - <_> \"<method> <path> <_>\" <status> <_>" for fixed formats such as access logs. After a parser, typed field filters, line formatting and structured metadata are available as ordinary fields:

{namespace="demo", container="hello"} | json | status >= 500 and duration_ms > 100
{namespace="demo", container="hello"} | json | line_format "{{.method}} {{.path}} {{.status}} {{.duration_ms}}ms"
{namespace="demo", container="hello"} | trace_id = "4bf92f3577b34da6a3ce929d0e0e4736"

Metric queries turn logs into time series, which can be graphed and even alerted on in the Loki ruler:

sum by (pod) (rate({namespace="demo", container="hello"} | json | status >= 500 [5m]))

quantile_over_time(0.95, {namespace="demo", container="hello"} | json | unwrap duration_ms [5m]) by (pod)

rate counts lines per second, count_over_time lines per window, and unwrap takes a numeric field from the line and lets you compute quantiles and sums over it. If a field with that name already exists in structured metadata, the json parser puts the value from the line next to it with a suffix: in our case trace_id_extracted appears.

API server audit

Application logs answer the question "what did the program do"; the kube-apiserver audit log answers "who did what to the cluster": every API request with the user, address, object, verb, authorization decision and, optionally, the request and response bodies. It is off by default in kubeadm.

What to write and in how much detail is set by the policy, a list of rules checked in order; the first match wins. The levels:

Level	What is written
`None`	nothing
`Metadata`	who, when, from where, which object, verb, response code; no bodies
`Request`	the same plus the request body
`RequestResponse`	the same plus the response body

Every request goes through stages: RequestReceived, ResponseStarted (for long requests like watch), ResponseComplete, Panic, and an event can be written at each of them. A reasonable policy to start with:

# /etc/kubernetes/audit/policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
  - RequestReceived           # the same event will come at ResponseComplete
rules:
  # noise: health checks, events, lease renewals
  - level: None
    nonResourceURLs: ["/healthz*", "/livez*", "/readyz*", "/version"]
  - level: None
    resources:
    - group: ""
      resources: ["events"]
    - group: "coordination.k8s.io"
      resources: ["leases"]
  - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]
  # secrets, ConfigMaps and tokens: metadata only, the bodies contain the secrets themselves
  - level: Metadata
    resources:
    - group: ""
      resources: ["secrets", "configmaps", "serviceaccounts/token"]
    - group: "authentication.k8s.io"
      resources: ["tokenreviews"]
  # changes: with the request body
  - level: Request
    verbs: ["create", "update", "patch", "delete", "deletecollection"]
  # everything else: metadata
  - level: Metadata

The order matters: the rule for secrets comes before the "changes with a body" rule, otherwise kubectl create secret would write the password to the audit log in clear text.

Auditing is turned on with kube-apiserver flags. In kubeadm they are set in ClusterConfiguration together with the volumes, because the API server is a static pod and sees only what has been mounted for it:

apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
apiServer:
  extraArgs:
  - name: audit-policy-file
    value: /etc/kubernetes/audit/policy.yaml
  - name: audit-log-path
    value: /var/log/kubernetes/audit/audit.log
  - name: audit-log-maxage
    value: "7"
  - name: audit-log-maxbackup
    value: "5"
  - name: audit-log-maxsize
    value: "100"
  extraVolumes:
  - name: audit-policy
    hostPath: /etc/kubernetes/audit
    mountPath: /etc/kubernetes/audit
    readOnly: true
    pathType: DirectoryOrCreate
  - name: audit-log
    hostPath: /var/log/kubernetes/audit
    mountPath: /var/log/kubernetes/audit
    pathType: DirectoryOrCreate

On a running cluster the policy file is placed in /etc/kubernetes/audit/ on the master, the configuration is saved to the kubeadm-config ConfigMap (sudo kubeadm init phase upload-config kubeadm --config kubeadm.yaml), and the manifest is recreated with sudo kubeadm init phase control-plane apiserver --config kubeadm.yaml; the kubelet notices the file change in /etc/kubernetes/manifests/ and restarts the API server. Editing the manifest by hand is pointless: on the next kubeadm upgrade (part 35) it will be regenerated from kubeadm-config.

Audit writes a lot. On our lab cluster of three nodes with almost no applications the file grows by 3.7 MB per minute, mostly because of controllers and Calico, so the audit-log-maxsize and audit-log-maxbackup limits above (a 100 MB file and five old ones) keep about three hours on disk. You cannot rely on the audit-log-maxage: 7 period here, which is one more reason to ship the audit log off the node right away.

The audit log is a file on the master, so it has to be collected by the same Alloy. The instance on k8s-master-1 is there thanks to the toleration in the values, and two components are added to the configuration:

local.file_match "audit" {
  path_targets = [{
    "__path__" = "/var/log/kubernetes/audit/audit.log",
    "job"      = "audit",
    "node"     = sys.env("HOSTNAME"),
  }]
}

loki.source.file "audit" {
  targets    = local.file_match.audit.targets
  forward_to = [loki.write.default.receiver]
}

Workers have no such file, and the Alloy instances on them simply read nothing. Each line is a JSON event:

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "Metadata",
  "auditID": "a32fbc03-5ff5-4de7-9f3f-70bf2a485ae8",
  "stage": "ResponseComplete",
  "requestURI": "/api/v1/namespaces/demo/secrets/hello-db",
  "verb": "get",
  "user": {
    "username": "alice",
    "groups": ["developers", "system:authenticated"],
    "extra": {
      "authentication.kubernetes.io/credential-id": ["X509SHA256=900edb442a1e478b5d2011e9678383508ec7165174129f232fc65e20895eb22e"]
    }
  },
  "sourceIPs": ["192.168.10.17"],
  "userAgent": "kubectl/v1.36.5 (linux/amd64) kubernetes/ad950d1",
  "objectRef": { "resource": "secrets", "namespace": "demo", "name": "hello-db", "apiVersion": "v1" },
  "responseStatus": { "metadata": {}, "code": 200 },
  "requestReceivedTimestamp": "2026-11-04T05:54:39.382006Z",
  "stageTimestamp": "2026-11-04T05:54:39.382953Z",
  "annotations": {
    "authorization.k8s.io/decision": "allow",
    "authorization.k8s.io/reason": "RBAC: allowed by RoleBinding \"dev-secrets/demo\" of Role \"secret-reader\" to Group \"developers\""
  }
}

The authorization annotations show which RBAC binding from part 29 allowed the request, which is the best way to find overly broad rights. A denial is visible the same way: alice's attempt to read someone else's secret has code 403 and authorization.k8s.io/decision: forbid. The value of the secret is not in the file: kubectl create secret is recorded at the Metadata level, without a body. In LogQL: {job="audit"} | json | objectRef_resource="secrets" and verb="get" | line_format "{{.user_username}} {{.objectRef_namespace}}/{{.objectRef_name}}".

Traces

A metric says: "the 95th percentile of response time has grown to 800 ms". A log says: "the request to /orders was handled in 812 ms". Neither answers where those 800 ms went if the request passed through Traefik, an orders service, a payments service and a database. A trace shows this.

A trace is a tree of operations performed for the sake of one external request. Each operation is a span: a name, start and end time, a span id, a parent span id, attributes (http.request.method, http.response.status_code, db.statement), events inside the span, a status. All spans of one trace carry a common trace id: 16 random bytes written as 32 hexadecimal characters.

 trace id 4bf92f3577b34da6a3ce929d0e0e4736
 ─────────────────────────────────────────────────────────────────────►  time
 [traefik: GET /orders                                          812 ms]
   [orders: GET /orders                                       798 ms]
     [orders: SELECT orders           31 ms]
     [orders: POST payments/check                          744 ms]
       [payments: POST /check                            739 ms]
         [payments: call bank API                      722 ms]

Context in the traceparent header

For the spans of different services to add up to one trace, each service has to pass the trace id and the id of its span to the next one. The format of this transfer is standardized by the W3C as Trace Context: the HTTP header traceparent of four dash-separated fields:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             │  │                                │                └─ flags: 01 means the trace is sampled
             │  │                                └─ span id of the caller (8 bytes)
             │  └─ trace id (16 bytes)
             └─ format version

A second header, tracestate, carries additional data of specific tracing systems. A service that receives a request with traceparent creates its span as a child of the given one and on every outgoing call sets traceparent with its own span id. If there is no header, the service starts a new trace. Ordinary proxies, including Traefik from part 18, pass the header through; Traefik with tracing enabled also adds two spans of its own: a server span for the incoming request and a client ReverseProxy span for the call to the service.

Sampling. Recording every trace is expensive, so you decide which ones to keep. Head sampling is a decision at the beginning of the trace, in the first service: for example, 10 % of traces, and the decision (the 01 flag) is passed on in traceparent so that a trace is either whole or absent. Tail sampling is a decision in the collector after all the spans of a trace have been gathered: "keep all traces with errors and slower than a second, and 5 % of the rest". The second is more precise, but requires all spans of a trace to reach one collector instance.

OpenTelemetry

OpenTelemetry (OTel) is a CNCF project that standardizes telemetry collection: APIs and SDKs for programming languages, the OTLP protocol (OpenTelemetry Protocol, transferring traces, metrics and logs over gRPC on port 4317 or HTTP on 4318), attribute naming conventions (semantic conventions) and a collector. An application is instrumented once, and where to send the data is decided by configuration.

API: the interfaces for creating spans, used by libraries and application code.
SDK: the implementation. It generates ids, makes sampling decisions, accumulates spans in batches and sends them with an exporter over OTLP. It is configured with OTEL_* environment variables.
Instrumentation libraries: wrappers for HTTP servers and clients, database drivers, gRPC. They create spans and carry traceparent without the programmer's involvement.
OpenTelemetry Collector: a separate service, a "receivers → processors → exporters" pipeline. Applications send data to it, and it enriches, filters and samples the data and forwards it to storage. The collector decouples applications from storage: replacing Tempo with something else does not require rebuilding services.

Tempo

Tempo is a trace store from Grafana Labs. Like Loki, it indexes almost nothing and keeps everything in object storage: spans are laid out by column in Apache Parquet files, and a query reads only the columns it needs. It accepts OTLP, Jaeger and Zipkin, and searches for a trace by trace id and by conditions in the TraceQL language.

At the time of writing the current branch is Tempo 3.x (2.x still gets fixes). The internal architecture changed in the third version: in microservices mode writes go through a Kafka-compatible queue that serves as a write-ahead log, fresh data is served by the live-store instead of the former ingesters, and compaction and deletion of old blocks are done by the backend scheduler and backend worker instead of the compactor. In monolithic mode (-target=all) Kafka is not needed: all components are in one process, and blocks go to S3. That is the one we take.

# tempo-values.yaml
tempo:
  retention: 168h
  extraArgs:
    config.expand-env: true
  extraEnvFrom:
  - secretRef:
      name: tempo-s3
  storage:
    trace:
      backend: s3
      s3:
        bucket: tempo-traces
        endpoint: s3.example.com:3900
        region: garage
        access_key: ${S3_ACCESS_KEY_ID}
        secret_key: ${S3_SECRET_ACCESS_KEY}
        forcepathstyle: true
        insecure: true
      wal:
        path: /var/tempo/wal

persistence:
  enabled: true
  storageClassName: local-path
  size: 10Gi

$ helm install tempo oci://ghcr.io/grafana-community/helm-charts/tempo \
    --version 3.2.1 -n monitoring -f tempo-values.yaml

The tempo-traces bucket, a separate key and the tempo-s3 Secret are created the same way as for Loki. Chart 3.2.1 installs Tempo 3.1.1. The default receivers are OTLP on ports 4317 and 4318 and several Jaeger protocols; you cannot turn Jaeger off with jaeger: null: the chart's service template expects its settings and fails with nil pointer evaluating interface {}.protocols.

If you enable the metrics-generator in Tempo, it will build RED metrics (requests, errors, duration) per service and a graph of calls between services from traces and send them to Prometheus over remote write; for this, reception is enabled in Prometheus (enableRemoteWriteReceiver: true in prometheusSpec).

The collector

We install the collector with the OpenTelemetry project's Helm chart in Deployment mode. The image is the opentelemetry-collector-k8s distribution, which includes the k8sattributes processor: by the IP address the spans came from it finds the pod and adds the attributes k8s.namespace.name, k8s.pod.name, k8s.deployment.name, k8s.node.name to them. Inside the cluster pods reach the collector without NAT (part 14), so the source address is the pod's address.

# otel-collector-values.yaml
fullnameOverride: otel-collector
mode: deployment
image:
  repository: otel/opentelemetry-collector-k8s
presets:
  kubernetesAttributes:
    enabled: true             # the k8sattributes processor and rights to read pods
config:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: ${env:MY_POD_IP}:4317
        http:
          endpoint: ${env:MY_POD_IP}:4318
  processors:
    memory_limiter:
      check_interval: 1s
      limit_percentage: 80
      spike_limit_percentage: 25
    batch: {}
  exporters:
    otlp_grpc/tempo:
      endpoint: tempo.monitoring.svc:4317
      tls:
        insecure: true
  service:
    pipelines:
      traces:
        receivers: [otlp]
        processors: [memory_limiter, k8sattributes, batch]
        exporters: [otlp_grpc/tempo]

$ helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
$ helm install otel-collector open-telemetry/opentelemetry-collector \
    --version 0.178.2 -n monitoring -f otel-collector-values.yaml

memory_limiter comes first: when memory runs short the collector starts refusing data instead of dying of OOM. batch gathers spans into batches before sending. The OTLP exporter over gRPC is called otlp_grpc in recent collector versions. Traffic between the collector and Tempo stays inside the cluster without TLS, so the Tempo ports are closed with NetworkPolicy so that only the collector can write to it.

Tracing in hello

Version 1.2 of hello gets SDK setup at startup, a wrapper around the HTTP handler and the trace id in logs. Along the way the handwritten /metrics from part 5 is replaced with the Prometheus client library: a histogram hello_request_duration_seconds joins the former hello_requests_total and hello_build_info, and only the library can attach exemplars, links to traces, to its points.

$ go get go.opentelemetry.io/otel \
    go.opentelemetry.io/otel/sdk \
    go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc \
    go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp \
    github.com/prometheus/client_golang/prometheus

import (
    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
    "go.opentelemetry.io/otel/propagation"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    "go.opentelemetry.io/otel/trace"
)

// The SDK takes the collector address, the service name and the sampler from the OTEL_* variables.
func setupTracing(ctx context.Context) (func(context.Context) error, error) {
    exp, err := otlptracegrpc.New(ctx)
    if err != nil {
        return nil, err
    }
    tp := sdktrace.NewTracerProvider(sdktrace.WithBatcher(exp))
    otel.SetTracerProvider(tp)
    // read and pass on the traceparent and baggage headers
    otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
        propagation.TraceContext{}, propagation.Baggage{}))
    return tp.Shutdown, nil
}

mux.HandleFunc("GET /{$}", func(w http.ResponseWriter, r *http.Request) {
    start := time.Now()
    ctx := r.Context()
    sc := trace.SpanContextFromContext(ctx) // the span created by otelhttp

    requests.Inc()
    w.Header().Set("X-App-Version", version)
    fmt.Fprintf(w, "hello from %s\n", hostname)

    elapsed := time.Since(start)
    if sc.IsSampled() {
        // exemplar: a histogram point with a link to the trace
        requestDuration.(prometheus.ExemplarObserver).ObserveWithExemplar(
            elapsed.Seconds(), prometheus.Labels{"trace_id": sc.TraceID().String()})
    } else {
        requestDuration.Observe(elapsed.Seconds())
    }
    slog.InfoContext(ctx, "request served",
        "method", r.Method, "path", r.URL.Path, "status", http.StatusOK,
        "duration_ms", float64(elapsed.Microseconds())/1000,
        "trace_id", sc.TraceID().String(), "span_id", sc.SpanID().String(),
        "trace_sampled", sc.IsSampled())
})

// kubelet probes and Prometheus scrapes have no place in traces
handler := otelhttp.NewHandler(mux, "hello", otelhttp.WithFilter(func(r *http.Request) bool {
    return r.URL.Path == "/"
}))

main calls setupTracing, and on stopping by SIGTERM the function it returned, to send the accumulated spans. On every incoming request the otelhttp wrapper reads traceparent, creates a server span with HTTP attributes and puts it in the request context; it takes the span name from the route pattern, so in traces the span is called GET /{$}. The Dockerfile from part 5 changes in one line: go.sum has appeared and has to be copied as well before go mod download. For the histogram to serve exemplars, the /metrics handler is created with promhttp.HandlerOpts{EnableOpenMetrics: true}: exemplars exist only in the OpenMetrics format, and Prometheus asks for it by itself. It looks like this:

hello_request_duration_seconds_bucket{le="0.0001"} 101 # {trace_id="482625a9a77fdc350ae275042e5007af"} 4.675e-06 1.793771092125748e+09
hello_request_duration_seconds_bucket{le="0.00025"} 101

At the end of the line, after #, are the trace id, the measured value and its time.

The settings go in the Deployment manifest, not in the code:

        env:
        - name: OTEL_SERVICE_NAME
          value: hello
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: http://otel-collector.monitoring.svc:4317
        - name: OTEL_TRACES_SAMPLER
          value: parentbased_traceidratio
        - name: OTEL_TRACES_SAMPLER_ARG
          value: "0.1"
        - name: OTEL_RESOURCE_ATTRIBUTES
          value: service.version=1.2

parentbased_traceidratio with the argument 0.1: if a request came with traceparent, obey the caller's decision; if without, keep 10 % of traces. Under load this shows in the numbers: hello served 261 requests per second, and TraceQL counted 27 spans per second in Tempo, right around 10 %. A trace id is generated for every request and goes into the log, but only a sampled trace ends up in Tempo; the trace_sampled field in the log tells you right away whether it is worth looking for. The http:// scheme in the collector address means a connection without TLS.

TraceQL

A TraceQL query selects spans by conditions in curly braces and returns the traces that contain them. Attributes are written with a scope: resource. for attributes of the source (service, pod), span. for attributes of the span; with no scope, both are searched.

{ resource.service.name = "hello" && span.http.response.status_code >= 500 }

{ resource.service.name = "hello" && duration > 200ms }

{ resource.k8s.namespace.name = "demo" } >> { span.db.system = "postgresql" && duration > 50ms }

{ resource.service.name = "hello" } | rate() by (resource.k8s.pod.name)

The third query uses the >> "descendant" operator: traces where under a span from demo there is a slow query to PostgreSQL. hello has no database, so on our lab it returns nothing, but in an application with an instrumented driver this is the most useful kind of query. The fourth is metrics from traces (TraceQL metrics): the rate of spans per pod, built directly from the stored traces.

Linking in Grafana: metrics, logs, traces

The three data sources become useful together when you can go from one to another by a common key. There are two keys: labels (namespace, pod are the same in Prometheus, Loki and the trace attributes thanks to k8sattributes) and the trace id.

 Prometheus                      Loki                              Tempo
 ┌─────────────────────┐ labels  ┌─────────────────────┐ derived   ┌─────────────────────┐
 │ hello latency       │ ns, pod │ {"msg":"request …", │ field     │ trace 4bf92f35…     │
 │ + exemplar trace_id ├────────►│  "trace_id":"4bf9…"}├──────────►│ spans by service    │
 └──────────┬──────────┘         └─────────────────────┘           └──┬──────────────────┤
            │                               ▲                         │                  ▲
            │                               │  trace to logs          │                  │
            │                               └─────────────────────────┘                  │
            │                 exemplar: straight to trace                                │
            └────────────────────────────────────────────────────────────────────────────┘

Exemplars: Prometheus keeps individual examples with a trace id next to histogram points (the exemplar-storage feature is needed), and Grafana draws them as points on the latency graph; a click leads to Tempo. From "a spike on the graph" to a specific slow request is one click.
Derived fields in the Loki data source: a regular expression that finds the trace id in a log line and turns it into a link to Tempo.
Trace to logs in the Tempo data source: from a span open the logs of the same pod for the same time, filtered by trace id.

All of these are Grafana data source settings, set in the kube-prometheus-stack values from the previous part:

# addition to kps-values.yaml
prometheus:
  prometheusSpec:
    enableFeatures: [exemplar-storage]

grafana:
  sidecar:
    datasources:
      exemplarTraceIdDestinations:
        datasourceUid: tempo
        traceIdLabelName: trace_id
  additionalDataSources:
  - name: Loki
    uid: loki
    type: loki
    url: http://loki-gateway.monitoring.svc
    jsonData:
      derivedFields:
      - name: TraceID
        matcherRegex: '"trace_id":"([0-9a-f]{32})"'
        datasourceUid: tempo
        url: '$${__value.raw}'
        urlDisplayLabel: Open trace
  - name: Tempo
    uid: tempo
    type: tempo
    url: http://tempo.monitoring.svc:3200
    jsonData:
      tracesToLogsV2:
        datasourceUid: loki
        filterByTraceID: true
        spanStartTimeShift: -5m
        spanEndTimeShift: 5m
        tags:
        - key: k8s.namespace.name
          value: namespace
        - key: k8s.pod.name
          value: pod
      serviceMap:
        datasourceUid: prometheus
      nodeGraph:
        enabled: true

$$ in url is escaping: in Grafana provisioning files a single $ means substituting an environment variable, and here we need a literal ${__value.raw}, the value found by the regular expression. Grafana removes one level of escaping, and ${__value.raw} is what remains in the data source settings. tags in tracesToLogsV2 map trace attributes to Loki labels: from a span with k8s.pod.name=hello-… Grafana builds a query to the stream {namespace="demo", pod="hello-…"} and keeps the lines in it where the trace id occurs, in the text or in structured metadata.

A hands-on look

Rotation on the node. The container directory after a couple of rotations:

user@k8s-node-1:~$ sudo ls -l /var/log/pods/demo_hello-5878b94bb8-mt56n_70a2bcc9-9f5f-4df4-99bf-cf2a35e2c18d/hello/
-rw-r----- 1 root root  4582200 Nov  4 06:10 0.log
-rw-r----- 1 root root  1720726 Nov  4 06:07 0.log.20261104-055949.gz
-rw-r----- 1 root root 10583775 Nov  4 06:07 0.log.20261104-060709
user@k8s-node-1:~$ ls -l /var/log/containers/ | grep hello-5878b94bb8-mt56n
lrwxrwxrwx 1 root root 90 Nov  4 05:41 hello-5878b94bb8-mt56n_demo_hello-33f551c709bffd09f6c5ac74e3cde6c6c7ea43b28744f82bf8c6cff1e0b0b2d7.log -> /var/log/pods/demo_hello-5878b94bb8-mt56n_70a2bcc9-9f5f-4df4-99bf-cf2a35e2c18d/hello/0.log

The current file, the last rotated one uncompressed (the agent may not have finished reading it) and an older one in gzip: JSON compresses sixfold. kubectl logs sees only the current file, and the tens of thousands of lines from the other two are out of its reach:

user@k8s-node-1:~$ sudo wc -l /var/log/pods/demo_hello-5878b94bb8-mt56n_70a2bcc9-9f5f-4df4-99bf-cf2a35e2c18d/hello/0.log
16622 /var/log/pods/demo_hello-5878b94bb8-mt56n_70a2bcc9-9f5f-4df4-99bf-cf2a35e2c18d/hello/0.log
$ kubectl logs -n demo hello-5878b94bb8-mt56n | wc -l
16622

A request with a given traceparent. Let us generate a trace id ourselves and send a request through Traefik with the "sample" flag:

$ TRACE=$(openssl rand -hex 16); SPAN=$(openssl rand -hex 8); echo $TRACE
94b7f00c2965404ab99ccf31b1a034f4
$ curl -s -H "traceparent: 00-$TRACE-$SPAN-01" https://hello.apps.example.com/
hello from hello-5878b94bb8-mt56n

From log to trace. logcli is the Loki console client, available as a separate file in Loki releases on GitHub; through a port-forward to the gateway we find the line by structured metadata:

$ kubectl port-forward -n monitoring svc/loki-gateway 3100:80 &
$ export LOKI_ADDR=http://localhost:3100
$ logcli query --quiet --since=10m '{namespace="demo", container="hello"} | trace_id="94b7f00c2965404ab99ccf31b1a034f4"'
2026-11-04T06:06:38Z {} {"time":"2026-11-04T06:06:38.434392517Z","level":"INFO","msg":"request served","method":"GET","path":"/","status":200,"duration_ms":0.003,"trace_id":"94b7f00c2965404ab99ccf31b1a034f4","span_id":"cd05e89b3ff79fed","trace_sampled":true}

{} in place of the labels means that all of them are common to the lines found: logcli prints them in a separate line, which --quiet removes. The trace_id in the log matches the given one: otelhttp read traceparent and continued our trace, and span_id is already the own span of hello. The trace in Tempo, through its HTTP API:

$ kubectl port-forward -n monitoring svc/tempo 3200 &
$ curl -s localhost:3200/api/v2/traces/94b7f00c2965404ab99ccf31b1a034f4 | jq -r '
    .trace.resourceSpans[] | .resource.attributes as $r | .scopeSpans[].spans[] |
    "\(($r[] | select(.key=="k8s.pod.name") | .value.stringValue)) \(.name) \(.kind)"'
hello-5878b94bb8-mt56n GET /{$} SPAN_KIND_SERVER

There is one span: our curl did not record a span of its own, and Traefik without tracing enabled only passed the header on. The k8s.pod.name attribute was added by the collector. If tracing is enabled in Traefik as well, the same request gives three of them:

traefik | GET | SPAN_KIND_SERVER | 973.824 us
traefik | ReverseProxy | SPAN_KIND_CLIENT | 831.232 us
hello | GET /{$} | SPAN_KIND_SERVER | 69.888 us

This also shows the main thing traces are for: of almost a millisecond, only 70 microseconds passed in hello itself. Searching with TraceQL uses the same API. hello answers so fast that a 1 ms threshold selects only rare requests:

$ curl -sG localhost:3200/api/search --data-urlencode 'q={ resource.service.name = "hello" && duration > 1ms }' --data-urlencode 'limit=3' \
    | jq -r '.traces[] | "\(.traceID) \(.rootTraceName) \(.durationMs) ms"'
59bb1283ad496d0f0281ae8b14b8a30a GET /{$} 1 ms
6e7da5bc93db26c51119f9c4bc2f22b9 GET /{$} 1 ms
4c2c2b5de9fe53ef061b0e6dd804e659 GET /{$} 1 ms

From graph to trace. Exemplars are also visible through the Prometheus API: trace ids attached to histogram points, with the value in seconds:

$ curl -sG localhost:9090/api/v1/query_exemplars \
    --data-urlencode 'query=hello_request_duration_seconds_bucket{namespace="demo"}' \
    --data-urlencode "start=$(date -d '-10 min' +%s)" --data-urlencode "end=$(date +%s)" \
    | jq -r '.data[0].exemplars[:2][] | "\(.labels.trace_id) \(.value)"'
6b205cf6da6e77d400deccb09a6e3307 0.000001889
27ac9e1ff24eb94d1548bbd19dfbbebc 0.000004743

In Grafana it looks like this: on a panel with histogram_quantile over the latency of hello, diamonds of exemplars run over the line, and the exemplarTraceIdDestinations setting leads from them to Tempo by the trace_id field. On the trace page the "Logs for this trace" button opens a Loki panel alongside with the log line of the same pod and the same trace id.

Metrics from logs. The same hello requests can also be counted from logs:

$ logcli instant-query --quiet 'sum by (pod) (rate({namespace="demo", container="hello"} | json | status >= 200 [5m]))' | jq -c '.[] | [.metric.pod, .value[1]]'
["hello-5878b94bb8-hkxd2","87.10666666666667"]
["hello-5878b94bb8-jpbmw","86.85"]
["hello-5878b94bb8-mt56n","86.50666666666666"]

The sum agrees with the 261 requests per second shown by hello_requests_total in Prometheus.

Events. A pod with a nonexistent image has already been deleted, but its events remain:

$ logcli query --quiet --since=5m --limit=2 '{job="kubernetes-events"} |= "ev-test"'
2026-11-04T05:51:04Z {detected_level="error"}   {"count":2,"eventRV":"2007593","kind":"Pod","msg":"Error: ImagePullBackOff","name":"ev-test","objectAPIversion":"v1","objectRV":"2007513","reason":"Failed","reportingcontroller":"kubelet","reportinginstance":"k8s-node-1","sourcecomponent":"kubelet","sourcehost":"k8s-node-1","type":"Warning"}
2026-11-04T05:51:04Z {detected_level="unknown"} {"count":2,"eventRV":"2007590","kind":"Pod","msg":"Back-off pulling image \"busybox:1.37-does-not-exist\"","name":"ev-test","objectAPIversion":"v1","objectRV":"2007513","reason":"BackOff","reportingcontroller":"kubelet","reportinginstance":"k8s-node-1","sourcecomponent":"kubelet","sourcehost":"k8s-node-1","type":"Normal"}

Audit. Who read the hello-db secret in the last half hour:

$ logcli query --quiet --since=30m '{job="audit"}
    | json user="user.username", verb="verb", ip="sourceIPs[0]", res="objectRef.resource", name="objectRef.name"
    | res="secrets" and name="hello-db" and user="alice"
    | line_format "{{.user}} {{.verb}} {{.ip}}"'
2026-11-04T05:54:39Z {} alice get 192.168.10.17

The json parser with expressions takes only the needed fields, including nested ones (user.username) and array elements (sourceIPs[0]); without expressions it extracts all fields, joining nested names with an underscore (objectRef_resource), but skips arrays.

Metrics, logs and traces are now stored off the nodes and linked to each other, and cluster events and the API audit outlive the hour and a pod restart. The observability block is finished. The next one is about operating the cluster itself, and it starts with the component without which there is no cluster at all.

Sources

Kubernetes: Logging Architecture: node log files, containerLogMaxSize, containerLogMaxFiles.
Kubernetes: Auditing: stages, levels, policy, kube-apiserver flags.
Grafana Alloy, migrating from Promtail, loki.source.kubernetes_events.
Loki: deployment modes, structured metadata, LogQL.
Tempo: what's new in version 3.0, TraceQL.
grafana-community charts: Loki, Tempo and Grafana.
W3C Trace Context: the traceparent and tracestate headers.
OpenTelemetry: collector, SDK for Go, OTEL_* environment variables.
Prometheus: feature flags: exemplar-storage.

Next — part 34: etcd. How it stores the cluster state, what Raft and quorum are, how to work with it through etcdctl, take snapshots and restore from them.