29 October 2026, 10:42Виталий Лещенко

Kubernetes for beginners. Part 16: How does DNS work in the cluster?

All parts of the series “Kubernetes for beginners”

In the previous part the hello service got a stable address from the service network, and clients in the cluster reached it by name — hello, hello.demo, hello.demo.svc.cluster.local. The cluster DNS, CoreDNS, turns the name into the address. Today we look at how it works: the Deployment and the kube-dns Service, the Corefile and its plugins, which records CoreDNS serves for services, pods, headless services and StatefulSets, what kubelet writes into every pod's /etc/resolv.conf, and why ndots:5 turns one query for an external name into several. The DNS protocol itself — queries and answers, record types, recursion and caching — is covered in part 12 of the networking series; here we only look at what Kubernetes does with it.

The demo namespace with the three-pod hello Deployment is rebuilt from part 11 and the hello service from the previous part, so its ClusterIP is different: 10.102.84.146. Yours will be different too — take it from kubectl -n demo get svc hello.

CoreDNS in the cluster

CoreDNS is a DNS server written in Go, a CNCF project built from plugins: each plugin does one job (answers from the Kubernetes API, forwarding, caching, logging), and the configuration file, the Corefile, describes their chain for each zone. kubeadm installs it during init as an addon:

$ kubectl -n kube-system get deployment,svc -l k8s-app=kube-dns
NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/coredns   2/2     2            2           4d10h

NAME               TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
service/kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   4d10h
$ kubectl -n kube-system get pods -l k8s-app=kube-dns -o wide
NAME                       READY   STATUS    RESTARTS   AGE     IP              NODE           NOMINATED NODE   READINESS GATES
coredns-589f44dc88-bzxpn   1/1     Running   0          4d10h   172.20.70.194   k8s-master-1   <none>           <none>
coredns-589f44dc88-lv5bl   1/1     Running   0          4d10h   172.20.70.197   k8s-master-1   <none>           <none>

The coredns Deployment — two replicas of registry.k8s.io/coredns/coredns:v1.14.2 (the version pinned by kubeadm 1.36). The pods have the system-cluster-critical priority class, a request of 100m CPU and 70Mi of memory, a memory limit of 170Mi, dnsPolicy: Default (more on it below), tolerations for the control plane taint and CriticalAddonsOnly, and a soft anti-affinity across nodes (preferredDuringScheduling… with weight 100). The container runs with a read-only root filesystem and no capabilities except NET_BIND_SERVICE — the right to listen on port 53. The liveness probe goes to /health on port 8080, the readiness probe to /ready on port 8181.
The kube-dns Service with the fixed address 10.96.0.10 — the tenth address of the service network. kubeadm sets it explicitly, from the lower band of the range, which automatic allocation uses last (part 15). The name kube-dns is historical: that was the name of the cluster DNS server before CoreDNS, and the service kept it for compatibility. Port 9153 is metrics.

Both replicas are on the master: kubeadm creates the Deployment during init, the pods start as soon as the pod network appears — and there are no workers yet at that moment (on the lab CoreDNS was ready two minutes before k8s-node-1 joined). The scheduler does not move running pods (part 13), and the anti-affinity here is soft, so it fixes nothing by itself: if the master is down, the cluster DNS goes down with it. After the workers join, recreate the CoreDNS pods with kubectl -n kube-system rollout restart deployment coredns: a new ReplicaSet creates the pods again, and the soft anti-affinity spreads them over different nodes. We did not do this on the lab for this article, so in the output below both replicas live on the master.

For such a critical component it also makes sense to add a PodDisruptionBudget with maxUnavailable: 1, so that drain does not evict both replicas at once. kubeadm does not create one:

$ kubectl -n kube-system get pdb
No resources found in kube-system namespace.

CoreDNS keeps no cluster records in files: the kubernetes plugin holds a watch on Services and EndpointSlices (and, in pods verified mode, on pods) and builds answers from that in-memory cache. A new service answers by name right after it is created: on the lab the very first dig run right after kubectl create service got the address, and after the service was deleted the name stopped resolving at once.

The Corefile

The configuration is the coredns ConfigMap in kube-system. This is the Corefile kubeadm 1.36 writes (the kubernetes plugin block is shortened to "…" in the output; we go through it below):

$ kubectl -n kube-system get configmap coredns -o jsonpath='{.data.Corefile}'
.:53 {
    errors
    health {
       lameduck 5s
    }
    ready
    …
    prometheus :9153
    forward . /etc/resolv.conf {
       max_concurrent 1000
    }
    cache 30 {
       disable success cluster.local
       disable denial cluster.local
    }
    loop
    reload
    loadbalance
}

.:53 is a server block: the zone . (the root, that is, every name) on port 53. Inside are plugins. The order in which they run is set not by the order of lines in the file but when CoreDNS is built (the plugin.cfg file in the sources); in the Corefile only which plugins are enabled and with which parameters matters.

Plugin	What it does
`errors`	logs query processing errors to the pod log
`health`	HTTP `:8080/health` for the liveness probe; `lameduck 5s` — on shutdown keep working for 5 more seconds, and `/health` answers 200 all that time
`ready`	HTTP `:8181/ready` for the readiness probe: 200 once all plugins are ready (including `kubernetes` having synced with the API); in `lameduck` mode it stops answering OK, and the pod leaves the EndpointSlice while it still serves queries
`kubernetes`	answers names in `cluster.local` and reverse queries from API data
`prometheus`	metrics in the format of Prometheus — a monitoring system that periodically scrapes them over HTTP (part 32) — on `:9153`
`forward`	sends everything the previous plugins did not handle to the upstream servers from the CoreDNS pod's `/etc/resolv.conf`
`cache`	caches answers for at most 30 seconds, except answers about `cluster.local`
`loop`	detects a forwarding loop at startup and stops CoreDNS
`reload`	rereads the Corefile when the ConfigMap changes (without restarting the pod)
`loadbalance`	shuffles the order of records in the answer so that clients do not always go to the first address

Which records exist

Name	Type	Answer
`hello.demo.svc.cluster.local`	A	the service ClusterIP: `10.102.84.146`
`hello-headless.demo.svc.cluster.local`	A	the addresses of all ready pods of the headless service
`_http._tcp.hello.demo.svc.cluster.local`	SRV	port 80 and the service name
`_http._tcp.hello-headless.demo.svc.cluster.local`	SRV	one record per pod: port 8080 and the pod name
`web-0.web.demo.svc.cluster.local`	A	the address of pod `web-0` of the `web` StatefulSet
`172-20-172-24.hello.demo.svc.cluster.local`	A	`172.20.172.24` (a pod behind a service, name from the address)
`172-20-172-24.demo.pod.cluster.local`	A	`172.20.172.24` (a pod record by address)
`registry.demo.svc.cluster.local`	CNAME	`ghcr.io` (an `ExternalName` service)
`146.84.102.10.in-addr.arpa`	PTR	`hello.demo.svc.cluster.local`
`24.172.20.172.in-addr.arpa`	PTR	one name per service that selects the pod

Services. The name is <service>.<namespace>.svc.<cluster domain>. A normal service gets an A record with the ClusterIP. That is a single address however many pods there are: kube-proxy balances, not DNS.

SRV for named ports. SRV is a record type that gives not only the address but also the port of a service: _<port name>._<protocol>.<service>.<namespace>.svc.cluster.local returns priority, weight, port and target name. The records are created only for ports with a name — one more reason to name ports. Few clients look up SRV themselves (some database drivers and message brokers do), but this way you can learn the port of a service without knowing it in advance.

Headless services. A service with clusterIP: None has no address of its own, and the A record for its name returns the addresses of all ready pods from the EndpointSlice. The client gets the list and picks for itself; loadbalance shuffles the order every time. The SRV records of a headless service point at each pod separately, with its own port and equal weights.

StatefulSet. If a pod has spec.hostname and spec.subdomain set, and the namespace has a headless service named the same as subdomain, CoreDNS creates the name <hostname>.<subdomain>.<namespace>.svc.cluster.local. A StatefulSet fills these fields itself (hostname = pod name, subdomain = serviceName), hence the stable names web-0.web.demo.svc.cluster.local (part 8). An ordinary pod can do the same explicitly. The record appears only when the pod is ready (unless the service has publishNotReadyAddresses: true).

Names of pods behind a service. A pod without hostname gets a name made of its address with dashes under every service that selects it: 172-20-172-24.hello-headless.demo.svc.cluster.local. CoreDNS answers such names for normal ClusterIP services too (172-20-172-24.hello.demo.svc.cluster.local), although the documentation describes them as a CoreDNS feature rather than a required part of the specification.

Pod records by address. <address with dashes>.<namespace>.pod.cluster.local. With pods insecure CoreDNS answers any such name with the address taken from the name itself, without checking whether the pod exists: 10-1-2-3.demo.pod.cluster.local resolves to 10.1.2.3. They are useful mostly for TLS certificates — TLS being the connection encryption protocol (part 23 of the networking series) — where a name is needed and the pod only has an address.

ExternalName. A service of type ExternalName has neither an address nor pods: CoreDNS answers its name with a CNAME record — an alias that says "this name is another name", and the resolver goes on looking up that one (part 15).

Reverse records (PTR). PTR is an "address → name" record, looked up by the address written backwards in the in-addr.arpa zone. For a service ClusterIP CoreDNS answers with the service name, and for a pod address with the names from the previous paragraph, one record for every service that selects the pod: a hello pod has two, from hello and from hello-headless. For addresses that are in no service, fallthrough passes the query to the upstream server, which usually answers NXDOMAIN.

What is not there. No records by pod name (hello-6d5c6db6b4-gn5n4) — only by address or by hostname; no records for a Deployment; no node names: CoreDNS resolves k8s-node-1 only through forward, if the upstream DNS knows it.

The pod's /etc/resolv.conf

kubelet decides which DNS server and which suffixes a pod sees when it creates the pod. It takes the parameters from its configuration:

user@k8s-node-1:~$ sudo grep -A1 -E 'clusterDNS|clusterDomain|resolvConf' /var/lib/kubelet/config.yaml
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
containerRuntimeEndpoint: unix:///var/run/containerd/containerd.sock
--
resolvConf: /run/systemd/resolve/resolv.conf
rotateCertificates: true

and writes a file that is mounted into every container of the pod:

$ kubectl -n demo exec net -- cat /etc/resolv.conf
search demo.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:5

nameserver 10.96.0.10 — the ClusterIP of the kube-dns service. Queries go through kube-proxy's DNAT and reach one of the CoreDNS pods.
search — the suffixes the resolver appends to incomplete names: first the pod's namespace (demo.svc.cluster.local), then all services (svc.cluster.local), then the cluster domain. After them kubelet adds the node's suffixes from the resolvConf file; on the lab the nodes have no domain of their own (search .), so the list ends at cluster.local. From demo plain hello works, from another namespace — hello.demo.
options ndots:5 — if a name has fewer than five dots, the resolver first tries it with every suffix from search and only then as is.

Why five: the name web-0.web.demo.svc (three dots) must resolve through the cluster.local suffix, and the longest in-cluster names like _http._tcp.hello.demo.svc have four dots. A threshold of 5 guarantees that any shortened cluster name is tried with the suffixes first.

Extra queries for external names

The flip side: the name api.github.com has two dots, fewer than five. The glibc resolver (the standard C library in Debian- and Ubuntu-based images and in the busybox image) sends, one after another:

api.github.com.demo.svc.cluster.local.   → NXDOMAIN (CoreDNS, from memory)
api.github.com.svc.cluster.local.        → NXDOMAIN (CoreDNS, from memory)
api.github.com.cluster.local.            → NXDOMAIN (CoreDNS, from memory)
api.github.com.                          → answer (CoreDNS → forward → network DNS server)

Four names — at least four queries. CoreDNS rejects the queries for the first three within tenths of a millisecond, but they still go through DNAT, conntrack and the network to the CoreDNS pod; the query for the last one goes outside and waits for the upstream server's answer. If the node has a search domain of its own (say, search example.com from DHCP), it becomes the fifth suffix in the list, and one more name, api.github.com.example.com., appears before the real answer, and its query goes outside as well. For an application that makes hundreds of outgoing requests per second to external APIs this is a noticeable delay and load on CoreDNS. The musl resolver (the C library in Alpine images) differs in details — for example, it queries all nameserver servers in parallel — but walks the search suffixes the same way: on the lab it went through the same four names in the same order.

Ways to fix it:

A fully qualified name with a trailing dot. api.github.com. is an absolute name, search is not applied: one name instead of four. The most reliable way for names of external services in application configuration; most HTTP clients and libraries understand the trailing dot. The downside is that the dot may end up in the Host header and in SNI (the server name a TLS client states at the start of the connection), and some servers handle it badly, so check it with the actual client.
dnsConfig in the pod — lower ndots:
```
spec:
  dnsConfig:
    options:
      - name: ndots
        value: "2"
```
kubelet replaces ndots:5 with ndots:2 and leaves the rest as it was. Now api.github.com (two dots) goes out as is right away, while hello and hello.demo (fewer than two dots) still get the suffixes. The price: a name like web-0.web.demo (two dots) is no longer tried with the suffixes first, so better write it in full. For pods that mostly talk to the outside world it is a good trade.
A cache on the node — NodeLocal DNSCache (below) does not reduce the number of queries but answers them from a cache on the same node, without DNAT and the network.

There is also the CoreDNS autopath plugin, which finds the client's pod and namespace by its address and answers the very first query of the chain with the final answer plus a CNAME to the found name. It requires pods verified, that is, a watch on all pods of the cluster, and is not part of the default configuration.

dnsPolicy and dnsConfig

How /etc/resolv.conf is built is set by the pod's spec.dnsPolicy field:

`dnsPolicy`	the pod's `/etc/resolv.conf`
`ClusterFirst` (default)	CoreDNS, cluster and node `search`, `ndots:5` — as above
`Default`	DNS servers and suffixes from the node's `resolvConf` file, cluster names do not resolve
`ClusterFirstWithHostNet`	like `ClusterFirst`, but for pods with `hostNetwork: true`
`None`	nothing automatic; everything comes from `dnsConfig`

The name Default is misleading: it is not the default but "as on the node". The CoreDNS pods themselves use it — otherwise CoreDNS would send queries to itself.

ClusterFirstWithHostNet exists because a pod in the node network (hostNetwork: true) with the ClusterFirst policy silently gets the Default settings, that is, the node's DNS, not the cluster's. If such a pod (a monitoring agent, an ingress controller in the node network) needs service names, it gets ClusterFirstWithHostNet. This policy is not supported on Windows nodes.

dnsConfig adds to what the policy generated or, with None, sets everything:

spec:
  dnsPolicy: None
  dnsConfig:
    nameservers:
      - 10.96.0.10
    searches:
      - demo.svc.cluster.local
      - svc.cluster.local
    options:
      - name: ndots
        value: "2"
      - name: timeout
        value: "2"

The API server checks the limits: at most three nameservers, at most 32 suffixes in searches and at most 2048 characters in them together; with None at least one server is required. Three servers is the limit of the glibc and musl resolvers themselves, which read only the first three nameserver lines.

Upstream servers and stub zones

Everything outside cluster.local CoreDNS forwards by forward . /etc/resolv.conf. And kubelet builds the CoreDNS pod's /etc/resolv.conf from the file named in its resolvConf parameter. This is where the loop happens on Ubuntu.

On Ubuntu Server names are resolved by systemd-resolved — a local caching service that listens on 127.0.0.53 and forwards queries to the real servers. The node's /etc/resolv.conf is a link to its "stub":

user@k8s-node-1:~$ ls -l /etc/resolv.conf
lrwxrwxrwx 1 root root 39 Aug 26 07:46 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf
user@k8s-node-1:~$ grep -v '^#' /etc/resolv.conf

nameserver 127.0.0.53
options edns0 trust-ad
search .

If kubelet passed this file to the CoreDNS pod, CoreDNS would forward queries to 127.0.0.53 — but in the pod's network namespace that is its own loopback, where there is no systemd-resolved (and in similar setups with 127.0.0.1 it is CoreDNS itself that listens there). So kubeadm checks during init and join whether systemd-resolved is running and, if it is, writes resolvConf: /run/systemd/resolve/resolv.conf into the node's /var/lib/kubelet/config.yaml (this line does not go into the shared kubelet-config ConfigMap). That is the file with the real upstream servers, which systemd-resolved maintains exactly for such programs:

user@k8s-node-1:~$ grep -v '^#' /run/systemd/resolve/resolv.conf

nameserver 192.168.10.2
nameserver 192.168.10.3
…
search .

There may be more than three nameserver lines in this file. Then kubelet keeps the first three for the pod and warns about the rest with a DNSConfigForming event (Nameserver limits were exceeded, some nameservers have been omitted) on every pod with dnsPolicy: Default or in the node network, CoreDNS included. If the nodes have more than three servers or some of them are unreachable from pods, it is more reliable to list the upstream servers in the Corefile explicitly: forward . 192.168.10.2 192.168.10.3.

If you see CoreDNS pods in CrashLoopBackOff with the error plugin/loop: Loop (127.0.0.1:<port> -> :53) detected for zone "." in the log, check exactly kubelet's resolvConf: the loop is almost always caused by a loopback address in the file kubelet gave to the CoreDNS pod.

Stub zones. If the network has a separate DNS server for an internal zone (say, corp.example.com is served by 192.168.10.53) and the ordinary servers do not know about it, a separate server block is added to the Corefile:

corp.example.com:53 {
    errors
    cache 30
    forward . 192.168.10.53
}

CoreDNS picks the block by the longest matching zone: queries for *.corp.example.com go to 192.168.10.53, everything else to the .:53 block. The block has its own set of plugins: without prometheus in it its queries do not show up in the coredns_dns_requests_total metrics, without log — in the log. Static records without a separate server — for example, the name of your own registry from the "Your own registry" section of part 5 — can be added with the hosts plugin:

    hosts {
       192.168.10.32 registry.example.com
       fallthrough
    }

in the .:53 block. Without an explicit file name hosts also reads the /etc/hosts of the CoreDNS pod itself, and fallthrough passes on down the chain everything that is not in these entries. But it is better to keep such names in the real DNS rather than in the cluster configuration: otherwise nodes and pods resolve the same name differently.

NodeLocal DNSCache

In large clusters DNS is limited not by CoreDNS but by the path to it. Every query is a UDP packet through kube-proxy's DNAT, and each one creates a conntrack entry that lives for 30 seconds (nf_conntrack_udp_timeout, 30 on the lab nodes). Worse, glibc may send several queries from one socket almost at once, and DNAT of such packets can race when inserting conntrack entries: one of the packets is lost, and the resolver waits for a timeout — 5 seconds by default. These "random five-second DNS delays" are a well-known problem of clusters with many queries.

NodeLocal DNSCache solves it with a caching agent on every node:

the node-local-dns DaemonSet in kube-system — a build of CoreDNS in cache mode (the k8s-dns-node-cache image), in the node network;
the agent brings up a dummy interface on the node with a link-local address (the documentation's example is 169.254.20.10) and listens on it;
for packets to and from that address it adds NOTRACK rules so that conntrack does not track them — and the DNAT race is gone;
cache misses for cluster.local and the reverse zones the agent forwards to CoreDNS over TCP (force_tcp in its Corefile), where TCP itself retransmits a lost packet within a fraction of a second instead of the resolver waiting out a five-second timeout; external names the agent sends straight to the node's upstream servers.

It is installed with the nodelocaldns.yaml manifest from the Kubernetes repository, in which sed fills in the cache address, the cluster domain and the kube-dns address. The documentation describes two variants: with kube-proxy in iptables mode the agent listens both on its own address and on 10.96.0.10, intercepting queries without changing kubelet; with IPVS it listens only on its own address, and kubelet's clusterDNS is changed to that address. For nftables mode, as on our lab, the documentation gives no separate variant; the second one is more reliable: the agent on its own address and an explicit change of clusterDNS in the kubelet configuration on all nodes, after which new pods get a nameserver with the cache address. One more warning from the documentation: if the agent's pod is OOM-killed, its rules stay on the node, and DNS on that node does not work until it restarts, so its memory limit is chosen from measured usage.

Three lab nodes do not need the cache, and we did not install it. It pays off when the cluster has hundreds of nodes or applications with heavy queries to external names, and DNS timeouts show up in the logs.

Hands on

A pod with tools — nicolaka/netshoot, which has dig (BIND 9.20):

$ kubectl -n demo run net --image=nicolaka/netshoot:v0.16 -- sleep 3600
pod/net created
$ kubectl -n demo exec net -- dig +noall +answer +stats hello.demo.svc.cluster.local
hello.demo.svc.cluster.local. 30 IN     A       10.102.84.146
;; Query time: 0 msec
;; SERVER: 10.96.0.10#53(10.96.0.10) (UDP)
;; WHEN: Wed Oct 28 21:02:09 UTC 2026
;; MSG SIZE  rcvd: 113

TTL 30 comes from ttl 30 in the Corefile. dig itself does not apply search from /etc/resolv.conf (unlike the application resolver): dig hello asks the external servers for hello. and gets SERVFAIL. A short name is queried with +search:

$ kubectl -n demo exec net -- dig +search +noall +answer hello
hello.demo.svc.cluster.local. 30 IN     A       10.102.84.146

SRV for the named port:

$ kubectl -n demo exec net -- dig +noall +answer SRV _http._tcp.hello.demo.svc.cluster.local
_http._tcp.hello.demo.svc.cluster.local. 30 IN SRV 0 100 80 hello.demo.svc.cluster.local.

Priority 0, weight 100, port 80 and the target name. Now a headless service over the same pods:

apiVersion: v1
kind: Service
metadata:
  name: hello-headless
  namespace: demo
spec:
  clusterIP: None
  selector:
    app.kubernetes.io/name: hello
  ports:
    - name: http
      port: 8080
      targetPort: http

$ kubectl apply -f hello-headless.yaml
service/hello-headless created
$ kubectl -n demo exec net -- dig +noall +answer hello-headless.demo.svc.cluster.local
hello-headless.demo.svc.cluster.local. 30 IN A  172.20.211.50
hello-headless.demo.svc.cluster.local. 30 IN A  172.20.172.24
hello-headless.demo.svc.cluster.local. 30 IN A  172.20.172.22
$ kubectl -n demo exec net -- dig +noall +answer hello-headless.demo.svc.cluster.local
hello-headless.demo.svc.cluster.local. 30 IN A  172.20.172.22
hello-headless.demo.svc.cluster.local. 30 IN A  172.20.211.50
hello-headless.demo.svc.cluster.local. 30 IN A  172.20.172.24
$ kubectl -n demo exec net -- dig +noall +answer SRV _http._tcp.hello-headless.demo.svc.cluster.local
_http._tcp.hello-headless.demo.svc.cluster.local. 30 IN SRV 0 33 8080 172-20-172-24.hello-headless.demo.svc.cluster.local.
_http._tcp.hello-headless.demo.svc.cluster.local. 30 IN SRV 0 33 8080 172-20-211-50.hello-headless.demo.svc.cluster.local.
_http._tcp.hello-headless.demo.svc.cluster.local. 30 IN SRV 0 33 8080 172-20-172-22.hello-headless.demo.svc.cluster.local.

Three pod addresses, and the order differs from query to query — that is loadbalance. In SRV, for pods without hostname the target name is made of the address with dashes, and the weight is split evenly. Reverse records of the ClusterIP and of a pod address:

$ kubectl -n demo exec net -- dig +noall +answer -x 10.102.84.146
146.84.102.10.in-addr.arpa. 30  IN      PTR     hello.demo.svc.cluster.local.
$ kubectl -n demo exec net -- dig +noall +answer -x 172.20.172.24
24.172.20.172.in-addr.arpa. 30  IN      PTR     172-20-172-24.hello.demo.svc.cluster.local.
24.172.20.172.in-addr.arpa. 30  IN      PTR     172-20-172-24.hello-headless.demo.svc.cluster.local.

The pod has two names — one from each service that selects it. For the address of net itself, which is in no service, dig -x 172.20.172.21 gets NXDOMAIN with the ra flag and without aa — the query went through fallthrough to the network's servers. Pod records by address:

$ kubectl -n demo exec net -- dig +noall +answer 172-20-172-24.demo.pod.cluster.local
172-20-172-24.demo.pod.cluster.local. 30 IN A   172.20.172.24
$ kubectl -n demo exec net -- dig +noall +answer 172-20-172-24.hello-headless.demo.svc.cluster.local
172-20-172-24.hello-headless.demo.svc.cluster.local. 30 IN A 172.20.172.24
$ kubectl -n demo exec net -- dig +noall +answer 10-1-2-3.demo.pod.cluster.local
10-1-2-3.demo.pod.cluster.local. 30 IN  A       10.1.2.3

The last answer is pods insecure at work: there is no pod with the address 10.1.2.3, yet the record exists. Now a StatefulSet with a headless service:

apiVersion: v1
kind: Service
metadata:
  name: web
  namespace: demo
spec:
  clusterIP: None
  selector:
    app.kubernetes.io/name: web
  ports:
    - name: http
      port: 8080
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
  namespace: demo
spec:
  serviceName: web
  replicas: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: web
  template:
    metadata:
      labels:
        app.kubernetes.io/name: web
    spec:
      containers:
      - name: web
        image: ghcr.io/example/hello:1.0
        ports:
        - name: http
          containerPort: 8080

$ kubectl apply -f web.yaml
service/web created
statefulset.apps/web created
$ kubectl -n demo get pod web-0 -o jsonpath='{.spec.hostname} {.spec.subdomain}{"\n"}'
web-0 web
$ kubectl -n demo exec net -- dig +noall +answer web-0.web.demo.svc.cluster.local
web-0.web.demo.svc.cluster.local. 30 IN A       172.20.211.63
$ kubectl -n demo exec net -- dig +noall +answer SRV _http._tcp.web.demo.svc.cluster.local
_http._tcp.web.demo.svc.cluster.local. 30 IN SRV 0 50 8080 web-0.web.demo.svc.cluster.local.
_http._tcp.web.demo.svc.cluster.local. 30 IN SRV 0 50 8080 web-1.web.demo.svc.cluster.local.

The StatefulSet set hostname and subdomain itself, and SRV now points at pod names rather than addresses. And an ExternalName service:

$ kubectl -n demo create service externalname registry --external-name=ghcr.io
service/registry created
$ kubectl -n demo exec net -- dig +noall +answer registry.demo.svc.cluster.local
registry.demo.svc.cluster.local. 30 IN  CNAME   ghcr.io.
ghcr.io.                30      IN      A       140.82.121.34

CoreDNS returned the CNAME and resolved ghcr.io itself through forward, putting both records into one answer.

nslookup from busybox, often used to check DNS in a cluster, works its own way: it does not use the libc resolver but sends queries with all suffixes at once and prints every failure. A pod bb with busybox:1.37 and the default policy; the -type=a option keeps only queries for A records:

$ kubectl -n demo run bb --image=busybox:1.37 -- sleep 3600
$ kubectl -n demo exec bb -- nslookup -type=a hello
Server:		10.96.0.10
Address:	10.96.0.10:53

** server can't find hello.cluster.local: NXDOMAIN

Name:	hello.demo.svc.cluster.local
Address: 10.102.84.146

** server can't find hello.svc.cluster.local: NXDOMAIN

command terminated with exit code 1

The address is found, yet the exit code is 1 — because of the failures for the other suffixes. In check scripts give busybox nslookup a full name; nslookup from netshoot (BIND) resolves short names like a normal resolver but prints the warning ;; Got recursion not available from 10.96.0.10: CoreDNS answers questions about cluster.local itself, without the recursion flag ra.

DNS policies. Four busybox pods with different settings — dnsPolicy: Default, hostNetwork: true without a policy, hostNetwork: true with ClusterFirstWithHostNet, and None with the dnsConfig from the example above:

apiVersion: v1
kind: Pod
metadata:
  name: dns-hostnet-cf
  namespace: demo
spec:
  hostNetwork: true
  dnsPolicy: ClusterFirstWithHostNet
  containers:
  - name: sh
    image: busybox:1.37
    command: ["sleep", "3600"]

$ for p in dns-default dns-hostnet dns-hostnet-cf dns-none; do echo "== $p"; kubectl -n demo exec $p -- cat /etc/resolv.conf; done
== dns-default
nameserver 192.168.10.2
nameserver 192.168.10.3
…
== dns-hostnet
nameserver 192.168.10.2
nameserver 192.168.10.3
…
== dns-hostnet-cf
search demo.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:5
== dns-none
search demo.svc.cluster.local svc.cluster.local
nameserver 10.96.0.10
options ndots:2 timeout:2
$ kubectl -n demo get pod dns-hostnet -o jsonpath='{.spec.dnsPolicy}{"\n"}'
ClusterFirst

dns-hostnet has ClusterFirst in its spec, yet the file is like Default's: a pod in the node network silently gets the node's DNS. kubelet does not carry the node's search . line into the pod. A fifth pod with only dnsConfig.options ndots: "2" and the default policy got the cluster file with options ndots:2. The API server checks the dnsConfig limits at once (here with kubectl apply --dry-run=server):

The Pod "dns-bad" is invalid: spec.dnsConfig.nameservers: Invalid value: ["10.96.0.10","192.0.2.1","192.0.2.2","192.0.2.3"]: must not have more than 3 nameservers
The Pod "dns-bad" is invalid: spec.dnsConfig.nameservers: Required value: must provide at least one DNS nameserver when `dnsPolicy` is None
The Pod "dns-bad" is invalid: spec.dnsConfig: Required value: must provide `dnsConfig` when `dnsPolicy` is None

and with 33 suffixes — must not have more than 32 search paths with the whole list in the error text.

Extra queries for an external name. Let us save the Corefile, enable the query log — a log line after errors in the .:53 block — and wait for reload to apply the configuration. Instead of the interactive kubectl edit the change is easier to make as a patch from the copy:

$ kubectl -n kube-system get configmap coredns -o json | jq '{data:{Corefile:.data.Corefile}}' > coredns-orig.json
$ jq '.data.Corefile |= sub("    errors\n"; "    errors\n    log\n")' coredns-orig.json > coredns-log.json
$ kubectl -n kube-system patch configmap coredns --type merge --patch-file coredns-log.json
configmap/coredns patched
$ kubectl -n kube-system logs -l k8s-app=kube-dns --since=3m --prefix
[pod/coredns-589f44dc88-bzxpn/coredns] [INFO] Reloading
[pod/coredns-589f44dc88-bzxpn/coredns] [INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[pod/coredns-589f44dc88-bzxpn/coredns] [INFO] plugin/reload: Running configuration SHA512 = 2dd49c56b94cd5672aafc99f92d4b5ff824b6ae87878b18330ea3f5199d47baffba03feab068065d91ea11f9ff9de9eb155741708e79bfb0228360463ef6dbde
[pod/coredns-589f44dc88-bzxpn/coredns] [INFO] Reloading complete
[pod/coredns-589f44dc88-bzxpn/coredns] [INFO] 127.0.0.1:52498 - 19161 "HINFO IN 3633825480454799534.3516841135501091470. udp 57 false 512" NXDOMAIN qr,rd,ra 132 0.029861796s

The last line is the loop plugin's probe after the reload: an HINFO query with a random name went to the network's servers and came back NXDOMAIN, no loop. The second pod reread the configuration a minute later — we read the log of both. Now let us resolve an external name through the glibc resolver — ping in busybox; the -4 option keeps only queries for A records:

$ kubectl -n demo exec bb -- ping -4 -c1 -W2 api.github.com
PING api.github.com (140.82.121.5): 56 data bytes
64 bytes from 140.82.121.5: seq=0 ttl=54 time=27.561 ms

--- api.github.com ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 27.561/27.561/27.561 ms
$ kubectl -n kube-system logs -l k8s-app=kube-dns --since=30s --prefix | grep api.github.com
[pod/coredns-589f44dc88-lv5bl/coredns] [INFO] 172.20.211.8:52478 - 29223 "A IN api.github.com.demo.svc.cluster.local. udp 55 false 512" NXDOMAIN qr,aa,rd 148 0.000131424s
[pod/coredns-589f44dc88-lv5bl/coredns] [INFO] 172.20.211.8:33691 - 45563 "A IN api.github.com.svc.cluster.local. udp 50 false 512" NXDOMAIN qr,aa,rd 143 0.000112023s
[pod/coredns-589f44dc88-lv5bl/coredns] [INFO] 172.20.211.8:49009 - 24720 "A IN api.github.com.cluster.local. udp 46 false 512" NXDOMAIN qr,aa,rd 139 0.000057053s
[pod/coredns-589f44dc88-lv5bl/coredns] [INFO] 172.20.211.8:51928 - 16629 "A IN api.github.com. udp 32 false 512" NOERROR qr,rd,ra 62 0.039123878s

Four queries for one name. The line format: client address and port (the bb pod), query ID, type, name, protocol, query size, the DO flag (readiness to accept DNSSEC signatures — DNSSEC being the DNS extension where records are signed cryptographically), the UDP buffer size the client announced via EDNS (the DNS extension for answers larger than 512 bytes); then the response code, the flags (aa — an authoritative answer from the cluster zone, ra — recursion available, that is, the answer came through forward), the answer size and the time. CoreDNS rejected the first three queries from memory in 0.06–0.13 ms, the last one went to the network's servers and cost 39 ms. The same experiment from netshoot (getent ahostsv4 api.github.com, the musl resolver) went through the same four names in the same order. With a trailing dot one query remains:

$ kubectl -n demo exec bb -- ping -4 -c1 -W2 api.github.com. | head -1
PING api.github.com. (140.82.121.6): 56 data bytes
$ kubectl -n kube-system logs -l k8s-app=kube-dns --since=10s --prefix | grep api.github.com
[pod/coredns-589f44dc88-bzxpn/coredns] [INFO] 172.20.211.8:42586 - 45704 "A IN api.github.com. udp 32 false 512" NOERROR qr,aa,rd,ra 62 0.000387958s

This query landed on the second CoreDNS pod, which already had the name cached after the netshoot experiment — the answer took 0.4 ms instead of 39. CoreDNS also marks an answer from the cache with the aa flag, so what tells it apart from an answer from the cluster zone (qr,aa,rd) is the ra flag, and from an answer forwarded outside (qr,rd,ra) — the aa flag and the time. The TTL in the cache counts down, and every CoreDNS pod has its own cache:

$ kubectl -n demo exec net -- dig +noall +answer api.github.com
api.github.com.         30      IN      A       140.82.121.5
$ kubectl -n demo exec net -- dig +noall +answer api.github.com
api.github.com.         25      IN      A       140.82.121.5
$ kubectl -n demo exec net -- dig +noall +answer +stats api.github.com | grep -E 'IN|Query'
api.github.com.         30      IN      A       140.82.121.5
;; Query time: 3 msec

The second query five seconds later came from the same pod's cache (TTL 25), the third landed on the other pod and went outside again. The TTL is no more than 30 even on the first answer: cache 30 cuts down the external record's TTL if it is larger.

A stub zone and hosts. Let us add to the Corefile a block for corp.example.com that forwards queries only to the first network server (both servers know the zone, so the point of the experiment is to check how the block is chosen), and hosts to the .:53 block:

    hosts {
       192.168.10.32 registry.example.com
       fallthrough
    }
    loop
    reload
    loadbalance
}
corp.example.com:53 {
    errors
    log
    cache 30
    forward . 192.168.10.2
}

$ kubectl -n demo exec net -- dig +noall +answer registry.example.com
registry.example.com.   30      IN      A       192.168.10.32
$ kubectl -n demo exec net -- dig +noall +answer www.corp.example.com
www.corp.example.com.   30      IN      CNAME   web.corp.example.com.
web.corp.example.com.   30      IN      A       192.168.10.80
$ kubectl -n kube-system logs -l k8s-app=kube-dns --since=30s --prefix | grep registry
[pod/coredns-589f44dc88-bzxpn/coredns] [INFO] 172.20.172.21:43345 - 64115 "A IN registry.example.com. udp 61 false 1232" NOERROR qr,aa,rd 74 0.000309068s
$ kubectl -n demo exec net -- curl -s http://172.20.70.194:9153/metrics | grep 'zones="corp'
coredns_cache_misses_total{server="dns://:53",view="",zones="corp.example.com."} 1
coredns_cache_requests_total{server="dns://:53",view="",zones="corp.example.com."} 1

The record from hosts is authoritative (aa), and the cache of the .:53 block cut its TTL to 30 (hosts defaults to an hour). The query for corp.example.com was handled by the separate block: this shows in the metrics of its cache with zones="corp.example.com.", while coredns_dns_requests_total does not have it — we did not enable prometheus in that block.

And if we make a mistake? Let us add a nonexistent plugin nosuchplugin to the .:53 block:

$ kubectl -n kube-system logs coredns-589f44dc88-bzxpn --since=2m
[INFO] Reloading
[ERROR] Restart failed: /etc/coredns/Corefile:28 - Error during parsing: Unknown directive 'nosuchplugin'
[ERROR] plugin/reload: Corefile changed but reload failed: starting with listener file descriptors: /etc/coredns/Corefile:28 - Error during parsing: Unknown directive 'nosuchplugin'
$ kubectl -n kube-system get pods -l k8s-app=kube-dns
NAME                       READY   STATUS    RESTARTS   AGE
coredns-589f44dc88-bzxpn   1/1     Running   0          4d10h
coredns-589f44dc88-lv5bl   1/1     Running   0          4d10h
$ kubectl -n demo exec net -- dig +short hello.demo.svc.cluster.local
10.102.84.146

CoreDNS keeps running with the old configuration and keeps answering. The danger is elsewhere: a pod that restarts with such a ConfigMap will not start any more. So after editing the Corefile always look at the log of both pods. Keeping the query log on in a production cluster is not a good idea either — that is a line per query. Let us return the original Corefile from the copy:

$ kubectl -n kube-system patch configmap coredns --type merge --patch-file coredns-orig.json
configmap/coredns patched

Within a minute and a half both pods log Running configuration SHA512 = 1b226df7… — the same hash as at the first start.

CoreDNS metrics are on port 9153 of any of its pods:

$ kubectl -n demo exec net -- curl -s http://172.20.70.194:9153/metrics | grep -E '^coredns_(dns_request_duration_seconds_count|cache_hits_total|proxy_request_duration_seconds_count)'
coredns_cache_hits_total{server="dns://:53",type="denial",view="",zones="."} 3
coredns_cache_hits_total{server="dns://:53",type="success",view="",zones="."} 2
coredns_dns_request_duration_seconds_count{server="dns://:53",view="",zone="."} 240
coredns_proxy_request_duration_seconds_count{proxy_name="forward",rcode="NOERROR",to="192.168.10.2:53"} 12
coredns_proxy_request_duration_seconds_count{proxy_name="forward",rcode="NOERROR",to="192.168.10.3:53"} 15
coredns_proxy_request_duration_seconds_count{proxy_name="forward",rcode="NXDOMAIN",to="192.168.10.2:53"} 4
coredns_proxy_request_duration_seconds_count{proxy_name="forward",rcode="NXDOMAIN",to="192.168.10.3:53"} 8
coredns_proxy_request_duration_seconds_count{proxy_name="forward",rcode="SERVFAIL",to="192.168.10.2:53"} 1
coredns_proxy_request_duration_seconds_count{proxy_name="forward",rcode="SERVFAIL",to="192.168.10.3:53"} 2

coredns_dns_request_duration_seconds_count is how many queries the pod has handled since it started, coredns_cache_hits_total is how many of them it answered from the cache (denial — negative answers such as NXDOMAIN). Forwarding is shown by the coredns_proxy_* metrics with proxy_name="forward" (the former coredns_forward_requests_total and its neighbours are no longer exported in CoreDNS 1.14): queries are split between the two servers at random.

To clean up: kubectl -n demo delete pod net bb dns-default dns-hostnet dns-hostnet-cf dns-none, kubectl -n demo delete svc hello-headless registry and kubectl delete -f web.yaml; the Corefile is already back.

Sources

DNS for Services and Pods — service and pod records, hostname and subdomain, dnsPolicy, dnsConfig and its limits.
Customizing DNS Service — the default Corefile, plugins, stub zones, the ConfigMap propagation delay.
Debugging DNS Resolution — checking DNS from a pod, the log plugin, known issues with resolvConf and systemd-resolved.
Using CoreDNS for Service Discovery — CoreDNS in kubeadm and keeping the configuration during upgrades.
Using NodeLocal DNSCache in Kubernetes Clusters — the cache architecture, the iptables and IPVS variants, memory.
KEP-1024: NodeLocal DNS Cache — why a cache on the node, NOTRACK rules, TCP to CoreDNS.
Kubernetes DNS-Based Service Discovery — the specification of A, SRV and PTR records for services.
kubernetes/kubernetes#56903 — the conntrack race on DNAT and five-second DNS delays.
CoreDNS plugins: kubernetes (the pods modes, PTR for pods behind services), forward (max_concurrent, health checks, the coredns_proxy_* metrics), cache, loop (the HINFO probe, systemd-resolved), reload, log (the line format), hosts, autopath, health (lameduck), metrics.
plugin.cfg — the order in which CoreDNS plugins run.
RFC 2782 — SRV records: priority, weight, port, target.
resolv.conf(5) — search, ndots, the limit of three servers.
nicolaka/netshoot — an image with network tools for debugging.

Next — LoadBalancer without a cloud: why a LoadBalancer service hangs in Pending on bare metal, how MetalLB gives it an address from the 192.168.10.200–220 pool and announces it in the network over ARP or BGP, which node answers for the address and what happens when that node fails.