Vitaliy Leschenko’s blog
Виталий Лещенко

Kubernetes for beginners. Part 14: How is the Kubernetes network built?

All parts of the series “Kubernetes for beginners”

So far we have used pod addresses without asking where they come from: in the output of kubectl get pods -o wide every pod has its own address from 172.20.0.0/16, and wget from one pod to another just works, whichever node the other one lives on. Today we look at why. First the network model Kubernetes requires from every cluster, and what it does not do itself. Then CNI, the interface through which containerd asks a network plugin to connect a pod. And then we follow the packet on our lab with Calico from part 4: how a pod gets its address, where the cali… veth and the /32 route on the node come from, who answers the pod's ARP, how a packet reaches a pod on the same node and on the neighbouring one, when VXLAN kicks in and how pods reach the internet. The pod's network namespace, held by the pause container, comes from part 6.

The Kubernetes network model

Kubernetes does not implement the pod network itself, but it puts three requirements on any implementation:

  1. Every pod has its own IP address, shared by all its containers. Containers of one pod live in one network namespace and see each other via localhost.
  2. Any pod reaches any other pod by its address without NAT — on the same node or on another one. Pod A sees the real address of B in packets from pod B.
  3. Agents on a node (kubelet, system daemons) reach all pods of that node. In practice, in Calico and most other implementations a node reaches all pods of the cluster, and pods reach node addresses.

One more property follows from this and is often stated separately: the address a pod sees on its interface is the same address the others see it under. Inside the cluster a pod has no "internal" and "external" address.

Why this is convenient is clear in contrast with Docker on a single machine. There containers get addresses from the private network of the docker0 bridge, and to reach them from outside you have to publish a node port (-p 8080:80). Two containers that need port 80 cannot get the same node port, so ports have to be allocated and written down somewhere. Kubernetes has no such task: every pod has its own address, all pods can listen on port 8080, and the application behaves as if on its own virtual machine. The price is that the cluster network must be able to deliver a packet to any of tens of thousands of pod addresses, and that is the network plugin's job.

What the model does not include:

  • a stable address. A pod is recreated — the address is new. A stable name and address come from a Service (part 15), not from the pod network;
  • a pod address outside the cluster. The router 192.168.10.1 knows nothing about the 172.20.0.0/16 network, and the administrator's laptop cannot reach a pod directly. How to let traffic in from outside is covered in parts 15, 17 and 18;
  • isolation. By default any pod can open a connection to any other. Restrictions are NetworkPolicy (part 20).

Who does what

A pod's network is created when the pod starts, along the chain kubelet → containerd → CNI plugin:

 kubelet                containerd                    CNI plugin (calico)
    │ RunPodSandbox (CRI) ─►│                                   │
    │                       │ create netns,                     │
    │                       │ start pause                       │
    │                       │ ADD: netns, interface name ──────►│ request an address (calico-ipam)
    │                       │                                   │ create veth, eth0 in the pod netns,
    │                       │                                   │ address, routes, route on the node
    │                       │◄──────── result: IP, interfaces ──┤
    │◄── sandbox ready, IP ─┤                                   │
    │ write status.podIP    │                                   │
    ▼                       ▼                                   ▼

Through CRI (the interface the kubelet uses to control containerd) the kubelet asks to create the pod sandbox. containerd creates a new network namespace (a file in /var/run/netns/cni-…), starts the pause container in it and calls the network plugin according to the CNI specification. The plugin returns an address, containerd hands it to the kubelet, and the kubelet writes it into status.podIP and status.podIPs. Everything that happens between the call and the answer is the plugin's business: Kubernetes does not know what a veth, a route or VXLAN is.

Kubernetes also has a mechanism for handing subnets to nodes: kube-controller-manager with the --allocate-node-cidrs flag (kubeadm turns it on together with --cluster-cidr when podSubnet is set at init) gives every node a piece of the pod network and writes it into spec.podCIDR of the Node object. Some plugins take addresses from this field. Calico does not use it by default: it has its own address management, described below. So the value in spec.podCIDR and the real pod addresses on a node do not match on our lab, and that is fine.

CNI

CNI (Container Network Interface) is a specification of a CNCF project (Cloud Native Computing Foundation — the foundation under which Kubernetes, containerd and related projects are developed) describing how a container runtime calls network plugins. It is deliberately primitive:

  • a plugin is an executable. It lives in the binary directory (on Kubernetes nodes — /opt/cni/bin), is started anew for every operation and exits;
  • the operation and parameters are in environment variables: CNI_COMMAND (ADD, DEL, CHECK, VERSION, and in spec version 1.1 also GC and STATUS), CNI_CONTAINERID, CNI_NETNS (the path to the network namespace), CNI_IFNAME (the interface name inside, usually eth0), CNI_ARGS (extra key-value pairs), CNI_PATH (where to look for other plugins);
  • the network configuration is JSON on standard input, the result is JSON on standard output: interfaces, addresses, routes, DNS;
  • a plugin chain. A .conflist file lists several plugins; the runtime calls them in turn, passing each the result of the previous one. The first creates the interface and the address, the next ones add to it: publish ports, limit bandwidth.

ADD connects a container to the network: creates an interface, assigns an address. DEL disconnects it and releases the address; it is called when a pod is deleted and must be idempotent (a repeated DEL is not an error). CHECK verifies that everything is in place. GC releases resources of containers that no longer exist. STATUS tells whether the plugin is ready to accept ADD. Calico's configuration on the lab declares "cniVersion": "1.0.0", so GC and STATUS are not sent to it.

Where containerd looks for all this is set in its configuration:

user@k8s-node-1:~$ grep -A4 "cri.v1.runtime'.cni]" /etc/containerd/config.toml
    [plugins.'io.containerd.cri.v1.runtime'.cni]
      bin_dir = ''
      bin_dirs = ['/opt/cni/bin']
      conf_dir = '/etc/cni/net.d'
      max_conf_num = 1

max_conf_num = 1 means containerd takes one, the alphabetically first configuration file from /etc/cni/net.d. Hence a common breakage: after experiments with another plugin its configuration stays in the directory, and if its name sorts before 10-calico.conflist — for example 05-cilium.conflist of Cilium, an eBPF-based network plugin (eBPF is described below) — pods get addresses from the wrong network. While the directory holds no configuration at all, containerd reports that the network is not ready, and the kubelet keeps the node Ready=False with the message container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized; no pods are scheduled to the node except those that run in the node's network (hostNetwork: true). That is how a freshly joined node behaves before Calico is installed — we saw it in part 4.

The binaries in /opt/cni/bin come from two places. The set of reference plugins of the CNI project (bridge, host-local, loopback, portmap, bandwidth and others) is installed by the kubernetes-cni package, which apt pulled in as a kubeadm dependency. Calico puts its own calico, calico-ipam and flannel binaries and updated copies of some reference plugins there itself: this is done by the install-cni init container of the calico-node pod, which the Tigera operator — the controller that installs and upgrades Calico — runs as a DaemonSet on every node. It also writes 10-calico.conflist. That is why a new node gets CNI by itself as soon as calico-node starts on it.

How Calico assigns an address

Addresses are handed out by the IPAM plugin calico-ipam, which the main calico plugin calls. It hands out addresses from an IP pool — a Calico resource the operator created at installation:

$ kubectl get ippool default-ipv4-ippool -o yaml | grep -A10 '^spec:'
spec:
  allowedUses:
  - Workload
  - Tunnel
  assignmentMode: Automatic
  blockSize: 26
  cidr: 172.20.0.0/16
  ipipMode: Never
  natOutgoing: true
  nodeSelector: all()
  vxlanMode: CrossSubnet

allowedUses allows taking addresses from the pool both for pods (Workload) and for node tunnel interfaces (Tunnel), and assignmentMode: Automatic means handing them out automatically rather than only on an explicit pod request. The pool is not handed out address by address. It is cut into blocks of blockSize — /26 by default, 64 addresses — and every block is tied to a node (affinity). When the first pod starts on a node, calico-ipam ties a free block to the node and takes an address from it; the next pods of that node get addresses from the same block until it runs out, and then a second block is tied to the node. If the pool has no free blocks left, a node can take an address from someone else's block (borrowing); this works, but breaks route aggregation and does not happen on a properly planned pool. Inside a block addresses are not necessarily handed out in order: the block keeps a list of free addresses, many pods have passed through the node in the previous parts, and on the lab pods of one node got .43, .45 and .53.

Why blocks: a route to a neighbouring node is needed per block, not per pod. A node with a hundred pods means two /26 routes on its neighbours instead of a hundred /32.

Calico keeps the IPAM state in the Kubernetes API, in its own CRDs of the crd.projectcalico.org group:

Resource What it stores
IPPool the pool, block size, encapsulation mode, natOutgoing
IPAMBlock one block: CIDR, owner node, used and free addresses, who took them
BlockAffinity the binding of a block to a node
IPAMHandle an allocation ID and its addresses, so that DEL releases exactly those: k8s-pod-network.<pod sandbox ID> for a pod, vxlan-tunnel-addr-<node> for a tunnel address

Calico also writes the address given to a pod into the pod's annotations — cni.projectcalico.org/podIP and cni.projectcalico.org/podIPs — together with the sandbox ID (cni.projectcalico.org/containerID). From the pod and these annotations Calico builds a WorkloadEndpoint — its own representation of "a pod's interface on a node", to which network policies are then attached. It is not stored as a separate object in the Kubernetes API, and kubectl does not see it (error: the server doesn't have a resource type "workloadendpoints"); it is shown by Calico's calicoctl utility, and a name like k8s--node--1-k8s-net-eth0 (node with doubled dashes, orchestrator, pod, interface) appears in the CNI log.

veth, /32 and 169.254.1.1

Having the address, the plugin connects the pod to the node. Calico does not use a bridge like docker0: every pod is connected to the node by its own veth pair, and from there ordinary routing in the node's kernel takes over. This is an L3 model: there is no shared L2 segment between pods.

            pod hello (netns cni-2880cb32…)                node k8s-node-1
 ┌───────────────────────────────────────┐   ┌───────────────────────────────────────────┐
 │ eth0@if172  172.20.211.43/32          │   │ calia68edcf6ff3  no address,              │
 │             MAC 4a:17:f0:d5:da:0a     │◄─►│                  MAC ee:ee:ee:ee:ee:ee    │
 │ default via 169.254.1.1 dev eth0      │   │                  proxy_arp = 1            │
 │ 169.254.1.1 dev eth0 scope link       │   │ 172.20.211.43 dev calia68edcf6ff3         │
 │                                       │   │              scope link                   │
 └───────────────────────────────────────┘   └───────────────────────────────────────────┘

The pod side. The eth0 interface with a /32 address — the pod has no "own subnet" in which it could look for neighbours via ARP. The default route points to 169.254.1.1, an address from the link-local range that in reality nobody has. It exists only because the default route has to point to some gateway, and Calico does not want to spend an address from the pool on a gateway.

The node side. The interface is cali plus the first 11 characters of the SHA-1 of the string <namespace>.<pod name>: for the pod net from the demo namespace below it is cali4d7612d46be, and printf demo.net | sha1sum starts with 4d7612d46be. It has no address, and all such interfaces have the same MAC address — ee:ee:ee:ee:ee:ee (it never leaves the pair, so there are no conflicts). On the node there is a /32 route to the pod's address through this interface: that is how the node's kernel knows which pipe to send the pod's packets into.

Who answers ARP for 169.254.1.1. The pod wants to send a packet anywhere — the pod's kernel looks up the route, sees the gateway 169.254.1.1 and asks for its MAC address via ARP (ARP is in part 7 of the networking series: a broadcast question "who has this IP" and an answer with the MAC address). The request arrives over the veth at the node's cali…. The node does not have the address 169.254.1.1, but proxy ARP is enabled on the interface — the kernel answers an ARP request with its own MAC address if it has a route to the requested address through another interface. The node has a default route, so the pod gets the answer "169.254.1.1 is ee:ee:ee:ee:ee:ee". From then on all the pod's packets go to this MAC, that is, into the node's kernel, which routes them like an ordinary router.

In short: the pod thinks there is a gateway behind its eth0; in fact the gateway is the node itself, and everything the pod sends ends up in the node's routing table and in the nftables rules where Felix applies network policies.

The CNI plugin creates the interface and the route on ADD, and from then on they are maintained by Felix — Calico's agent on the node: it sets the needed kernel parameters on cali… (proxy ARP, forwarding), makes sure the routes to the node's pods match the IPAM data, and attaches policy rules to the interface.

A packet to a pod on the same node

net on k8s-node-1 (172.20.211.53) talks to the pod hello on the same node (172.20.211.43):

  1. The kernel of the net pod: the destination is not in its own network, send to the gateway 169.254.1.1 via eth0; the gateway's MAC is ee:ee:ee:ee:ee:ee.
  2. The frame comes out of the veth on the node side — the cali4d7612d46be interface.
  3. The node's kernel: a packet to 172.20.211.43, the routing table has 172.20.211.43 dev calia68edcf6ff3 scope link. The packet passes the netfilter forward hook — this is where Calico's nftables chains check network policies — and goes out into calia68edcf6ff3.
  4. The hello pod receives the packet on its eth0 with source address 172.20.211.53. No NAT.

The packet does not leave the node and passes through no bridge — only two veths and kernel routing.

A packet to a pod on another node

Now net on k8s-node-1 talks to the hello pod on k8s-node-2 (172.20.172.6). The routes of k8s-node-1:

user@k8s-node-1:~$ ip route show default
default via 192.168.10.1 dev ens18 proto dhcp src 192.168.10.20 metric 100
user@k8s-node-1:~$ ip route | grep 172.20
172.20.70.192/26 via 192.168.10.17 dev ens18 proto 80 onlink
172.20.172.0/26 via 192.168.10.21 dev ens18 proto 80 onlink
blackhole 172.20.211.0/26 proto 80
172.20.211.1 dev cali01e594701c5 scope link metric 1024
172.20.211.43 dev calia68edcf6ff3 scope link metric 1024
172.20.211.53 dev cali4d7612d46be scope link metric 1024
  • /32 routes to the node's own pods — through their cali…;
  • blackhole for the node's own block 172.20.211.0/26 — if a packet arrives for an address from the node's block but there is no pod with that address (it was deleted), it is dropped here instead of going along the default route to the router. The more specific /32 routes to live pods override this route;
  • the neighbours' blocks /26 — the master's block 172.20.70.192/26 and the block of k8s-node-2 172.20.172.0/26, via their addresses on ens18; onlink tells the kernel to treat the next hop as reachable directly through this interface.

proto 80 is the number Felix marks its routes with. For our VXLAN pool it is Felix that writes the routes to the neighbours' blocks and the blackhole, from the IPAM data in the API: it knows which block is tied to which node and what that node's address is. The BGP daemon BIRD, which also runs in calico-node, does not write these routes into the kernel — we come back to it in the section on Calico's components.

The packet's path:

 pod net                  k8s-node-1                     k8s-node-2              pod hello
 172.20.211.53            192.168.10.20                  192.168.10.21           172.20.172.6
     │ eth0 → cali…  ─────►│ route 172.20.172.0/26         │                          │
     │                     │ via 192.168.10.21 dev ens18   │                          │
     │                     │ ──── Ethernet, no wrapper ───►│ route /32                │
     │                     │  src 172.20.211.53            │ dev cali… ──────────────►│ eth0
     │                     │  dst 172.20.172.6             │                          │

The nodes are in one network, 192.168.10.0/24, and to the kernel they are L2 neighbours: k8s-node-1 finds the MAC address of 192.168.10.21 via ARP and sends the frame straight to it, with the pod's IP header inside as is. k8s-node-2 sees a packet for 172.20.172.6, finds the /32 route to cali… and hands it to the pod. The router 192.168.10.1 takes no part in this and does not need to know about the pod network. This is direct routing — the fastest option: no extra headers and no reduced MTU.

VXLANCrossSubnet: when a wrapper is needed

Direct routing works while the nodes are L2 neighbours. If a node k8s-node-3 were in another subnet, 192.168.20.0/24, a packet for 172.20.x.x would have to be handed to the router, and the router knows nothing about the pod network and would send it along its default route. There are three ways out.

VXLAN encapsulation. VXLAN is a protocol that wraps a whole Ethernet frame in a UDP datagram (port 4789) with a header carrying a 24-bit segment number, the VNI; between the nodes ordinary UDP with node addresses flows, and the router sees only that. In detail — in part 18 of the networking series. Calico creates a vxlan.calico interface (VNI 4096) on every node with a tunnel address — the first address of the node's block — and Felix writes ARP entries and forwarding-table entries for the tunnel addresses of all other nodes onto it in advance: "the neighbour's tunnel MAC is such-and-such, send to its address in the node network". For blocks of nodes in another subnet Felix writes a route through vxlan.calico with the owner node's tunnel address as the gateway. For the imaginary k8s-node-3 with the block 172.20.88.0/26 such a route would look like this:

172.20.88.0/26 via 172.20.88.0 dev vxlan.calico onlink

The price is 50 bytes of headers on every packet (outer Ethernet, IP, UDP and VXLAN itself), so with VXLAN enabled the MTU of pod interfaces is 50 less than the node's: 1450 with the usual 1500.

Our pool's mode vxlanMode: CrossSubnet is a compromise: Felix compares the subnet of the destination node with its own and wraps only traffic between subnets in VXLAN, while neighbours in its own subnet get packets directly, as above. On the lab all nodes are in one network, so VXLAN is effectively unused: the vxlan.calico interface exists and its neighbour entries are ready, but there are no routes through it. Always mode would wrap everything; Never nothing.

IPIP encapsulation. IP-in-IP (IP protocol number 4) puts the pod's IP packet into one more IP header with node addresses — without UDP and without an Ethernet frame, so the overhead is smaller, 20 bytes. The interface on the node is tunl0. The limitation: some networks and clouds do not pass protocol 4 (Calico's documentation names Azure), while UDP goes everywhere. In Calico IPIP is turned on by the pool's ipipMode field with the same values Always and CrossSubnet, and by default the routes to the blocks for it are distributed and installed by the BGP daemon BIRD.

BGP without encapsulation. BGP is the protocol routers use to tell each other which networks are reachable through them (part 22 of the networking series). Calico can announce the blocks to the router as well, and then the router knows that 172.20.211.0/26 is via 192.168.10.20: packets between subnets travel without a wrapper, and the administrator's laptop can reach a pod directly. This needs a router that speaks BGP, and is covered in the series "Highly available Kubernetes on BGP".

Mode Between nodes of one subnet Between subnets Overhead What the network must allow
VXLAN CrossSubnet (lab) direct UDP 4789 50 bytes only between subnets UDP 4789 between nodes
VXLAN Always UDP 4789 UDP 4789 50 bytes always UDP 4789
IPIP CrossSubnet direct IP protocol 4 20 bytes between subnets protocol 4 and BGP between nodes
no encapsulation, BGP direct direct none a router with BGP

Pods going out: natOutgoing

The hello pod talks to a server on the local network, say 192.168.10.32, or to the internet. The packet leaves the node with source address 172.20.211.43, and the receiver cannot answer it: the router does not know where the 172.20.0.0/16 network is. That is why the pool has natOutgoing: true: Felix adds a masquerade rule — packets from the pool to destinations outside all Calico pools get the node's address as their source. This is ordinary source NAT, as in a home router (part 6 of the networking series): the kernel remembers the connection in conntrack and on the way back restores the pod's address as the destination.

Pod-to-pod traffic is not caught by this rule (the destination is inside the pool), so the "no NAT" requirement holds. But to external servers all pods of a node look like the node itself. This matters for security: if an NFS server allows access "from the cluster node addresses", it allows it to any pod on those nodes as well. We will come back to this trap in part 22.

Calico's components

$ kubectl get pods -n calico-system -o wide
NAME                                       READY   STATUS    RESTARTS        AGE     IP              NODE           NOMINATED NODE   READINESS GATES
calico-apiserver-7c56559b65-dm42w          1/1     Running   0               4d9h    172.20.70.198   k8s-master-1   <none>           <none>
calico-apiserver-7c56559b65-phtkx          1/1     Running   0               4d9h    172.20.70.196   k8s-master-1   <none>           <none>
calico-kube-controllers-6879897867-8jnzf   1/1     Running   0               4d9h    172.20.70.195   k8s-master-1   <none>           <none>
calico-node-6d2qb                          1/1     Running   0               4d9h    192.168.10.20   k8s-node-1     <none>           <none>
calico-node-7mx4w                          1/1     Running   2 (6h16m ago)   4d8h    192.168.10.21   k8s-node-2     <none>           <none>
calico-node-l9g44                          1/1     Running   0               4d9h    192.168.10.17   k8s-master-1   <none>           <none>
calico-typha-55d594d885-dsh7d              1/1     Running   0               4d9h    192.168.10.17   k8s-master-1   <none>           <none>
calico-typha-55d594d885-q7nvz              1/1     Running   0               4h10m   192.168.10.20   k8s-node-1     <none>           <none>
csi-node-driver-6nzck                      2/2     Running   4 (6h16m ago)   4d8h    172.20.172.16   k8s-node-2     <none>           <none>
csi-node-driver-7jlkp                      2/2     Running   0               4d9h    172.20.70.193   k8s-master-1   <none>           <none>
csi-node-driver-q6t25                      2/2     Running   0               4d9h    172.20.211.1    k8s-node-1     <none>           <none>

Pods with node addresses (192.168.10.x) run in the node's network, the rest are ordinary pods with addresses from the blocks.

calico-node (a DaemonSet, in the node's network) is the main agent. In its single container several processes run under the runsv supervisor: calico-node -felix, bird, calico-node -confd and the auxiliary calico-node -node-services.

  • Felix programs the node: creates and configures cali…, writes /32 routes to its pods and routes to the blocks of other nodes (directly or through vxlan.calico), brings up vxlan.calico, writes network policy rules and natOutgoing into nftables (in other dataplanes — into iptables or as eBPF programs), and reports its state to the API;
  • BIRD is a routing daemon that speaks BGP (TCP port 179). The operator enables BGP by default (spec.calicoNetwork.bgp: Enabled in the Installation), and every lab node keeps a BGP session with every other one (a full mesh): the nodes announce their blocks to each other. But BIRD does not write routes from a VXLAN pool into the kernel — Felix does that, and BIRD's configuration filter says so. In this configuration BGP is needed only to announce pod or service networks outward, to routers; Calico's documentation says plainly that VXLAN-only pools do not require BGP. In IPIP mode and without encapsulation it is BIRD that installs the routes to the blocks;
  • confd generates BIRD's configuration from Calico resources (AS number, peers, pools) and reloads it on changes.

calico-typha is a caching intermediary between the Kubernetes API and Felix. Without it every Felix would keep its own watches on pods, nodes, policies and Calico resources, and on hundreds of nodes that means hundreds of identical streams from kube-apiserver. Typha keeps one connection, filters the changes and fans them out to all Felix instances. The operator always installs Typha and picks the number of replicas by the number of nodes itself (two on the lab); the replicas are placed on different nodes and run in the node's network.

calico-kube-controllers are controllers that clean up after deleted objects: they release IPAM addresses and blocks of nodes that no longer exist, and sync node and ServiceAccount labels for policies.

calico-apiserver is Calico's API server from part 4: through it kubectl works with projectcalico.org/v3 resources (pools, policies, block affinities).

If the standard custom-resources.yaml is applied as a whole at installation, the namespace also gets Goldmane pods (an aggregator of the network flow logs Felix collects) and Whisker (a web interface to them); they do not affect how the network works, and the lab does not have them.

csi-node-driver is Calico's node CSI plugin through which pods get service volumes for application-layer policies; it has nothing to do with the pod network.

The Tigera operator itself lives in the tigera-operator namespace, reads the Installation resource and brings all these DaemonSets and Deployments to the state described there.

Dataplane. How Felix applies its rules is set by spec.calicoNetwork.linuxDataplane. On the lab it is Nftables (part 4): Felix writes its rules directly into its own nftables table, next to the tables of kube-proxy, which in our cluster also runs in nftables mode (Calico's documentation requires this combination). The operator's default is Iptables: the same rules through iptables. The third option is BPF: eBPF is a Linux kernel mechanism for loading verified packet-processing programs into the kernel, and Calico's programs of this kind replace both the netfilter rules and kube-proxy; the eBPF dataplane is covered in the BGP series.

Hands-on

Let's take the hello pod on k8s-node-1 and start a pod with network tools next to it. hello here is the three-replica Deployment in the demo namespace from part 11. The hello image is built without a shell (part 5), so we will look inside through busybox placed on the same node: we set the node explicitly with the nodeName field, bypassing the scheduler (part 13).

$ kubectl -n demo get pod -l app.kubernetes.io/name=hello -o wide --field-selector spec.nodeName=k8s-node-1
NAME                     READY   STATUS    RESTARTS   AGE   IP              NODE         NOMINATED NODE   READINESS GATES
hello-6d5c6db6b4-nqsp8   1/1     Running   0          15s   172.20.211.43   k8s-node-1   <none>           <none>
$ kubectl -n demo run net --image=busybox:1.37 --overrides='{"spec":{"nodeName":"k8s-node-1"}}' -- sleep 3600
pod/net created
$ kubectl -n demo get pod net -o wide
NAME   READY   STATUS    RESTARTS   AGE   IP              NODE         NOMINATED NODE   READINESS GATES
net    1/1     Running   0          6s    172.20.211.53   k8s-node-1   <none>           <none>
$ kubectl -n demo exec net -- ip addr show eth0
2: eth0@if173: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue qlen 1000
    link/ether 36:c6:f6:49:44:cf brd ff:ff:ff:ff:ff:ff
    inet 172.20.211.53/32 scope global eth0
       valid_lft forever preferred_lft forever
    …
$ kubectl -n demo exec net -- ip route
default via 169.254.1.1 dev eth0
169.254.1.1 dev eth0 scope link

eth0@if173 is a veth pair whose other end is interface number 173 on the node. A /32 address, MTU 1450: Calico computes the MTU itself from the node's MTU (1500) and the enabled encapsulations, and VXLAN is enabled in the pool even though the lab does not use it (the value is stored on the node in /var/lib/calico/mtu). Let's ping hello and look at the pod's ARP table:

$ kubectl -n demo exec net -- ping -c1 172.20.211.43
PING 172.20.211.43 (172.20.211.43): 56 data bytes
64 bytes from 172.20.211.43: seq=0 ttl=63 time=0.103 ms

--- 172.20.211.43 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.103/0.103/0.103 ms
$ kubectl -n demo exec net -- ip neigh
169.254.1.1 dev eth0 lladdr ee:ee:ee:ee:ee:ee ref 1 used 0/0/0 probes 4 REACHABLE

ttl=63 — the packet went through one router (the node's kernel), even though both pods are on the same node. The pod's ARP table has a single entry — the fake gateway with the cali… MAC address.

On the node — interfaces and parameters:

user@k8s-node-1:~$ ip -br link | grep -E 'cali|vxlan'
vxlan.calico     UNKNOWN        66:ee:f4:61:98:50 <BROADCAST,MULTICAST,UP,LOWER_UP>
cali01e594701c5@if2 UP             ee:ee:ee:ee:ee:ee <BROADCAST,MULTICAST,UP,LOWER_UP>
calia68edcf6ff3@if2 UP             ee:ee:ee:ee:ee:ee <BROADCAST,MULTICAST,UP,LOWER_UP>
cali4d7612d46be@if2 UP             ee:ee:ee:ee:ee:ee <BROADCAST,MULTICAST,UP,LOWER_UP>
user@k8s-node-1:~$ cat /proc/sys/net/ipv4/conf/cali4d7612d46be/proxy_arp
1
user@k8s-node-1:~$ ip route get 172.20.211.53
172.20.211.53 dev cali4d7612d46be src 192.168.10.20 uid 1000
    cache

All cali… have one MAC, and proxy ARP is on for each; @if2 is the number of eth0 inside its pod's namespace (in every pod it is the second one, after lo). calia68edcf6ff3 is hello, cali4d7612d46be is net, and cali01e594701c5 is the csi-node-driver pod with address 172.20.211.1. The tunnel interface:

user@k8s-node-1:~$ ip -d link show vxlan.calico | grep -o 'vxlan id .* nolearning'
vxlan id 4096 local 192.168.10.20 dev ens18 srcport 0 0 dstport 4789 ttl auto ageing 300 reserved_bits 0xf7ffffff000000ff nolearning
user@k8s-node-1:~$ ip addr show vxlan.calico | grep 'inet '
    inet 172.20.211.0/32 scope global vxlan.calico
user@k8s-node-1:~$ ip neigh show dev vxlan.calico
172.20.70.192 lladdr 66:79:81:e0:1a:7d PERMANENT
172.20.172.0 lladdr 66:05:38:46:ab:ea PERMANENT
user@k8s-node-1:~$ bridge fdb show dev vxlan.calico
66:05:38:46:ab:ea dst 192.168.10.21 self permanent
66:79:81:e0:1a:7d dst 192.168.10.17 self permanent

VNI 4096, port 4789, the tunnel address is the first address of the node's block, 172.20.211.0. nolearning means the kernel does not learn neighbour addresses from incoming packets: the PERMANENT entries for the tunnel addresses of the master and k8s-node-2 (the MAC of their vxlan.calico and where to send UDP) were written by Felix itself. But, as the ip route above shows, there are no routes through vxlan.calico: on the lab all neighbours are in the same subnet.

Packets. Let's start two tcpdumps on k8s-node-1 — on the net pod's veth and on ens18 — and ping the hello pod on k8s-node-2 from net:

$ kubectl -n demo exec net -- ping -c1 172.20.172.6
PING 172.20.172.6 (172.20.172.6): 56 data bytes
64 bytes from 172.20.172.6: seq=0 ttl=62 time=0.536 ms

--- 172.20.172.6 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.536/0.536/0.536 ms

user@k8s-node-1:~$ sudo tcpdump -eni cali4d7612d46be -c 4 'icmp or arp'
22:47:09.834467 36:c6:f6:49:44:cf > ee:ee:ee:ee:ee:ee, ethertype IPv4 (0x0800), length 98: 172.20.211.53 > 172.20.172.6: ICMP echo request, id 46, seq 0, length 64
22:47:09.834894 ee:ee:ee:ee:ee:ee > 36:c6:f6:49:44:cf, ethertype IPv4 (0x0800), length 98: 172.20.172.6 > 172.20.211.53: ICMP echo reply, id 46, seq 0, length 64
22:47:14.960527 ee:ee:ee:ee:ee:ee > 36:c6:f6:49:44:cf, ethertype ARP (0x0806), length 42: Request who-has 172.20.211.53 tell 192.168.10.20, length 28
22:47:14.960534 36:c6:f6:49:44:cf > ee:ee:ee:ee:ee:ee, ethertype ARP (0x0806), length 42: Request who-has 169.254.1.1 tell 172.20.211.53, length 28

user@k8s-node-1:~$ sudo tcpdump -eni ens18 -c 2 'icmp and host 172.20.172.6'
22:47:09.834514 00:15:5d:0a:00:02 > 00:15:5d:0a:00:03, ethertype IPv4 (0x0800), length 98: 172.20.211.53 > 172.20.172.6: ICMP echo request, id 46, seq 0, length 64
22:47:09.834865 00:15:5d:0a:00:03 > 00:15:5d:0a:00:02, ethertype IPv4 (0x0800), length 98: 172.20.172.6 > 172.20.211.53: ICMP echo reply, id 46, seq 0, length 64

On the veth — the pod's MAC and ee:ee:…, on ens18 — the MAC addresses of the two nodes, while the IP addresses in both cases are the pods', without NAT and without a wrapper. ttl=62 — two routers: the kernel of k8s-node-1 and the kernel of k8s-node-2. Five seconds after the exchange both sides of the veth re-check their neighbour: the node asks for the pod's address, the pod for its fake gateway. Were the receiver on a node in another subnet, ens18 would carry UDP datagrams to port 4789 between node addresses instead of ICMP between pod addresses, with the pod's packet inside; the lab has no such node, so this cannot be shown live.

How proxy ARP answers a pod is best seen on a new pod: its first ARP request goes out while it starts. We listen on all interfaces and create net2:

user@k8s-node-1:~$ sudo tcpdump -eni any -c 2 'arp and host 169.254.1.1'
22:47:50.883928 calie01d6b79688 B   ifindex 174 f2:36:0d:6b:6f:f8 ethertype ARP (0x0806), length 48: Request who-has 169.254.1.1 tell 172.20.211.45, length 28
22:47:50.883945 calie01d6b79688 Out ifindex 174 ee:ee:ee:ee:ee:ee ethertype ARP (0x0806), length 48: Reply 169.254.1.1 is-at ee:ee:ee:ee:ee:ee, length 28

$ kubectl -n demo run net2 --image=busybox:1.37 --overrides='{"spec":{"nodeName":"k8s-node-1"}}' -- sleep 3600
pod/net2 created

A broadcast (B) question "who has 169.254.1.1" from the new pod 172.20.211.45 and, 17 microseconds later, the node's answer: ee:ee:ee:ee:ee:ee. Deleting a pod is visible on the node too: after kubectl -n demo delete pod net2 (DEL) the calie01d6b79688 interface, the route to 172.20.211.45 and the network namespace disappear, and the address goes back to the block — ip route | grep 172.20.211 again shows only the blackhole and three /32.

CNI configuration and binaries:

user@k8s-node-1:~$ ls /opt/cni/bin | xargs
LICENSE README.md bandwidth bridge calico calico-ipam dhcp dummy firewall flannel host-device host-local ipvlan loopback macvlan portmap ptp sbr static tap tuning vlan vrf
user@k8s-node-1:~$ ls -l /opt/cni/bin/bandwidth /opt/cni/bin/calico
-rwxr-xr-x 1 root root  5042926 Apr 24  2026 /opt/cni/bin/bandwidth
-rwxr-xr-x 1 root root 95727950 Oct 24 13:39 /opt/cni/bin/calico
user@k8s-node-1:~$ ls /etc/cni/net.d
ls: cannot open directory '/etc/cni/net.d': Permission denied
user@k8s-node-1:~$ sudo ls /etc/cni/net.d
10-calico.conflist
calico-kubeconfig
user@k8s-node-1:~$ sudo jq '.plugins[] | {type, ipam, capabilities}' /etc/cni/net.d/10-calico.conflist
{
  "type": "calico",
  "ipam": {
    …
    "type": "calico-ipam"
  },
  "capabilities": null
}
{
  "type": "portmap",
  "ipam": null,
  "capabilities": {
    "portMappings": true
  }
}

The configuration directory is closed to everyone but root (mode 700): calico-kubeconfig holds the credentials the plugin uses to talk to the API (the calico-cni-plugin ServiceAccount in calico-system). The chain has two plugins: calico (interface and address via calico-ipam) and portmap (hostPort support, enabled by hostPorts: Enabled in the Installation). bandwidth is in the binary directory but not in the chain, so the pod annotations kubernetes.io/ingress-bandwidth and kubernetes.io/egress-bandwidth it handles limit nothing on the lab. Our configuration does not use the other plugins in /opt/cni/bin. The modification time shows who put what there: bandwidth has the package build date, while calico, calico-ipam, flannel, host-local, loopback, portmap and tuning have the moment Calico was installed on the node — they were written by install-cni.

What the plugin did on ADD of the net pod is in its log:

user@k8s-node-1:~$ sudo grep 1d36abcf9f7f /var/log/calico/cni/cni.log | grep -oE '(Calico CNI IPAM assigned addresses IPv4=\[[^]]*\]|Setting the host side veth name to [a-z0-9]+|Adding IPv4 route \(inside container\)|Disabling IPv4 forwarding|Wrote updated endpoint to datastore)'
Calico CNI IPAM assigned addresses IPv4=[172.20.211.53/26]
Setting the host side veth name to cali4d7612d46be
Adding IPv4 route (inside container)
Disabling IPv4 forwarding
Wrote updated endpoint to datastore

Exactly the steps of the diagram at the beginning: an address from calico-ipam, a veth named cali…, a route inside the pod, forwarding turned off inside the pod, and the WorkloadEndpoint written. 1d36abcf9f7f is the beginning of the pod's sandbox ID, which we will now find with crictl.

The pod's network namespace from the node — through crictl and nsenter:

user@k8s-node-1:~$ POD=$(sudo crictl pods --name '^net$' -q)
user@k8s-node-1:~$ sudo crictl inspectp $POD | jq -r '.info.runtimeSpec.linux.namespaces[] | select(.type=="network").path'
/var/run/netns/cni-0d0c14d6-a330-4d39-54cf-8c500df46f20
user@k8s-node-1:~$ sudo nsenter --net=/var/run/netns/cni-0d0c14d6-a330-4d39-54cf-8c500df46f20 ip -br -4 addr
lo               UNKNOWN        127.0.0.1/8
eth0@if173       UP             172.20.211.53/32

The namespace is held by the sandbox's pause container, and the net container only joins it:

user@k8s-node-1:~$ sudo crictl inspectp $POD | jq -r '.info.image, .info.pid'
registry.k8s.io/pause:3.10.2
657193
user@k8s-node-1:~$ sudo crictl inspect $(sudo crictl ps --pod $POD -q) | jq -r .info.pid
657233
user@k8s-node-1:~$ ps -o pid,args -p 657193
    PID COMMAND
 657193 /pause
user@k8s-node-1:~$ ps -o pid,args -p 657233
    PID COMMAND
 657233 sleep 3600
user@k8s-node-1:~$ sudo readlink /proc/657193/ns/net /proc/657233/ns/net
net:[4026532358]
net:[4026532358]

/pause and sleep 3600 have the same namespace number: restarting the net container touches neither the address nor the veth; they live as long as the sandbox does.

IPAM. Calico's calicoctl utility (installed as a separate binary of the same version as Calico) shows the IPAM state as a summary — calicoctl ipam show --show-blocks — but all of it is visible through kubectl too, via Calico resources:

$ kubectl get blockaffinities
NAME                            CREATED AT
k8s-master-1-172-20-70-192-26   2026-10-24T10:34:56Z
k8s-node-1-172-20-211-0-26      2026-10-24T10:39:23Z
k8s-node-2-172-20-172-0-26      2026-10-24T10:51:40Z
$ kubectl get ipamblock 172-20-211-0-26 -o json | jq -c '.spec | {cidr, affinity, free: (.unallocated|length)}, (.attributes[] | {handle_id, pod: .secondary.pod, type: .secondary.type})'
{"cidr":"172.20.211.0/26","affinity":"host:k8s-node-1","free":60}
{"handle_id":"vxlan-tunnel-addr-k8s-node-1","pod":null,"type":"vxlanTunnelAddress"}
{"handle_id":"k8s-pod-network.9457999e6ebeb782c8e51628afc15ebd3a4cec62d1e67040f6862822c9a5bfb9","pod":"csi-node-driver-q6t25","type":null}
{"handle_id":"k8s-pod-network.167ac68289d80f244a698f115447dc85e2a75fa9e2c11671dd2ee994dba24a2d","pod":"hello-6d5c6db6b4-nqsp8","type":null}
{"handle_id":"k8s-pod-network.1d36abcf9f7fb65581a2a7228afe555a954a9abfff43f909aaa686840325b880","pod":"net","type":null}

The k8s-node-1 block has 64 addresses, four used, 60 free. The first address of every block (172.20.70.192, 172.20.211.0, 172.20.172.0) is taken by its node's vxlan.calico tunnel interface: in pools with VXLAN Calico takes the address for it from the same block. A pod's allocation ID is the network name from the CNI configuration (k8s-pod-network) plus the sandbox ID. The pod's annotations:

$ kubectl -n demo get pod net -o jsonpath='{.metadata.annotations}' | jq
{
  "cni.projectcalico.org/containerID": "1d36abcf9f7fb65581a2a7228afe555a954a9abfff43f909aaa686840325b880",
  "cni.projectcalico.org/podIP": "172.20.211.53/32",
  "cni.projectcalico.org/podIPs": "172.20.211.53/32",
  "kubernetes.io/limit-ranger": "LimitRanger plugin set: cpu, memory request for container net; memory limit for container net"
}

The last annotation was set by the LimitRange of the demo namespace from part 11, the rest by Calico.

The nodes' spec.podCIDR allocated by kube-controller-manager, for comparison:

$ kubectl get nodes -o custom-columns=NAME:.metadata.name,PODCIDR:.spec.podCIDR
NAME           PODCIDR
k8s-master-1   172.20.0.0/24
k8s-node-1     172.20.1.0/24
k8s-node-2     172.20.3.0/24

Nothing in common with Calico's real blocks — Calico does not read this field.

BGP on the node. BIRD's sessions and what it knows about the block of k8s-node-2:

user@k8s-node-1:~$ C=$(sudo crictl ps --name calico-node -q)
user@k8s-node-1:~$ sudo crictl exec $C birdcl show protocols
BIRD v0.3.3+birdv1.6.8 ready.
name     proto    table    state  since       info
static1  Static   master   up     2026-10-24
kernel1  Kernel   master   up     2026-10-24
device1  Device   master   up     2026-10-24
direct1  Direct   master   up     2026-10-24
Mesh_192_168_10_17 BGP      master   up     2026-10-24  Established
Mesh_192_168_10_21 BGP      master   up     13:33:41    Established
user@k8s-node-1:~$ sudo crictl exec $C birdcl show route 172.20.172.0/26
BIRD v0.3.3+birdv1.6.8 ready.
172.20.172.0/26    via 192.168.10.21 on ens18 [Mesh_192_168_10_17 2026-10-24 from 192.168.10.17] * (200/0) [i]
                   via 192.168.10.21 on ens18 [Mesh_192_168_10_21 13:33:41] (200/0) [i]
                   via 192.168.10.21 on ens18 [Mesh_192_168_10_21 2026-10-24] (100/0) [i]
                   via 192.168.10.21 on ens18 [kernel1 2026-10-24] (10)
user@k8s-node-1:~$ sudo crictl exec $C grep 'handled by Felix' /etc/calico/confd/config/bird_ipam.cfg
  if (net ~ 172.20.0.0/16) then { reject; } # VXLAN routes are handled by Felix.

The sessions with the master and k8s-node-2 are up (Established), and BIRD knows the neighbour's block both from BGP peers (Mesh_…) and from the kernel (kernel1 is the route Felix wrote). But it hands nothing from the pool to the kernel: the calico_kernel_programming filter that confd generated for a VXLAN pool rejects all 172.20.0.0/16 addresses, with the explanation right in the comment.

And the natOutgoing rule. First let's check that going out is masqueraded: in tcpdump on ens18, when a pod connects to port 443 of a server on the local network, the source address is the node's:

user@k8s-node-1:~$ sudo tcpdump -ni ens18 -c 2 'host 192.168.10.32 and tcp[tcpflags] & tcp-syn != 0'
22:49:46.116936 IP 192.168.10.20.60722 > 192.168.10.32.443: Flags [S], seq 1035344350, win 64860, options [mss 1410,sackOK,TS val 1430924388 ecr 0,nop,wscale 9], length 0
22:49:46.117256 IP 192.168.10.32.443 > 192.168.10.20.60722: Flags [S.], seq 587632908, ack 1035344351, win 65535, options [mss 1460,nop,wscale 8,sackOK,TS val 1277288463 ecr 1430924388], length 0

$ kubectl -n demo exec net -- nc -z -w 2 192.168.10.32 443

nc -z prints nothing and exits with code 0 — the port is open. The pod's MTU shows up here too: mss 1410 is 1450 minus 40 bytes of IP and TCP headers.

Where this rule lives. In Nftables mode Felix keeps all its rules in one table, ip calico, next to the kube-proxy tables:

user@k8s-node-1:~$ sudo nft list tables | grep -E 'table (ip|arp) '
table ip mangle
table ip filter
table ip nat
table ip kube-proxy
table ip calico
table arp calico-arp

mangle, filter and nat are tables of the iptables-nft compatibility layer, through which the kubelet writes its rules (part 4). calico-arp is a small auxiliary table in the arp family: through it Felix stops the node from answering a pod's ARP for the pod's own address. It has a chain per cali…:

user@k8s-node-1:~$ sudo nft list chain arp calico-arp filter-cali-arp-cali4d7612d46be
table arp calico-arp {
	chain filter-cali-arp-cali4d7612d46be {
		oifname "cali4d7612d46be" arp operation reply arp saddr ip 172.20.211.53 counter packets 0 bytes 0 drop comment "cali:NLnSDII0c5kZJj0t;"
	}
}

"An ARP reply going out into cali4d7612d46be on behalf of 172.20.211.53 — drop it": otherwise proxy ARP would answer the net pod's question about its own address too. Inside ip calico the logic is the same as Calico's in iptables mode, but instead of four iptables tables (raw, mangle, nat, filter) there is one table, and chain names got the former table as a prefix. The base chains (filter-FORWARD, nat-POSTROUTING and others) are hooked into netfilter with priorities chosen to interleave correctly with the kube-proxy chains; the working ones are called filter-cali-…, nat-cali-…. Instead of ipsets there are nftables sets in the same table. The masquerade chain and the way to it:

user@k8s-node-1:~$ sudo nft list chain ip calico nat-POSTROUTING
table ip calico {
	chain nat-POSTROUTING {
		type nat hook postrouting priority srcnat + 10; policy accept;
		counter packets 41613 bytes 2566587 jump nat-cali-POSTROUTING comment "cali:z057H0Z77Fcko47w;"
	}
}
user@k8s-node-1:~$ sudo nft list chain ip calico nat-cali-nat-outgoing
table ip calico {
	chain nat-cali-nat-outgoing {
		ip saddr @cali40masq-ipam-pools ip daddr != @cali40network-ip-pools counter packets 4 bytes 240 masquerade fully-random comment "cali:oMLt0S09xBKlYFS5;"
	}
}
user@k8s-node-1:~$ sudo nft list set ip calico cali40masq-ipam-pools
table ip calico {
	set cali40masq-ipam-pools {
		type ipv4_addr
		flags interval
		elements = { 172.20.0.0/16 }
	}
}

"Source in pools with natOutgoing (cali40masq-ipam-pools), destination not in Calico pools (cali40network-ip-pools, also 172.20.0.0/16): masquerade". A packet gets here along the chain nat-POSTROUTING → nat-cali-POSTROUTING → nat-cali-nat-outgoing; the base nat-POSTROUTING sits on the postrouting hook with priority srcnat + 10, that is, slightly after the kube-proxy NAT chain with priority srcnat. The rule's counter grows with every new outgoing connection — a NAT rule fires only on the first packet of a connection, and conntrack handles the rest itself. The cali40… sets are what would be ipsets in iptables mode: when pools change, Felix changes the set elements without touching the rules. The cali:… comment is a hash of the rule by which Felix recognises its rules and notices if someone changed them. What lives in the kube-proxy tables — in the next part.

Clean up: kubectl -n demo delete pod net.

Sources

Next — Service and kube-proxy: why pods need a stable address, how a virtual ClusterIP that exists on no interface turns into DNAT to a pod address, how EndpointSlice tracks ready pods and what kube-proxy in nftables mode writes on every node; NodePort, traffic policies, headless services, and LoadBalancer, which without a cloud stays Pending.

Only registered users can leave a comment.

Sign in or register to leave a comment.

© 2006-2026 Vitaliy Leschenko