Vitaliy Leschenko’s blog
Виталий Лещенко

Kubernetes for beginners. Part 20: How does NetworkPolicy work?

All parts of the series “Kubernetes for beginners”

In part 14 we saw the main rule of the Kubernetes network model: any pod can reach any other pod by its address, without NAT. Since then Services, DNS, an external address from MetalLB, a Traefik gateway and certificates from the previous part have been added — traffic from the outside is in order. But inside the cluster there is still not a single wall: a pod from another namespace can open a connection to the database, a compromised frontend can go anywhere in the local network. Today we look at NetworkPolicy — the standard Kubernetes object for restricting traffic between pods — then at Calico's own policies, which can do more, and at what all of it turns into on the node.

Everything is allowed by default

While a namespace has no policies at all, its pods are not isolated: they accept connections from anyone and go anywhere themselves — to other pods, to the node network, to the internet through the node's NAT. This is by design: a cluster has to work right after installation, without configuring a firewall.

Isolation is turned on by a policy that selects the pod. As soon as a pod is selected by at least one NetworkPolicy of type Ingress, it accepts only what one of the policies selecting it explicitly allows. The same goes for Egress separately. The directions are independent: an ingress-only policy does not touch the pod's outgoing traffic.

Three properties of the model follow from this and are worth remembering:

  • A policy selects the pods it applies to, not traffic. The spec.podSelector field is "to whom", the rules inside are "what is allowed".
  • Policies are additive. The allows of all policies selecting a pod add up. There is no order, no priorities, and the standard API has no deny rules at all: a deny is the absence of an allow for an isolated pod. You cannot write "everyone may, except this address".
  • Policies are connection-aware. If a connection from A to B is allowed, reply packets from B to A pass without a separate rule — the network plugin relies on conntrack, the kernel's connection tracking table.

And three exceptions spelled out in the Kubernetes documentation:

  • traffic between a pod and its node is always allowed: otherwise kubelet could not run liveness and readiness probes;
  • a pod cannot block access to itself;
  • the behaviour for pods with hostNetwork: true is undefined, and most plugins, Calico included, simply do not see them: such a pod lives in the node's network namespace, and its traffic looks like node traffic.

NetworkPolicy is implemented by the network plugin (CNI), not by Kubernetes itself: the API server accepts the object in any cluster, but if the plugin cannot do policies (like flannel, for example — a simple CNI plugin that only provides connectivity between pods), nothing happens and nobody warns you. Calico, which we have had since part 4, can.

The NetworkPolicy object

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: hello-from-traefik
  namespace: demo
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: hello
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: traefik
          podSelector:
            matchLabels:
              app.kubernetes.io/name: traefik
      ports:
        - protocol: TCP
          port: 8080

podSelector — which pods of its own namespace the policy applies to. An empty selector {} means all pods of the namespace. A policy never selects pods of another namespace.

policyTypes — Ingress, Egress or both. If the field is omitted, Ingress is always implied, and Egress only if the policy has an egress section. It is better to write it explicitly: the "deny all outgoing" policy looks exactly like policyTypes: [Egress] without an egress section, and without the explicit field its meaning is lost.

ingress[].from and egress[].to — a list of peers, each of one of these kinds:

  • podSelector — pods of its own namespace with these labels;
  • namespaceSelector — all pods of namespaces with these labels. A namespace is selected by name with the kubernetes.io/metadata.name label, which the control plane puts on every namespace automatically and does not let you change;
  • namespaceSelector and podSelector in one element — pods with these labels in namespaces with these labels;
  • ipBlock — an address range (cidr) with exceptions (except).

The most common mistake is an extra dash. Compare:

# one peer: traefik pods IN the traefik namespace (AND)
- from:
    - namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: traefik}}
      podSelector: {matchLabels: {app.kubernetes.io/name: traefik}}

# two peers: ALL pods of the traefik namespace OR traefik pods of its own namespace
- from:
    - namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: traefik}}
    - podSelector: {matchLabels: {app.kubernetes.io/name: traefik}}

The difference is one character, and the second form opens more than it seems.

ports — the protocol (TCP, UDP or SCTP — a transport protocol with several streams in one connection, found mostly in telecom equipment), the port as a number or the name of a container port, a range via endPort (numbers only). The list of peers and the list of ports inside one rule are joined by AND: "from these peers to these ports". A rule without from allows from everyone, a rule without ports — to all ports.

Two subtle points:

  • Ports and addresses — after DNAT. The Kubernetes documentation leaves it to the plugin whether a policy sees the address before or after rewriting, and Calico checks the packet after kube-proxy has replaced the service's ClusterIP with a pod address. So you write the container port (8080), not the service port (80), and select pods by labels, not by ClusterIP: an ipBlock with an address from the service network 10.96.0.0/12 will not work. Below, in "Hands-on", this shows up on the kubernetes service.
  • ipBlock is for addresses outside the cluster. Pod addresses change on every restart; there are selectors for them. And remember NAT: a pod going outside gets the node's address in our cluster (natOutgoing: true in Calico), so an external server sees the node, not the pod, while the policy on the cluster side, on the contrary, sees the pod's real address before NAT.

Whether a new policy closes connections that are already open is also left to the plugin by the Kubernetes documentation. In Calico the first rule of a pod's chain passes packets of established connections, so connections that are already open keep working, and a new policy applies to new ones.

Default deny and what breaks after it

The right starting point for a namespace with applications is to deny everything and open what is needed. The deny is a policy that selects all pods and allows nothing:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: demo
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

Right after it everything in demo stops working, and the first thing is DNS. A pod looks up postgres.demo.svc.cluster.local, sends a query to 10.96.0.10:53, kube-proxy rewrites the address to a CoreDNS pod in kube-system, and the pod's outgoing traffic is denied. The application sees not "connection refused" but "name not found" or a timeout, and people usually look for the problem in the wrong place. So the second policy is always an allow for DNS:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: demo
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

TCP is needed too: DNS switches to TCP when an answer does not fit into a UDP datagram. The k8s-app: kube-dns label is on the CoreDNS pods in a kubeadm cluster for historical reasons.

What else breaks after default deny and is worth thinking about in advance:

What Why How to open
incoming traffic from Traefik pods became isolated for ingress a policy from the traefik namespace
pods talking to each other inside demo isolation applies inside the namespace too rules for specific pairs
the application talks to the Kubernetes API 10.96.0.1:443 after DNAT is the control plane address, 192.168.10.17:6443 ipBlock with the API server address and port 6443
the application goes to the outside world (updates, external APIs) outgoing is denied ipBlock with the needed addresses, or giving that up altogether
Prometheus metrics scraping incoming to the metrics port a policy from the monitoring namespace
kubelet probes do not break: traffic from the node to its pod is always allowed —

A policy set for demo

demo runs hello (accepts HTTP from Traefik on port 8080) and postgres — a lab database that only hello talks to:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres
  namespace: demo
  labels:
    app.kubernetes.io/name: postgres
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: postgres
  template:
    metadata:
      labels:
        app.kubernetes.io/name: postgres
    spec:
      containers:
      - name: postgres
        image: postgres:18.6-alpine
        env:
        - name: POSTGRES_HOST_AUTH_METHOD   # lab only: login without a password
          value: trust
        ports:
        - name: postgres
          containerPort: 5432
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            memory: 256Mi
---
apiVersion: v1
kind: Service
metadata:
  name: postgres
  namespace: demo
spec:
  selector:
    app.kubernetes.io/name: postgres
  ports:
  - name: postgres
    port: 5432
    targetPort: postgres

A real database is not run without a password: the password goes into a Secret (part 12), and the data onto a volume, which is the subject of the next part. Here the database is only needed as a server on port 5432.

The default-deny and allow-dns policies are already there. Let's add three more.

Ingress to hello from Traefik is the hello-from-traefik policy from the beginning. Traefik, as we saw in part 18, goes straight to pod addresses rather than through the ClusterIP, so the source is the Traefik pod, and a selector on its namespace and labels works. If Traefik ran with hostNetwork: true, the source would be the node address, and you would have to allow the node addresses with ipBlock.

Egress from hello to the database:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: hello-to-postgres
  namespace: demo
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: hello
  policyTypes:
    - Egress
  egress:
    - to:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: postgres
      ports:
        - protocol: TCP
          port: 5432

And ingress to the database from hello:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: postgres-from-hello
  namespace: demo
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: postgres
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: hello
      ports:
        - protocol: TCP
          port: 5432

With default deny in both directions a connection has to be allowed twice: on egress at the client and on ingress at the server. Forgetting one of the two halves is the second most common mistake after DNS.

If the database is not in the cluster but on a separate server, hello-to-postgres gets an ipBlock with its /32 address instead of the selector, and ingress on the server is restricted by its own firewall — by node addresses, because from the outside pods appear with the node addresses.

The result for the namespace:

                 ┌──────────────── demo (default deny) ────────────────┐
 Traefik ───────►│ hello :8080 ──────────────────► postgres :5432      │
 (ns traefik)    │   │                                                 │
                 │   └──► CoreDNS :53 (kube-system)                    │
                 │                                                     │
                 │ everything else: ingress and egress denied          │
                 └─────────────────────────────────────────────────────┘

Namespace policies are the responsibility of the application team: whoever has rights on demo can change and delete them. Restrictions that must apply to everyone, always, regardless of what the teams write, cannot be expressed by the standard API: it is namespaced and allow-only. For that you need Calico policies.

How Calico applies policies

Calico runs on every node as the calico-node DaemonSet, and its main component inside is Felix, the agent that programs the node's data plane: routes to pods and filtering rules. Policies reach it like this:

 kubectl apply  ──►  kube-apiserver  ──►  etcd
                          │ watch
                          ▼
                     Typha (calico-system)          fans changes out to many nodes
                          │                         to spare the API server
                          ▼
                     Felix in calico-node on every node
                          │ recalculates which policies select
                          │ which pods of THIS node
                          ▼
         chains and address sets in the nftables table ip calico
         for the caliXXXXXXXXXXX interface of every pod

Typha is an intermediate cache between the API server and the Felix agents: in large clusters hundreds of Felix instances do not each hold their own watch on the API server but get changes from a few Typha replicas.

Felix sees every pod as a WorkloadEndpoint — an internal Calico object with the pod's address, its labels, the namespace labels and the name of the veth interface on the node (cali plus the first 11 characters of the SHA-1 of the string <namespace>.<pod>, as we computed in part 14). These objects are not visible through kubectl. Felix translates a Kubernetes NetworkPolicy into its internal model, where it is stored next to Calico policies and differs by kind, and turns all of it together into nftables rules. On the lab Calico runs in linuxDataplane: Nftables mode (part 4), and all rules live in its table ip calico — the same one that holds the natOutgoing masquerading from part 14. Chain names in it carry the prefix of the former iptables table, filter-:

  • two verdict maps (an nftables hash table that returns an action for a key right away; we saw them at kube-proxy in part 15): filter-cali-to-wl-dispatch and filter-cali-from-wl-dispatch, where the key is the name of a cali… interface and the value is a jump to that pod's chain; the chains of the same name apply them with a rule like oifname vmap @filter-cali-to-wl-dispatch (for traffic leaving a pod — by the input interface) and drop a packet for an unknown interface;
  • two chains per pod interface: filter-cali-tw-<interface> (to workload — traffic into the pod, that is ingress) and filter-cali-fw-<interface> (from workload — traffic out of the pod, egress);
  • per policy — the chains filter-cali-pi-<kind>/<namespace>/<name> (policy ingress) and filter-cali-po-… (policy egress), where the kind is knp for a Kubernetes NetworkPolicy, np for a Calico NetworkPolicy and gnp for a GlobalNetworkPolicy; the interface chains of the selected pods jump to them. If several policies of one tier have the same selector, Felix collects them into a group chain filter-cali-gi-…/filter-cali-go-… and jumps to that instead;
  • after the policies a pod's chain has profiles filter-cali-pri-kns.<namespace> and filter-cali-pri-ksa.<namespace>.<ServiceAccount> (pro- for egress) — Calico creates them for every namespace and ServiceAccount, and they allow everything. This is how "everything is allowed by default" works: if no policy selected the pod, the packet reaches the profile and is accepted;
  • selectors in rules ("pods with the label app.kubernetes.io/name=traefik in the traefik namespace") turn into nftables address sets named cali40s-<selector hash>. When a new pod with such a label appears, Felix adds its address to the set and leaves the chains alone.

Calico's other data planes work the same way in principle: in Iptables mode the same chains (cali-tw-…, cali-pi-…) live in the iptables filter table and the sets in ipset; in eBPF mode (eBPF is a Linux kernel mechanism for loading verified packet-processing programs into the kernel) the rules are executed by such programs on the interfaces, without netfilter; that mode is covered in part 5 of the series on Kubernetes over BGP.

Thanks to the two chains per interface a policy is checked on the sender's node (egress) and on the receiver's node (ingress) — each at its own end, even when the pods are on different nodes.

Calico policies

Calico adds its own resources in the projectcalico.org/v3 group. The Calico 3.32 operator installs the Calico API server by default — an extension of the Kubernetes API through which these resources are available to plain kubectl (without it you would need the calicoctl utility). To check: kubectl get tigerastatus apiserver. This brings a naming trap: networkpolicies exists both in networking.k8s.io and in projectcalico.org, and the short name cnp is taken by two resources at once. kubectl get networkpolicies shows only Kubernetes policies, so Calico policies are requested by the full resource name:

$ kubectl get networkpolicies -A
No resources found
$ kubectl get networkpolicies.projectcalico.org -A
NAMESPACE       NAME                                   CREATED AT
calico-system   calico-system.apiserver-access         2026-10-24T10:37:10Z
calico-system   calico-system.default-deny             2026-10-24T10:37:10Z
calico-system   calico-system.kube-controller-access   2026-10-24T10:37:10Z
kube-system     calico-system.cluster-dns              2026-10-24T10:36:50Z

These four policies were created by the Calico operator for its own components — more on them below.

Kubernetes NetworkPolicy Calico NetworkPolicy Calico GlobalNetworkPolicy
Scope namespace namespace the whole cluster, nodes included (HostEndpoint)
Actions allow only Allow, Deny, Pass, Log Allow, Deny, Pass, Log
Order none, everything adds up the order field the order field
Selectors matchLabels, matchExpressions expressions: app.kubernetes.io/name == 'hello' && tier != 'test' the same, plus namespaceSelector
Other ports, ipBlock ServiceAccount, ICMP, nets, network sets the same

HostEndpoint is a Calico object for an interface of the node itself: with it, policies start filtering node traffic, not only pod traffic. Right after a HostEndpoint is created without policies, Calico denies everything on the interface except service exceptions (failsafe), so on a working cluster they are introduced carefully; today we do not touch them.

Actions:

  • Allow — let through, check no further;
  • Deny — drop, check no further;
  • Log — write the packet to the kernel log (with the calico-packet prefix, set by the logPrefix field of FelixConfiguration) and continue with the next rule;
  • Pass — stop checking the policies of the current level (tier) and go on to the next one.

order is a number; policies with a lower value are checked first, rules inside a policy top to bottom. The first Allow or Deny that matches decides the packet's fate.

Tiers are levels of policies. Policies are grouped into tiers, tiers are checked by their order from lower to higher, and inside each one its policies by their order. This lets security administrators keep their rules at a level that is checked before the teams' policies and that the teams cannot change (rights on a tier are granted separately through RBAC). The built-in tiers in Calico 3.32:

Tier order defaultAction What is in it
kube-admin 1,000 Pass ClusterNetworkPolicy with tier: Admin (more on them below)
default 1,000,000 Deny all Kubernetes NetworkPolicies and Calico policies without an explicit tier
kube-baseline 10,000,000 Pass ClusterNetworkPolicy with tier: Baseline

According to the Calico documentation the order of the default tier cannot be changed, and kube-admin and kube-baseline have the Pass action fixed. Besides these, the Calico operator creates the calico-system tier with order: 100 for its own policies — the four we saw above. calico-system.default-deny closes the pods in calico-system except calico-apiserver, the two other policies in calico-system open the paths the components need, and calico-system.cluster-dns in kube-system selects the CoreDNS pods: it allows them all outgoing traffic and incoming traffic from Calico components, and passes the rest of the incoming traffic on with a Pass rule. There is no need to touch this tier.

The implicit deny at the end of a tier. This is the main thing to understand about tiers. If at least one policy of a tier selected the pod and no rule in that tier made a decision, the packet is dropped — like for an isolated pod in the standard API. A tier where no policy selected the pod is simply skipped. For a packet that did not match any rule to go on to be checked further, you either set the tier's defaultAction: Pass or end the policy with an explicit action: Pass rule.

Closing the path from pods to a service server

Now a general technique that we will need in part 22. The node network has servers that nodes must reach but pods must not: the NFS server nfs.example.com (192.168.10.30), which we will set up in part 22, the file server files.example.com (192.168.10.31) with shared folders over SMB (the Windows file access protocol, port 445) for part 24, the backup server, hardware management interfaces. On such servers access is usually restricted to the node addresses 192.168.10.17, .20 and .21. But a pod going to the external network leaves through the node's NAT in our cluster — with the node's address. For the server it is indistinguishable from the node itself, and the restriction by node addresses stops meaning anything.

This has to be closed from the cluster side: deny all pods outgoing traffic to the service addresses. Team NetworkPolicies are no good for this — they are namespaced, allow-only, and a team can delete any of them. You need a GlobalNetworkPolicy in a separate tier that is checked before the teams' policies.

Why this does not break the volumes themselves: network volumes are mounted by the node's kernel (who gives it the command and how is in the next part), and the CSI drivers that manage this run with hostNetwork: true. Their traffic is node traffic, not a WorkloadEndpoint, and pod policies do not apply to it.

First the tier:

apiVersion: projectcalico.org/v3
kind: Tier
metadata:
  name: infra
spec:
  order: 200
  defaultAction: Pass

order: 200 — after the service tier calico-system (100), but before kube-admin (1,000) and default (1,000,000). defaultAction: Pass — packets that matched no rule of this tier go on to the ordinary policies.

The policy:

apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
  name: infra.deny-pods-to-service-hosts
spec:
  tier: infra
  order: 100
  namespaceSelector: all()       # pods only: nodes have no namespace
  types:
    - Egress
  egress:
    - action: Log
      destination:
        nets:
          - 192.168.10.30/32     # nfs.example.com
          - 192.168.10.31/32     # files.example.com
    - action: Deny
      destination:
        nets:
          - 192.168.10.30/32
          - 192.168.10.31/32
    - action: Pass               # everything else goes on to be checked further

Let's see why exactly like this.

  • namespaceSelector: all() selects endpoints in all namespaces — that is, pods. Without it, the all() selector of a GlobalNetworkPolicy would also select resources without a namespace — node endpoints (HostEndpoint), if they ever appear — and then the policy would close the way to the service servers for the nodes themselves. Calico has a separate selector for those, global().
  • Log before Deny — every attempt lands in the node's kernel log, and you can see which pod tried to reach the server. When the log is no longer needed, the rule is removed.
  • The last rule Pass is mandatory, even with defaultAction: Pass. This policy selects all pods of the cluster, and without an explicit Pass the fate of "the rest" of the traffic would be decided by the tier's default behaviour. If you put such a policy into the default tier with only a Deny rule, the implicit deny at the end of the tier drops all outgoing traffic of all pods except what the teams' policies allow — and in namespaces without policies the cluster loses connectivity at once. An explicit Pass makes the intent obvious and does not depend on the tier's settings.
  • If some pod does need access (a backup job, for example), it gets a separate Allow rule with a selector before the Deny — and that is an administrator's decision, visible in one place.

There are usually more than two addresses; it is convenient to collect them into a GlobalNetworkSet — a named set of networks with labels — and refer to it with a selector (destination: selector: role == 'service-hosts') instead of repeating the addresses in every rule.

The same way Calico also does a cluster-wide default deny, which the standard API does not have: a GlobalNetworkPolicy with a namespaceSelector that excludes the system namespaces (kubernetes.io/metadata.name not in {"calico-system", "kube-public", "kube-system", "tigera-operator"}) and a rule allowing DNS — the Calico documentation has such an example.

ClusterNetworkPolicy

What we did with a Calico policy — administrator rules on top of the teams' rules — has long been wanted in the standard API. In SIG Network this is the job of the Network Policy API project. First it released a pair of resources, AdminNetworkPolicy and BaselineAdminNetworkPolicy (version v1alpha1), and in April 2026, in release v0.2.0, it merged them into one resource, ClusterNetworkPolicy (policy.networking.k8s.io/v1alpha2), with a tier field: Admin is checked before NetworkPolicy and can deny, Baseline is checked after it and sets default rules that teams can override. Within a tier policies are ordered by the priority field from 0 to 1000; the actions are Accept, Deny and Pass (hand the decision on).

Status at the time of writing: it is alpha, not part of Kubernetes itself and installed as separate CRDs, and not all network plugins support it. Calico 3.32 understands it and installs the CRD itself — on our cluster clusternetworkpolicies.policy.networking.k8s.io has been there since installation — and puts the policies into the kube-admin and kube-baseline tiers, translating priority into order. On a learning cluster it is interesting to try, but relying in production on an alpha API whose fields are still changing (between v1alpha1 and v1alpha2 both the resource names and the actions changed: Allow became Accept) is premature — Calico policies are more reliable here.

Hands-on

The lab is as at the end of the previous part: Traefik in the traefik namespace serves hello.apps.example.com from demo, which runs three hello pods and postgres from the manifest above. Let's prepare a probe pod in demo and a second one in the sandbox namespace, which has no policies:

$ kubectl -n demo run probe --image=busybox:1.37 --labels=app.kubernetes.io/name=probe -- sleep 1d
pod/probe created
$ kubectl create namespace sandbox
namespace/sandbox created
$ kubectl -n sandbox run probe --image=busybox:1.37 -- sleep 1d
pod/probe created
$ kubectl -n demo get pods -o wide
NAME                       READY   STATUS    RESTARTS   AGE   IP              NODE         NOMINATED NODE   READINESS GATES
hello-6d5c6db6b4-2wvbm     1/1     Running   0          54s   172.20.211.30   k8s-node-1   <none>           <none>
hello-6d5c6db6b4-nt56v     1/1     Running   0          54s   172.20.172.44   k8s-node-2   <none>           <none>
hello-6d5c6db6b4-snq5t     1/1     Running   0          54s   172.20.211.27   k8s-node-1   <none>           <none>
postgres-dbd894d99-cqrv5   1/1     Running   0          53s   172.20.211.34   k8s-node-1   <none>           <none>
probe                      1/1     Running   0          41s   172.20.172.50   k8s-node-2   <none>           <none>
$ kubectl -n traefik get pods -o wide
NAME                       READY   STATUS    RESTARTS   AGE   IP              NODE         NOMINATED NODE   READINESS GATES
traefik-6b9b5d5f95-j9rhw   1/1     Running   0          69s   172.20.172.47   k8s-node-2   <none>           <none>
traefik-6b9b5d5f95-vvfzh   1/1     Running   0          69s   172.20.211.35   k8s-node-1   <none>           <none>

Before the policies everything is open:

$ kubectl -n sandbox exec probe -- nc -zv -w 2 postgres.demo 5432
postgres.demo (10.106.39.155:5432) open
$ kubectl -n demo exec probe -- wget -qO- -T 2 http://hello.demo/
hello from hello-6d5c6db6b4-nt56v
$ kubectl -n demo exec probe -- nc -zv -w 2 192.168.10.31 445
192.168.10.31 (192.168.10.31:445) open

A pod from another namespace opened a connection to the database, and a pod from demo — to the file server. We apply default-deny:

$ kubectl apply -f default-deny.yaml
networkpolicy.networking.k8s.io/default-deny created
$ kubectl -n demo exec probe -- nslookup hello.demo.svc.cluster.local
;; connection timed out; no servers could be reached

command terminated with exit code 1
$ kubectl -n demo exec probe -- wget -qO- -T 2 http://hello.demo/
wget: bad address 'hello.demo'
command terminated with exit code 1

DNS broke first, as expected, and wget complains not about the connection but about the name. We add allow-dns and the other policies:

$ kubectl apply -f allow-dns.yaml
networkpolicy.networking.k8s.io/allow-dns created
$ kubectl apply -f hello-from-traefik.yaml -f hello-to-postgres.yaml -f postgres-from-hello.yaml
networkpolicy.networking.k8s.io/hello-from-traefik created
networkpolicy.networking.k8s.io/hello-to-postgres created
networkpolicy.networking.k8s.io/postgres-from-hello created
$ kubectl -n demo get networkpolicy
NAME                  POD-SELECTOR                      AGE
allow-dns             <none>                            40s
default-deny          <none>                            91s
hello-from-traefik    app.kubernetes.io/name=hello      20s
hello-to-postgres     app.kubernetes.io/name=hello      20s
postgres-from-hello   app.kubernetes.io/name=postgres   20s
$ kubectl -n demo exec probe -- nslookup hello.demo.svc.cluster.local
Server:		10.96.0.10
Address:	10.96.0.10:53


Name:	hello.demo.svc.cluster.local
Address: 10.103.28.1

$ kubectl -n demo exec probe -- wget -qO- -T 2 http://hello.demo/
wget: download timed out
command terminated with exit code 1
$ kubectl -n sandbox exec probe -- nc -zv -w 2 postgres.demo 5432
nc: postgres.demo (10.106.39.155:5432): Connection timed out
command terminated with exit code 1

The full name in nslookup is no accident: with the short hello.demo busybox prints NXDOMAIN for every search suffix and exits with an error even after finding the address (part 16). DNS works; probe cannot reach hello (it has no egress allow, and hello has no ingress allow from it), and from sandbox the database cannot be reached either. Note: the denial looks like a timeout, not "connection refused" — the packet is dropped silently.

Now let's check that hello reaches the database. The hello image is built on distroless and has neither a shell nor nc, so we add an ephemeral container to the pod with kubectl debug. It starts inside the already running pod and shares its network namespace — it has the same address and the same policies. An ephemeral container cannot be removed from a pod; it goes away together with the pod:

$ kubectl -n demo debug hello-6d5c6db6b4-nt56v --image=busybox:1.37 --profile=general -c nettest -- sleep 1d
$ kubectl -n demo exec hello-6d5c6db6b4-nt56v -c nettest -- nc -zv -w 2 postgres 5432
postgres (10.106.39.155:5432) open
$ kubectl -n demo exec hello-6d5c6db6b4-nt56v -c nettest -- nc -zv -w 2 192.168.10.32 443
nc: 192.168.10.32 (192.168.10.32:443): Connection timed out
command terminated with exit code 1

The database is reachable, the server in the local network 192.168.10.32 is not: hello's outgoing traffic is under default-deny too. And here is what happens if you forget one half of a connection:

$ kubectl -n demo delete networkpolicy postgres-from-hello
networkpolicy.networking.k8s.io "postgres-from-hello" deleted from demo namespace
$ kubectl -n demo exec hello-6d5c6db6b4-nt56v -c nettest -- nc -zv -w 2 postgres 5432
nc: postgres (10.106.39.155:5432): Connection timed out
command terminated with exit code 1
$ kubectl apply -f postgres-from-hello.yaml
networkpolicy.networking.k8s.io/postgres-from-hello created
$ kubectl -n demo exec hello-6d5c6db6b4-nt56v -c nettest -- nc -zv -w 2 postgres 5432
postgres (10.106.39.155:5432) open

From the outside the site works — Traefik gets into hello by the hello-from-traefik allow. In this check we reach Traefik directly through the NodePort of its service (32486 on k8s-node-1), from the laptop and with a Host header; the full path through the MetalLB address and TLS is the one built in part 17 and the two after it:

$ curl -s -m 5 -H 'Host: hello.apps.example.com' http://192.168.10.20:32486/
hello from hello-6d5c6db6b4-nt56v

Through the gateway with a certificate the command would be curl -s https://hello.apps.example.com/; its output was not captured here.

A policy sees addresses after DNAT. Let's allow probe egress to the Kubernetes API — first by the address of the kubernetes service, as intuition suggests:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: probe-to-api
  namespace: demo
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: probe
  policyTypes:
    - Egress
  egress:
    - to:
        - ipBlock:
            cidr: 10.96.0.1/32
      ports:
        - protocol: TCP
          port: 443

$ kubectl apply -f probe-to-api.yaml
networkpolicy.networking.k8s.io/probe-to-api created
$ kubectl -n demo exec probe -- nc -zv -w 2 kubernetes.default.svc.cluster.local 443
nc: kubernetes.default.svc.cluster.local (10.96.0.1:443): Connection timed out
command terminated with exit code 1

No way through. We change cidr in the policy to 192.168.10.17/32 and port to 6443 — the API server address and port that kube-proxy turns 10.96.0.1:443 into:

$ kubectl apply -f probe-to-api.yaml
networkpolicy.networking.k8s.io/probe-to-api configured
$ kubectl -n demo exec probe -- nc -zv -w 2 kubernetes.default.svc.cluster.local 443
kubernetes.default.svc.cluster.local (10.96.0.1:443) open
$ kubectl -n demo delete -f probe-to-api.yaml
networkpolicy.networking.k8s.io "probe-to-api" deleted from demo namespace

The pod still connects to 10.96.0.1:443, while the policy already compares 192.168.10.17:6443.

What is on the node. The hello-6d5c6db6b4-nt56v pod runs on k8s-node-2 with the address 172.20.172.44. We find its interface by the route and check it against the name hash:

user@k8s-node-2:~$ ip route | grep 172.20.172.44
172.20.172.44 dev calid81a2fd6ead scope link metric 1024
user@k8s-node-2:~$ printf demo.hello-6d5c6db6b4-nt56v | sha1sum
d81a2fd6ead57c74c790ee4269d803e58bb91e4f  -

A packet that the node's kernel sends into the pod's interface passes the forward hook, Calico's base chain filter-FORWARD, the filter-cali-FORWARD chain and reaches the dispatcher — the verdict map where the pod's chain is found right away by the name of the output interface. There are four pods on the node — hello and probe from demo, Traefik and a Calico service pod:

user@k8s-node-2:~$ sudo nft list map ip calico filter-cali-to-wl-dispatch
table ip calico {
	map filter-cali-to-wl-dispatch {
		type ifname : verdict
		elements = { "cali5ed29fc4313" : goto filter-cali-tw-cali5ed29fc4313,
			     "calia463c8b45d5" : goto filter-cali-tw-calia463c8b45d5,
			     "cali1f20149a837" : goto filter-cali-tw-cali1f20149a837,
			     "calid81a2fd6ead" : goto filter-cali-tw-calid81a2fd6ead }
	}
}

The pod's chain itself (the hashes in the comments differ on every node):

user@k8s-node-2:~$ sudo nft list chain ip calico filter-cali-tw-calid81a2fd6ead
table ip calico {
	chain filter-cali-tw-calid81a2fd6ead {
		ct state established,related counter packets 48 bytes 4780 accept comment "cali:J8USgVKtKmeIe9eC;"
		ct state invalid counter packets 0 bytes 0 drop comment "cali:tpzDHJWaPoRJ9c-x;"
		counter packets 5 bytes 300 meta mark set meta mark & 0xfffcffff comment "cali:mOiJXmMqCk_6D0uA;"
		counter packets 3 bytes 180 meta mark set meta mark & 0xfffdffff comment "cali:-BrknvkZT89OBg5i; Start of tier default"
		meta mark & 0x00020000 == 0x00000000 counter packets 3 bytes 180 jump filter-cali-pi-knp/demo/default-deny comment "cali:wS87dAYgY8Qv1scI;"
		meta mark & 0x00010000 == 0x00010000 counter packets 0 bytes 0 return comment "cali:rJjBzICTlwANm-rK; Return if policy accepted"
		meta mark & 0x00020000 == 0x00000000 counter packets 3 bytes 180 jump filter-cali-pi-knp/demo/hello-from-traefik comment "cali:Ob1dmafZJNlYN6IN;"
		meta mark & 0x00010000 == 0x00010000 counter packets 3 bytes 180 return comment "cali:XT75spmzaSoDqrIn; Return if policy accepted"
		meta mark & 0x00020000 == 0x00000000 counter packets 0 bytes 0 drop comment "cali:QhISMjA7w6n0c7jC; End of tier default. Drop if no policies passed packet"
		counter packets 0 bytes 0 jump filter-cali-pri-kns.demo comment "cali:t5R3VD0h1d7cyDqK;"
		meta mark & 0x00010000 == 0x00010000 counter packets 0 bytes 0 return comment "cali:xOh-51vkZftxinOv; Return if profile accepted"
		counter packets 0 bytes 0 jump filter-cali-pri-ksa.demo.default comment "cali:-WMoRlMBSgE6l-hC;"
		meta mark & 0x00010000 == 0x00010000 counter packets 0 bytes 0 return comment "cali:tYnNxJTkcWQ0zGdu; Return if profile accepted"
		counter packets 0 bytes 0 drop comment "cali:xPRB56C7OTlIahZq; Drop if no profiles matched"
	}
}

The whole model is here in about fifteen lines. First, packets of established connections pass (conntrack) — this is how "the reply is allowed" works and how open connections survive new policies — and packets that conntrack considers invalid are dropped. Then the result mark bits are cleared: 0x10000 means "accepted by a policy", 0x20000 means "passed on" (Pass). The chains of the policies selecting this pod are called in turn: default-deny and hello-from-traefik; the name shows both the kind (knp — Kubernetes NetworkPolicy) and the namespace. The policy that allowed the packet sets bit 0x10000, and the pod's chain returns — after that the packet is accepted by the base chain filter-FORWARD (Policy explicitly accepted packet). If none allowed it and nobody set Pass — End of tier default ... drop: that very implicit deny at the end of the tier. Below are the profiles kns.demo and ksa.demo.default, which would accept everything if no policy selected the pod; for our pod no packet gets down to them. The counters show how many packets went through each rule: all new connections that reached the pod were accepted by hello-from-traefik. The attempts of probe did not get here at all — they were dropped by its own filter-cali-fw-… chain on egress.

The chain of the hello-from-traefik policy is one rule: source from an address set, port 8080, set the "accepted" bit:

user@k8s-node-2:~$ sudo nft list chain ip calico filter-cali-pi-knp/demo/hello-from-traefik
table ip calico {
	chain filter-cali-pi-knp/demo/hello-from-traefik {
		ip saddr @cali40s-rob4WbEGIc5jlJUd4ap62IG tcp dport 8080 counter packets 1 bytes 60 meta mark set meta mark | 0x00010000 comment "cali:xmq1Y6-XZkIVNcn7; KubernetesNetworkPolicy demo/hello-from-traefik ingress"
	}
}
user@k8s-node-2:~$ sudo nft list set ip calico cali40s-rob4WbEGIc5jlJUd4ap62IG
table ip calico {
	set cali40s-rob4WbEGIc5jlJUd4ap62IG {
		type ipv4_addr
		flags interval
		elements = { 172.20.172.47, 172.20.211.35 }
	}
}

The set holds the addresses of the two Traefik pods, one on each worker node. Scale Traefik to three and Felix adds a third address to the set, while the chains stay as they are. The default-deny chain for ingress is empty in meaning — it has a single continue rule: the policy selects the pod but allows nothing.

Now the egress of the probe pod (interface cali5ed29fc4313 on the same node):

user@k8s-node-2:~$ sudo nft list chain ip calico filter-cali-fw-cali5ed29fc4313
table ip calico {
	chain filter-cali-fw-cali5ed29fc4313 {
		ct state established,related counter packets 41 bytes 2278 accept comment "cali:AeGE1uIZEs6IHBCc;"
		ct state invalid counter packets 0 bytes 0 drop comment "cali:ul9cvPWeiE6970Y4;"
		counter packets 84 bytes 6128 meta mark set meta mark & 0xfffcffff comment "cali:-epCTaGIhf4ACahG;"
		udp dport 4789 counter packets 0 bytes 0 drop comment "cali:2zjF5A7i-ubwEJFH; Drop VXLAN encapped packets originating in workloads"
		meta l4proto ipencap counter packets 0 bytes 0 drop comment "cali:o6BUzHCylnAha-3F; Drop IPinIP encapped packets originating in workloads"
		counter packets 49 bytes 3732 meta mark set meta mark & 0xfffdffff comment "cali:5ZJaeiYdH_1iS0vr; Start of tier default"
		meta mark & 0x00020000 == 0x00000000 counter packets 33 bytes 2672 jump filter-cali-go-iBOnLIGKDZQ62MbnzC50 comment "cali:QPSvYligqE5NnYoT;"
		meta mark & 0x00010000 == 0x00010000 counter packets 26 bytes 2252 return comment "cali:k937F-wsU9jm-T4y; Return if policy accepted"
		meta mark & 0x00020000 == 0x00000000 counter packets 0 bytes 0 drop comment "cali:Y9PXwujeD4Iy_Mqj; End of tier default. Drop if no policies passed packet"
		counter packets 0 bytes 0 jump filter-cali-pro-kns.demo comment "cali:KUve0gdJD1lTvy1H;"
		meta mark & 0x00010000 == 0x00010000 counter packets 0 bytes 0 return comment "cali:8XgaEsHkxyfJbRYq; Return if profile accepted"
		counter packets 0 bytes 0 jump filter-cali-pro-ksa.demo.default comment "cali:fzqYFm0Vr-1cSXG0;"
		meta mark & 0x00010000 == 0x00010000 counter packets 0 bytes 0 return comment "cali:nbx0mCNhhs4-y0pv; Return if profile accepted"
		counter packets 0 bytes 0 drop comment "cali:GSMTU3rHcQPwWEoo; Drop if no profiles matched"
	}
}
user@k8s-node-2:~$ sudo nft list chain ip calico filter-cali-go-iBOnLIGKDZQ62MbnzC50
table ip calico {
	chain filter-cali-go-iBOnLIGKDZQ62MbnzC50 {
		counter packets 61 bytes 4590 jump filter-cali-po-knp/demo/allow-dns comment "cali:4kvO45nccx68evnZ;"
		meta mark & 0x00030000 == 0x00000000 counter packets 21 bytes 1260 jump filter-cali-po-knp/demo/default-deny comment "cali:GE12kpl8uZ68stOw;"
	}
}

Egress has two peculiarities. Two drop rules before the policies drop packets that the pod itself wrapped into VXLAN (UDP 4789) or IP-in-IP — the tunnels in which Calico carries pod packets between nodes (part 14): this way a pod cannot pass its packet off as tunnel traffic between nodes. And the allow-dns and default-deny policies select pods for egress with the same selector {}, so Felix collected them into the group filter-cali-go-…: allow-dns marks packets to the CoreDNS pods as "accepted" (their addresses are in a cali40s-… set of their own), the rest go through the empty default-deny, get neither "accepted" nor Pass and are dropped by the End of tier default rule. The counters of neighbouring rules do not add up here: when policies change, Felix rewrites the rules of the chains, and a new rule starts counting from zero.

GlobalNetworkPolicy. We apply the infra tier and the policy and repeat the check from sandbox, which has no policies of its own:

$ kubectl get tiers.projectcalico.org -o custom-columns=NAME:.metadata.name,ORDER:.spec.order,DEFAULT:.spec.defaultAction
NAME            ORDER      DEFAULT
calico-system   100        Deny
default         1000000    Deny
infra           200        Pass
kube-admin      1000       Pass
kube-baseline   10000000   Pass
$ kubectl -n sandbox exec probe -- nc -zv -w 2 192.168.10.31 445
nc: 192.168.10.31 (192.168.10.31:445): Connection timed out
command terminated with exit code 1
$ kubectl -n sandbox exec probe -- nc -zv -w 2 192.168.10.32 443
192.168.10.32 (192.168.10.32:443) open

The file server cannot be reached from sandbox, the rest of the network can: Pass handed the decision on, and sandbox has no policies. The probe pod in sandbox runs on k8s-node-1 with the address 172.20.211.36, and that node's kernel log shows every attempt — the first SYN and its retransmission a second later:

user@k8s-node-1:~$ sudo journalctl -k --since -3min | grep calico-packet
Oct 29 17:14:27 k8s-node-1 kernel: calico-packet: IN=cali798b0e33dc9 OUT=ens18 MAC=ee:ee:ee:ee:ee:ee:e6:ba:dd:df:35:bf:08:00 SRC=172.20.211.36 DST=192.168.10.31 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=40761 DF PROTO=TCP SPT=33081 DPT=445 WINDOW=64860 RES=0x00 SYN URGP=0 MARK=0x80000 
Oct 29 17:14:28 k8s-node-1 kernel: calico-packet: IN=cali798b0e33dc9 OUT=ens18 MAC=ee:ee:ee:ee:ee:ee:e6:ba:dd:df:35:bf:08:00 SRC=172.20.211.36 DST=192.168.10.31 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=40762 DF PROTO=TCP SPT=33081 DPT=445 WINDOW=64860 RES=0x00 SYN URGP=0 MARK=0x80000

SRC=172.20.211.36 is the pod's address, still before NAT: the policy fires before natOutgoing replaces it with the node's address. In the pod's egress chain the infra tier now comes first:

user@k8s-node-1:~$ sudo nft list chain ip calico filter-cali-fw-cali798b0e33dc9
table ip calico {
	chain filter-cali-fw-cali798b0e33dc9 {
		ct state established,related counter packets 6 bytes 312 accept comment "cali:J_LfkfwvC54l83v2;"
		ct state invalid counter packets 0 bytes 0 drop comment "cali:MfiUREkd1FQP_HzX;"
		counter packets 4 bytes 240 meta mark set meta mark & 0xfffcffff comment "cali:tz4G1cxLiOD-ZEMK;"
		udp dport 4789 counter packets 0 bytes 0 drop comment "cali:kavdU9UVVc0pYmKY; Drop VXLAN encapped packets originating in workloads"
		meta l4proto ipencap counter packets 0 bytes 0 drop comment "cali:-6_2lSk8w-Om-V8p; Drop IPinIP encapped packets originating in workloads"
		counter packets 3 bytes 180 meta mark set meta mark & 0xfffdffff comment "cali:CLeo9XtaTkaB1LqV; Start of tier infra"
		meta mark & 0x00020000 == 0x00000000 counter packets 3 bytes 180 jump filter-cali-po-gnp/infra.deny-pods-to-service-hosts comment "cali:0jLFQEYCXuKDMW2-;"
		meta mark & 0x00010000 == 0x00010000 counter packets 0 bytes 0 return comment "cali:THVQXgMg_INj4vLP; Return if policy accepted"
		counter packets 1 bytes 60 jump filter-cali-pro-kns.sandbox comment "cali:w9NEpwcoZm1K-J81;"
		meta mark & 0x00010000 == 0x00010000 counter packets 1 bytes 60 return comment "cali:uw0CZIYtsHoUOiew; Return if profile accepted"
		counter packets 0 bytes 0 jump filter-cali-pro-ksa.sandbox.default comment "cali:RmTNeXNdE1VRzbIu;"
		meta mark & 0x00010000 == 0x00010000 counter packets 0 bytes 0 return comment "cali:mGWnLQLcplCSpp5i; Return if profile accepted"
		counter packets 0 bytes 0 drop comment "cali:CFU8WikkcUpyKa4J; Drop if no profiles matched"
	}
}
user@k8s-node-1:~$ sudo nft list chain ip calico filter-cali-po-gnp/infra.deny-pods-to-service-hosts
table ip calico {
	chain filter-cali-po-gnp/infra.deny-pods-to-service-hosts {
		counter packets 3 bytes 180 meta mark set meta mark & 0xffe7ffff comment "cali:tPsxEs9yNevw57sV; GlobalNetworkPolicy infra.deny-pods-to-service-hosts egress"
		ip daddr 192.168.10.30 counter packets 0 bytes 0 meta mark set meta mark | 0x00080000 comment "cali:BevwyTLT-AsDpjvJ;"
		ip daddr 192.168.10.31 counter packets 2 bytes 120 meta mark set meta mark | 0x00080000 comment "cali:8c-HdlA1pEPopaz0;"
		meta mark & 0x00080000 == 0x00080000 counter packets 2 bytes 120 log prefix "calico-packet: " level info comment "cali:TS4pb3hyChf-kKaL;"
		counter packets 3 bytes 180 meta mark set meta mark & 0xffe7ffff comment "cali:IbLmPGYDsdd-1P9P;"
		ip daddr 192.168.10.30 counter packets 0 bytes 0 meta mark set meta mark | 0x00080000 comment "cali:r3dpDZl_FK_647kN;"
		ip daddr 192.168.10.31 counter packets 2 bytes 120 meta mark set meta mark | 0x00080000 comment "cali:4IemNuPhSwSiXvWz;"
		meta mark & 0x00080000 == 0x00080000 counter packets 2 bytes 120 meta mark set meta mark | 0x00040000 comment "cali:9cqIl-YWHB6_XRPC;"
		meta mark & 0x00040000 == 0x00040000 counter packets 2 bytes 120 drop comment "cali:KbiVF72qasWlTRW8;"
		counter packets 1 bytes 60 meta mark set meta mark | 0x00020000 comment "cali:BV9Sd-zYRt1FO0Bv;"
	}
}

Felix unrolls every policy rule into several lines: the addresses from nets one by one mark the packet with the service bit 0x80000 ("matched"), and then the action is taken on that bit — log for the Log rule (hence MARK=0x80000 in the log), bit 0x40000 and drop for Deny. The last line is our Pass: bit 0x20000. After it the pod's chain has no End of tier infra ... drop (the tier has defaultAction: Pass), and since in sandbox no other policy selects the packet, it reaches the kns.sandbox profile and is accepted. And a check from the node itself — the service server is reachable for it as before:

user@k8s-node-1:~$ nc -zv -w 2 192.168.10.31 445
Connection to 192.168.10.31 445 port [tcp/microsoft-ds] succeeded!

Sources

Now there are walls inside the cluster: the demo namespace is closed by default and open exactly as much as needed, and the service servers of the node network are closed to all pods by a policy that the teams cannot get around.

Next — volumes and CSI: where a pod keeps data that has to survive a restart, how PersistentVolume, PersistentVolumeClaim and StorageClass work, and what a CSI driver does so that a mounted directory appears on the node.

Only registered users can leave a comment.

Sign in or register to leave a comment.

© 2006-2026 Vitaliy Leschenko