Kubernetes for beginners. Part 38: Building a hybrid cluster with kubeadm
All parts of the series “Kubernetes for beginners”In part 36 we looked at how a Windows node differs from a Linux node, and in the previous part at how its network works and why a separate cluster built for Calico for Windows from the start is more convenient for Windows workers than migrating a running one. Today we build that cluster and look live at everything the two previous parts talked about. The whole hybrid cluster block runs on it, while the series' main lab cluster stays as it was — with VXLANCrossSubnet and BGP.
The part has three sections. The first is the Linux base of the test cluster: the master and two Linux workers are installed exactly as in part 4, so only the differences are here, and Calico is installed from day one in a mode that Windows understands. The second is the Windows worker k8s-lab-win-1.example.com on Windows Server 2025 in the Server Core installation. The path is the same as for Linux nodes: prepare the OS, install containerd, kubelet and kubeadm, run kubeadm join, give the node a network. The differences are in the details: Windows services instead of systemd, HostProcess pods instead of privileged containers, HNS instead of netfilter, and Linux pods have to be explicitly kept off the new node. The third is checking and taking it apart: a Windows pod, connectivity between the OSes, and what all of it looks like from inside the node.
The lab
| Node | Address | Role | Resources |
|---|---|---|---|
k8s-lab-master-1.example.com |
192.168.10.23 | control plane, Ubuntu Server 26.04 | 2 vCPU, 4 GB, 32 GB |
k8s-lab-node-1.example.com |
192.168.10.24 | Linux worker, Ubuntu Server 26.04 | 2 vCPU, 8 GB, 32 GB |
k8s-lab-node-2.example.com |
192.168.10.26 | Linux worker, Ubuntu Server 26.04 | 2 vCPU, 8 GB, 32 GB |
k8s-lab-win-1.example.com |
192.168.10.25 | Windows worker, Windows Server 2025 Core | 4 vCPU, 8 GB, 64 GB |
| Network | Test cluster | Main cluster (for comparison) |
|---|---|---|
| node network | 192.168.10.0/24, gateway 192.168.10.1 (router.example.com) |
the same |
| pod network | 172.21.0.0/16, /26 blocks |
172.20.0.0/16 |
| service network | 10.96.0.0/12, API 10.96.0.1, DNS 10.96.0.10 |
the same |
| API address | k8s-lab-api.example.com:6443 → 192.168.10.23 |
k8s-api.example.com:6443 |
| delivery between nodes | always VXLAN, BGP off | VXLANCrossSubnet and BIRD |
| Component | Linux nodes | Windows node |
|---|---|---|
| OS | Ubuntu Server 26.04 | Windows Server 2025 Datacenter, Server Core, build 10.0.26100 |
| Kubernetes | 1.37.2 | 1.37.2 |
| containerd | 2.2.2 from the Ubuntu packages | 2.2.2, a Windows service |
| Calico | 3.33.0, Tigera operator, nftables dataplane | 3.33.0 (Felix and CNI for Windows), a HostProcess pod |
| kube-proxy | nftables mode |
kernelspace (winkernel) mode, a HostProcess pod from a DaemonSet |
The test cluster has its own pod network so that its addresses in the output are not confused with those of the main one; the service network is the same — the clusters do not route between each other, and overlapping service ranges do not bother them. Kubernetes is 1.37.2 right away, the version the main cluster moved to in part 35: there is no point installing 1.36 only to upgrade it straight away.
Why the test cluster has two Linux workers. A Windows node can run only Windows containers, while almost everything that serves the cluster exists only as Linux images: CoreDNS, the Tigera operator, Typha and calico-kube-controllers, webhooks (cert-manager, Calico, later the gMSA webhook), metrics-server, CSI controllers. The Calico for Windows documentation directly requires at least one Linux worker that runs Calico's cluster-wide components. All of this could be kept on the master by removing its taint, but then the test cluster would differ from the main one in the most important thing — the separation of node roles. One worker would be enough for the cluster to work, but then it could not be maintained: kubectl drain of the only Linux worker (part 35) would leave CoreDNS, the webhooks and the controllers with nowhere to go. So next to the master there are two Linux workers, k8s-lab-node-1 and k8s-lab-node-2, as in the main cluster; for Typha high availability Calico recommends three Linux workers, but two are enough for a test cluster.
workstation: kubectl, context lab
│ 6443
▼
┌─ k8s-lab-master-1 192.168.10.23 ──────────────────────────┐
│ etcd, kube-apiserver, scheduler, controller-manager │
│ API name: k8s-lab-api.example.com │
└────────────────────────────┬──────────────────────────────┘
┌────────────────────────┼────────────────────────┐
▼ ▼ ▼
┌─ k8s-lab-node-1 .24 ─┐ ┌─ k8s-lab-node-2 .26 ─┐ ┌─ k8s-lab-win-1 .25 ──┐
│ Ubuntu Server 26.04 │ │ Ubuntu Server 26.04 │ │ Windows Server 2025 │
│ system pods │ │ system pods │ │ Core, aspnetapp │
│ and hello pods │ │ and hello pods │ │ taint os=windows │
│ 172.21.211.0/26 │ │ 172.21.172.0/26 │ │ 172.21.41.0/26 │
└──────────────────────┘ └──────────────────────┘ └──────────────────────┘
between all nodes — VXLAN: UDP 4789, VNI 4096
The node addresses in the diagram are the last octet in the 192.168.10.0/24 network, and below them are the pod address blocks Calico will give the nodes. How to install Server Core and work with it from PowerShell is covered in the first part of the Windows Server series. The Windows node has twice the CPUs of the Linux workers: the Kubernetes documentation names 4 cores, 8 GB of memory and 50 GB of disk as a reasonable starting point — the OS itself takes more, and Windows images weigh hundreds of megabytes to gigabytes. The Linux workers get as much memory, with room to spare: in part 39 Prometheus and Loki will run on them.
DNS needs records for the four nodes and k8s-lab-api.example.com → 192.168.10.23 — the same separate API name as in part 4: if the test cluster ever gets a second master, the name will be pointed at a virtual address without rebuilding the cluster.
Linux nodes: as in part 4
Preparation, containerd and packages
On k8s-lab-master-1, k8s-lab-node-1 and k8s-lab-node-2 everything is done as in part 4, sections "Preparing the nodes", "containerd" and "kubeadm, kubelet and kubectl", with these differences:
- names and addresses —
hostnamectl set-hostname k8s-lab-master-1,k8s-lab-node-1andk8s-lab-node-2, addresses 192.168.10.23, .24 and .26 in netplan; the uniqueness checks ofproduct_uuidand/etc/machine-idmatter especially if the machines are cloned from the same template as the main cluster's nodes; - swap, the
overlayandbr_netfiltermodules, sysctl, chrony withrouter.example.com— unchanged; - containerd — the same
containerdpackage from the Ubuntu repository,SystemdCgroup = true; thesandboximage is set to whatkubeadm config images list --kubernetes-version v1.37.2names — for the same reason as in part 4: kubeadm warns about a mismatched pause image; - registry trust — nothing to configure: the test cluster takes the
helloimages from the same private package inghcr.ioas the main one, and ghcr.io has a public certificate (part 5); - package repository —
v1.37right away:
user@k8s-lab-node-1:~$ curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.37/deb/Release.key \
| sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
user@k8s-lab-node-1:~$ echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.37/deb/ /' \
| sudo tee /etc/apt/sources.list.d/kubernetes.list
user@k8s-lab-node-1:~$ sudo apt-get update && sudo apt-get install -y kubelet kubeadm kubectl
user@k8s-lab-node-1:~$ sudo apt-mark hold kubelet kubeadm kubectl
user@k8s-lab-node-1:~$ kubeadm version -o short
v1.37.2
The nodes' firewall is off, as in the main cluster; of the port table in part 4 the test cluster does not need 179/TCP — there will be no BGP here — while 4789/UDP is now needed between all nodes, not only between subnets.
Configuration and kubeadm init
kubeadm-config.yaml on the master differs from the part 4 file in the address, names, version and pod network:
apiVersion: kubeadm.k8s.io/v1beta4
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 192.168.10.23
bindPort: 6443
nodeRegistration:
name: k8s-lab-master-1
criSocket: unix:///run/containerd/containerd.sock
---
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
kubernetesVersion: v1.37.2
controlPlaneEndpoint: k8s-lab-api.example.com:6443
networking:
podSubnet: 172.21.0.0/16
serviceSubnet: 10.96.0.0/12
dnsDomain: cluster.local
apiServer:
certSANs:
- k8s-lab-api.example.com
- k8s-lab-master-1.example.com
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd
serverTLSBootstrap: true
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: nftables
The kubeadm configuration format in 1.37 is still v1beta4; the meaning of all four documents is covered in part 4, section "Cluster configuration". Two fields matter here for the Windows node. serverTLSBootstrap: true goes into the shared KubeletConfiguration and onto the Windows node too: its kubelet will request a serving certificate from the cluster CA, and metrics-server will be able to scrape metrics from it while verifying the certificate. controlPlaneEndpoint is the name that not only nodes but also the Calico components on Windows will use; more on this below.
user@k8s-lab-master-1:~$ sudo kubeadm config validate --config kubeadm-config.yaml
ok
user@k8s-lab-master-1:~$ sudo kubeadm init --config kubeadm-config.yaml
[init] Using Kubernetes version: v1.37.2
[preflight] Running pre-flight checks
...
[certs] apiserver serving cert is signed for DNS names [k8s-lab-api.example.com k8s-lab-master-1 k8s-lab-master-1.example.com kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.10.23]
...
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
...
kubeadm join k8s-lab-api.example.com:6443 --token 3h8q2w.k5n9v1c7x4m6z0p8 \
--discovery-token-ca-cert-hash sha256:7b2e9d4a1c6f8b3e0d5a2c7f9e4b1d6a8c3f0e5b2d7a9c4e1f6b8d3a0c5e2f7b
The phases are the same as in part 4, section "kubeadm init and its phases". The test cluster has its own CA — a new root of trust — so the sha256:… hash is different too: the certificates of the main and test clusters do not trust each other. super-admin.conf is likewise taken off the master to a safe place.
A second context in the kubeconfig
From the workstation we now need to reach two clusters. That is what the kubeconfig is designed for: one file holds several clusters, users and contexts, kubectl works with the current context, and the --context flag picks another one for a single command.
The first idea is to put the test cluster's admin.conf next to the existing file and list both files in the KUBECONFIG variable. Don't. kubeadm names the entries the same way in both clusters: cluster kubernetes, user kubernetes-admin, context kubernetes-admin@kubernetes. When the files are merged, the first of identical names wins, and the "second" context silently gets the address and credentials of the first cluster. So we move the contents of admin.conf under our own names — with kubectl config commands.
We copy admin.conf from the master as in part 4, section "Access from the workstation":
user@k8s-lab-master-1:~$ sudo install -m 600 -o user -g user /etc/kubernetes/admin.conf ~/admin.conf
$ LAB=~/.kube/lab-admin.conf
$ scp user@k8s-lab-master-1.example.com:admin.conf "$LAB" && chmod 600 "$LAB"
$ ssh user@k8s-lab-master-1.example.com rm admin.conf
$ field() { kubectl --kubeconfig "$LAB" config view --raw -o jsonpath="$1"; }
$ kubectl config set-cluster lab --server=https://k8s-lab-api.example.com:6443
Cluster "lab" set.
$ kubectl config set clusters.lab.certificate-authority-data "$(field '{.clusters[0].cluster.certificate-authority-data}')"
Property "clusters.lab.certificate-authority-data" set.
$ kubectl config set users.lab-admin.client-certificate-data "$(field '{.users[0].user.client-certificate-data}')"
Property "users.lab-admin.client-certificate-data" set.
$ kubectl config set users.lab-admin.client-key-data "$(field '{.users[0].user.client-key-data}')"
Property "users.lab-admin.client-key-data" set.
$ kubectl config set-context lab --cluster=lab --user=lab-admin
Context "lab" created.
$ shred -u "$LAB"
set-cluster lab— an entry in theclusterslist with the test cluster's API address;certificate-authority-datais its CA, whichkubectluses to verify kube-apiserver's certificate.users.lab-admin— the test cluster administrator's client certificate and key. For the*-datafieldskubectl config setexpects base64 and decodes it itself before writing, and inadmin.confthe values are already base64 — so we pass them as is.set-context lab— a "cluster — user" pair under a short name.shred -u— the copy ofadmin.confwith the private key is no longer needed; the key is already in~/.kube/config.
$ kubectl config get-contexts
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
* kubernetes-admin@kubernetes kubernetes kubernetes-admin
lab lab lab-admin
$ kubectl --context lab cluster-info
Kubernetes control plane is running at https://k8s-lab-api.example.com:6443
CoreDNS is running at https://k8s-lab-api.example.com:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
$ kubectl config use-context lab
Switched to context "lab".
From here on in this and the following parts of the block, kubectl commands without --context run in the test cluster: the current context is lab. You have to keep the current context in mind all the time: it is stored in the file and shared by all terminal windows, and kubectl delete in "the wrong" cluster is a classic mistake. That is why scripts specify the context explicitly (kubectl --context lab …, for Helm --kube-context lab), and many people put the current context's name into their shell prompt.
Calico ready for Windows from the start
The node is still NotReady — there is no pod network yet, as in part 4 before Calico was installed. We install Calico with the same operator, but with everything a Windows node will need from the start: no VXLANCrossSubnet, no BGP and no migration later.
CRDs and the operator
$ kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.33.0/manifests/v3_projectcalico_org.yaml
customresourcedefinition.apiextensions.k8s.io/apiservers.operator.tigera.io created
...
customresourcedefinition.apiextensions.k8s.io/ipamconfigurations.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.projectcalico.org created
...
$ kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.33.0/manifests/tigera-operator.yaml
namespace/tigera-operator created
...
deployment.apps/tigera-operator created
The difference from part 4 is in the CRD file. There, Calico resources (IPPool, Calico NetworkPolicy and others) were stored in internal CRDs of the crd.projectcalico.org group, and the "real" projectcalico.org/v3 API with validation and defaults was served by a separate aggregated Calico API server. Starting with Calico 3.33, new installations on Kubernetes 1.36 and later use native projectcalico.org/v3 CRDs by default: Calico resources are ordinary CRDs that kubectl works with directly, and validation and defaults are handled by admission policies (MutatingAdmissionPolicy and ValidatingAdmissionPolicy — rules in the CEL language that kube-apiserver evaluates itself, without calling an external webhook) and a small webhook pod. For Kubernetes 1.34–1.35 Calico has a separate CRD file, and kubectl create instead of apply is needed for the same reason as in part 4: the CRDs are too large for the client-side apply annotation.
The API address for Calico
Before installing Calico we create a ConfigMap through which the operator passes the API address to Calico components:
kind: ConfigMap
apiVersion: v1
metadata:
name: kubernetes-services-endpoint
namespace: tigera-operator
data:
KUBERNETES_SERVICE_HOST: k8s-lab-api.example.com
KUBERNETES_SERVICE_PORT: "6443"
$ kubectl apply -f kubernetes-services-endpoint.yaml
configmap/kubernetes-services-endpoint created
Normally pods reach the API at the address of the kubernetes service — 10.96.0.1. On a Windows node this does not work: Calico starts in a HostProcess pod in the host's network, and a Windows host cannot reach service addresses — VFP does their load balancing on pod ports, not on the host itself. Besides, calico-node-windows starts before kube-proxy sets up service load balancing: kube-proxy for Windows waits for the network that Calico creates — without a direct API address Calico would never start. So the operator passes the KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT variables from this ConfigMap to Calico components, and they go to the API directly — by the shared name. The ConfigMap also applies to the Linux components: on Linux nodes calico-node also goes to k8s-lab-api.example.com rather than through kube-proxy and does not depend on it at startup. In the main cluster the ConfigMap would have had to be added after the fact, restarting every calico-node; here it exists from the first minute.
Installation
calico-installation.yaml:
apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
name: default
spec:
serviceCIDRs:
- 10.96.0.0/12
calicoNetwork:
bgp: Disabled
linuxDataplane: Nftables
windowsDataplane: HNS
ipPools:
- name: default-ipv4-ippool
cidr: 172.21.0.0/16
blockSize: 26
encapsulation: VXLAN
natOutgoing: Enabled
nodeSelector: all()
---
apiVersion: operator.tigera.io/v1
kind: APIServer
metadata:
name: default
spec: {}
Every field here was chosen in the previous part; a short reminder:
bgp: Disabled— calico-node does not start BIRD; Felix writes the routes to other nodes' blocks on Linux too, from Calico's resources. Calico for Windows cannot do VXLAN together with BGP.encapsulation: VXLANinstead ofVXLANCrossSubnet— packets between pods on different nodes always go through a VXLAN tunnel, even within one subnet: the pod's L2 frame is wrapped into a UDP datagram to port 4789 between node addresses (part 18 of the networking series). The price is 50 bytes of headers per packet and a pod MTU of 1450 instead of 1500. VNI 4096 and port 4789 are the defaults, and Windows will not accept others.windowsDataplane: HNS— the only Calico dataplane for Windows. Until the field is set, the operator does not serve Windows nodes; with it, it creates thecalico-node-windowsDaemonSet — empty for now, there are no Windows nodes yet.serviceCIDRs— the service network, matchingserviceSubnetfrom the kubeadm configuration: on Windows the CNI plugin attaches an outbound NAT policy (OutBoundNAT) with exceptions — the pod network and the service network — to each pod's HNS endpoint, so that traffic to pods and services leaves with the pod's address.cidr: 172.21.0.0/16,blockSize: 26,natOutgoing: Enabled— as in part 4, only our own network; a /26 block on a Windows node means 60 addresses for pods, more on this in thekubeadm joinsection.linuxDataplane: NftablesandAPIServer— as in part 4. With native CRDs,APIServerdeploys not an aggregated API server but the validating webhook pod.
$ kubectl create -f calico-installation.yaml
installation.operator.tigera.io/default created
apiserver.operator.tigera.io/default created
$ kubectl get tigerastatus
NAME AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
apiserver True False False 3m45s All objects available
calico True False False 3m10s All objects available
calico-windows True Invalid StrictAffinity, it must be set to 'true' when using Calico IPAM on Windows: StrictAffinity is false, it must be set to 'true' in the default IPAMConfiguration when using Calico IPAM on Windows
ippools True False False 4m50s All objects available
tiers True False False 4m50s All objects available
There are more rows here than in part 4: tiers are the Calico policy tiers from part 20, and calico-windows is the state of the Windows part, which the operator shows because the Installation has windowsDataplane. It is DEGRADED and says plainly what is missing.
strictAffinity
The last IPAM setting — IPAM is the Calico module that hands out addresses to pods:
$ kubectl patch ipamconfigurations default --type merge --patch='{"spec": {"strictAffinity": true}}'
ipamconfiguration.projectcalico.org/default patched
$ kubectl get ipamconfigurations default -o jsonpath='{.spec.strictAffinity}{"\n"}'
true
$ kubectl get tigerastatus calico-windows
NAME AVAILABLE PROGRESSING DEGRADED SINCE
calico-windows True False False 25s
strictAffinity: true forbids all nodes to take addresses from other nodes' blocks: a Windows node holds exactly one block, and an address from it on another node would break routing (details in the previous part). We set it now, while there is no Windows node and no Linux node has had a chance to take someone else's address. Calico creates the IPAMConfiguration resource named default itself during installation, and the native CRD lets us change it directly with kubectl.
The Calico pods on the master:
$ kubectl -n calico-system get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
calico-kube-controllers-7cbbcdd58d-2h4x8 1/1 Running 0 4m50s 172.21.70.196 k8s-lab-master-1
calico-node-k74mn 1/1 Running 0 4m53s 192.168.10.23 k8s-lab-master-1
calico-typha-7c68577f9-hjzzm 1/1 Running 0 4m54s 192.168.10.23 k8s-lab-master-1
calico-webhooks-648955b74-2m5p7 1/1 Running 0 4m57s 172.21.70.195 k8s-lab-master-1
calico-webhooks-648955b74-zqrkl 1/1 Running 0 4m57s 172.21.70.194 k8s-lab-master-1
csi-node-driver-gjfgd 2/2 Running 0 4m52s 172.21.70.197 k8s-lab-master-1
calico-webhooks is that native-CRD webhook pod instead of calico-apiserver. While the cluster has one node, everything runs on the master: Calico tolerates its taint.
The Linux workers
We join k8s-lab-node-1 and k8s-lab-node-2 with the command from the init output, as in part 4, section "Worker nodes: kubeadm join", and approve the kubelet serving CSRs of the master and the workers after checking names and addresses, as in the "Certificate for kubelet HTTPS" section. The command is the same on both workers; here is the first one:
user@k8s-lab-node-1:~$ sudo kubeadm join k8s-lab-api.example.com:6443 --token 3h8q2w.k5n9v1c7x4m6z0p8 \
--discovery-token-ca-cert-hash sha256:7b2e9d4a1c6f8b3e0d5a2c7f9e4b1d6a8c3f0e5b2d7a9c4e1f6b8d3a0c5e2f7b
...
This node has joined the cluster:
...
$ kubectl get csr --field-selector spec.signerName=kubernetes.io/kubelet-serving
NAME AGE SIGNERNAME REQUESTOR REQUESTEDDURATION CONDITION
csr-4wm2t 7m2s kubernetes.io/kubelet-serving system:node:k8s-lab-master-1 <none> Pending
csr-7672m 18s kubernetes.io/kubelet-serving system:node:k8s-lab-node-1 <none> Pending
csr-gsll8 9s kubernetes.io/kubelet-serving system:node:k8s-lab-node-2 <none> Pending
csr-lp4tq 7m21s kubernetes.io/kubelet-serving system:node:k8s-lab-master-1 <none> Pending
$ kubectl certificate approve csr-4wm2t csr-7672m csr-gsll8 csr-lp4tq
certificatesigningrequest.certificates.k8s.io/csr-4wm2t approved
certificatesigningrequest.certificates.k8s.io/csr-7672m approved
certificatesigningrequest.certificates.k8s.io/csr-gsll8 approved
certificatesigningrequest.certificates.k8s.io/csr-lp4tq approved
There are two requests from the master: the kubelet repeats the CSR while there is no answer to the first one — both can be approved, and once a certificate is issued the kubelet takes one. Until the CSRs are approved, the kubelet has no serving certificate at all: kubectl logs for the node's pods fails with remote error: tls: internal error, and metrics-server does not pass its readiness check.
Both CoreDNS replicas started on the master before the workers appeared: there were no other nodes, and CoreDNS tolerates the master's taint. They will not move by themselves — the scheduler does not touch running pods. Let's restart the Deployment:
$ kubectl -n kube-system rollout restart deployment coredns
deployment.apps/coredns restarted
$ kubectl -n kube-system get pods -l k8s-app=kube-dns -o wide
NAME READY STATUS RESTARTS AGE IP NODE
coredns-8f97fdbd9-9gc9h 1/1 Running 0 27s 172.21.172.3 k8s-lab-node-2
coredns-8f97fdbd9-vr6vv 1/1 Running 0 26s 172.21.70.199 k8s-lab-master-1
The replicas went to different nodes — that is the anti-affinity rule in the kubeadm manifest: "kube-dns replicas preferably on different nodes". But one of them is on the master again. The rule is only preferred, CoreDNS tolerates the master's taint, and the scheduler added the anti-affinity to the other node scores. What matters for DNS is that the replicas are on different nodes: shutting down any one of them leaves the cluster with working DNS.
Checking the Linux part
$ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k8s-lab-master-1 Ready control-plane 8m2s v1.37.2 192.168.10.23 <none> Ubuntu 26.04.1 LTS 7.0.0-42-generic (amd64) containerd://2.2.2
k8s-lab-node-1 Ready <none> 60s v1.37.2 192.168.10.24 <none> Ubuntu 26.04.1 LTS 7.0.0-42-generic (amd64) containerd://2.2.2
k8s-lab-node-2 Ready <none> 51s v1.37.2 192.168.10.26 <none> Ubuntu 26.04.1 LTS 7.0.0-42-generic (amd64) containerd://2.2.2
$ kubectl -n calico-system get ds
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
calico-node 3 3 3 3 3 kubernetes.io/os=linux 6m49s
calico-node-windows 0 0 0 0 0 kubernetes.io/os=windows 107s
csi-node-driver 3 3 3 3 3 kubernetes.io/os=linux 6m48s
$ kubectl -n calico-system get pods -l k8s-app=calico-typha -o wide
NAME READY STATUS RESTARTS AGE IP NODE
calico-typha-7c68577f9-8gjrr 1/1 Running 0 50s 192.168.10.24 k8s-lab-node-1
calico-typha-7c68577f9-hjzzm 1/1 Running 0 6m51s 192.168.10.23 k8s-lab-master-1
In the KERNEL-VERSION column kubectl 1.37 appends the node's architecture to the kernel version. calico-node-windows already exists with zero desired pods — windowsDataplane: HNS created it. Typha is the intermediary between the API and the Calico agents on the nodes: it alone holds a watch on kube-apiserver and distributes changes to all Felix instances, so that hundreds of nodes do not each load the API with their own watch. The operator picks the number of replicas by the number of nodes: while there was one node, one replica ran; with three nodes there are two, on different nodes, in the node network (the pod address is the node address). Felix on the Windows node will connect to one of them as well.
The main difference from the main cluster is visible in a Linux worker's routing table:
user@k8s-lab-node-2:~$ ip route
default via 192.168.10.1 dev ens18 proto static
172.21.70.192/26 via 172.21.70.192 dev vxlan.calico onlink mtu 1450
blackhole 172.21.172.0/26 proto 80
172.21.172.2 dev calid60012519f0 scope link metric 1024
172.21.172.3 dev caliad8a24266f7 scope link metric 1024
172.21.211.0/26 via 172.21.211.0 dev vxlan.calico onlink mtu 1450
192.168.10.0/24 dev ens18 proto kernel scope link src 192.168.10.26
user@k8s-lab-node-2:~$ ip -d link show vxlan.calico | grep -o 'vxlan id [0-9]* .*dstport [0-9]*'
vxlan id 4096 local 192.168.10.26 dev ens18 srcport 0 0 dstport 4789
user@k8s-lab-node-2:~$ pgrep -a bird || echo 'BIRD is not running'
BIRD is not running
user@k8s-lab-node-2:~$ sudo ss -ltn 'sport = :179'
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
In part 4 the route to a neighbour's block went via 192.168.10.17 dev ens18 proto 80 — directly via the node's address. Here the blocks of the master, 172.21.70.192/26, and of the first worker, 172.21.211.0/26, are reached through the vxlan.calico interface with those nodes' tunnel addresses (the first address of a block — 172.21.70.192, 172.21.211.0 — Calico gives to the node's VXLAN interface), and the routes have their own MTU of 1450 — room for the VXLAN headers. Such routes have no proto 80 mark: Felix puts it only on the blackhole of its own block. onlink tells the kernel to consider the gateway reachable directly through this interface even though it is not in the interface's subnet. VNI 4096, port 4789, nobody listens on BGP port 179. The worker's own block is 172.21.172.0/26; its .0 address is taken by the tunnel address, .2 and .3 are a Calico pod (csi-node-driver) and CoreDNS.
metrics-server and cert-manager
The test cluster needs two add-ons right away.
metrics-server — as in part 32, section "metrics-server", with the same manifest and without --kubelet-insecure-tls: the kubelet serving certificates are signed by the test cluster's CA, and we have just approved the CSRs. If you forget to approve them, the metrics-server pod will be Running but not Ready: its readiness check returns 500 until it can scrape metrics from at least one kubelet.
$ kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.9.0/components.yaml
...
$ kubectl get apiservice v1beta1.metrics.k8s.io
NAME SERVICE AVAILABLE AGE
v1beta1.metrics.k8s.io kube-system/metrics-server True 2m35s
cert-manager — as in part 19, section "Installation", with the same manifest but without the --enable-gateway-api flag: the test cluster needs neither Gateway API nor Traefik. Nor does it need Let's Encrypt issuers — cert-manager is here for the gMSA webhook from part 40, which needs an HTTPS certificate issued inside the cluster.
$ kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.21.2/cert-manager.yaml
...
$ kubectl -n cert-manager rollout status deploy cert-manager-webhook
deployment "cert-manager-webhook" successfully rolled out
Both manifests give their Deployments nodeSelector: kubernetes.io/os: linux — the Windows node will not interest them even if it ever loses its taint. We will install Prometheus, Grafana and csi-driver-smb into the test cluster in part 39, and Traefik, MetalLB and the rest of the main cluster are not needed here.
hello for the checks
To check connectivity between the OSes we need a Linux service. Let's take hello from part 5 — from the same private package in ghcr.io, with the ghcr-pull pull secret bound to the namespace's default ServiceAccount, as in that part:
$ kubectl create namespace demo
$ kubectl label namespace demo pod-security.kubernetes.io/enforce=restricted
$ read -rsp 'GitHub token (read:packages): ' GHCR_TOKEN; echo
$ kubectl -n demo create secret docker-registry ghcr-pull \
--docker-server=ghcr.io --docker-username=example --docker-password="$GHCR_TOKEN"
$ unset GHCR_TOKEN
$ kubectl -n demo patch serviceaccount default -p '{"imagePullSecrets":[{"name":"ghcr-pull"}]}'
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello
namespace: demo
labels:
app.kubernetes.io/name: hello
spec:
replicas: 2
selector:
matchLabels:
app.kubernetes.io/name: hello
template:
metadata:
labels:
app.kubernetes.io/name: hello
spec:
os:
name: linux
securityContext:
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
containers:
- name: hello
image: ghcr.io/example/hello:1.2
ports:
- name: http
containerPort: 8080
readinessProbe:
httpGet: {path: /readyz, port: http}
livenessProbe:
httpGet: {path: /healthz, port: http}
resources:
requests: {cpu: 100m, memory: 32Mi}
limits: {memory: 64Mi}
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
---
apiVersion: v1
kind: Service
metadata:
name: hello
namespace: demo
spec:
selector:
app.kubernetes.io/name: hello
ports:
- name: http
port: 80
targetPort: http
$ kubectl apply -f hello.yaml
deployment.apps/hello created
service/hello created
$ kubectl -n demo get pods,svc -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/hello-55fb8f8684-69mvv 1/1 Running 0 45s 172.21.211.25 k8s-lab-node-1 <none> <none>
pod/hello-55fb8f8684-nztv2 1/1 Running 0 45s 172.21.172.4 k8s-lab-node-2 <none> <none>
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/hello ClusterIP 10.97.141.163 <none> 80/TCP 45s app.kubernetes.io/name=hello
The replicas spread over the two Linux workers: the master is closed by the control-plane taint, and by default the scheduler spreads the pods of one Deployment across different nodes. The Linux base is ready — Windows is next.
Preparing Windows Server 2025 Core
The cast from the two previous parts, in brief:
- SCM (Service Control Manager) — the Windows service manager, in this role the analogue of systemd: it starts services at boot, restarts them, takes dependencies into account.
- HCS (Host Compute Service, the
vmcomputeservice) — the Windows API for creating containers. A process-isolated container is a job object (a group of processes with shared memory and CPU limits — the role of cgroups), a server silo (a job object with its own namespaces of objects, registry and files — the role of namespaces) and a file system made of image layers. - HNS (Host Networking Service, the
hnsservice) — creates virtual networks and endpoints (container network cards) on a Hyper-V virtual switch (vSwitch) and attaches policies to them; VFP (Virtual Filtering Platform) — a vSwitch extension that enforces policies on every port: NAT, ACL, load balancing, encapsulation. This is the place of netfilter. - HostProcess container — an ordinary host process inside a job object, without file system and network isolation. This is how Windows runs the network plugin and kube-proxy, for which Linux uses privileged pods.
This is what will be on the node by the end of the article:
k8s-lab-win-1.example.com — 192.168.10.25, Windows Server 2025 Core
┌───────────────────────────────────────────────────────────────┐
│ Windows services (SCM) │
│ containerd ◄──── CRI, named pipe ──── kubelet (nssm) │
│ │ │
│ ▼ containerd-shim-runhcs-v1 │
│ HCS: job object, image layers; container network via HNS │
│ │
│ HostProcess pods (host processes in the host network) │
│ calico-node-windows: node, felix ──► HNS: network, ACL │
│ kube-proxy-windows: winkernel ──► HNS: load balancers │
│ │
│ HNS + vSwitch with the VFP extension │
│ network "Calico": Overlay, 172.21.41.0/26, VNI 4096 │
│ gateway 172.21.41.1 — also the node's tunnel address│
│ Calico_ep 172.21.41.2 — the host's pod-network address│
│ pods 172.21.41.5, .6 — aspnetapp │
│ vEthernet (Ethernet) 192.168.10.25 — the host's own address │
└───────────────────────────────┬───────────────────────────────┘
│ Ethernet; VXLAN — UDP 4789
▼
node network 192.168.10.0/24
Installing containerd, kubelet and kubeadm is handled by two PowerShell scripts from the SIG Windows sig-windows-tools repository; the official guide Adding Windows worker nodes relies on them too. The scripts do little, and everything they change is covered below.
Everything in this section is done on k8s-lab-win-1 in PowerShell as an administrator. After logon Server Core opens sconfig — a text menu of basic settings; everything in it is also available as commands, and commands can be repeated on the next node.
Name and address
PS C:\> New-NetIPAddress -InterfaceAlias Ethernet -IPAddress 192.168.10.25 -PrefixLength 24 -DefaultGateway 192.168.10.1
PS C:\> Set-DnsClientServerAddress -InterfaceAlias Ethernet -ServerAddresses 192.168.10.1
PS C:\> Set-ItemProperty 'HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters' -Name 'NV Domain' -Value 'example.com'
PS C:\> Rename-Computer -NewName k8s-lab-win-1 -Restart
...
PS C:\> [System.Net.Dns]::GetHostEntry('').HostName
k8s-lab-win-1.example.com
PS C:\> (Resolve-DnsName k8s-lab-api.example.com -Type A).IPAddress
192.168.10.23
The node's name in the cluster comes from the computer name, so it must be unique and stable; a computer's NetBIOS name is limited to 15 characters, and k8s-lab-win-1 fits. NV Domain is the primary DNS suffix: the node is not in an Active Directory domain (whether a Windows node needs one is a question for part 40), and without the suffix its full name would be just k8s-lab-win-1. Addresses and DNS from PowerShell are covered in detail in the second part of the Windows Server series. A virtual machine template for cloning is prepared with sysprep /generalize, which resets the installation identifiers — the analogue of clearing /etc/machine-id.
Updates
On Windows, OS updates affect which images can run: a process-isolated container runs on the host's kernel, and the OS build in the image must match the host's build — 10.0.26100 for Windows Server 2025 and the ltsc2025 tags. The monthly update number (UBR) does not have to match, but Microsoft tests new image revisions on updated hosts, so the host is updated before installing the components (the "Install updates" item in sconfig) and then every month:
PS C:\> Get-ItemProperty 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion' |
Select-Object ProductName, InstallationType, CurrentBuild, UBR
ProductName InstallationType CurrentBuild UBR
----------- ---------------- ------------ ---
Windows Server 2025 Datacenter Server Core 26100 33438
On a server freshly installed from an ISO, updates may not install on the first try. A cumulative update requires a recent servicing stack (the component that installs updates itself), and if the stack is installed in the same pass, the cumulative update fails with error 0x800F0823; after a reboot the installation is run again, and then it goes through. The cumulative update itself takes tens of minutes to install and reboots the server twice.
Set the update mode in sconfig to Manual: a cluster node is rebooted only through drain and uncordon, as in part 35. Updating a running Windows node is a topic for part 39.
Time
Time on Windows is synchronized by the W32Time service — an NTP client and server, NTP being the protocol for syncing clocks with an accurate time source. In a workgroup it goes to time.windows.com by default; let's point it at router.example.com, like the Linux nodes:
PS C:\> w32tm /config /manualpeerlist:"router.example.com,0x8" /syncfromflags:manual /update
The command completed successfully.
PS C:\> Restart-Service w32time; w32tm /resync
The command completed successfully.
PS C:\> w32tm /query /status | Select-String 'Stratum|Source'
Stratum: 3 (secondary reference - syncd by (S)NTP)
Source: router.example.com,0x8
The 0x8 flag is client mode. A domain-joined node would get time through the domain hierarchy (part 12 of the Windows Server series). The consequences of a drifting clock are the same as on Linux: kubelet certificates "not yet valid" or "already expired".
The Containers feature
PS C:\> Install-WindowsFeature -Name Containers
Success Restart Needed Exit Code Feature Result
------- -------------- --------- --------------
True Yes SuccessRest... {Containers}
WARNING: You must restart this server to finish the installation process.
PS C:\> Restart-Computer
The Containers feature takes the place of the overlay and br_netfilter modules from part 4: it brings the HCS (vmcompute) and HNS (hns) services, the container file system filter drivers and the virtual switch; ip_forward is not needed — the vSwitch does the forwarding.
Firewall
Windows Defender Firewall, unlike ufw on Ubuntu Server, is on by default. The node needs 10250/TCP for the kubelet API (PrepareNode.ps1 will create that rule) and 4789/UDP for VXLAN. The VXLAN rule does not need to be created by hand: when Calico creates the overlay network on the node, HNS adds the inbound rule OverlayTraffic4789UDP ("Overlay Traffic 4789 UDP") itself — we will see it below. Optionally there is 10256/TCP for the kube-proxy health check used by external load balancers:
PS C:\> New-NetFirewallRule -Name kube-proxy-health -DisplayName 'kube-proxy health' -Direction Inbound `
-Protocol TCP -LocalPort 10256 -RemoteAddress 192.168.10.0/24 -Action Allow
The script will create the kubelet rule open to all addresses; it is worth narrowing it the same way: Set-NetFirewallRule -Name kubelet -RemoteAddress 192.168.10.0/24.
containerd
PS C:\> curl.exe -LO https://raw.githubusercontent.com/kubernetes-sigs/sig-windows-tools/master/hostprocess/Install-Containerd.ps1
PS C:\> .\Install-Containerd.ps1 -ContainerDVersion 2.2.2 -crictlVersion 1.37.0 -skipHypervisorSupportCheck
Windows feature 'Containers' is already installed.
WARNING: Windows optional feature 'Microsoft-Hyper-V' is not enabled. Enabling it now...
Deployment Image Servicing and Management tool
...
The operation completed successfully.
WARNING: Windows optional feature 'Microsoft-Hyper-V-Online' is not disabled. Disabling it now...
...
The operation completed successfully.
WARNING: The Hyper-V features was installed without checking the CPU for Hypervisor support. You may not be able to host Hyper-V isolated containers if CPU doesn't support hypervisors.
Please reboot and re-run this script.
PS C:\> Restart-Computer
The script lives in the master branch, which changes; it is better to download it by a link to a specific commit, read it and keep it next to the other installation files. It takes the containerd version without the letter v; the crictl version is best set to match the cluster version.
The first run does not get as far as containerd. Without the -skipHypervisorSupportCheck switch the script would install the Hyper-V roles in full, with a CPU check; with the switch it enables the Microsoft-Hyper-V feature through DISM (the Windows image servicing tool), disables Microsoft-Hyper-V-Online and asks for a reboot and a re-run. Our virtual machine does not need nested virtualization: Kubernetes does not support Hyper-V isolation, and the CPU warning can be ignored. The second run:
PS C:\> .\Install-Containerd.ps1 -ContainerDVersion 2.2.2 -crictlVersion 1.37.0 -skipHypervisorSupportCheck
Windows feature 'Containers' is already installed.
Windows optional feature 'Microsoft-Hyper-V' is already enabled.
Windows optional feature 'Microsoft-Hyper-V-Online' is already disabled.
WARNING: The Hyper-V features was installed without checking the CPU for Hypervisor support. You may not be able to host Hyper-V isolated containers if CPU doesn't support hypervisors.
Getting ContainerD binaries
Downloading https://github.com/containerd/containerd/releases/download/v2.2.2/containerd-2.2.2-windows-amd64.tar.gz to C:\Program Files\containerd\containerd.tar.gz
Configuring ContainerD CNI section: [plugins.'io.containerd.cri.v1.runtime'.cni]
Registering ContainerD as a service
Starting ContainerD service
Downloading https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.37.0/crictl-v1.37.0-windows-amd64.tar.gz to C:\Program Files\containerd\crictl.tar.gz
...
Done - please remember to add '--cri-socket "npipe:////./pipe/containerd-containerd"' to your kubeadm join command if your kubernetes version is below 1.25!
What the script did:
- checked the Containers feature and the Hyper-V features — they are in place now;
- unpacked containerd for Windows from the GitHub release into
C:\Program Files\containerdand added the directory to the systemPATH; - generated
config.tomlwithcontainerd.exe config defaultand set the directoriesc:/opt/cni/binandc:/etc/cni/net.din the CNI section — paths as on Linux, with a drive letter; this is where Calico expects them; - registered containerd as a service (
containerd.exe --register-service): containerd can answer the SCM itself and runs asLocalSystem; - installed
crictlwith the configuration$HOME\.crictl\crictl.yamlof the user who ran the script, where the runtime endpoint isnpipe://./pipe/containerd-containerd.
Run the script in a separate PowerShell window rather than redirecting its output to a file: tar.exe writes the list of unpacked files to the error stream, and with a *>&1 redirect Windows PowerShell 5.1 turns those lines into errors and stops the script.
CRI on Windows works not over a unix socket but over a named pipe \\.\pipe\containerd-containerd — a kernel object for exchanging data between processes, addressed by name. The effective configuration:
PS C:\> containerd.exe config dump | Select-String "^version|default_runtime_name|runhcs-wcow-process\]|runtime_type|bin_dirs|conf_dir|sandbox ="
version = 3
sandbox = 'registry.k8s.io/pause:3.10.1'
default_runtime_name = 'runhcs-wcow-process'
runtime_type = 'io.containerd.runhcs.v1'
cni_conf_dir = ''
[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runhcs-wcow-process]
runtime_type = 'io.containerd.runhcs.v1'
cni_conf_dir = ''
bin_dirs = ['c:/opt/cni/bin']
conf_dir = 'c:/etc/cni/net.d'
runhcs-wcow-process— the default runtime handler: WCOW is Windows Containers on Windows,processis process isolation. It starts thecontainerd-shim-runhcs-v1.exeshim from the hcsshim project — the intermediary between containerd and HCS that does runc's job on Windows. This name will be needed in the RuntimeClass. Next to it the configuration also has therunhcs-wcow-hypervisorhandler for Hyper-V isolation, which we do not use.- No
SystemdCgroup: there are no cgroups; the container's resources are limited by a job object. sandbox— the pause container image.registry.k8s.io/pauseis multi-platform; its manifest list has Windows variants, and containerd picks the one matching the host's build.
PS C:\> crictl version | Select-String Runtime
RuntimeName: containerd
RuntimeVersion: v2.2.2
RuntimeApiVersion: v1
kubelet and kubeadm as Windows services
PS C:\> curl.exe -LO https://raw.githubusercontent.com/kubernetes-sigs/sig-windows-tools/master/hostprocess/PrepareNode.ps1
PS C:\> .\PrepareNode.ps1 -KubernetesVersion v1.37.2
Using Kubernetes version: v1.37.2
Downloading https://dl.k8s.io/v1.37.2/bin/windows/amd64/kubelet.exe to C:\k\kubelet.exe
Downloading https://dl.k8s.io/v1.37.2/bin/windows/amd64/kubeadm.exe to C:\k\kubeadm.exe
...
Installing nssm
Registering kubelet service
...
Here the version is customarily written with the letter v, although the script would add it itself. There are no packages and no pkgs.k8s.io for Windows; instead of apt-get install the script:
- checks that the containerd pipe is in place;
- downloads
kubelet.exeandkubeadm.exefromdl.k8s.io/v1.37.2/bin/windows/amd64/intoC:\kand addsC:\ktoPATH; - creates
C:\var\log\kubelet,C:\etc\kubernetes\pki,C:\etc\kubernetes\manifestsandC:\var\lib\kubelet\etc\kuberneteswith apkisymbolic link toC:\etc\kubernetes\pki: paths like/var/lib/kubeletin the kubelet and kubeadm configuration mean these directories on Windows; - writes the
C:\k\StartKubelet.ps1wrapper, which readsC:\var\lib\kubelet\kubeadm-flags.envand startskubelet.exewith those flags plus permanent ones; - installs nssm and registers through it the
kubeletservice, depending oncontainerd, with output toC:\var\log\kubelet\kubelet.out.logandkubelet.err.logand rotation of these files daily or at 10 MB; - creates the
kubeletfirewall rule for 10250/TCP.
nssm (Non-Sucking Service Manager) turns any console program into a Windows service: it answers the SCM on the program's behalf, restarts it when it crashes and writes its output to files. The nssm service and StartKubelet.ps1 play the role of the kubelet.service unit with the 10-kubeadm.conf drop-in from part 4: the wrapper likewise mixes in the flags from kubeadm-flags.env. Of the permanent flags, the interesting ones are --cgroups-per-qos=false and --enforce-node-allocatable="" — there is no cgroup hierarchy and the kubelet does not build one (which is why systemReserved on Windows only reduces the amount available to the scheduler), --resolv-conf="" — the kubelet passes DNS settings for pods straight into the HNS endpoint, and --hostname-override with the computer name.
The wrapper is not the only way: with the --windows-service flag kubelet.exe answers the SCM itself and can be registered with New-Service. But then the flags from kubeadm-flags.env would have to be copied into the service parameters by hand after every join and upgrade, whereas the wrapper re-reads them on every start. We follow the official guide — nssm. apt-mark hold is not needed: nobody but us will update the binaries in C:\k.
Unlike Linux, where the kubelet restarts in a loop before join, here the service is registered but not started — kubeadm join will start it:
PS C:\> Get-Service kubelet | Format-Table Status, Name, StartType
Status Name StartType
------ ---- ---------
Stopped kubelet Automatic
kubeadm join
For a Windows node the command from the init output is not enough: the node must appear in the cluster with the taint already on, otherwise between registration and kubectl taint the scheduler will manage to send Linux pods to it that will not start (a Linux image has no Windows variant). So — a configuration file, like kubeadm-config.yaml in part 4. We create a short-lived token on the master:
user@k8s-lab-master-1:~$ sudo kubeadm token create --ttl 1h --description 'join k8s-lab-win-1'
6w1r8t.p3k9m2x7c5v0b4n1
On the node — C:\k\join.yaml:
apiVersion: kubeadm.k8s.io/v1beta4
kind: JoinConfiguration
discovery:
bootstrapToken:
apiServerEndpoint: k8s-lab-api.example.com:6443
token: 6w1r8t.p3k9m2x7c5v0b4n1
caCertHashes:
- sha256:7b2e9d4a1c6f8b3e0d5a2c7f9e4b1d6a8c3f0e5b2d7a9c4e1f6b8d3a0c5e2f7b
nodeRegistration:
name: k8s-lab-win-1
criSocket: npipe:////./pipe/containerd-containerd
taints:
- key: os
value: windows
effect: NoSchedule
patches:
directory: C:\k\patches
and the patch C:\k\patches\kubeletconfiguration.yaml:
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
maxPods: 60
discovery.bootstrapToken— the same as--tokenand--discovery-token-ca-cert-hashin part 4: a token for TLS bootstrapping and the hash of the CA public key by which the node verifies the cluster's authenticity. The hash is the same as in the test cluster'sinitoutput.criSocket— the CRI address: thenpipe://scheme and the pipe path. kubeadm on Windows substitutes it itself, but an explicit value reads better.taints— the kubelet registers the Node with theos=windows:NoScheduletaint right away. The key and value are a convention from the Kubernetes documentation, not a built-in name.patches— a directory with patches to the configuration from the cluster; the file name sets the target,kubeletconfigurationis the sharedKubeletConfigurationfrom thekubelet-configConfigMap. The patch applies only to this node.
Why maxPods: 60: Calico on Windows holds one address block per node and reserves four addresses in it (the network address, the gateway, Calico_ep and the last address), as covered in the previous part. Of the 64 addresses of a /26 block, 60 remain for pods, while the kubelet by default expects 110 pods — the sixty-first would get a node but no address. kubeadm does not store patches in the cluster, so when the node is upgraded the directory is passed again: kubeadm upgrade node --patches C:\k\patches.
PS C:\k> kubeadm join --config C:\k\join.yaml
[preflight] Running pre-flight checks
[preflight] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[preflight] Use 'kubeadm init phase upload-config kubeadm --config your-config-file' to re-upload it.
W1106 23:16:41.134109 5016 utils.go:69] The recommended value for "authentication.x509.clientCAFile" in "KubeletConfiguration" is: \etc\kubernetes\pki\ca.crt; the provided value is: /etc/kubernetes/pki/ca.crt
[kubelet-start] Writing kubelet configuration to file "\\var\\lib\\kubelet\\instance-config.yaml"
[patches] Reading patches from path "C:\\k\\patches"
[patches] Found the following patch files: [kubeletconfiguration.yaml]
[patches] Applied patch of type "application/strategic-merge-patch+json" to target "kubeletconfiguration"
[patches] Applied patch of type "application/strategic-merge-patch+json" to target "kubeletconfiguration"
[kubelet-start] Writing kubelet configuration to file "\\var\\lib\\kubelet\\config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "\\var\\lib\\kubelet\\kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[patches] Applied patch of type "application/strategic-merge-patch+json" to target "kubeletconfiguration"
[kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s
[kubelet-check] The kubelet is healthy after 3.0008224s
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
PS C:\k> Remove-Item C:\k\join.yaml
The mechanism is the same as in part 4: an anonymous read of cluster-info with a check of the JWS signature and the CA hash, a temporary bootstrap-kubelet.conf with the token, a CSR for the system:node:k8s-lab-win-1 client certificate, a permanent C:\etc\kubernetes\kubelet.conf. Only kubeadm restarts the kubelet through the SCM rather than systemd. The clientCAFile warning is harmless: kubeadm compares path strings, and /etc/kubernetes/pki/ca.crt on Windows means C:\etc\kubernetes\pki\ca.crt anyway. The patch was applied three times — to the shared configuration, to the node's config.yaml and to its instance-config.yaml, where kubeadm has recently started putting the parameters of a particular node. We delete join.yaml with the token right away.
The kubelet serving certificate (serverTLSBootstrap: true from the shared configuration) waits for approval, as with the Linux nodes. We check the names and approve:
$ kubectl get csr --field-selector spec.signerName=kubernetes.io/kubelet-serving | grep Pending
csr-ljq25 9s kubernetes.io/kubelet-serving system:node:k8s-lab-win-1 <none> Pending
$ kubectl get csr csr-ljq25 -o jsonpath='{.spec.request}' | base64 -d | openssl req -noout -text | grep -A1 'Subject Alternative'
X509v3 Subject Alternative Name:
DNS:k8s-lab-win-1, IP Address:192.168.10.25
$ kubectl certificate approve csr-ljq25
certificatesigningrequest.certificates.k8s.io/csr-ljq25 approved
The node is in the cluster but, like the master before Calico was installed, NotReady:
$ kubectl get node k8s-lab-win-1 -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k8s-lab-win-1 NotReady <none> 10s v1.37.2 192.168.10.25 <none> Windows Server 2025 Datacenter 10.0.26100.33438 (amd64) containerd://2.2.2
$ kubectl describe node k8s-lab-win-1 | grep -E 'NetworkReady'
Ready False Fri, 06 Nov 2026 23:16:46 +0300 Fri, 06 Nov 2026 23:16:46 +0300 KubeletNotReady [container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized, CSINode is not yet initialized]
$ kubectl get node k8s-lab-win-1 -o jsonpath='{.spec.taints}{"\n"}{.status.allocatable.pods}{"\n"}'
[{"effect":"NoSchedule","key":"os","value":"windows"},{"effect":"NoSchedule","key":"node.kubernetes.io/not-ready"},{"effect":"NoExecute","key":"node.kubernetes.io/not-ready","timeAdded":"2026-11-06T20:16:50Z"}]
60
The os=windows:NoSchedule taint is there from the first second, next to the internal not-ready taints, which will go away by themselves. The node's capacity is 60 pods — the patch worked.
Calico for Windows
What is already in place in the cluster
In the main cluster we would have to change the Installation and create the ConfigMap at this point. We built the test cluster for Windows from the start, so we only check:
$ kubectl get installation default -o jsonpath='{.spec.calicoNetwork.bgp} {.spec.calicoNetwork.ipPools[0].encapsulation} {.spec.calicoNetwork.windowsDataplane} {.spec.serviceCIDRs}{"\n"}'
Disabled VXLAN HNS ["10.96.0.0/12"]
$ kubectl get ipamconfigurations default -o jsonpath='{.spec.strictAffinity}{"\n"}'
true
$ kubectl -n tigera-operator get configmap kubernetes-services-endpoint -o jsonpath='{.data}{"\n"}'
{"KUBERNETES_SERVICE_HOST":"k8s-lab-api.example.com","KUBERNETES_SERVICE_PORT":"6443"}
All of Calico for Windows' conditions are met: VXLAN without CrossSubnet, BGP off, the HNS dataplane on, the service network known, strictAffinity and a direct API address. Nothing else needs doing: as soon as a node with the kubernetes.io/os=windows label appeared in the cluster, the calico-node-windows DaemonSet got a desired pod.
About the OS version. The Calico for Windows requirements page lists Windows Server 1809 and 2022 at the time of writing, and Windows Server 2025 is not on it; the same page says the supported versions depend on the Kubernetes version, with a link to the Kubernetes table where 2025 is listed. For us the combination of Calico 3.33.0 and Windows Server 2025 (26100.33438) works, but formally Calico does not explicitly confirm it. For a production cluster this is a reason to check the release notes of your Calico version, or to keep Windows nodes on Windows Server 2022 until 2025 appears on the list.
What appears on the node
$ kubectl -n calico-system get ds calico-node-windows
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
calico-node-windows 1 1 0 1 0 kubernetes.io/os=windows 83m
$ kubectl -n calico-system get pod -l k8s-app=calico-node-windows -o json \
| jq -c '.items[0].spec | {init: [.initContainers[].name], containers: [.containers[] | {name, windowsOptions: .securityContext.windowsOptions}]}'
{"init":["uninstall-calico","install-cni"],"containers":[{"name":"felix","windowsOptions":{"hostProcess":true,"runAsUserName":"NT AUTHORITY\\system"}},{"name":"node","windowsOptions":{"hostProcess":true,"runAsUserName":"NT AUTHORITY\\system"}}]}
Let's take the pod apart:
hostProcess: trueandrunAsUserName: NT AUTHORITY\systemare set on each container rather than on the pod — that works too: the containers run as host processes as SYSTEM, the most privileged local account, in the host's network (hostNetwork: true, the pod has the node's address). HCS still creates a job object for them (this is how the kubelet accounts for resources and stops the pod as a whole), and mounts the image's file system into a directory whose path is in theCONTAINER_SANDBOX_MOUNT_POINTvariable. The process sees the host's disks, registry, services and network, which is why such pods are forbidden by thebaselineandrestrictedlevels of Pod Security Admission (part 30), and the operator labels thecalico-systemnamespace with theprivilegedlevel.uninstall-calicoremoves Calico installed the old way (with scripts and Windows services);install-cniputs the plugins intoC:\opt\cni\bin, and the CNI configuration and the plugin's kubeconfig intoC:\etc\cni\net.d.noderegisters the node in Calico's data, gets an address block and creates the HNS network.felixconnects to Typha on the Linux nodes, turns NetworkPolicy (part 20) into ACL policies on HNS endpoints, which VFP enforces, and writes routes to other nodes' pod blocks via VXLAN into the HNS network. There are no policies for the nodes themselves (host endpoints) on Windows.
In BGP mode there would also be a confd container and the RemoteAccess role on the node. The pod is ready in three to four minutes:
$ kubectl -n calico-system describe pod -l k8s-app=calico-node-windows | grep -E 'Pulled|Unhealthy'
Normal Pulled 116s kubelet spec.initContainers{uninstall-calico}: Successfully pulled image "quay.io/calico/node-windows:v3.33.0" in 1m0.951s (1m0.951s including waiting). Image size: 146560272 bytes.
Normal Pulled 92s kubelet spec.initContainers{install-cni}: Successfully pulled image "quay.io/calico/cni-windows:v3.33.0" in 17.528s (17.528s including waiting). Image size: 149137842 bytes.
Warning Unhealthy 20s (x5 over 58s) kubelet spec.containers{felix}: Liveness probe failed: calico/node is not ready: Felix is not live: Get "http://localhost:9099/liveness": dial tcp [::1]:9099: connectex: No connection could be made because the target machine actively refused it.
$ kubectl -n calico-system get pods -l k8s-app=calico-node-windows
NAME READY STATUS RESTARTS AGE
calico-node-windows-q54h2 2/2 Running 0 4m6s
$ kubectl get node k8s-lab-win-1
NAME STATUS ROLES AGE VERSION
k8s-lab-win-1 Ready <none> 4m6s v1.37.2
The calico/node-windows and calico/cni-windows images weigh a little over 140 MB each. For the first minute after start Felix does not answer its probes — that is normal while it builds the network, and the pod is not restarted: the kubelet waits until the liveness probe fails the required number of times in a row.
HNS networks and a new network adapter have appeared on the node:
PS C:\> Get-HnsNetwork | Select-Object Name, Type,
@{n='Subnet';e={$_.Subnets.AddressPrefix}}, @{n='Gateway';e={$_.Subnets.GatewayAddress}}, ManagementIP |
Format-Table
Name Type Subnet Gateway ManagementIP
---- ---- ------ ------- ------------
External Overlay 192.168.255.0/30 192.168.255.1 192.168.10.25
Calico Overlay 172.21.41.0/26 172.21.41.1 192.168.10.25
PS C:\> Get-NetIPAddress -AddressFamily IPv4 | Where-Object PrefixOrigin -ne WellKnown |
Format-Table InterfaceAlias, IPAddress, PrefixLength
InterfaceAlias IPAddress PrefixLength
-------------- --------- ------------
vEthernet (Ethernet) 192.168.10.25 24
PS C:\> Get-HnsEndpoint | Where-Object Name -eq Calico_ep | Format-List Name, IPAddress, VirtualNetworkName
Name : Calico_ep
IPAddress : 172.21.41.2
VirtualNetworkName : Calico
PS C:\> Get-NetFirewallRule -Name OverlayTraffic4789UDP | Format-Table Name, DisplayName, Enabled, Direction, Action
Name DisplayName Enabled Direction Action
---- ----------- ------- --------- ------
OverlayTraffic4789UDP Overlay Traffic 4789 UDP True Inbound Allow
- The
Externalnetwork — an auxiliary Overlay network with a tiny subnet192.168.255.0/30: Calico creates it first, and together with it HNS brings up the external vSwitch on top of theEthernetadapter. - The
Caliconetwork of typeOverlay— with the block subnet172.21.41.0/26that Calico allocated to the node. The network's gateway172.21.41.1is the vSwitch's distributed router port; Calico also records it as the node's VXLAN tunnel address. In Overlay mode VFP wraps traffic to other nodes' pods into VXLAN. vEthernet (Ethernet)— when creating the first network, HNS moved the host's address to a virtual adapter in the switch. At the moment of the move the host's network disappears for a few seconds, and the kubelet sometimes does not restore its connection to the API by itself (see the troubleshooting section).Calico_ep— the endpoint of the node itself in the pod network: through it the host reaches the pods, and kube-proxy uses its address as the source VIP.OverlayTraffic4789UDP— the very firewall rule for VXLAN that HNS added together with the overlay network.
kube-proxy for Windows
The kube-proxy DaemonSet from kubeadm will not land on the Windows node: it has nodeSelector: kubernetes.io/os: linux and a Linux-only image. Windows needs a separate DaemonSet in HostProcess pods; the manifest for Calico is in sig-windows-tools, and we replace the KUBE_PROXY_VERSION placeholder with the cluster version:
$ curl -sL https://raw.githubusercontent.com/kubernetes-sigs/sig-windows-tools/master/hostprocess/calico/kube-proxy/kube-proxy.yml \
| sed 's/KUBE_PROXY_VERSION/v1.37.2/g' > kube-proxy-windows.yaml
$ grep -E 'image:|hostProcess:|kubernetes.io/os' kube-proxy-windows.yaml
hostProcess: true
- image: sigwindowstools/kube-proxy:v1.37.2-calico-hostprocess
kubernetes.io/os: windows
$ kubectl apply -f kube-proxy-windows.yaml
daemonset.apps/kube-proxy-windows created
The manifest creates only the kube-proxy-windows DaemonSet in kube-system; it borrows the kube-proxy service account with its RBAC rights and the kube-proxy ConfigMap with the kubeconfig from its Linux sibling. config.conf from the same ConfigMap — our KubeProxyConfiguration with mode: nftables — is not used on Windows: the parameters are set with flags. The operator: Exists toleration lets the pod onto a node with any taint.
The image is built by SIG Windows: inside are the official kube-proxy.exe and a start.ps1 script that waits for the Calico HNS network to appear and passes its name in the KUBE_NETWORK variable; for an Overlay network it waits for the Calico_ep endpoint and passes its address with the --source-vip flag; it deletes old load balancing policies in HNS (kube-proxy cannot reconcile with them) and starts kube-proxy with --proxy-mode=kernelspace and --enable-dsr=true. What the kernelspace mode, source VIP and DSR are was covered in the previous part.
$ kubectl -n kube-system get pods -o wide --field-selector spec.nodeName=k8s-lab-win-1
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-proxy-windows-z9qdq 1/1 Running 0 30s 192.168.10.25 k8s-lab-win-1 <none> <none>
$ kubectl -n kube-system logs -l k8s-app=kube-proxy-windows | head -6
WARNING: The names of some imported commands from the module 'hns' include unapproved verbs that might make them less
discoverable. To find the commands with unapproved verbs, run the Import-Module command again with the Verbose
parameter. For a list of approved verbs, type Get-Verb.
Running kub-proxy service.
Waiting for HNS network Calico to be created...
HNS network Calico found.
The warning about "unapproved verbs" comes from the auxiliary hns module with old command names, and the kub-proxy typo is in the script itself; after that you can see it wait for the Calico network and start.
Taint, RuntimeClass and Linux DaemonSets
Kubernetes itself puts the kubernetes.io/os and kubernetes.io/arch labels on nodes, and node.kubernetes.io/windows-build on a Windows node too:
$ kubectl get node k8s-lab-win-1 -L kubernetes.io/os,node.kubernetes.io/windows-build
NAME STATUS ROLES AGE VERSION OS WINDOWS-BUILD
k8s-lab-win-1 Ready <none> 6m6s v1.37.2 windows 10.0.26100
Pods are spread across nodes by three mechanisms from part 13, each with its own job.
The os=windows:NoSchedule taint protects the Windows node from everything that does not know about Windows: a Deployment without a nodeSelector, or a chart written for Linux, will not land on it — their pods have no toleration.
RuntimeClass saves Windows pods from repeating the nodeSelector and toleration in every manifest. It is a cluster-scoped object that ties a name to a containerd handler and placement conditions:
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
name: windows-2025
handler: runhcs-wcow-process
scheduling:
nodeSelector:
kubernetes.io/os: windows
kubernetes.io/arch: amd64
node.kubernetes.io/windows-build: "10.0.26100"
tolerations:
- key: os
operator: Equal
value: windows
effect: NoSchedule
$ kubectl apply -f runtimeclass-windows-2025.yaml
runtimeclass.node.k8s.io/windows-2025 created
The RuntimeClass admission plugin writes the selector and toleration from scheduling into a pod with runtimeClassName: windows-2025, and the kubelet passes the handler name to containerd. The build in the selector keeps the class name honest: an ltsc2025 image will not start on Windows Server 2022, and nodes of each version get their own RuntimeClass. RuntimeClass does not set the spec.os.name: windows field; it is written explicitly: by it the API server rejects the Linux fields of securityContext, and Pod Security Admission relaxes the checks that do not apply to Windows.
nodeSelector: kubernetes.io/os: linux on Linux DaemonSets is the third part, which is easy to forget. System DaemonSets often tolerate everything (operator: Exists) to run on any node; the taint will not stop them, and only the selector keeps them on Linux. Let's check what the test cluster has:
$ kubectl get ds -A -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,NODE-SELECTOR:.spec.template.spec.nodeSelector,TOLERATIONS:.spec.template.spec.tolerations[*].operator'
NAMESPACE NAME NODE-SELECTOR TOLERATIONS
calico-system calico-node map[kubernetes.io/os:linux] Exists,Exists,Exists
calico-system calico-node-windows map[kubernetes.io/os:windows] Exists,Exists,Exists
calico-system csi-node-driver map[kubernetes.io/os:linux] Exists,Exists,Exists
kube-system kube-proxy map[kubernetes.io/os:linux] Exists
kube-system kube-proxy-windows map[kubernetes.io/os:windows] Exists,Exists
The selector on kube-proxy is written in the kubeadm manifest; on calico-node and csi-node-driver it is set by the Tigera operator — each component gets the OS it supports. The test cluster is young and has no other DaemonSets yet; they will appear in part 39 together with monitoring and storage. A good example of why the rule matters is node-exporter from kube-prometheus-stack (part 32). Its chart by default tolerates any taint with the NoSchedule effect — to collect metrics from masters too — and therefore os=windows:NoSchedule as well:
$ helm show values oci://ghcr.io/prometheus-community/charts/prometheus-node-exporter | grep -A3 -E '^(nodeSelector|tolerations):'
nodeSelector:
kubernetes.io/os: linux
# kubernetes.io/arch: amd64
tolerations:
- effect: NoSchedule
operator: Exists
On Linux it is held only by nodeSelector: kubernetes.io/os: linux, which the chart fortunately sets itself. Whoever overrides nodeSelector in their values, for example to add an architecture label, and forgets the OS, will get a node-exporter pod with a Linux image on the Windows node — an eternal ErrImagePull.
The rule: every DaemonSet with a Linux image must have a kubernetes.io/os: linux selector, even if the taint saves it for now; for every new chart this is checked in the default values (helm show values) and after installation with the same kubectl get ds -A. Log and metric agents for the Windows nodes themselves are in part 39.
Checking: a Windows pod, Service, DNS and connectivity
Let's repeat the check from part 4 across OSes: a pod on a Linux node reaches a Windows pod and vice versa — by pod addresses, through a Service and by DNS names. For the Windows application we take a ready one — a sample .NET web application that Microsoft publishes in a Nano Server-based image, mcr.microsoft.com/dotnet/samples:aspnetapp-nanoserver-ltsc2025. The tag floats and points to the newest .NET version — at the time of writing it is .NET 11 RC; production applications get images built by their owners, which is what part 39 is about. The image weighs about 260 MB — versus 2.4 GB for IIS on Server Core — but we pull it onto the node in advance anyway:
PS C:\> crictl pull mcr.microsoft.com/dotnet/samples:aspnetapp-nanoserver-ltsc2025
Image is up to date for sha256:35169c38821e233b05473c07337b96034a71bbab3cda564a2817d5e9bdb0fd00
PS C:\> crictl images | Select-String samples
mcr.microsoft.com/dotnet/samples aspnetapp-nanoserver-ltsc2025 35169c38821e2 258MB
Nano Server images run as the unprivileged ContainerUser account, so the win-demo namespace can get the same restricted Pod Security Admission level as demo — as covered in part 36, for a Windows pod it requires only runAsNonRoot:
apiVersion: v1
kind: Namespace
metadata:
name: win-demo
labels:
pod-security.kubernetes.io/enforce: restricted
pod-security.kubernetes.io/enforce-version: v1.37
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: aspnetapp
namespace: win-demo
labels:
app.kubernetes.io/name: aspnetapp
spec:
replicas: 2
selector:
matchLabels:
app.kubernetes.io/name: aspnetapp
template:
metadata:
labels:
app.kubernetes.io/name: aspnetapp
spec:
os:
name: windows
runtimeClassName: windows-2025
securityContext:
runAsNonRoot: true
windowsOptions:
runAsUserName: ContainerUser
containers:
- name: aspnetapp
image: mcr.microsoft.com/dotnet/samples:aspnetapp-nanoserver-ltsc2025
ports:
- name: http
containerPort: 8080
readinessProbe:
httpGet:
path: /
port: http
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: "1"
memory: 512Mi
---
apiVersion: v1
kind: Service
metadata:
name: aspnetapp
namespace: win-demo
spec:
selector:
app.kubernetes.io/name: aspnetapp
ports:
- name: http
port: 80
targetPort: http
$ kubectl apply -f aspnetapp.yaml
namespace/win-demo created
deployment.apps/aspnetapp created
service/aspnetapp created
$ kubectl -n win-demo get pods,svc -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/aspnetapp-769559d8ff-dkp9z 1/1 Running 0 56s 172.21.41.6 k8s-lab-win-1 <none> <none>
pod/aspnetapp-769559d8ff-vk445 1/1 Running 0 57s 172.21.41.5 k8s-lab-win-1 <none> <none>
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/aspnetapp ClusterIP 10.102.98.171 <none> 80/TCP 57s app.kubernetes.io/name=aspnetapp
$ kubectl -n win-demo get pod aspnetapp-769559d8ff-dkp9z -o json | jq '{nodeSelector: .spec.nodeSelector, tolerations: [.spec.tolerations[] | select(.key == "os")]}'
{
"nodeSelector": {
"kubernetes.io/arch": "amd64",
"kubernetes.io/os": "windows",
"node.kubernetes.io/windows-build": "10.0.26100"
},
"tolerations": [
{
"effect": "NoSchedule",
"key": "os",
"operator": "Equal",
"value": "windows"
}
]
}
$ kubectl -n win-demo exec aspnetapp-769559d8ff-dkp9z -- cmd /c "echo %USERNAME% & ver"
ContainerUser
Microsoft Windows [Version 10.0.26100.33674]
The pod passed restricted without a single warning. The manifest had neither a nodeSelector nor the os=windows toleration — the RuntimeClass admission plugin added them; the pod's other tolerations (not-ready, unreachable) are added by Kubernetes to every pod, so we pick out the one we need with a filter. Inside is ContainerUser, and the Windows revision in the image (26100.33674) is newer than on the host (26100.33438): only the build has to match. Nano Server has no whoami and no PowerShell, hence cmd /c. The service addresses of the 172.21.41.0/26 block do not go to pods: .0 is the network address, .1 the HNS network's gateway, .2 Calico_ep, .63 the last address of the block; the pods got .5 and .6, while .3 and .4 had already been held by other pods — addresses are not reused right away.
From Linux to Windows
We create the check pod in the default namespace: it has no Pod Security restrictions, and ping in busybox needs root privileges that restricted would not give. Without a RuntimeClass the taint keeps the pod off the Windows node — it ends up on one of the Linux workers, in our case k8s-lab-node-2:
$ kubectl run -i --rm check --image=busybox:1.37 --restart=Never -- sh -c '
nslookup aspnetapp.win-demo.svc.cluster.local | tail -3
wget -qO- http://aspnetapp.win-demo.svc.cluster.local | grep -o "<title>[^<]*</title>"
ping -c 2 172.21.41.6'
Name: aspnetapp.win-demo.svc.cluster.local
Address: 10.102.98.171
<title>Welcome to .NET - AspNetCoreRazorApp</title>
PING 172.21.41.6 (172.21.41.6): 56 data bytes
64 bytes from 172.21.41.6: seq=0 ttl=126 time=0.405 ms
64 bytes from 172.21.41.6: seq=1 ttl=126 time=0.489 ms
...
pod "check" deleted from default namespace
k8s-lab-node-2 (Linux) k8s-lab-win-1 (Windows)
┌────────────────────────────┐ ┌────────────────────────────┐
│ pod check on this node │ │ pod aspnetapp 172.21.41.6 │
│ │ aspnetapp.win-demo... │ │ ▲ │
│ │ CoreDNS → 10.102.98.171│ │ │ VFP: ACL from Felix │
│ ▼ │ │ │ │
│ nftables kube-proxy: DNAT │ │ vSwitch, network "Calico": │
│ 10.102.98.171:80 → │ │ strip the VXLAN header │
│ 172.21.41.6:8080 │ │ ▲ │
│ ▼ │ UDP 4789 │ │ │
│ vxlan.calico, VNI 4096 ────┼──────────────►┼───┘ │
└────────────────────────────┘ .26 → .25 └────────────────────────────┘
CoreDNS resolved the service name (part 16), and kube-proxy on the Linux node turned the ClusterIP into the Windows pod's address — with nftables rules; it does not care which OS the pod runs on. The packet to the 172.21.41.0/26 block went into vxlan.calico — Felix wrote the route to it as soon as the Windows node got its block — and on the Windows node VFP stripped the VXLAN header and delivered the packet to the pod's port. TTL 126: Windows sets 128, and on the way back the reply passes two routing hops — the HNS network's gateway on the Windows node and the Linux node's kernel. A ping of the same pod from the Linux node itself would show ttl=127.
From Windows to Linux
Now from the Windows pod to hello in the demo namespace, which we deployed on the Linux workers. Nano Server has no nslookup, but it has curl.exe and ping; curl.exe itself shows which address the client got from DNS:
$ kubectl -n win-demo exec deploy/aspnetapp -- curl.exe -s -o NUL -w "%{remote_ip} %{http_code}\n" http://hello.demo.svc.cluster.local/healthz
10.97.141.163 200
$ kubectl -n win-demo exec deploy/aspnetapp -- ping -n 2 172.21.172.4
Pinging 172.21.172.4 with 32 bytes of data:
Reply from 172.21.172.4: bytes=32 time=13ms TTL=63
Reply from 172.21.172.4: bytes=32 time<1ms TTL=63
...
Here the hello service's ClusterIP was turned into the Linux pod's address by kube-proxy for Windows — with an HNS load balancing policy on the pod's port. The peculiarities of Windows pods discussed in the previous part, in action:
$ kubectl -n win-demo exec deploy/aspnetapp -- curl.exe -s -o NUL -w "%{remote_ip} %{http_code}\n" --max-time 5 http://hello.demo/healthz
000
command terminated with exit code 6
$ kubectl -n win-demo exec deploy/aspnetapp -- curl.exe -s -o NUL -w "%{remote_ip} %{http_code}\n" http://aspnetapp/
10.102.98.171 200
$ kubectl -n win-demo exec deploy/aspnetapp -- ping -n 2 1.1.1.1 | tail -3
Packets: Sent = 2, Received = 2, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 9ms, Maximum = 9ms, Average = 9ms
- A name with a dot is not completed with search suffixes. Exit code 6 from
curl.exemeans "couldn't resolve host": the Windows DNS client does not completehello.demo, whileaspnetappwithout a dot gets completed. Services in other namespaces are reached from Windows pods by their full name. - Outbound
pingworks for us. The Kubernetes documentation lists outbound ICMP through NAT among the limitations, but on Windows Server 2025 with Calico in VXLAN mode the reply came back. Checking outbound connectivity over TCP — withcurl.exe— is still more reliable.
And a limitation of the node itself: the Windows host does not reach service addresses — curl.exe http://10.102.98.171 in PowerShell on k8s-lab-win-1 will not get an answer, while one to a pod address will.
Hands-on
Now that the cluster is built, let's look live at what parts 36 and 37 took apart in diagrams.
The whole cluster
$ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k8s-lab-master-1 Ready control-plane 153m v1.37.2 192.168.10.23 <none> Ubuntu 26.04.1 LTS 7.0.0-42-generic (amd64) containerd://2.2.2
k8s-lab-node-1 Ready <none> 146m v1.37.2 192.168.10.24 <none> Ubuntu 26.04.1 LTS 7.0.0-42-generic (amd64) containerd://2.2.2
k8s-lab-node-2 Ready <none> 146m v1.37.2 192.168.10.26 <none> Ubuntu 26.04.1 LTS 7.0.0-42-generic (amd64) containerd://2.2.2
k8s-lab-win-1 Ready <none> 64m v1.37.2 192.168.10.25 <none> Windows Server 2025 Datacenter 10.0.26100.33438 (amd64) containerd://2.2.2
$ kubectl top nodes
NAME CPU(cores) CPU(%) MEMORY(bytes) MEMORY(%)
k8s-lab-master-1 121m 6% 1588Mi 48%
k8s-lab-node-1 34m 1% 1960Mi 26%
k8s-lab-node-2 55m 2% 1514Mi 20%
k8s-lab-win-1 8m 0% 1292Mi 16%
The OS-IMAGE and KERNEL-VERSION columns of the Windows node are the edition name and the full OS version with the revision. The node name is the short k8s-lab-win-1, like the Linux nodes': the kubelet registers the node by the computer name. kubectl top sees the Windows node too: the kubelet on Windows serves /metrics/resource just as on Linux, and metrics-server verifies its serving certificate issued from the approved CSR.
Services and pods on the node
PS C:\> Get-Service kubelet, containerd, vmcompute, hns | Format-Table -AutoSize Status, Name, StartType
Status Name StartType
------ ---- ---------
Running containerd Automatic
Running hns Manual
Running kubelet Automatic
Running vmcompute Manual
PS C:\> crictl pods
POD ID CREATED STATE NAME NAMESPACE ATTEMPT RUNTIME
df1924ad959c6 4 minutes ago Ready aspnetapp-769559d8ff-dkp9z win-demo 0 runhcs-wcow-process
690ccfd48f5c4 4 minutes ago Ready aspnetapp-769559d8ff-vk445 win-demo 0 runhcs-wcow-process
7fdf200e6eea1 59 minutes ago Ready kube-proxy-windows-z9qdq kube-system 0 (default)
b1da3ea54ae81 About an hour ago Ready calico-node-windows-q54h2 calico-system 0 (default)
vmcompute (HCS) and hns start on demand, when containerd or the CNI plugin calls them. In the RUNTIME column the aspnetapp pods have the runhcs-wcow-process handler that the kubelet took from the RuntimeClass, while (default) on the HostProcess pods is the default handler, the same runhcs-wcow-process in our configuration.
The container's processes on the host
With process isolation, a container's processes are host processes. Each container has its own session, and the session number (SI) makes them easy to tell apart. Let's find the application's session and see what else runs in it:
PS C:\> Get-Process -Name AspNetCoreRazorApp | Format-Table -AutoSize Id, SI, ProcessName
Id SI ProcessName
-- -- -----------
144 5 AspNetCoreRazorApp
8228 4 AspNetCoreRazorApp
PS C:\> Get-Process | Where-Object SessionId -eq 5 | Sort-Object ProcessName | Format-Table -AutoSize Id, SI, ProcessName
Id SI ProcessName
-- -- -----------
144 5 AspNetCoreRazorApp
8872 5 CExecSvc
8736 5 csrss
912 5 lsass
6168 5 services
192 5 svchost
...
3528 5 wininit
Two application processes — two pods, each in its own session. The container's session has its own csrss, wininit, services, lsass and a dozen svchost, the CExecSvc execution service and the application itself — exactly the picture from the diagram in part 36. The session manager smss is not in this selection, although the container has its own and it lives the whole time: it runs in session 0, and we selected processes by the container's session. HostProcess containers, on the other hand, are ordinary host processes in session 0, next to the node's own system services, and they run from the directory where HCS mounted the image contents:
PS C:\> Get-Process calico, kube-proxy | Format-Table -AutoSize Id, SI, ProcessName, Path
Id SI ProcessName Path
-- -- ----------- ----
852 0 calico C:\hpc\CalicoWindows\calico.exe
3856 0 calico C:\hpc\CalicoWindows\calico.exe
4308 0 kube-proxy C:\hpc\kube-proxy\kube-proxy.exe
Two calico processes — the two containers of the calico-node-windows pod, felix and node: Calico for Windows is built as one executable with subcommands. C:\hpc is the value of CONTAINER_SANDBOX_MOUNT_POINT for these pods.
The pod network in HNS
The node's network adapters:
PS C:\> Get-NetAdapter | Format-Table Name, InterfaceDescription, MacAddress
Name InterfaceDescription MacAddress
---- -------------------- ----------
Ethernet Microsoft Hyper-V Network Adapter 00-15-5D-0A-00-05
vEthernet (Ethernet) Hyper-V Virtual Ethernet Adapter 00-15-5D-0A-00-05
vEthernet (DrPort 1803C339-EAB3-4933-8F1F-62E7D36C6E21) Hyper-V Virtual Ethernet Container Adapter 00-15-5D-EB-90-AE
vEthernet (df1924ad959c693c6dc07f1f797f63458fa53dcc81f8126755c3a4509e87a15e_Calico) Hyper-V Virtual Ethernet Container Adapter #3 0E-2A-AC-15-29-06
...
vEthernet (DrPort …) is the distributed router port in the vSwitch: it is the 172.21.41.1 gateway of the Calico network. The pods' adapters are named after the pod sandbox ID, and the last four bytes of their MAC address are the pod's address in hexadecimal: AC-15-29-06 is 172.21.41.6.
The Calico HNS network and its routes to other nodes' blocks:
PS C:\> $net = Get-HnsNetwork | Where-Object Name -eq Calico
PS C:\> $net.Subnets.Policies | ConvertTo-Json -Compress
{"Type":"VSID","VSID":4096}
PS C:\> $net.Policies | Where-Object Type -eq RemoteSubnetRoute |
Format-Table DestinationPrefix, ProviderAddress, DistributedRouterMacAddress, IsolationId
DestinationPrefix ProviderAddress DistributedRouterMacAddress IsolationId
----------------- --------------- --------------------------- -----------
172.21.211.0/26 192.168.10.24 66-ee-f4-61-98-50 4096
172.21.70.192/26 192.168.10.23 66-79-81-e0-1a-7d 4096
172.21.172.0/26 192.168.10.26 66-05-38-46-ab-ea 4096
These are the RemoteSubnetRoute policies from the previous part: a neighbour's block, its node's address and the MAC of its VTEP — the vxlan.calico interface on the Linux node. Let's compare with the Linux side:
user@k8s-lab-node-1:~$ ip -d link show vxlan.calico | head -2
5: vxlan.calico: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/ether 66:ee:f4:61:98:50 brd ff:ff:ff:ff:ff:ff promiscuity 0 allmulti 0 minmtu 68 maxmtu 65535
user@k8s-lab-node-1:~$ ip route | grep 172.21.41
172.21.41.0/26 via 172.21.41.1 dev vxlan.calico onlink mtu 1450
user@k8s-lab-node-1:~$ ip neigh show dev vxlan.calico
172.21.70.192 lladdr 66:79:81:e0:1a:7d PERMANENT
172.21.172.0 lladdr 66:05:38:46:ab:ea PERMANENT
172.21.41.1 lladdr 00:15:5d:eb:90:ae PERMANENT
user@k8s-lab-node-1:~$ bridge fdb show dev vxlan.calico
66:05:38:46:ab:ea dst 192.168.10.26 self permanent
66:79:81:e0:1a:7d dst 192.168.10.23 self permanent
00:15:5d:eb:90:ae dst 192.168.10.25 self permanent
The MAC 66:ee:f4:61:98:50 from ip -d link is the one recorded on the Windows node for the k8s-lab-node-1 block. In the other direction: the Windows node's block 172.21.41.0/26 goes through vxlan.calico to the gateway 172.21.41.1, the ARP entry leads it to the MAC of the DrPort port, and the FDB entry to the Windows node's address. Where Felix on Linux gets these values from is visible in the Node object's annotations:
$ kubectl get node k8s-lab-win-1 -o json | jq '.metadata.annotations | with_entries(select(.key | test("VXLAN")))'
{
"projectcalico.org/IPv4VXLANTunnelAddr": "172.21.41.1",
"projectcalico.org/VXLANTunnelMACAddr": "00:15:5d:eb:90:ae"
}
On the Linux nodes the tunnel address is the first address of the block (172.21.70.192, 172.21.211.0, 172.21.172.0), while on the Windows node it is the HNS network's gateway .1, and Calico records the MAC of its VTEP explicitly: on Windows it is not a kernel interface but a vSwitch port.
HNS endpoints — local ones (the pods) and remote ones that kube-proxy created for the backends and for service addresses:
PS C:\> Get-HnsEndpoint | Sort-Object IsRemoteEndpoint, IPAddress | Format-Table IPAddress, MacAddress, IsRemoteEndpoint
IPAddress MacAddress IsRemoteEndpoint
--------- ---------- ----------------
10.102.98.171 00:15:5d:0a:00:05 True
10.96.0.1 00:15:5d:0a:00:05 True
10.96.0.10 00:15:5d:0a:00:05 True
10.97.141.163 00:15:5d:0a:00:05 True
...
172.21.172.3 02-11-ac-15-ac-03 True
172.21.172.4 02-11-ac-15-ac-04 True
172.21.211.25 02-11-ac-15-d3-19 True
172.21.41.2 00:15:5d:0a:00:05 True
172.21.70.199 02-11-ac-15-46-c7 True
...
172.21.41.5 0E-2A-ac-15-29-05
172.21.41.6 0E-2A-ac-15-29-06
172.21.211.25 and 172.21.172.4 are the hello pods, 172.21.172.3 and 172.21.70.199 are CoreDNS; there are remote endpoints for the ClusterIPs themselves and for Calico_ep too. The MAC of the remote pod endpoints is not a real one: kube-proxy builds it from the 02-11 prefix and the four bytes of the address (ac-15-d3-19 is 172.21.211.25), while the ClusterIPs and Calico_ep carry the node's own MAC. They are reached not through a switch port but through the network's routing policies. The policies of a pod's endpoint:
PS C:\> $ep = Get-HnsEndpoint | Where-Object IPAddress -eq 172.21.41.6
PS C:\> $ep.Policies | Group-Object Type | Format-Table Count, Name
Count Name
----- ----
2 OutBoundNAT
1 ROUTE
1 PA
7 ACL
PS C:\> ($ep.Policies | Where-Object { $_.Type -eq 'OutBoundNAT' -and $_.ExceptionList }).ExceptionList
10.96.0.0/12
172.21.0.0/16
OutBoundNAT with exceptions is natOutgoing: everything that goes neither to the pod network nor to the service network gets the node's address; the service network in the list is the very serviceCIDRs from the Installation. PA (provider address) and ROUTE tie the endpoint to the node's address in the VXLAN network; the ACL entries are the policies Felix laid out on this endpoint — without a NetworkPolicy in the namespace these are "allow" rules and service rules for the node.
kube-proxy's load balancers are shown by hnsdiag, the HNS diagnostic utility that ships with Windows Server:
PS C:\> hnsdiag list loadbalancers
Load Balancer : 996c2141-38f4-4d34-b971-781a68c61678
Virtual IPs : 10.96.0.10
Direct IP IDs : 2b87a046-19b9-458c-a681-5ff88360106d
0f8dc4db-f502-48d3-8845-29772ed1ea03
...
Load Balancer : 6d740400-5fdb-4792-9d80-f05533d763e1
Virtual IPs : 10.96.0.10
Direct IP IDs : 2b87a046-19b9-458c-a681-5ff88360106d
0f8dc4db-f502-48d3-8845-29772ed1ea03
...
Every port of every service is a separate load balancer with a list of backend endpoints: what lives in the ip kube-proxy nftables table on a Linux node. DNS (10.96.0.10) has two of them — for UDP and TCP — and both refer to the same CoreDNS pod endpoints.
Linux fields in a Windows pod
And the last thing from part 36 — the os.name check in the API. A pod into which a securityContext from a Linux node has been copied out of habit:
$ kubectl -n win-demo apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: bad
spec:
os:
name: windows
runtimeClassName: windows-2025
securityContext:
runAsUser: 10001
seccompProfile:
type: RuntimeDefault
containers:
- name: shell
image: mcr.microsoft.com/windows/nanoserver:ltsc2025
EOF
The Pod "bad" is invalid:
* spec.securityContext.seccompProfile: Forbidden: cannot be set for a windows pod
* spec.securityContext.runAsUser: Forbidden: cannot be set for a windows pod
The API refused before anything was written to etcd. Without os.name the same pod would have been accepted — Pod Security would have checked it as a Linux pod and issued warnings about allowPrivilegeEscalation, capabilities and runAsNonRoot — and the Linux fields would have been silently ignored on the node. So os.name is worth setting always, for Linux pods too.
If the node is NotReady
Start, as on Linux, with the node's conditions: kubectl describe node k8s-lab-win-1, the Conditions section.
NetworkPluginNotReady ... cni plugin not initialized — there is no configuration in C:\etc\cni\net.d, so calico-node-windows has not done its job:
- there is no pod at all — check
windowsDataplane: HNSin the Installation andkubectl get tigerastatus: the operator writes the reason there, as in thestrictAffinitycase above; - the pod hangs in
InitorContainerCreating—kubectl -n calico-system describe pod <pod>: most often it is a long image download or a download error; - the pod restarts — logs per container:
-c install-cni, then-c nodeand-c felix. The errordial tcp 10.96.0.1:443: i/o timeoutmeans thekubernetes-services-endpointConfigMap is missing. Failing Felix probes in the first minute after start are normal.
Kubelet stopped posting node status or the Unknown status — the kubelet could not reach the API. On the node: Get-Service kubelet, containerd, Get-Content C:\var\log\kubelet\kubelet.err.log -Tail 30 and Test-NetConnection k8s-lab-api.example.com -Port 6443. Common causes:
- the connection was lost when the vSwitch was created — the host's address moved to
vEthernet (Ethernet), and the kubelet got stuck on the old connection; the Calico documentation directly recommendsRestart-Service kubeletafter installation; - the clock —
x509: certificate has expired or is not yet validin the log; look atw32tm /query /status; - containerd is not working —
kubeletdepends on it and does not start without it;crictl infoshows whether containerd answers, and the System log with theService Control Managersource (Get-WinEvent) shows why the service failed.
The node is Ready but pods do not work — sort it out right there:
The container operating system does not match the host operating systemin the pod's events — the image build does not match the node's build; the cure is the right image tag or RuntimeClass;kube-proxy-windowsinCrashLoopBackOffwith waiting for theCaliconetwork in the log — fix Calico first;kubectl logsandexecfail withtls: internal error— thekubelet-servingCSR is not approved; with a timeout — port 10250 is closed;- Windows pods cannot see the pods of Linux nodes while everything works within the node — UDP 4789 is closed on one side: in the Linux node's firewall or in the hypervisor's firewall between the virtual machines.
While there is nothing valuable on the node, reinstalling it is cheaper than fixing it:
$ kubectl drain k8s-lab-win-1 --ignore-daemonsets --delete-emptydir-data
$ kubectl delete node k8s-lab-win-1
PS C:\> kubeadm reset -f
PS C:\> Get-HnsNetwork | Remove-HnsNetwork
PS C:\> Remove-Item C:\etc\cni\net.d\* -Force
PS C:\> Restart-Computer
kubeadm reset on Windows, as on Linux, does not touch what is not its own: the HNS network and the CNI configuration are removed by hand, and the reboot removes the virtual switch and returns the address to the physical adapter. If it is the Linux base that went wrong and the test cluster has nothing in it yet, the simplest thing is to go through the "If something went wrong" section of part 4 on all three Linux nodes and start again from kubeadm init — that is what a test cluster is for. When deleting /etc/cni/net.d/* on a Linux node, keep in mind that the directory is closed to ordinary users: the * in sudo rm /etc/cni/net.d/* is expanded by a non-root shell, and the files stay — you need sudo sh -c 'rm -f /etc/cni/net.d/*'.
Sources
- Kubernetes: Adding Windows worker nodes, kubeadm Configuration (v1beta4), Windows containers in Kubernetes, Guide for running Windows containers, Networking on Windows, Create a Windows HostProcess Pod, Pod Security Standards, kubectl config set, Organizing cluster access using kubeconfig files.
- sig-windows-tools:
Install-Containerd.ps1,PrepareNode.ps1,calico/kube-proxy. - Calico: installing on your own cluster with the operator, native v3 CRDs, Installation reference.
- Calico for Windows: installation with the operator, requirements, limitations, troubleshooting.
- metrics-server, installing cert-manager, prometheus-node-exporter chart values.
- Microsoft: Windows container networking, Windows container version compatibility, Windows Time service tools and settings, .NET sample images.
Next — workloads, storage and observability on Windows: servercore and nanoserver images and multi-platform manifests with os.version, building hello for windows/amd64, how requests and limits on Windows differ from Linux, probes and graceful shutdown, CSI on Windows nodes through csi-proxy and HostProcess plugins, csi-driver-smb in the test cluster and on the Windows node and why NFS stays with the Linux nodes, kube-prometheus-stack with windows-exporter in the test cluster, Loki in the test cluster and the Windows node's logs in it, and monthly node updates.
Only registered users can leave a comment.
Sign in or register to leave a comment.