Vitaliy Leschenko’s blog
Виталий Лещенко

Kubernetes for beginners. Part 36: How do Windows nodes work in a cluster?

All parts of the series “Kubernetes for beginners”

In the previous part we upgraded the lab to Kubernetes 1.37.2 and learned to take nodes out for maintenance one at a time. So far all the nodes have been the same: Ubuntu Server, containerd, a Linux kernel. Today a block about a hybrid cluster begins, where a Windows Server worker runs next to the Linux workers.

Why bring Windows into a cluster at all? Because far from everything applications are built from is cross-platform, and a program is only as cross-platform as its least portable dependency. Modern .NET (5 and later) has long run on Linux, and the code builds for any OS, but the application runs only where everything it pulls in is present. And it can pull in things that exist only for Windows: a COM component registered in the registry, a library for documents or printing built on the Windows graphics subsystem, an SDK for hardware or a third-party service shipped as a Windows DLL. One such dependency, and a "cross-platform" service becomes a Windows service. If the dependency can be replaced, a Linux container will be simpler and cheaper. But when there is no replacement and the rest of the infrastructure already lives in Kubernetes, Windows nodes let you run such services the same way: Deployment, Service, Ingress, the same kubectl and the same health checks.

Heavy .NET Framework 4.x monoliths with IIS are usually not moved into containers: a Server Core image with IIS and .NET Framework weighs several gigabytes, takes a long time to download to every new node and is updated together with the OS every month. Such applications more often stay on virtual machines, while the cluster's Windows nodes go to small services that need Windows because of one or two dependencies. That is why the examples in this block are built on the compact Nano Server base rather than on Server Core.

The block has five parts. This one is about how a Windows node is built: what runs on it instead of namespaces, cgroups and netfilter, why the image version must match the OS version, how pods get to "their" nodes and what Windows lacks. Part 37 covers the network of Windows nodes and the choice of a network plugin, part 38 builds the hybrid cluster with kubeadm, part 39 covers workloads, storage and observability, part 40 covers gMSA domain accounts for pods. How the kubelet, CRI and a pod work on Linux was covered in part 2 and part 6; here we will compare with them all the time.

The series' lab cluster stays as it was in this block. The whole block runs on a separate test cluster: a Linux control plane k8s-lab-master-1 (192.168.10.23), two Linux workers k8s-lab-node-1 (192.168.10.24) and k8s-lab-node-2 (192.168.10.26), all on Ubuntu Server 26.04, and one Windows worker k8s-lab-win-1.example.com (192.168.10.25) on Windows Server 2025 Datacenter in the Server Core installation (no graphical shell, build 10.0.26100). The test cluster has its own pod network, 172.21.0.0/16, so that its addresses in the output are not confused with those of the main cluster (172.20.0.0/16). Why Windows needed a separate cluster rather than one more node in the lab cluster is in part 37; how to build that cluster is in part 38. Today there is no cluster yet, and this part is about the design: what a Windows node consists of, what replaces the mechanisms familiar from Linux and which rules follow from that. We will see everything described here with our own eyes in part 38, when we build the cluster and join a Windows node to it.

The big picture

 ┌─ k8s-lab-master-1 (Linux) ─────────────────────────────────────────┐
 │ etcd, kube-apiserver, kube-scheduler, kube-controller-manager      │
 │ static pods; kubelet and containerd are systemd units              │
 └───────────────────────────────┬────────────────────────────────────┘
                                 │ 6443: watch, status, Lease
                                 │ 10250: logs, exec, port-forward
                ┌────────────────┴──────────────────┐
                ▼                                   ▼
 ┌─ k8s-lab-node-1 ──────────────────┐ ┌─ k8s-lab-win-1 ───────────────────┐
 │ Ubuntu Server 26.04               │ │ Windows Server 2025, Server Core  │
 │ kubelet, containerd: systemd      │ │ kubelet, containerd: services     │
 │ shim-runc-v2 → runc               │ │ shim-runhcs-v1 → HCS              │
 │ namespaces, cgroups               │ │ server silo, job object           │
 │ network plugin, kube-proxy:       │ │ network plugin, kube-proxy:       │
 │   privileged, hostNetwork         │ │   HostProcess containers          │
 │ nftables rules                    │ │ HNS → VFP in the Hyper-V vSwitch  │
 └───────────────────────────────────┘ └───────────────────────────────────┘

The main thing the diagram shows: from the API's point of view a Windows node is just a node. Its kubelet connects to kube-apiserver on the same port 6443 with the same kind of certificate, system:node:k8s-lab-win-1, renews its Lease in kube-node-lease, receives through a watch the pods with its name in spec.nodeName and reports their status. The scheduler, controllers, EndpointSlice and RBAC work with it unchanged. The differences are below the API level: how the kubelet is started, what creates containers, who sets up the network.

The control plane runs only on Linux. The Kubernetes documentation says plainly that Windows nodes can only be workers. There are several reasons. In a kubeadm cluster the control plane components are static pods with hostNetwork: true (part 2), and an ordinary Windows container cannot run in the node's network. etcd, kube-apiserver and the other components are built and tested by the project for Linux. Finally, most add-ons — CoreDNS, metrics-server, cert-manager, ingress controllers, the cluster-wide components of network plugins — exist only as Linux images. So a hybrid cluster is always Linux masters, Linux workers for system components and Windows workers for Windows applications. In the test cluster that role is played by k8s-lab-node-1 and k8s-lab-node-2.

Windows Server versions

Kubernetes 1.37 supports Windows nodes on two Windows Server versions:

Version Build Base image tag Microsoft mainstream support Extended
Windows Server 2022 10.0.20348 ltsc2022 until 13 October 2026 until 14 October 2031
Windows Server 2025 10.0.26100 ltsc2025 until November 2029 until November 2034

Both are LTSC (Long-Term Servicing Channel) releases: long-supported versions that come out every few years and get monthly security updates. A Windows version number has four parts: 10.0.26100.<revision>. The build (26100) changes only with a new OS version; the revision grows with every monthly cumulative update. As we will see below, for containers it is the build that matters.

Windows Server 2022 is already in extended support (security fixes only), so new nodes are best installed on 2025, as in the test cluster.

Server Core suits a node better: without the graphical shell the OS takes less disk and memory, gets fewer updates and reboots, and the attack surface is smaller. It is managed from PowerShell — locally, through PowerShell Remoting (part 14 of the "Windows Server for beginners" series) or over SSH; installing Server Core is covered in the first part of that series.

How a Windows container is built

On Linux a container is assembled from kernel primitives that any process with the right privileges can use: runc calls clone() with flags for new namespaces, creates a cgroup in /sys/fs/cgroup, mounts overlayfs and starts the process. Windows has no such open primitives. A container is created by a system service — HCS (Host Compute Service, the vmcompute service, "Hyper-V Host Compute Service"). It has an API: "create a compute system with these layers, this network endpoint, these limits", "start a process in it", "stop it". Everything that runs containers on Windows — Docker, containerd — works through HCS, usually via Microsoft's Go library hcsshim.

What HCS does inside. The foundation is the job object, a Windows kernel mechanism that existed long before containers. A job object groups processes: every process spawned by a member of the job joins it too, limits can be put on the job as a whole — a share of CPU time, a memory limit, the number of processes, I/O priority — and all its processes can be terminated in one operation. In its role it is closest to a cgroup in Linux (and has nothing to do with the Kubernetes Job object from part 9).

For a container the job object is extended to a server silo — Windows's name for a job object that has its own copies of the OS namespaces (just a silo below, for short):

  • file system: the wcifs filter driver (Windows Container Isolation File System) assembles the container's C: drive from the image layers and the container's writable layer — in meaning the same as overlayfs on Linux;
  • registry: its own hives HKLM\SOFTWARE, HKLM\SYSTEM and the rest, taken from the image; changes in the container's registry are not visible to the host;
  • Object Manager namespace: named kernel objects (mutexes, events, memory sections, symbolic links like \DosDevices\C:) are per container;
  • process and thread IDs;
  • network: its own network compartment — an isolated set of interfaces, routes and tables, the analogue of a network namespace; more on it in part 37.

Microsoft's documentation calls this namespace isolation, and in meaning these really are namespaces, only gathered into one object and created by one service rather than by a set of system calls.

Its own system processes

There is also a fundamental difference from Linux. In Linux the boundary between the kernel and user space is the system call interface, and it has been stable for decades: a binary built for an old kernel runs on a new one. In Windows the boundary lies elsewhere: applications call not the kernel but system DLLs (ntdll.dll, kernel32.dll and others) and system services, and the protocol between them and the kernel changes from build to build. So a Windows container carries the whole user-mode part of the OS in its image — system libraries and services — and runs its own set of system processes:

 ┌─ host k8s-lab-win-1 ─────────────────────────────────────────────────┐
 │ ┌─ server silo: container aspnetapp ───────────────────────────────┐ │
 │ │ smss.exe   csrss.exe   wininit.exe   services.exe   lsass.exe    │ │
 │ │ svchost.exe …   CExecSvc.exe   AspNetCoreRazorApp.exe            │ │
 │ │                                                                  │ │
 │ │ C:\         wcifs: image layers + the container's writable layer │ │
 │ │ registry    its own hives from the image                         │ │
 │ │ objects     its own Object Manager namespace                     │ │
 │ │ network     the pod's compartment, vNIC in the Hyper-V vSwitch   │ │
 │ │ resources   job object: CPU and memory limits                    │ │
 │ └──────────────────────────────────────────────────────────────────┘ │
 │ shared kernel of build 10.0.26100: ntoskrnl.exe, drivers             │
 └──────────────────────────────────────────────────────────────────────┘

csrss.exe (the Win32 subsystem), wininit.exe (session startup), services.exe (the service control manager), lsass.exe (authentication and security policy) are the same processes that start when an ordinary Windows boots, only inside the silo. As in ordinary Windows, they are started by the session manager smss.exe, and the container has its own: it lives as long as the container does and belongs to its job object. It runs in session 0, so selecting processes by the container's session will not show it. CExecSvc.exe (Container Execution Service) is the service through which HCS starts new processes in the container: on a kubectl exec request it is what creates cmd.exe or another program from the image inside. And AspNetCoreRazorApp.exe is the application itself: a sample .NET web application from Microsoft's Nano Server-based image, which we will use in this block. Two consequences follow. Windows images are big: the Server Core base image weighs several gigabytes, the minimal Nano Server about 200 MB compressed, but it has no PowerShell, no .NET Framework and no IIS. And the version of the user-mode part in the image must fit the host's kernel — that is the next section.

Process isolation and Hyper-V

Windows containers have two isolation modes:

  • process isolation — what was described above: a silo on the host's shared kernel. The container's processes are ordinary host processes, visible in Get-Process on the node, only in their own session (the SI column). This is the same approach as Linux containers;
  • Hyper-V isolation — each container runs in a lightweight utility virtual machine with its own Windows kernel. The boundary is the hypervisor rather than the kernel, and the kernel version in the VM is chosen to fit the image, so the image may be older than the host.

Kubernetes supports only process isolation. The documentation says it plainly: Windows containers means containers with process isolation; Hyper-V isolation is not supported. containerd's configuration for Windows has both handlers — runhcs-wcow-process (process isolation, the default) and runhcs-wcow-hypervisor (Hyper-V) — and technically a RuntimeClass can point to the second one, but the Kubernetes project neither tests nor supports such a configuration: resource limits, metrics and volumes do not work as intended. Everything below is about process isolation.

Image version and host version

Since the image carries the user-mode part of the OS and the kernel is shared, they must match. Microsoft's rule for process isolation: the OS build in the image must match the host's build. Revisions do not have to match (that requirement was dropped starting with Windows Server 1809): a node with the October update will run an image built on the September base. But an ltsc2022 image (10.0.20348) on a 10.0.26100 host is a different build.

There is one documented exception to the rule: Windows Server 2025 runs Windows Server 2022 images with process isolation. It does not work the other way round: a 2022 host will not run an ltsc2025 image, and 2019 and 2016 images on a 2025 host run only with Hyper-V isolation, that is, not at all in Kubernetes. The Kubernetes documentation states the rule strictly: the host OS version must match the base image version. That is what we stick to in the test cluster: images for k8s-lab-win-1 are built on ltsc2025. The exception is useful as headroom during a migration from 2022 to 2025, but a permanent setup should not rely on it.

Host \ image ltsc2019 ltsc2022 ltsc2025
Windows Server 2022 no yes no
Windows Server 2025 no yes (exception) yes

If a pod does get an unsuitable image, the error shows up in the pod's events. When the image has a manifest for several platforms, containerd finds no suitable variant already at download time (no match for platform in manifest). When the image is single-platform, it downloads, but HCS refuses to start the container with the error "The operating system of the container does not match the operating system of the host" (0xc0370101), and the pod goes into CrashLoopBackOff.

There are two defences. The first is scheduling: on every Windows node the kubelet sets the node.kubernetes.io/windows-build label with the build (10.0.26100 for us), and pods choose nodes by it — below, RuntimeClass will do this. The second is multi-platform images: one tag with variants for Linux and for different Windows builds under it, with the os.version field in the manifest of each Windows variant. containerd on the node picks the variant for its build. How to build such images is in part 39, along with what to do with images when nodes move to a new Windows Server version.

kubelet and containerd on Windows

In part 2 the kubelet was a systemd unit and containerd listened on a Unix socket. On Windows the roles are the same, but the mechanisms are different.

        Linux node                              Windows node
 kubelet: systemd unit                   kubelet: Windows service
    │ CRI over /run/containerd/             │ CRI over a named pipe
    │ containerd.sock                       │ \\.\pipe\containerd-containerd
    ▼                                       ▼
 containerd: systemd unit                containerd: Windows service
    │                                       │
    ▼                                       ▼
 containerd-shim-runc-v2                 containerd-shim-runhcs-v1.exe
    │ starts runc                           │ hcsshim: HCS calls
    ▼                                       ▼
 runc: clone(), cgroupfs                 HCS: the vmcompute service
    │                                       │
    ▼                                       ▼
 kernel: namespaces, cgroups             kernel: server silo, job object,
                                         wcifs, network compartment
 CNI → veth, route, nftables             CNI → HNS: the hns service → VFP

Windows services. The Windows analogue of systemd is the Service Control Manager (SCM, the services.exe process). A service is an executable registered with the SCM with parameters: start type (automatic at boot, manual, disabled), the account it runs as, dependencies on other services and recovery actions (restart after N seconds — what Restart=always does in systemd). The service process must be able to talk to the SCM: report that it has started and accept stop commands. containerd registers itself with containerd.exe --register-service and runs as LocalSystem, a built-in account with full rights on the machine, like root. The kubelet can also run as a service (the --windows-service flag); the common installation scripts from the sig-windows-tools project instead run it through the nssm wrapper, which turns any program into a service. Which option to choose and how to make the kubelet depend on containerd is in part 38.

CRI over a named pipe. Instead of a Unix socket, containerd on Windows accepts CRI gRPC requests on the named pipe \\.\pipe\containerd-containerd. A named pipe is a Windows kernel object for exchanging data between processes, with a name in the \\.\pipe\ namespace and an access list — in meaning the same as a Unix socket in the file system. In the kubelet configuration it is written as npipe:////./pipe/containerd-containerd. The CRI protocol itself is the same: RunPodSandbox, CreateContainer, PullImage and so on.

The shim, and no runc. For each pod containerd starts containerd-shim-runhcs-v1.exe. Its role is the same as that of containerd-shim-runc-v2 on Linux: it is the parent of the pod's containers, holds their I/O and survives a containerd restart. But there is no runc: through hcsshim the shim asks HCS to create a silo and start a process in it. Which handler to use is passed by the kubelet in RunPodSandbox — it is the handler field of a RuntimeClass, covered below. By default containerd for Windows uses runhcs-wcow-process (WCOW — Windows Containers on Windows).

The pause container exists here too: the registry.k8s.io/pause image is multi-platform, with variants for Windows. The pod sandbox is a pause container with a network compartment and an HNS endpoint that the pod's other containers join. The containers of a Windows pod can share only the network: shared IPC, a shared process namespace (shareProcessNamespace) and UTS, as in part 6, do not exist on Windows.

Directories. The kubelet's paths repeat Linux with a drive letter in front: C:\var\lib\kubelet (pod directories, the config.yaml configuration, certificates in pki), C:\etc\kubernetes (kubelet.conf, the cluster CA), C:\var\log\pods (container logs in the same CRI format as on Linux), CNI plugins in C:\opt\cni\bin, their configuration in C:\etc\cni\net.d. containerd keeps images and layers in C:\ProgramData\containerd.

kubelet differences. Some kubelet features do not work on Windows because they rely on cgroups:

  • there is no pod cgroup and no QoS hierarchy: the kubelet runs with cgroupsPerQOS: false and an empty enforceNodeAllocatable;
  • kubeReserved and systemReserved only reduce the node's allocatable for the scheduler but reserve nothing at the OS level: no boundary stops pods from eating that memory;
  • the kubelet does not evict pods under memory pressure (OOM eviction) and does not set the PIDPressure condition;
  • on the other hand there is the --windows-priorityclass flag: the priority of the kubelet process in the Windows scheduler. The documentation recommends ABOVE_NORMAL_PRIORITY_CLASS, so that busy pods do not starve the kubelet of CPU time and it does not stop renewing its Lease.

What about the network?

The network is the most unfamiliar thing in a Windows node, and it gets a part of its own. Here it is enough to know who is responsible for it. On Linux a pod's network is a veth pair, a route and nftables rules that the CNI plugin and kube-proxy create themselves. Windows has neither netfilter nor veth: the container network is built by the system service HNS (Host Networking Service, the hns service) on a Hyper-V virtual switch — just as HCS creates containers — and the rules on the switch ports are enforced by its extension VFP (Virtual Filtering Platform). On a Windows node the CNI plugin and kube-proxy do not write rules themselves but ask HNS. How this works, which network plugins can do it and what follows for our cluster is in part 37.

HostProcess containers

The network plugin's agent and kube-proxy on Linux nodes are privileged pods with hostNetwork: true (part 2, part 30): they need to configure the node's own network. Windows has no privileged containers: namespace isolation in a silo is always on and cannot be switched off with a flag. So that system components can still be deployed as DaemonSets, Kubernetes has HostProcess containers (stable since 1.26).

A HostProcess container is not a silo but an ordinary host process inside a job object. From a container it keeps the image (delivering the program through a registry), the pod lifecycle, volumes and the job's resource limits. There is no isolation: the process sees the node's file system, registry, processes and network and runs as a host service account. The image contents are available to the process at the path in the CONTAINER_SANDBOX_MOUNT_POINT variable, and with containerd 1.7 and later volumes are mounted at their mountPath, as in ordinary containers.

apiVersion: v1
kind: Pod
metadata:
  name: hpc-shell
  namespace: kube-system
spec:
  os:
    name: windows
  securityContext:
    windowsOptions:
      hostProcess: true
      runAsUserName: "NT AUTHORITY\\SYSTEM"
  hostNetwork: true
  nodeSelector:
    kubernetes.io/os: windows
  tolerations:
  - key: os
    operator: Equal
    value: windows
    effect: NoSchedule
  containers:
  - name: shell
    image: mcr.microsoft.com/oss/kubernetes/windows-host-process-containers-base-image:v1.0.0
    command: ["powershell.exe", "-Command", "Start-Sleep -Seconds 3600"]

Required fields:

  • windowsOptions.hostProcess: true — at the pod level or on every container; HostProcess and ordinary containers cannot be mixed in one pod;
  • hostNetwork: true — the only case where a Windows pod runs in the node's network;
  • runAsUserName — one of the host's service accounts: NT AUTHORITY\SYSTEM (LocalSystem, full rights), NT AUTHORITY\Local service (minimal rights, anonymous on the network), NT AUTHORITY\NetworkService (minimal rights on the machine, uses the computer account on the network) or, with containerd 1.7 and later, a local group: then the pod gets a temporary user in that group. The documentation asks you to choose LocalService wherever its rights are enough.

The windows-host-process-containers-base-image base image is almost empty: the host's programs, including powershell.exe, are available to the process anyway, and the image build does not matter because there is no silo. Such a pod is the node administrator's console through the API, equivalent to a privileged pod on Linux. Pod Security forbids hostProcess already at the baseline level; such pods are allowed only in namespaces at the privileged level, such as kube-system.

How pods get to their nodes

A hybrid cluster has two tasks: Windows pods must land only on Windows nodes of a suitable build, and Linux pods must not land on Windows nodes. The mechanisms are the same labels, taints and nodeSelector from part 13, plus two new fields.

Node labels and .spec.os.name

The kubelet itself labels the node with kubernetes.io/os (linux or windows), kubernetes.io/arch (amd64) and, on Windows, node.kubernetes.io/windows-build with the OS build. The values for the two supported versions: 10.0.20348 (2022) and 10.0.26100 (2025).

The pod field .spec.os.name (linux or windows) says which OS the pod is written for. It does three things:

  • API validation: for a pod with os.name: windows, kube-apiserver rejects fields that make no sense on Windows — hostPID, hostIPC, shareProcessNamespace, the Linux fields of securityContext (the list is below). The error comes straight away on kubectl apply, not as a mysterious failure on the node;
  • node check: the kubelet refuses to run a pod whose os.name does not match the node's OS — the pod gets the Failed status with the reason PodOSNotSupported;
  • Pod Security: for pods with os.name: windows, the restricted level does not require three Linux checks — allowPrivilegeEscalation: false, a seccomp profile and capabilities.drop: [ALL].

What this field does not do is affect scheduling. The scheduler does not read it: a pod with os.name: windows and no nodeSelector will happily be assigned to a Linux node and fail there. Node selection is still labels and taints.

A taint on Windows nodes

The reverse task matters more than it seems. A working cluster has dozens of manifests without a nodeSelector — in the series' main cluster that is hello, Traefik, cert-manager, everything from Helm charts. Once a Windows node appears, the scheduler considers it suitable for any of those pods, the kubelet finds no Windows variant in a Linux image, and the pod hangs in ErrImagePull. Rewriting all the charts is slow and unreliable, so the documentation recommends putting a taint on Windows nodes and giving Windows pods a toleration for it. The key and value are a convention, not a built-in name; the common choice is os=windows with the NoSchedule effect. In part 38 the node will get it right at registration in the cluster, so that not a single foreign pod lands on it between the node's appearance and the taint being set.

With such a taint ordinary pods will not land on a Windows node. The exception is pods that tolerate everything (operator: Exists without a key), and those are exactly the system DaemonSets: log collectors, CSI node plugins, the MetalLB speaker. The taint does not stop them, and on a Windows node they get a pod with a Linux image that will never start. Such DaemonSets need nodeSelector: kubernetes.io/os: linux. kube-proxy from kubeadm already has it; we will check the test cluster's other DaemonSets in part 38.

RuntimeClass

A Windows pod now needs three things: a nodeSelector on the OS, a nodeSelector on the build and a toleration for the taint. Writing them into every manifest is inconvenient, and when nodes on another Windows version appear, all manifests would have to change. This is solved by RuntimeClass, a cluster-scoped object that describes a runtime environment:

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: windows-2025
handler: runhcs-wcow-process
scheduling:
  nodeSelector:
    kubernetes.io/os: windows
    kubernetes.io/arch: amd64
    node.kubernetes.io/windows-build: "10.0.26100"
  tolerations:
  - key: os
    operator: Equal
    value: windows
    effect: NoSchedule
  • handler is the name of the handler in the containerd configuration that the kubelet passes in RunPodSandbox. runhcs-wcow-process means process isolation. On Linux the same mechanism selects, for example, gVisor or Kata Containers instead of runc;
  • scheduling.nodeSelector and scheduling.tolerations are what will be added to the pod.

A pod refers to the class with one line, runtimeClassName: windows-2025. When the pod is created, the RuntimeClass admission plugin in kube-apiserver merges the class's nodeSelector with the pod's nodeSelector (if keys conflict, the pod is rejected) and adds the class's tolerations to the pod's tolerations. The scheduler sees a ready pod with ordinary fields:

 Deployment web: runtimeClassName: windows-2025, os.name: windows
        │ the ReplicaSet creates a pod
        ▼
 kube-apiserver, RuntimeClass admission
   + nodeSelector: kubernetes.io/os=windows, kubernetes.io/arch=amd64,
                   node.kubernetes.io/windows-build=10.0.26100
   + tolerations:  os=windows:NoSchedule
        │
        ▼
 kube-scheduler: NodeAffinity and TaintToleration filters
   k8s-lab-master-1   label os=linux, control-plane taint  — does not fit
   k8s-lab-node-1     label os=linux                       — does not fit
   k8s-lab-win-1      labels match, taint tolerated        — fits
        │
        ▼
 kubelet k8s-lab-win-1: os.name matches the node's OS
   CRI RunPodSandbox, handler runhcs-wcow-process

When nodes on Windows Server 2022 appear in the cluster, a windows-2022 class with windows-build: "10.0.20348" will sit next to it, and each application will choose the class for its image's build. When nodes move to a new version, the class in the manifest changes together with the image tag — one line instead of three fields.

What Windows lacks

A summary of what does not work or works differently in a Windows pod. Some of it is rejected by the API server (if os.name: windows is set), some simply has no effect.

Feature On Windows Instead
privileged: true no: silo isolation cannot be switched off HostProcess containers
hostNetwork: true only for HostProcess —
hostPID, hostIPC, shareProcessNamespace no, rejected by the API the pod's containers share only the network
runAsUser, runAsGroup, supplementalGroups, fsGroup no, rejected by the API windowsOptions.runAsUserName
capabilities, seccompProfile, seLinuxOptions, procMount, allowPrivilegeEscalation, sysctls no, rejected by the API rights come from the account and its groups
AppArmor, user namespaces (hostUsers: false) no —
readOnlyRootFilesystem no: the container's system processes need to write to the registry and the disk —
HugePages no —
emptyDir with medium: Memory no an ordinary emptyDir on disk
block volumes volumeDevices, mountPropagation no —
Secret in a volume written to the node's disk in plain text, not to tmpfs an ACL on the directory, BitLocker on the node's disk
OOM killer no allocating memory beyond the limit fails with an error

The container user. Instead of a UID a Windows container has an account name. The base images have two built-in ones: ContainerAdministrator (an administrator inside the container) and ContainerUser (an ordinary user). Images based on Server Core run as ContainerAdministrator by default, those based on Nano Server as ContainerUser. This can be changed with the securityContext.windowsOptions.runAsUserName field. These accounts exist only inside the container: the container has its own account database (SAM), and ContainerAdministrator is not an administrator of the host. A pod gets a domain account through gMSA — part 40.

runAsNonRoot: true also works on Windows: the kubelet treats the ContainerAdministrator account as "root" and refuses to start a container as it. And restricted requires runAsNonRoot for Windows pods too. So an image on Server Core that runs as the container administrator (for example, IIS with the W3SVC service) will not pass into a namespace at the restricted level, like demo from part 30, while an image on Nano Server running as ContainerUser will, if the pod sets runAsNonRoot: true. Our Windows examples will be on Nano Server, and their win-demo namespace will get the same restricted level as demo.

Resources: job objects instead of cgroups

On Linux the kubelet turns requests and limits into cgroup parameters (part 10): cpu.weight for the guaranteed CPU share, cpu.max for the ceiling, memory.max for the memory limit beyond which the OOM killer kills the container. On Windows all of this is properties of the container's job object, and the set is poorer:

  • limits.cpu becomes a ceiling on the job's CPU time: a share of all the node's processors in units of 1/10000. A 500m limit on a node with four processors is 1250, that is 12.5% of the total time;
  • limits.memory is a limit on the job's committed memory. A process that tries to allocate memory beyond the limit gets an allocation error, and what happens next is up to the application itself; usually it crashes with an out-of-memory exception;
  • requests are needed only by the scheduler: Windows cannot give a guaranteed minimum CPU share, and the kubelet sets no weights like cpu.weight.

Windows does not overcommit memory, and a page file is mandatory: a node where pods without limits have eaten all the memory does not kill processes but starts paging and slows down as a whole — kubelet included. So the documentation recommends always giving Windows pods limits, leaving the system at least 2 GiB through systemReserved and raising the kubelet's priority.

A Windows node joins the cluster at the API level just like a Linux node, but inside everything is its own: services instead of systemd units, HCS and silos instead of runc and namespaces, job objects instead of cgroups, HostProcess instead of privileged pods. The most unfamiliar part of it is the network.

Sources

Next — the network of Windows nodes and the choice of a network plugin: how HNS and VFP build the pod network, how the l2bridge and overlay modes differ from Transparent and NAT, which plugins work on Windows and in which modes — Flannel, Calico, Antrea, ovn-kubernetes, the cloud Azure CNI and Amazon VPC CNI — kube-proxy in winkernel mode and DSR, the limitations of the Windows node network, and why for the hybrid cluster we chose a separate test cluster built in VXLAN without BGP from the start, leaving the migration of the lab cluster off VXLANCrossSubnet and BGP as an alternative with its own risks.

Only registered users can leave a comment.

Sign in or register to leave a comment.

© 2006-2026 Vitaliy Leschenko