Windows containers by hand. Part 6: Hyper-V isolation: a container in its own virtual machine
All parts of the series “Windows containers by hand”So far every container in this series has run with process isolation: on the host's kernel, inside a server silo. In part 1 Hyper-V isolation was only mentioned: a container with it failed to start because the lab did not have the Hyper-V feature. Today we will install that feature and see how a container in its own virtual machine differs from a regular one: what is visible from the host, what is visible from inside, where it gets its kernel and why all of this is needed.
How Hyper-V isolation differs from process isolation at the level of concepts is explained in part 36 of the Kubernetes series. Here we will see it live.
The Hyper-V feature
Hyper-V isolation needs a hypervisor. Our lab is itself a Hyper-V virtual machine, so a hypervisor inside it works only with nested virtualization. On the Hyper-V host where the lab lives:
Get-VMProcessor wincont-lab | Format-List ExposeVirtualizationExtensions
Get-VM wincont-lab | Format-List MemoryStartup, DynamicMemoryEnabled, ProcessorCount
ExposeVirtualizationExtensions : True
MemoryStartup : 8589934592
DynamicMemoryEnabled : False
ProcessorCount : 4
Nested virtualization is enabled (Set-VMProcessor -ExposeVirtualizationExtensions $true on a powered-off machine) and the memory is static, 8 GB: nested virtualization does not work with dynamic memory. On the lab itself:
Install-WindowsFeature Hyper-V
Restart-Computer
WARNING: You must restart this server to finish the installation process.
Success Restart Needed Exit Code Feature Result
------- -------------- --------- --------------
True Yes SuccessRest... {Hyper-V}
The PowerShell module for Hyper-V (Get-VM and the rest) is not installed by this command, and we will not need it: the utility VMs of containers are visible through HCS.
The first container with Hyper-V isolation
In nerdctl, isolation is set with the --isolation flag. Let's compare the start time of the same ver, downloading the image in advance so that the measurement shows only the start:
$nano = 'mcr.microsoft.com/windows/nanoserver:ltsc2025'
nerdctl pull $nano
"process: {0:N1} s" -f (Measure-Command { nerdctl run --rm $nano cmd /c ver }).TotalSeconds
"hyperv: {0:N1} s" -f (Measure-Command { nerdctl run --rm --isolation hyperv $nano cmd /c ver }).TotalSeconds
process: 2,2 s
hyperv: 5,4 s
Three seconds longer: that is the virtual machine booting. Let's start a long-running container and look at it from the host:
nerdctl run -d --name hv --isolation hyperv $nano ping -t localhost
hcsdiag list
Get-Process vmmem, vmwp | Format-Table Name, Id, SessionId, @{n='WS(MB)';e={[int]($_.WorkingSet64/1MB)}}
"ping processes on the host: $(@(Get-Process ping -ErrorAction SilentlyContinue).Count)"
95bec69df9de5357ab538b3f073767fd857027075871faf65e8f0ca9cda6d203
95bec69df9de5357ab538b3f073767fd857027075871faf65e8f0ca9cda6d203@vm
VM, Running, 60E95BC8-FAA0-54CA-B79E-CBE1CABD3D2F, containerd-shim-runhcs-v1.exe
Name Id SessionId WS(MB)
---- -- --------- ------
vmmem 6244 0 375
vmwp 5604 0 31
ping processes on the host: 0
Three differences from a container with process isolation:
hcsdiag listshowsVMinstead ofWindows Server Container, with the container's ID and the suffix@vm. The third field is the ID of the virtual machine in Hyper-V.- Two Hyper-V processes have appeared on the host:
vmwp, the worker process that serves the virtual machine, andvmmem, its memory, 375 MB. - There is no
pingprocess on the host. With process isolation, the container's processes were ordinary host processes in their own session, but here they run inside the virtual machine, and the host does not see them.
What HCS knows about this system is shown by hcsctl from part 5:
$id = (nerdctl inspect hv | ConvertFrom-Json).Id
hcsctl props "$id@vm"
hcsctl props $id
{
"Id": "95bec69df9de5357ab538b3f073767fd857027075871faf65e8f0ca9cda6d203@vm",
"SystemType": "VirtualMachine",
"Owner": "containerd-shim-runhcs-v1.exe",
"RuntimeId": "60e95bc8-faa0-54ca-b79e-cbe1cabd3d2f",
"State": "Running"
}
hcsctl: HcsOpenComputeSystem 95bec69df9de5357ab538b3f073767fd857027075871faf65e8f0ca9cda6d203: 0x8037010E
The host has only the virtual machine, a system of type VirtualMachine: the utility VM. The container itself is not among the host's systems: it was created inside the virtual machine, and it cannot be opened by name from outside (0x8037010E means the system was not found). Starting a process in the utility VM itself will not work either:
hcsdiag exec -uvm "$id@vm" cmd /c ver
The request is not supported.
The -uvm flag is listed in the hcsdiag help, but the utility VM of a Windows container does not support starting processes. The only way in is into the container itself, through nerdctl exec.
From inside
Let's see what the computer looks like from inside the container. We need PowerShell, so we will take Server Core, and mount a small script, also available as probe.ps1 in the repository, from a host folder:
New-Item -ItemType Directory C:\lab\probe -Force | Out-Null
@'
$cs = Get-CimInstance Win32_ComputerSystem
"Model: $($cs.Model); CPUs: $($cs.NumberOfLogicalProcessors); memory: $([int]($cs.TotalPhysicalMemory / 1MB)) MB"
"Kernel: $((Get-Item C:\Windows\System32\ntoskrnl.exe).VersionInfo.ProductVersion)"
'@ | Set-Content C:\lab\probe\probe.ps1 -Encoding ascii
$sc = 'mcr.microsoft.com/windows/servercore:ltsc2025'
$m = '-v', 'C:\lab\probe:C:\probe:ro'
nerdctl run --rm @m $sc powershell -NoProfile -File C:\probe\probe.ps1
nerdctl run --rm --isolation hyperv @m $sc powershell -NoProfile -File C:\probe\probe.ps1
Model: Virtual Machine; CPUs: 4; memory: 8191 MB
Kernel: 10.0.26100.33648
Model: Virtual Machine; CPUs: 2; memory: 1023 MB
Kernel: 10.0.26100.33648
Virtual Machine in both cases is the lab itself, which is a virtual machine. But the container with process isolation sees all 4 CPUs and the host's 8 GB, while the container with Hyper-V isolation sees 2 CPUs and 1 GB: that is what the utility VM gets by default. And it has its own complete Windows, from the very start of boot:
nerdctl run --rm --isolation hyperv $sc powershell -NoProfile -Command "Get-Process | Sort-Object Id | Select-Object -First 6 | Format-Table Name, Id, SessionId"
Name Id SessionId
---- -- ---------
Idle 0 0
System 4 0
smss 276 0
csrss 484 1
wininit 540 1
lsass 700 1
Idle and System with IDs 0 and 4 prove nothing by themselves: a container with process isolation sees them too, because they are processes of the kernel it shares with the host. The difference is in what follows them: smss with a small ID and session 1. This is the first boot of its own Windows, not one more session on the host.
Utility VM resources
--cpus sets the number of CPUs of the utility VM:
nerdctl run --rm --isolation hyperv --cpus 4 @m $sc powershell -NoProfile -File C:\probe\probe.ps1
Model: Virtual Machine; CPUs: 4; memory: 1023 MB
Kernel: 10.0.26100.33648
With --memory, however, the utility VM is not created at all on the lab, whatever the size: I tried 1, 1.5, 2 and 4 GB:
nerdctl run --rm --isolation hyperv --memory 2g @m $sc powershell -NoProfile -File C:\probe\probe.ps1
time="2026-11-08T12:57:05+03:00" level=error msg="failed to accept stdout connection on \\\\.\\pipe\\ctr-62eacd9a3fc1666d106349ba1f1e4e16a8fefeb835acdfd9163bfaf08be49d39-stdout" error="use of closed network connection"
time="2026-11-08T12:57:05+03:00" level=error msg="failed to accept stderr connection on \\\\.\\pipe\\ctr-62eacd9a3fc1666d106349ba1f1e4e16a8fefeb835acdfd9163bfaf08be49d39-stderr" error="use of closed network connection"
time="2026-11-08T12:57:05+03:00" level=fatal msg="failed to create shim task: error while creating the compute system: hcs::CreateComputeSystem 62eacd9a3fc1666d106349ba1f1e4e16a8fefeb835acdfd9163bfaf08be49d39@vm: winapi error #2147749931"
The code 0x8004102B explains nothing by itself, but the reason is in the HCS log:
Get-WinEvent -LogName Microsoft-Windows-Hyper-V-Compute-Admin -MaxEvents 3 | Sort-Object RecordId | Format-Table Id, LevelDisplayName, Message -AutoSize
Id LevelDisplayName Message
-- ---------------- -------
40014 Error Memory virtual quantity above maximum.
40024 Error Memory limit above maximum.
40030 Error Memory reservation above maximum.
More memory was requested than is allowed, even though we are asking for only 2 GB. Let's see what nerdctl wrote to the container's spec:
$a = 'io.microsoft.virtualmachine.computetopology.memory.sizeinmb'
nerdctl create --name m1 --isolation hyperv --memory 2g $nano cmd /c ver | Out-Null
$id = (nerdctl inspect m1 | ConvertFrom-Json).Id
$spec = (ctr containers info $id | ConvertFrom-Json).Spec
$spec.annotations.$a
$spec.windows.resources.memory.limit
nerdctl rm -f m1
2147483648
2147483648
There is the bug, and it is in nerdctl, not in Hyper-V. hcsshim takes the size of the utility VM from the annotation …memory.sizeinmb, and the value there is in megabytes. nerdctl writes to it the same number as to resources.memory.limit, that is, bytes (line 73 in run_windows.go), and HCS receives a request to create a machine of 2,147,483,648 MB, two petabytes. This is easy to check: "2048 bytes" give a 2 GB machine.
nerdctl run --rm --isolation hyperv --memory 2048 @m $sc powershell -NoProfile -File C:\probe\probe.ps1
Model: Virtual Machine; CPUs: 2; memory: 2047 MB
Kernel: 10.0.26100.33648
The error can be worked around by setting the annotation explicitly: it overrides the value nerdctl wrote.
nerdctl run --rm --isolation hyperv --memory 2g --annotation "$a=2048" @m $sc powershell -NoProfile -File C:\probe\probe.ps1
Model: Virtual Machine; CPUs: 2; memory: 2047 MB
Kernel: 10.0.26100.33648
Where its kernel comes from
So far the Kernel line has been the same everywhere, 10.0.26100.33648, like the host's. This is the version of the ntoskrnl.exe file on the container's disk, and by itself it says nothing about the running kernel. The utility VM, however, has its own kernel, and it comes from the image, not from the host. In part 3 we saw the UtilityVM folder in the base layer, and in part 4 that it arrives in the layer archive next to Files. This is the system of the utility VM. Let's download three more older images and look at the kernels in all the layers that have UtilityVM:
nerdctl pull mcr.microsoft.com/windows/nanoserver:ltsc2022
nerdctl pull --platform windows/amd64 mcr.microsoft.com/windows/nanoserver:1809
nerdctl pull --platform windows/amd64 mcr.microsoft.com/windows/servercore:ltsc2019
$s = 'C:\ProgramData\containerd\root\io.containerd.snapshotter.v1.windows\snapshots'
Get-ChildItem $s -Directory | Where-Object { Test-Path "$($_.FullName)\UtilityVM" } | Sort-Object { [int]$_.Name } | ForEach-Object {
[pscustomobject]@{ Snapshot = $_.Name; Kernel = (Get-Item "$($_.FullName)\UtilityVM\Files\Windows\System32\ntoskrnl.exe").VersionInfo.ProductVersion }
} | Format-Table
Snapshot Kernel
-------- ------
127 10.0.26100.33648
134 10.0.26100.32230
135 10.0.26100.33648
144 10.0.20348.5773
145 10.0.17763.9376
146 10.0.17763.7919
147 10.0.17763.9376
Snapshot numbers and images:
| Snapshot | Image | Layer |
|---|---|---|
| 127 | nanoserver:ltsc2025 |
the only one |
| 134, 135 | servercore:ltsc2025 |
base and cumulative update |
| 144 | nanoserver:ltsc2022 |
the only one |
| 145 | nanoserver:1809 |
the only one |
| 146, 147 | servercore:ltsc2019 |
base and cumulative update |
Server Core has two layers with UtilityVM: the base layer and the update layer, each with its own kernel. The Kernel line of the Server Core 2025 container above showed 33648, the version of the file from the top layer, 135, not from the base layer, 134.
The --platform windows/amd64 flag is needed for the 2019 images because without it containerd refuses to download them: "no match for platform in manifest". In part 1 we saw the line platforms = ['windows(10.0.26100)/amd64', ...] in config.toml, and an image of build 17763 does not match it, while a 2022 image does: Windows Server 2025 runs it with process isolation too. Now the 2019 image in both modes:
$old = 'mcr.microsoft.com/windows/servercore:ltsc2019'
nerdctl run --rm --platform windows/amd64 @m $old powershell -NoProfile -File C:\probe\probe.ps1
nerdctl run --rm --platform windows/amd64 --isolation hyperv @m $old powershell -NoProfile -File C:\probe\probe.ps1
time="2026-11-08T12:57:16+03:00" level=fatal msg="hcs::System::Start 5981c0690b763d5bf7303b831a0113044e613fe33b6ab6c21524c4432e4343e7: The container operating system does not match the host operating system."
Model: Virtual Machine; CPUs: 2; memory: 1023 MB
Kernel: 10.0.17763.9376
This is why Hyper-V isolation is needed. With process isolation, a Windows Server 2019 image does not run on a Windows Server 2025 host: the user-mode part of build 17763 does not work with the kernel of build 26100. With Hyper-V isolation it runs, and the Kernel line shows 10.0.17763.9376, the same version as the kernel in UtilityVM of the image's top layer, snapshot 147. But the Kernel line is still a file on the container's disk. To see which kernel is actually running, we ask the kernel itself: it writes its build number to KUSER_SHARED_DATA, a memory page shared with all processes, at address 0x7FFE0260:
$k = '[Runtime.InteropServices.Marshal]::ReadInt32([IntPtr]0x7FFE0260)'
nerdctl run --rm $sc powershell -NoProfile -Command $k
nerdctl run --rm --platform windows/amd64 --isolation hyperv $old powershell -NoProfile -Command $k
26100
17763
The container with process isolation runs on the host's kernel, build 26100, and the container with Hyper-V isolation runs on its own, 17763. The utility VM boots Windows from the container's image, so the kernel and the user-mode part are always of the same build. The rules for which images run on which hosts are covered in part 39 of the Kubernetes series.
One machine per container
nerdctl run -d --name hv2 --isolation hyperv --platform windows/amd64 mcr.microsoft.com/windows/nanoserver:1809 ping -t localhost
hcsdiag list
Get-Process vmmem, vmwp | Format-Table Name, Id, @{n='WS(MB)';e={[int]($_.WorkingSet64/1MB)}}
nerdctl rm -f hv hv2
df35c8aa56f8f91e526df0c81a1c9c257bc9408530f0b45345a1cfabf390d531
95bec69df9de5357ab538b3f073767fd857027075871faf65e8f0ca9cda6d203@vm
VM, Running, 60E95BC8-FAA0-54CA-B79E-CBE1CABD3D2F, containerd-shim-runhcs-v1.exe
df35c8aa56f8f91e526df0c81a1c9c257bc9408530f0b45345a1cfabf390d531@vm
VM, Running, 7B8A9D5B-C2F0-573B-A3C8-9A75B1F2A804, containerd-shim-runhcs-v1.exe
Name Id WS(MB)
---- -- ------
vmmem 6176 264
vmmem 6244 474
vmwp 5604 30
vmwp 6460 30
Two containers, two utility VMs, each with its own vmmem and vmwp. Containers of different builds, 2025 and 2019, run side by side, each on its own kernel. The price is memory and start time: hundreds of megabytes and several seconds per container, versus almost nothing with process isolation.
Cleaning up
We no longer need the images and the folder from this part:
nerdctl rmi mcr.microsoft.com/windows/nanoserver:ltsc2025 mcr.microsoft.com/windows/servercore:ltsc2025 mcr.microsoft.com/windows/nanoserver:ltsc2022 mcr.microsoft.com/windows/nanoserver:1809 mcr.microsoft.com/windows/servercore:ltsc2019
Remove-Item C:\lab -Recurse -Force
For the 2019 images nerdctl prints the warning failed to enumerate rootfs: their platform does not match the host's. It does not affect the removal. The Hyper-V feature stays on the server.
Can we do this ourselves?
In part 5 we created a container with process isolation from a single HCS document. With Hyper-V isolation the shim does more: it creates the utility VM from the image's UtilityVM, waits for it to boot, attaches the layers to it and only then creates the container inside it, through a service inside the utility VM rather than through the host's HCS. We have seen this: the container is not among the host's systems, and HCS does not let us start a process in the machine itself. Repeating this with hcsctl would mean repeating the shim's protocol with the utility VM, and that is no longer "by hand" but a shim of your own. So here we will stop at what can be seen from outside and from inside.
Summary
- Hyper-V isolation needs the Hyper-V feature, and on a virtual machine also nested virtualization with static memory.
- The container runs in a utility VM: the host shows
vmwpandvmmembut not the container's processes, and the host's HCS knows only the…@vmmachine of typeVirtualMachine. - Inside is its own Windows from the very start of boot:
smsswith a small ID and session 1, 2 CPUs and 1 GB by default.--cpuschanges the number of CPUs, while--memoryin nerdctl 2.4.2 writes bytes to the annotationio.microsoft.virtualmachine.computetopology.memory.sizeinmb, where megabytes are expected, and the machine is not created. An explicit annotation with the size in megabytes helps. - The kernel of the utility VM comes from
UtilityVMof the image's top layer. That is why a Windows Server 2019 image, which does not start with process isolation on Windows Server 2025, works with Hyper-V isolation on its own 17763 kernel. - Each container is its own machine: hundreds of megabytes of memory and several seconds of startup.
Sources
- Isolation modes — Microsoft Learn: process isolation and Hyper-V isolation.
- Windows container version compatibility — Microsoft Learn: which images run on which hosts and with which isolation.
- Run Hyper-V in a Virtual Machine with Nested Virtualization — Microsoft Learn: nested virtualization.
- microsoft/hcsshim — the containerd shim that creates utility VMs and the
io.microsoft.virtualmachine.*annotations.
Next — an image without Docker. We will build a layer from a folder, write a manifest and an index with os.version, and push the image to an image registry with the same protocol we used to download it in part 4.
Only registered users can leave a comment.
Sign in or register to leave a comment.