Windows containers by hand. Part 6: Hyper-V isolation: a container in its own virtual machine

All parts of the series “Windows containers by hand”

So far every container in this series has run with process isolation: on the host's kernel, inside a server silo. In part 1 Hyper-V isolation was only mentioned: a container with it failed to start because the lab did not have the Hyper-V feature. Today we will install that feature and see how a container in its own virtual machine differs from a regular one: what is visible from the host, what is visible from inside, where it gets its kernel and why all of this is needed.

How Hyper-V isolation differs from process isolation at the level of concepts is explained in part 36 of the Kubernetes series. Here we will see it live.


The Hyper-V feature

Hyper-V isolation needs a hypervisor. Our lab is itself a Hyper-V virtual machine, so a hypervisor inside it works only with nested virtualization. On the Hyper-V host where the lab lives:

Get-VMProcessor wincont-lab | Format-List ExposeVirtualizationExtensions
Get-VM wincont-lab | Format-List MemoryStartup, DynamicMemoryEnabled, ProcessorCount
ExposeVirtualizationExtensions : True

MemoryStartup        : 8589934592
DynamicMemoryEnabled : False
ProcessorCount       : 4

Nested virtualization is enabled (Set-VMProcessor -ExposeVirtualizationExtensions $true on a powered-off machine) and the memory is static, 8 GB: nested virtualization does not work with dynamic memory. On the lab itself:

Install-WindowsFeature Hyper-V
Restart-Computer
WARNING: You must restart this server to finish the installation process.

Success Restart Needed Exit Code      Feature Result
------- -------------- ---------      --------------
True    Yes            SuccessRest... {Hyper-V}

The PowerShell module for Hyper-V (Get-VM and the rest) is not installed by this command, and we will not need it: the utility VMs of containers are visible through HCS.


The first container with Hyper-V isolation

In nerdctl, isolation is set with the --isolation flag. Let's compare the start time of the same ver, downloading the image in advance so that the measurement shows only the start:

$nano = 'mcr.microsoft.com/windows/nanoserver:ltsc2025'
nerdctl pull $nano
"process: {0:N1} s" -f (Measure-Command { nerdctl run --rm $nano cmd /c ver }).TotalSeconds
"hyperv:  {0:N1} s" -f (Measure-Command { nerdctl run --rm --isolation hyperv $nano cmd /c ver }).TotalSeconds
process: 2,2 s
hyperv:  5,4 s

Three seconds longer: that is the virtual machine booting. Let's start a long-running container and look at it from the host:

nerdctl run -d --name hv --isolation hyperv $nano ping -t localhost
hcsdiag list
Get-Process vmmem, vmwp | Format-Table Name, Id, SessionId, @{n='WS(MB)';e={[int]($_.WorkingSet64/1MB)}}
"ping processes on the host: $(@(Get-Process ping -ErrorAction SilentlyContinue).Count)"
95bec69df9de5357ab538b3f073767fd857027075871faf65e8f0ca9cda6d203
95bec69df9de5357ab538b3f073767fd857027075871faf65e8f0ca9cda6d203@vm
    VM,                       	Running, 60E95BC8-FAA0-54CA-B79E-CBE1CABD3D2F, containerd-shim-runhcs-v1.exe

Name    Id SessionId WS(MB)
----    -- --------- ------
vmmem 6244         0    375
vmwp  5604         0     31

ping processes on the host: 0

Three differences from a container with process isolation:

  • hcsdiag list shows VM instead of Windows Server Container, with the container's ID and the suffix @vm. The third field is the ID of the virtual machine in Hyper-V.
  • Two Hyper-V processes have appeared on the host: vmwp, the worker process that serves the virtual machine, and vmmem, its memory, 375 MB.
  • There is no ping process on the host. With process isolation, the container's processes were ordinary host processes in their own session, but here they run inside the virtual machine, and the host does not see them.

What HCS knows about this system is shown by hcsctl from part 5:

$id = (nerdctl inspect hv | ConvertFrom-Json).Id
hcsctl props "$id@vm"
hcsctl props $id
{
    "Id": "95bec69df9de5357ab538b3f073767fd857027075871faf65e8f0ca9cda6d203@vm",
    "SystemType": "VirtualMachine",
    "Owner": "containerd-shim-runhcs-v1.exe",
    "RuntimeId": "60e95bc8-faa0-54ca-b79e-cbe1cabd3d2f",
    "State": "Running"
}
hcsctl: HcsOpenComputeSystem 95bec69df9de5357ab538b3f073767fd857027075871faf65e8f0ca9cda6d203: 0x8037010E

The host has only the virtual machine, a system of type VirtualMachine: the utility VM. The container itself is not among the host's systems: it was created inside the virtual machine, and it cannot be opened by name from outside (0x8037010E means the system was not found). Starting a process in the utility VM itself will not work either:

hcsdiag exec -uvm "$id@vm" cmd /c ver
The request is not supported.

The -uvm flag is listed in the hcsdiag help, but the utility VM of a Windows container does not support starting processes. The only way in is into the container itself, through nerdctl exec.


From inside

Let's see what the computer looks like from inside the container. We need PowerShell, so we will take Server Core, and mount a small script, also available as probe.ps1 in the repository, from a host folder:

New-Item -ItemType Directory C:\lab\probe -Force | Out-Null
@'
$cs = Get-CimInstance Win32_ComputerSystem
"Model: $($cs.Model); CPUs: $($cs.NumberOfLogicalProcessors); memory: $([int]($cs.TotalPhysicalMemory / 1MB)) MB"
"Kernel: $((Get-Item C:\Windows\System32\ntoskrnl.exe).VersionInfo.ProductVersion)"
'@ | Set-Content C:\lab\probe\probe.ps1 -Encoding ascii
$sc = 'mcr.microsoft.com/windows/servercore:ltsc2025'
$m = '-v', 'C:\lab\probe:C:\probe:ro'
nerdctl run --rm @m $sc powershell -NoProfile -File C:\probe\probe.ps1
nerdctl run --rm --isolation hyperv @m $sc powershell -NoProfile -File C:\probe\probe.ps1
Model: Virtual Machine; CPUs: 4; memory: 8191 MB
Kernel: 10.0.26100.33648
Model: Virtual Machine; CPUs: 2; memory: 1023 MB
Kernel: 10.0.26100.33648

Virtual Machine in both cases is the lab itself, which is a virtual machine. But the container with process isolation sees all 4 CPUs and the host's 8 GB, while the container with Hyper-V isolation sees 2 CPUs and 1 GB: that is what the utility VM gets by default. And it has its own complete Windows, from the very start of boot:

nerdctl run --rm --isolation hyperv $sc powershell -NoProfile -Command "Get-Process | Sort-Object Id | Select-Object -First 6 | Format-Table Name, Id, SessionId"
Name     Id SessionId
----     -- ---------
Idle      0         0
System    4         0
smss    276         0
csrss   484         1
wininit 540         1
lsass   700         1

Idle and System with IDs 0 and 4 prove nothing by themselves: a container with process isolation sees them too, because they are processes of the kernel it shares with the host. The difference is in what follows them: smss with a small ID and session 1. This is the first boot of its own Windows, not one more session on the host.

Utility VM resources

--cpus sets the number of CPUs of the utility VM:

nerdctl run --rm --isolation hyperv --cpus 4 @m $sc powershell -NoProfile -File C:\probe\probe.ps1
Model: Virtual Machine; CPUs: 4; memory: 1023 MB
Kernel: 10.0.26100.33648

With --memory, however, the utility VM is not created at all on the lab, whatever the size: I tried 1, 1.5, 2 and 4 GB:

nerdctl run --rm --isolation hyperv --memory 2g @m $sc powershell -NoProfile -File C:\probe\probe.ps1
time="2026-11-08T12:57:05+03:00" level=error msg="failed to accept stdout connection on \\\\.\\pipe\\ctr-62eacd9a3fc1666d106349ba1f1e4e16a8fefeb835acdfd9163bfaf08be49d39-stdout" error="use of closed network connection"
time="2026-11-08T12:57:05+03:00" level=error msg="failed to accept stderr connection on \\\\.\\pipe\\ctr-62eacd9a3fc1666d106349ba1f1e4e16a8fefeb835acdfd9163bfaf08be49d39-stderr" error="use of closed network connection"
time="2026-11-08T12:57:05+03:00" level=fatal msg="failed to create shim task: error while creating the compute system: hcs::CreateComputeSystem 62eacd9a3fc1666d106349ba1f1e4e16a8fefeb835acdfd9163bfaf08be49d39@vm: winapi error #2147749931"

The code 0x8004102B explains nothing by itself, but the reason is in the HCS log:

Get-WinEvent -LogName Microsoft-Windows-Hyper-V-Compute-Admin -MaxEvents 3 | Sort-Object RecordId | Format-Table Id, LevelDisplayName, Message -AutoSize
   Id LevelDisplayName Message
   -- ---------------- -------
40014 Error            Memory virtual quantity above maximum.
40024 Error            Memory limit above maximum.
40030 Error            Memory reservation above maximum.

More memory was requested than is allowed, even though we are asking for only 2 GB. Let's see what nerdctl wrote to the container's spec:

$a = 'io.microsoft.virtualmachine.computetopology.memory.sizeinmb'
nerdctl create --name m1 --isolation hyperv --memory 2g $nano cmd /c ver | Out-Null
$id = (nerdctl inspect m1 | ConvertFrom-Json).Id
$spec = (ctr containers info $id | ConvertFrom-Json).Spec
$spec.annotations.$a
$spec.windows.resources.memory.limit
nerdctl rm -f m1
2147483648
2147483648

There is the bug, and it is in nerdctl, not in Hyper-V. hcsshim takes the size of the utility VM from the annotation …memory.sizeinmb, and the value there is in megabytes. nerdctl writes to it the same number as to resources.memory.limit, that is, bytes (line 73 in run_windows.go), and HCS receives a request to create a machine of 2,147,483,648 MB, two petabytes. This is easy to check: "2048 bytes" give a 2 GB machine.

nerdctl run --rm --isolation hyperv --memory 2048 @m $sc powershell -NoProfile -File C:\probe\probe.ps1
Model: Virtual Machine; CPUs: 2; memory: 2047 MB
Kernel: 10.0.26100.33648

The error can be worked around by setting the annotation explicitly: it overrides the value nerdctl wrote.

nerdctl run --rm --isolation hyperv --memory 2g --annotation "$a=2048" @m $sc powershell -NoProfile -File C:\probe\probe.ps1
Model: Virtual Machine; CPUs: 2; memory: 2047 MB
Kernel: 10.0.26100.33648

Where its kernel comes from

So far the Kernel line has been the same everywhere, 10.0.26100.33648, like the host's. This is the version of the ntoskrnl.exe file on the container's disk, and by itself it says nothing about the running kernel. The utility VM, however, has its own kernel, and it comes from the image, not from the host. In part 3 we saw the UtilityVM folder in the base layer, and in part 4 that it arrives in the layer archive next to Files. This is the system of the utility VM. Let's download three more older images and look at the kernels in all the layers that have UtilityVM:

nerdctl pull mcr.microsoft.com/windows/nanoserver:ltsc2022
nerdctl pull --platform windows/amd64 mcr.microsoft.com/windows/nanoserver:1809
nerdctl pull --platform windows/amd64 mcr.microsoft.com/windows/servercore:ltsc2019
$s = 'C:\ProgramData\containerd\root\io.containerd.snapshotter.v1.windows\snapshots'
Get-ChildItem $s -Directory | Where-Object { Test-Path "$($_.FullName)\UtilityVM" } | Sort-Object { [int]$_.Name } | ForEach-Object {
    [pscustomobject]@{ Snapshot = $_.Name; Kernel = (Get-Item "$($_.FullName)\UtilityVM\Files\Windows\System32\ntoskrnl.exe").VersionInfo.ProductVersion }
} | Format-Table
Snapshot Kernel
-------- ------
127      10.0.26100.33648
134      10.0.26100.32230
135      10.0.26100.33648
144      10.0.20348.5773
145      10.0.17763.9376
146      10.0.17763.7919
147      10.0.17763.9376

Snapshot numbers and images:

Snapshot Image Layer
127 nanoserver:ltsc2025 the only one
134, 135 servercore:ltsc2025 base and cumulative update
144 nanoserver:ltsc2022 the only one
145 nanoserver:1809 the only one
146, 147 servercore:ltsc2019 base and cumulative update

Server Core has two layers with UtilityVM: the base layer and the update layer, each with its own kernel. The Kernel line of the Server Core 2025 container above showed 33648, the version of the file from the top layer, 135, not from the base layer, 134.

The --platform windows/amd64 flag is needed for the 2019 images because without it containerd refuses to download them: "no match for platform in manifest". In part 1 we saw the line platforms = ['windows(10.0.26100)/amd64', ...] in config.toml, and an image of build 17763 does not match it, while a 2022 image does: Windows Server 2025 runs it with process isolation too. Now the 2019 image in both modes:

$old = 'mcr.microsoft.com/windows/servercore:ltsc2019'
nerdctl run --rm --platform windows/amd64 @m $old powershell -NoProfile -File C:\probe\probe.ps1
nerdctl run --rm --platform windows/amd64 --isolation hyperv @m $old powershell -NoProfile -File C:\probe\probe.ps1
time="2026-11-08T12:57:16+03:00" level=fatal msg="hcs::System::Start 5981c0690b763d5bf7303b831a0113044e613fe33b6ab6c21524c4432e4343e7: The container operating system does not match the host operating system."
Model: Virtual Machine; CPUs: 2; memory: 1023 MB
Kernel: 10.0.17763.9376

This is why Hyper-V isolation is needed. With process isolation, a Windows Server 2019 image does not run on a Windows Server 2025 host: the user-mode part of build 17763 does not work with the kernel of build 26100. With Hyper-V isolation it runs, and the Kernel line shows 10.0.17763.9376, the same version as the kernel in UtilityVM of the image's top layer, snapshot 147. But the Kernel line is still a file on the container's disk. To see which kernel is actually running, we ask the kernel itself: it writes its build number to KUSER_SHARED_DATA, a memory page shared with all processes, at address 0x7FFE0260:

$k = '[Runtime.InteropServices.Marshal]::ReadInt32([IntPtr]0x7FFE0260)'
nerdctl run --rm $sc powershell -NoProfile -Command $k
nerdctl run --rm --platform windows/amd64 --isolation hyperv $old powershell -NoProfile -Command $k
26100
17763

The container with process isolation runs on the host's kernel, build 26100, and the container with Hyper-V isolation runs on its own, 17763. The utility VM boots Windows from the container's image, so the kernel and the user-mode part are always of the same build. The rules for which images run on which hosts are covered in part 39 of the Kubernetes series.

One machine per container

nerdctl run -d --name hv2 --isolation hyperv --platform windows/amd64 mcr.microsoft.com/windows/nanoserver:1809 ping -t localhost
hcsdiag list
Get-Process vmmem, vmwp | Format-Table Name, Id, @{n='WS(MB)';e={[int]($_.WorkingSet64/1MB)}}
nerdctl rm -f hv hv2
df35c8aa56f8f91e526df0c81a1c9c257bc9408530f0b45345a1cfabf390d531
95bec69df9de5357ab538b3f073767fd857027075871faf65e8f0ca9cda6d203@vm
    VM,                       	Running, 60E95BC8-FAA0-54CA-B79E-CBE1CABD3D2F, containerd-shim-runhcs-v1.exe
df35c8aa56f8f91e526df0c81a1c9c257bc9408530f0b45345a1cfabf390d531@vm
    VM,                       	Running, 7B8A9D5B-C2F0-573B-A3C8-9A75B1F2A804, containerd-shim-runhcs-v1.exe

Name    Id WS(MB)
----    -- ------
vmmem 6176    264
vmmem 6244    474
vmwp  5604     30
vmwp  6460     30

Two containers, two utility VMs, each with its own vmmem and vmwp. Containers of different builds, 2025 and 2019, run side by side, each on its own kernel. The price is memory and start time: hundreds of megabytes and several seconds per container, versus almost nothing with process isolation.


Cleaning up

We no longer need the images and the folder from this part:

nerdctl rmi mcr.microsoft.com/windows/nanoserver:ltsc2025 mcr.microsoft.com/windows/servercore:ltsc2025 mcr.microsoft.com/windows/nanoserver:ltsc2022 mcr.microsoft.com/windows/nanoserver:1809 mcr.microsoft.com/windows/servercore:ltsc2019
Remove-Item C:\lab -Recurse -Force

For the 2019 images nerdctl prints the warning failed to enumerate rootfs: their platform does not match the host's. It does not affect the removal. The Hyper-V feature stays on the server.


Can we do this ourselves?

In part 5 we created a container with process isolation from a single HCS document. With Hyper-V isolation the shim does more: it creates the utility VM from the image's UtilityVM, waits for it to boot, attaches the layers to it and only then creates the container inside it, through a service inside the utility VM rather than through the host's HCS. We have seen this: the container is not among the host's systems, and HCS does not let us start a process in the machine itself. Repeating this with hcsctl would mean repeating the shim's protocol with the utility VM, and that is no longer "by hand" but a shim of your own. So here we will stop at what can be seen from outside and from inside.


Summary

  • Hyper-V isolation needs the Hyper-V feature, and on a virtual machine also nested virtualization with static memory.
  • The container runs in a utility VM: the host shows vmwp and vmmem but not the container's processes, and the host's HCS knows only the …@vm machine of type VirtualMachine.
  • Inside is its own Windows from the very start of boot: smss with a small ID and session 1, 2 CPUs and 1 GB by default. --cpus changes the number of CPUs, while --memory in nerdctl 2.4.2 writes bytes to the annotation io.microsoft.virtualmachine.computetopology.memory.sizeinmb, where megabytes are expected, and the machine is not created. An explicit annotation with the size in megabytes helps.
  • The kernel of the utility VM comes from UtilityVM of the image's top layer. That is why a Windows Server 2019 image, which does not start with process isolation on Windows Server 2025, works with Hyper-V isolation on its own 17763 kernel.
  • Each container is its own machine: hundreds of megabytes of memory and several seconds of startup.

Sources


Next — an image without Docker. We will build a layer from a folder, write a manifest and an index with os.version, and push the image to an image registry with the same protocol we used to download it in part 4.

Only registered users can leave a comment.

Sign in or register to leave a comment.