Windows containers by hand. Part 4: Pulling an image and preparing layers without containerd

All parts of the series “Windows containers by hand”

In part three we took a running container apart and found that containerd does not build the container itself: it downloads the image, lays the layers out in folders and hands the shim a description, while the silo, the C: drive and the registry are created by HCS. Today we will do containerd's job by hand. We will download an image from a registry with ordinary HTTP requests, verify every byte against checksums, lay out the layers with the wclayer utility from hcsshim, mount the writable layer as an ordinary folder and look at what containerd passes to the shim. In the next part we will pass the same thing to HCS ourselves, without containerd.

The script from this part is in the windows-containers-by-hand repository, in the 04-image folder.


How a registry serves an image

An image registry is an HTTP server that speaks the OCI Distribution protocol. Its ancestor is the Docker Registry HTTP API V2, which is why all the addresses start with /v2/. Three kinds of requests are enough to download an image:

  • GET /v2/ — a check that this is a registry and what authorization it needs;
  • GET /v2/<repository>/manifests/<tag or digest> — a description of the image, a manifest or an index;
  • GET /v2/<repository>/blobs/<digest> — content: the image configuration or a layer archive.

A digest is sha256: followed by the SHA-256 hash of the content. A registry stores everything by content: the ltsc2025 tag is a mutable label, while a digest unambiguously names specific bytes, and the client must check that it received exactly those. The client says which document it is ready to accept with the Accept header, and the registry replies with the document's type in Content-Type.

An image is a chain of documents:

  GET /v2/windows/nanoserver/manifests/ltsc2025
                       │
                       ▼
  index: one manifest per platform
                       │ digest of the windows 10.0.26100.* variant
                       ▼
  manifest: configuration and layers
          │                          │
          │ config.digest            │ layers[].digest
          ▼                          ▼
  image configuration           tar.gz layer archives
  os.version, Cmd, diff_ids     GET /v2/.../blobs/<digest>

Let's check that mcr.microsoft.com is a registry and lets us in without authorization. First we prepare a working folder:

Set-Location (New-Item -ItemType Directory C:\lab\image -Force)
curl.exe -si https://mcr.microsoft.com/v2/ | Select-String '^(HTTP|Content-Length|Docker-Distribution)'
HTTP/1.1 200 OK
Content-Length: 2
Docker-Distribution-Api-Version: registry/2.0

Code 200: we can download without a token. A registry that needs authorization replies 401 with a WWW-Authenticate header; we will come back to this when we get to the script.


The index

Let's request the ltsc2025 tag and say that we accept an index in OCI format and in Docker format:

$reg = 'https://mcr.microsoft.com/v2/windows/nanoserver'
curl.exe -s -o index.json -H 'Accept: application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json' "$reg/manifests/ltsc2025"
$index = Get-Content index.json | ConvertFrom-Json
$index.mediaType
$index.manifests | ForEach-Object { [pscustomobject]@{ os = $_.platform.os; version = $_.platform.'os.version'; mediaType = $_.mediaType; digest = $_.digest } } | Format-List
application/vnd.docker.distribution.manifest.list.v2+json

os        : windows
version   : 10.0.26100.33674
mediaType : application/vnd.docker.distribution.manifest.v2+json
digest    : sha256:6020b6edd804e7cf8cbca9142c351f597a59f53c5c7f07a6caa2261dcc1d8e8f

The index (Docker calls it a manifest list) lists the image variants for different platforms. The ltsc2025 tag has one variant: Windows, build 26100, update number 33674. The host's update number is different:

Get-ItemProperty 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion' | Format-List CurrentBuild, UBR
CurrentBuild : 26100
UBR          : 33650

For process isolation the build (26100) must match, while the update number may differ: the container runs on the host's kernel, and the kernel of a build is compatible with the user-mode part of that build. Hence the selection rule that containerd applies on its own: it takes the variant from the index whose os.version has the form 10.0.<host build>.*. The version compatibility rules are covered in detail in part 39 of the Kubernetes series.


Manifest and configuration

We request the manifest by the digest from the index and immediately check that exactly these bytes arrived:

$m = $index.manifests | Where-Object { $_.platform.'os.version' -like '10.0.26100.*' }
curl.exe -s -o manifest.json -H "Accept: $($m.mediaType)" "$reg/manifests/$($m.digest)"
'sha256:' + (Get-FileHash manifest.json).Hash.ToLower()
$manifest = Get-Content manifest.json | ConvertFrom-Json
$manifest.config | Format-List
$manifest.layers | Format-List
sha256:6020b6edd804e7cf8cbca9142c351f597a59f53c5c7f07a6caa2261dcc1d8e8f

mediaType : application/vnd.docker.container.image.v1+json
size      : 640
digest    : sha256:bd66d3cd23188368edf1629b326ae63d85cc339df59a695da701782a6568b62f

mediaType : application/vnd.docker.image.rootfs.diff.tar.gzip
size      : 194222443
digest    : sha256:a7eba7a59d48cb7e97e4eeb9fc22cf6a070f73a398171e5da84ae4aecc15cbde

The file's hash matches the digest from the index. The manifest refers to two objects: the image configuration (640 bytes) and a single layer, a tar archive compressed with gzip, 194 MB. We download the configuration as a blob:

curl.exe -sL -o config.json "$reg/blobs/$($manifest.config.digest)"
'sha256:' + (Get-FileHash config.json).Hash.ToLower()
$config = Get-Content config.json | ConvertFrom-Json
$config | Format-List os, 'os.version', created
$config.config | Format-List User, Cmd
$config.rootfs.diff_ids
$config.history.created_by
sha256:bd66d3cd23188368edf1629b326ae63d85cc339df59a695da701782a6568b62f

os         : windows
os.version : 10.0.26100.33674
created    : 2026-10-24T23:26:15.8743218Z

User : ContainerUser
Cmd  : {c:\windows\system32\cmd.exe}

sha256:9a4c0752c6bda5fc00f886a9acf0c0afed0750082f7999c331df978fb9f97d2a
Apply image 10.0.26100.33674

The -L switch allows following redirects: many registries do not serve a blob themselves but redirect to a storage service. mcr.microsoft.com does this too: it answers a blob request with code 307 and an address on a *.data.mcr.microsoft.com host, so without -L the file will not be downloaded.

The configuration holds what is needed to start: the ContainerUser user and the default command cmd.exe. Environment variables and the working folder also go here if they were set during the build. rootfs.diff_ids has one hash per layer, and this hash is a different one, not the one in the manifest.


A layer: digest and diff_id

Let's download the layer and check the digest:

$layer = $manifest.layers[0]
curl.exe -sL -o base.tar.gz "$reg/blobs/$($layer.digest)"
'sha256:' + (Get-FileHash base.tar.gz).Hash.ToLower()
sha256:a7eba7a59d48cb7e97e4eeb9fc22cf6a070f73a398171e5da84ae4aecc15cbde

The digest in the manifest is the hash of the compressed archive as the registry stores it. The diff_id in the configuration is the hash of the same archive without compression. Let's compute it, decompressing the gzip on the fly:

$gz = [IO.Compression.GZipStream]::new([IO.File]::OpenRead("$PWD\base.tar.gz"), [IO.Compression.CompressionMode]::Decompress)
$hash = [Security.Cryptography.SHA256]::Create().ComputeHash($gz); $gz.Dispose()
'sha256:' + (($hash | ForEach-Object { $_.ToString('x2') }) -join '')
sha256:9a4c0752c6bda5fc00f886a9acf0c0afed0750082f7999c331df978fb9f97d2a

It matches rootfs.diff_ids. Two hashes are needed because the same layer can be compressed in different ways. Microsoft's ASP.NET sample image is built on the same Nano Server, but its base layer in the registry weighs 199,959,116 bytes rather than 194,222,443: a different gzip, a different digest. Its diff_id, however, is the same 9a4c0752…. This is the very hash that stood at the root of ctr snapshots tree in part three: for a base layer the chain ID is the same as diff_id. containerd recognizes a layer by its uncompressed content and does not store it twice.

Let's look inside the archive:

$files = tar -tzf base.tar.gz
$files | Group-Object { ($_ -split '/')[0] } | Format-Table Count, Name
$files | Where-Object { $_ -cmatch '^Files/Windows/System32/config/[A-Z]+$' }
Count Name
----- ----
 8166 Files
 2150 UtilityVM

Files/Windows/System32/config/BBI
Files/Windows/System32/config/COMPONENTS
Files/Windows/System32/config/DEFAULT
Files/Windows/System32/config/DRIVERS
Files/Windows/System32/config/ELAM
Files/Windows/System32/config/SAM
Files/Windows/System32/config/SECURITY
Files/Windows/System32/config/SOFTWARE
Files/Windows/System32/config/SYSTEM

The archive has two folders: Files, the container's file system, and UtilityVM, the files of the utility VM for Hyper-V isolation, which we will talk about in part six. The Hives folder we saw in containerd's layers is not in the archive: the registry hives are where they are in ordinary Windows, in Files/Windows/System32/config. tar has been in Windows since 2018; it is bsdtar.


wclayer

Simply unpacking the archive is not enough: a base layer needs registry hives in Hives and blank virtual disks, and the files in the archive carry their own permissions and attributes, which have to be restored. All of this is done by the layer import function in HCS, and it is called by the wclayer utility from hcsshim, the same library the containerd shim is built on. hcsshim releases do not include a ready-made wclayer.exe, so we will build it from source. We will need Go:

New-Item -ItemType Directory C:\build -Force | Out-Null; Set-Location C:\build
curl.exe -sLo go.zip https://go.dev/dl/go1.27.3.windows-amd64.zip
if ((Get-FileHash go.zip).Hash -ne 'f26081f438319757f8a1b46d7d0b2c5ec7583d4038e485e0aa725cfde7885fff') { throw 'go.zip: sha256 mismatch' }
Expand-Archive go.zip C:\build\tools
curl.exe -sLo hcsshim.zip https://github.com/microsoft/hcsshim/archive/refs/tags/v0.15.0.zip
Expand-Archive hcsshim.zip C:\build\src
Set-Location C:\build\src\hcsshim-0.15.0
C:\build\tools\go\bin\go.exe build -o C:\hcs\ ./cmd/wclayer
$p = [Environment]::GetEnvironmentVariable('Path', 'Machine')
[Environment]::SetEnvironmentVariable('Path', "$p;C:\hcs", 'Machine')
$env:Path += ';C:\hcs'

The hash of the Go archive is published at go.dev/dl. go install github.com/Microsoft/hcsshim/cmd/wclayer@v0.15.0 would be shorter, but it will not work: hcsshim's go.mod has a replace directive, and go install with a version does not build such modules. So we download the source archive of the tag and build it with go build.

wclayer has seven commands; we will need four:

wclayer --help
...
COMMANDS:
   create         creates a new writable container layer
   export         exports a layer to a tar file
   import         imports a layer from a tar file
   makebaselayer  converts a directory containing 'Files/' into a base layer
   mount          activates a scratch, optionally mounted to provided target
   remove         permanently removes a layer directory in its entirety
   unmount        deactivates a scratch, optionally unmounting
   help, h        Shows a list of commands or help for one command
...

Importing a layer

Set-Location C:\lab\image
wclayer import -i base.tar.gz C:\layers\nanoserver
Get-ChildItem C:\layers\nanoserver -Force | Format-Table Mode, Length, Name
Get-ChildItem C:\layers\nanoserver\Hives | Format-Table Length, Name
Get-Content C:\layers\nanoserver\layout
Mode   Length   Name
----   ------   ----
d-----          Files
d-----          Hives
d-----          UtilityVM
-a---- 16384    bcd.bak
-a---- 16384    bcd.log.bak
-a---- 0        bcd.log1.bak
-a---- 0        bcd.log2.bak
-a---- 41943040 blank-base.vhdx
-a---- 4194304  blank.vhdx
-a---- 15       layout

 Length Name
 ------ ----
   8192 DEFAULTUSER_BASE
   8192 SAM_BASE
   8192 SECURITY_BASE
4190208 SOFTWARE_BASE
2781184 SYSTEM_BASE

vhd-with-hives

wclayer accepts a compressed archive and makes from it exactly what containerd's snapshotter puts in the base layer's folder. Hives holds the registry hives of the base layer, which the import prepares from the hives in Files\Windows\System32\config. blank.vhdx and blank-base.vhdx are blank virtual disks for writable layers. layout, containing vhd-with-hives, describes how the layer is organized. The bcd*.bak files belong to the boot data of the utility VM.

The other image layers are imported the same way, only with -l switches: each layer goes on top of its parents, and wclayer needs to know all of them. The script below does this.

The writable layer

wclayer create -l C:\layers\nanoserver C:\layers\c1
Get-ChildItem C:\layers\c1 | Format-Table Length, Name
 Length Name
 ------ ----
4194304 sandbox.vhdx

This is the same 4 MB sandbox.vhdx we saw next to the running container. For now it is an empty disk. Let's mount it, together with the image layer, to a folder:

New-Item -ItemType Directory C:\mnt\c1 -Force | Out-Null
wclayer mount -l C:\layers\nanoserver C:\layers\c1 C:\mnt\c1
Get-ChildItem C:\mnt\c1 -Force | Format-Table Mode, Name
Get-Item C:\mnt\c1 | Format-List LinkType, Target
Get-ChildItem C:\mnt\c1\WcSandboxState -Recurse -File -Force | Format-Table Length, FullName
C:\mnt\c1

Mode   Name
----   ----
d--h-- ProgramData
d-r--- Users
d--hs- WcSandboxState
d----- Windows
-a---- License.txt

LinkType : Junction
Target   : {Volume{ee82a57b-3830-4167-8b04-5c2863dc20fd}\}

Length FullName
------ --------
     0 C:\mnt\c1\WcSandboxState\initialized
  8192 C:\mnt\c1\WcSandboxState\Hives\DefaultUser_Delta
  8192 C:\mnt\c1\WcSandboxState\Hives\Sam_Delta
  8192 C:\mnt\c1\WcSandboxState\Hives\Security_Delta
  8192 C:\mnt\c1\WcSandboxState\Hives\Software_Delta
  8192 C:\mnt\c1\WcSandboxState\Hives\System_Delta

Without a target folder, mount only prints the volume path \\?\Volume{…}\. With a target, the folder becomes a volume mount point, and it shows the container's future C: file system: the files of the image layer and WcSandboxState with empty differencing registry hives. This is what the shim does before starting a container: it attaches sandbox.vhdx and puts the wcifs filter with the layers on the volume. The filter is visible:

fltmc volumes | Select-String -SimpleMatch 'C:\mnt'
fltmc instances -f wcifs
C:\mnt\c1                       \Device\VhdHardDisk{ee82a57b-3830-4167-8b04-5c2863dc20fd}  NTFS

Instances for wcifs filter:

Volume Name                              Altitude        Instance Name       Frame  VlStatus
-------------------------------------  ------------  ----------------------  -----  --------
C:                                        189900     wcifs Instance            0
C:                                        189899     wcifs Outer Instance      0
C:\mnt\c1                                 189900     wcifs Instance            0

C:\mnt\c1 is our volume. There is no container here, but the disk for it is already assembled, and copy-on-write works the same way as in part three:

$hosts = 'Windows\System32\drivers\etc\hosts'
Add-Content "C:\mnt\c1\$hosts" '10.0.0.5 changed-in-layer'
Get-Content "C:\mnt\c1\$hosts" -Tail 1
Get-Content "C:\layers\nanoserver\Files\$hosts" -Tail 1
10.0.0.5 changed-in-layer
#	::1             localhost

The new line is visible through the mounted folder, and the file in the image layer has not changed: wcifs copied hosts to sandbox.vhdx. Let's unmount:

mountvol C:\mnt\c1 /D
wclayer unmount C:\layers\c1
(Get-Item C:\layers\c1\sandbox.vhdx).Length
11534336

sandbox.vhdx has grown from 4 to 11 MB. First we remove the mount point, then wclayer unmount deactivates the layer. wclayer unmount with a folder as the second argument does both by itself, but on the test machine it once reported "UnprepareLayer failed in Win32: The device is not ready" (although the volume was unmounted), so I remove the mount point separately. After another mount the line in hosts is still there: the writable layer keeps everything written to it until it is deleted with wclayer remove.


The whole image: pull-image.ps1

Nano Server has one layer. Application images have more, and each one has to be downloaded, verified and imported on top of the previous ones. The pull-image.ps1 script does this, repeating the steps above:

  1. it requests the tag with an Accept header for the index and the manifest; if an index arrives, it picks the windows/amd64 variant for the host's build;
  2. it downloads the manifest and the configuration, checking the digest of each document;
  3. for each layer it downloads the blob, checks the digest and the diff_id and imports it with wclayer import with -l switches for all the layers below;
  4. it writes manifest.json (the manifest), image.json (the configuration) and layers.json, the layer folders from top to bottom, in the order that wclayer -l and HCS expect.

The script takes the Windows build from the host, and the -Build switch lets you pick a different one from the index; we will need this in part seven.

Let's download the script to C:\hcs, next to wclayer.exe, and use it to pull Microsoft's ASP.NET sample image:

curl.exe -sSLo C:\hcs\pull-image.ps1 https://raw.githubusercontent.com/vitaliy-leschenko/windows-containers-by-hand/main/04-image/pull-image.ps1
pull-image.ps1 mcr.microsoft.com/dotnet/samples:aspnetapp-nanoserver-ltsc2025 C:\images\samples
layer 1/13 sha256:9a4c0752c6bda5fc00f886a9acf0c0afed0750082f7999c331df978fb9f97d2a imported, 199959116 bytes
layer 2/13 sha256:1ccca4faa1fd48b30b12b03949c7cfcd9fd8a8feb5f4a11fafd554f260539bfa imported, 1006 bytes
layer 3/13 sha256:a1bcfaf182cf90cf3cfb6bb45fb9ab14d606bf062b326bd58f3be65b2fb4a972 imported, 1033 bytes
layer 4/13 sha256:0ad7102766069a128eef445c2d8b18885f9a7a4b0b3e2f6c2d89520062d7875b imported, 76140 bytes
layer 5/13 sha256:9487ccd8505172053f0fabff9f17ac700f987adbbb0553f94097a7a0a2dc853b imported, 1004 bytes
layer 6/13 sha256:8c4be1711ee2c9b5a130ee6bf912e66f710b5c38d7e415158e2044980a4c4e5e imported, 39443598 bytes
layer 7/13 sha256:fcbb4b24f1613c5b0f6094018844d4dfaf4b49a89340fbb2d241753d82f680ec imported, 1006 bytes
layer 8/13 sha256:10f876ddcb18f3447e9ff2d05fd8889f16b2c4d6de76a85750e1f66ea8735bf0 imported, 13276405 bytes
layer 9/13 sha256:ccd9cce9ded83b318000841197b3e659b78c8209d44fec4c2e5a5e5473703bc6 imported, 1009 bytes
layer 10/13 sha256:4b2fc071b094e3faef94b1bb8aed91d8606953b0c9c3c5ec011475e9c9652456 imported, 1240 bytes
layer 11/13 sha256:0819b5a7014851744e8b2ce6cd6616e184e1448dea994ae40a400d47e83c371c imported, 5465816 bytes
layer 12/13 sha256:1af6674ae0862643ef080f90b1bd5628326de142698cd628330a7aa700739a26 imported, 1004 bytes
layer 13/13 sha256:3d76ac9a2cbce3a19d3f5d553423e52ea9bf5dd9317664524f7cb75f402291b2 imported, 1010 bytes
mcr.microsoft.com/dotnet/samples:aspnetapp-nanoserver-ltsc2025 -> C:\images\samples, 13 layers

When a tag leads to an index, the script prints an index line. There is none here: this tag leads straight to a manifest. There are thirteen layers, and most of them are about a kilobyte. This is not a mistake: the image is built so that every Dockerfile instruction became a layer, even ENV, EXPOSE and ENTRYPOINT; the image history has 13 entries, and none of them is marked as an entry without a layer. A layer from such an instruction contains only differencing registry hives in Hives, with no files.

Anonymous token

mcr.microsoft.com serves images without authorization. Most other registries require a token, even for public images, and the script gets one. The protocol is as follows:

  1. The registry answers a request with code 401 and the header WWW-Authenticate: Bearer realm="https://auth.example.com/token",service="registry.example.com",scope="repository:library/hello-world:pull".
  2. The client makes a GET <realm>?service=…&scope=… request and receives JSON with a token field; for a public image no login or password is needed.
  3. The client repeats the request with the Authorization: Bearer <token> header.

Docker Hub works exactly this way:

pull-image.ps1 docker.io/hello-world:nanoserver-ltsc2025 C:\images\hello
index sha256:e78dec6982395f4f83172fd5473e2f0a3cb00e2f2bfd2c2b8854c2e4ff1aa625 -> 10.0.26100.33674 sha256:20e26fb7982c0d02a2bf39d3eb60a64baf70e96892c28259ba477b335db24a4e
layer 1/3 sha256:9a4c0752c6bda5fc00f886a9acf0c0afed0750082f7999c331df978fb9f97d2a imported, 199959116 bytes
layer 2/3 sha256:16ead3e57b23384a31f3303741aba8b0d6e7402c76ea83845a8034cbafacac52 imported, 1800 bytes
layer 3/3 sha256:30713c3b5279166a80970fbd9a80cb7cb60c37de665062dc3c5158c8eb8213b3 imported, 1009 bytes
docker.io/hello-world:nanoserver-ltsc2025 -> C:\images\hello, 3 layers

Here too the base is the same 9a4c0752… layer. The script puts the layers of each image in that image's own folder, so we now have the base layer three times. containerd stores it once, and images refer to a shared snapshot.

The image files

Let's mount a writable layer on top of all thirteen layers:

$layers = Get-Content C:\images\samples\layers.json | ConvertFrom-Json
$l = foreach ($p in $layers) { '-l'; $p }
wclayer create @l C:\layers\web
New-Item -ItemType Directory C:\mnt\web -Force | Out-Null
wclayer mount @l C:\layers\web C:\mnt\web
Get-ChildItem C:\mnt\web -Force | Format-Table Mode, Name
Get-ChildItem C:\mnt\web\app -Filter *.exe | Format-Table Length, Name
(Get-Content C:\images\samples\image.json | ConvertFrom-Json).config | Format-List User, Entrypoint, WorkingDir, Env
mountvol C:\mnt\web /D
wclayer unmount C:\layers\web
C:\mnt\web

Mode   Name
----   ----
d----- app
d----- Program Files
d--h-- ProgramData
d-r--- Users
d--hs- WcSandboxState
d----- Windows
-a---- License.txt

Length Name
------ ----
 73728 AspNetCoreRazorApp.exe

User       : ContainerUser
Entrypoint : {AspNetCoreRazorApp}
WorkingDir : C:\app
Env        : {ASPNETCORE_HTTP_PORTS=8080, DOTNET_RUNNING_IN_CONTAINER=true, DOTNET_VERSION=11.0.0-rc.3.26522.148, ASPNET_VERSION=11.0.0-rc.3.26522.148}

wcifs has merged all the layers: C:\app with the application, Program Files\dotnet with the .NET runtime and Windows from the base layer. The configuration tells us what to run: AspNetCoreRazorApp in C:\app as ContainerUser with four environment variables. The disk, the command and the environment are ready. What is missing is the container itself: the silo, the network and the process inside.


What containerd passes to the shim

Let's see how containerd does the same work. We will start the same ASP.NET sample through it and look at the container's metadata:

nerdctl run -d --name web -p 8080:8080 mcr.microsoft.com/dotnet/samples:aspnetapp-nanoserver-ltsc2025
$id = (nerdctl inspect web | ConvertFrom-Json).Id
$info = ctr containers info $id | ConvertFrom-Json
$info | Format-List Runtime, Snapshotter, SnapshotKey
Runtime     : @{Name=io.containerd.runhcs.v1; Options=}
Snapshotter : windows
SnapshotKey : 1107b8f92b9a659e90bcf754a29d4e914ad08a22adaba342167a9e084b3299d7

The io.containerd.runhcs.v1 runtime is containerd-shim-runhcs-v1.exe. The container's disk is a snapshot whose key matches the container ID. ctr can show what this snapshot means in the form of a mount command:

ctr snapshots mounts C:\mnt\x $info.SnapshotKey
mount -t windows-layer C:\ProgramData\containerd\root\io.containerd.snapshotter.v1.windows\snapshots\111 C:\mnt\x -o rw,parentLayerPaths=["C:\\ProgramData\\containerd\\root\\io.containerd.snapshotter.v1.windows\\snapshots\\109","C:\\ProgramData\\containerd\\root\\io.containerd.snapshotter.v1.windows\\snapshots\\108","C:\\ProgramData\\containerd\\root\\io.containerd.snapshotter.v1.windows\\snapshots\\107","C:\\ProgramData\\containerd\\root\\io.containerd.snapshotter.v1.windows\\snapshots\\106","C:\\ProgramData\\containerd\\root\\io.containerd.snapshotter.v1.windows\\snapshots\\105","C:\\ProgramData\\containerd\\root\\io.containerd.snapshotter.v1.windows\\snapshots\\104","C:\\ProgramData\\containerd\\root\\io.containerd.snapshotter.v1.windows\\snapshots\\103","C:\\ProgramData\\containerd\\root\\io.containerd.snapshotter.v1.windows\\snapshots\\102","C:\\ProgramData\\containerd\\root\\io.containerd.snapshotter.v1.windows\\snapshots\\101","C:\\ProgramData\\containerd\\root\\io.containerd.snapshotter.v1.windows\\snapshots\\100","C:\\ProgramData\\containerd\\root\\io.containerd.snapshotter.v1.windows\\snapshots\\99","C:\\ProgramData\\containerd\\root\\io.containerd.snapshotter.v1.windows\\snapshots\\98","C:\\ProgramData\\containerd\\root\\io.containerd.snapshotter.v1.windows\\snapshots\\97"]

This is our wclayer mount written differently: snapshots\111 with sandbox.vhdx is the writable layer, and parentLayerPaths are the image layers from top to bottom, like -l and layers.json. ctr snapshots mounts mounts nothing, it only prints. containerd passes this description to the shim along with the request to create the container, and it is the shim that mounts the layers.

The second half is the OCI specification, config.json from the OCI Runtime standard. nerdctl assembles it from the image configuration and the nerdctl run options:

$info.Spec | Select-Object ociVersion, process, root, windows | ConvertTo-Json -Depth 5
{
    "ociVersion": "1.3.0",
    "process": {
        "user": {
            "uid": 0,
            "gid": 0,
            "username": "ContainerUser"
        },
        "args": [
            "AspNetCoreRazorApp"
        ],
        "env": [
            "ASPNETCORE_HTTP_PORTS=8080",
            "DOTNET_RUNNING_IN_CONTAINER=true",
            "DOTNET_VERSION=11.0.0-rc.3.26522.148",
            "ASPNET_VERSION=11.0.0-rc.3.26522.148",
            "HOSTNAME="
        ],
        "cwd": "C:\\app"
    },
    "root": {
        "path": ""
    },
    "windows": {
        "layerFolders": null,
        "ignoreFlushesDuringBoot": true,
        "network": {
            "allowUnqualifiedDNSQuery": true,
            "networkNamespace": "ab27b154-916c-4b72-b91a-6078a5d23833"
        }
    }
}

The rest of the specification is nerdctl/* annotations, in which nerdctl keeps the name, the ports and its other settings. Let's go through the fields:

  • process is what to run: the command from Entrypoint, the variables from Env and the folder from WorkingDir of the image configuration, and the user by name. uid and gid are not used on Windows.
  • root.path is empty, and windows.layerFolders is null. The OCI standard allows listing the layers right in layerFolders, but containerd does not do that: the layers come separately, in the windows-layer description we saw above.
  • windows.network.networkNamespace is the ID of the HNS namespace. The network is ready by this point: nerdctl has created the namespace, the nat CNI plugin has created an endpoint in it, and the shim only attaches the namespace to the container.
hnsdiag list endpoints
Endpoint         : ba8fa62b-4f5a-44b4-a1ad-c2c9255a2ac9
    Name             : 1107b8f92b9a659e90bcf754a29d4e914ad08a22adaba342167a9e084b3299d7_nat
    IP Address       : 172.30.0.10

Having received the specification and the layers, the shim mounts the layers (like our wclayer mount), turns the OCI specification into an HCS JSON document and calls HCS: create a compute system, start it, start a process in it. This HCS document is what we will write ourselves in the next part.


What about runhcs?

hcsshim has one more program, runhcs.exe, a runtime with a command line like that of runc. It accepts an OCI bundle: a folder with a config.json in which the layers are listed in windows.layerFolders. It would seem that this is the next step down. I built it the same way as wclayer and tried it on the test machine:

  • in hcsshim 0.13 and later, including 0.15.0, runhcs does not mount the layers from layerFolders and fails with the error "failed to mount container storage: invalid layer type <nil>": the new layer mounting code expects a description that only containerd prepares;
  • in 0.12.9 the layers are mounted, but runhcs itself crashes with a null pointer dereference;
  • 0.11.8 starts the container; exec, ps and kill work, and so do mounted folders. But the container's output does not reach the terminal, mounted sandbox.vhdx disks are left behind after failures, and a container started with run -d from a WinRM session exits together with the session.

runhcs was an early attempt to give Windows a runtime in the style of runc. hcsshim now develops around the containerd shim, the new layer mounting code is designed for it, and runhcs has fallen behind. That is why it is not in this series: instead of a runtime that hides HCS behind an OCI bundle and does it poorly, we will go to HCS directly.


Cleaning up

We no longer need the container, the containerd image, the layers and the working folders from this part. wclayer remove deletes the layer folders, and an ordinary Remove-Item handles the rest:

Set-Location C:\
nerdctl rm -f web
nerdctl rmi mcr.microsoft.com/dotnet/samples:aspnetapp-nanoserver-ltsc2025
foreach ($d in 'C:\layers\web', 'C:\layers\c1', 'C:\layers\nanoserver') { wclayer remove $d }
Get-ChildItem C:\images\*\layers -Directory | ForEach-Object { wclayer remove $_.FullName }
Remove-Item C:\images, C:\layers, C:\mnt, C:\lab -Recurse -Force

wclayer, pull-image.ps1 and Go stay on the server.


Summary

  • A registry is an HTTP server with three kinds of requests: /v2/, manifests and blobs. A tag leads to an index or a manifest, a manifest leads to a configuration and layers, and everything is addressed by the hash of its content.
  • The variant for the host's build is taken from the index. The update number may differ.
  • A layer has two hashes: the digest of the compressed archive in the manifest and the diff_id of the uncompressed one in the configuration. containerd recognizes identical layers by their uncompressed content: snapshots are named by chain IDs built from diff_id.
  • wclayer import turns a layer archive into a folder with Files and Hives, create makes sandbox.vhdx, and mount mounts it with the layers through wcifs, to a folder if you specify one.
  • containerd passes the layers to the shim as a windows-layer description with parentLayerPaths, and the OCI specification with an empty layerFolders and a ready HNS namespace.

Sources


Next — a container without a runtime: we write an HCS JSON document for the layers from this part, create a compute system through computecore.dll, attach an HNS endpoint and start a process in the container with redirected output.

Only registered users can leave a comment.

Sign in or register to leave a comment.