Windows containers by hand. Part 3: What is inside a running Windows container?

All parts of the series “Windows containers by hand”

In the first part we installed containerd and ran containers without once looking inside. Today we will take a running container apart: where the image layers are on disk, where the container's C: drive comes from, where what it writes goes, where its registry is and what a server silo looks like, the isolated instance of the Windows namespaces in which its processes run. What a silo and a job object are is explained in part 36 of the Kubernetes series. Here we will find them in the system.

Our subject will be a web container with Microsoft's ASP.NET sample, and for the experiments next to it we will download the Nano Server image:

nerdctl pull mcr.microsoft.com/windows/nanoserver:ltsc2025
nerdctl run -d --name web -p 8080:8080 mcr.microsoft.com/dotnet/samples:aspnetapp-nanoserver-ltsc2025
nerdctl ps
nerdctl images
CONTAINER ID    IMAGE                                                             COMMAND                 CREATED           STATUS    PORTS                     NAMES
eaf37bc3911c    mcr.microsoft.com/dotnet/samples:aspnetapp-nanoserver-ltsc2025    "AspNetCoreRazorApp"    31 seconds ago    Up        0.0.0.0:8080->8080/tcp    web

IMAGE                                                             ID              DISK USAGE    CONTENT SIZE    EXTRA
mcr.microsoft.com/dotnet/samples:aspnetapp-nanoserver-ltsc2025    3bbeaca15f52    1.76GB        258MB           U
mcr.microsoft.com/windows/nanoserver:ltsc2025                     05d1a7fb562f    1.28GB        194MB

The letter U in the last column means that a container is using the image.

Here is what we will find by the end of this part:

   ┌─ server silo 176 = job object \Container_eaf37bc3… ─────────┐
   │ processes: smss, csrss, wininit, services, lsass, svchost,  │
   │ CExecSvc, AspNetCoreRazorApp                                │
   │ object namespace: \Silos\176                                │
   │ registry: \Registry\WC\Silo<GUID>software, system, …        │
   └──────────────────────────────┬──────────────────────────────┘
                                  │ C: drive
                                  ▼
   volume \Device\VhdHardDisk{…} = sandbox.vhdx (writable layer)
                                  │ wcifs merges
                                  ▼
   image layers: snapshots\90 … snapshots\79 → snapshots\78 (Nano Server)

Image layers on disk

containerd lays out images with the windows snapshotter, and each layer is a folder:

$s = 'C:\ProgramData\containerd\root\io.containerd.snapshotter.v1.windows\snapshots'
Get-ChildItem $s | Sort-Object { [int]$_.Name } | ForEach-Object { [pscustomobject]@{ Id = $_.Name; Items = (Get-ChildItem $_.FullName -Force).Name -join ', ' } } | Format-Table -Wrap
Id Items
-- -----
78 Files, Hives, UtilityVM, bcd.bak, bcd.log.bak, bcd.log1.bak, bcd.log2.bak, blank-base.vhdx, blank.vhdx, layout
79 Files, Hives
80 Files, Hives
81 Files, Hives
82 Files, Hives
83 Files, Hives
84 Files, Hives
85 Files, Hives
86 Files, Hives
87 Files, Hives
88 Files, Hives
89 Files, Hives
90 Files, Hives
92 sandbox.vhdx

The numbers are the snapshotter's internal identifiers; yours will be different. ctr shows which layer sits on top of which:

ctr snapshots tree
 sha256:9a4c0752c6bda5fc00f886a9acf0c0afed0750082f7999c331df978fb9f97d2a
  \_ sha256:2f4cd251c20ce7719ee74faca600a3f16ea3e4c9a169058e4f92a954297b1f89
    \_ sha256:457013bd3cc91bc4b0c95f8a0e7ee85418943b86908641752c0ff19e0e1f6d43
      \_ sha256:671dbb4dab6a1cc74d85f182ae92b51a68d0bbf9ecdf00ff1b14269ddcc8d689
        \_ sha256:da4528a2f2f8d4895b4d8d77173c51066e09380e09f6da1628643d2e05d9c004
          \_ sha256:dafc3418f71a768f6316fe5ae7f63b7a6601fcbb717b0b9a9432a47eec696de1
            \_ sha256:de8d1c2d7c4b906829b712e279a9db239eb44aa60db172ce758dd053757b585b
              \_ sha256:1b7391ac0924cb70d9e88e1002259f1d7fcc9c782e35c9e0e91c0df82ac1c927
                \_ sha256:f232f4713f8765f11e0b5f8a21c3658d87bf0c41a669f1c689d54488138ba4f5
                  \_ sha256:4dc44b2c1ea7bbd9323c389f33af8e26001e763ad7402b41ed00734b948deafe
                    \_ sha256:e19b92aeb8f6f5e84102b94c1831d9716ef0fc66c8d7062763f362533f930ef4
                      \_ sha256:168d07b469cd924103b7646269f48e1c031f4941bcc6b0d1114ef1002e8d0ea4
                        \_ sha256:fd22f6f5f332667aac115cf69b809bb2d6a33caad504258d85c493cbdd18be22
                          \_ eaf37bc3911cbe5f1e11ad0c603f85fe3a3e4891011add150b7dd8cd605a5e51

At the top is the Nano Server base layer, below it are the twelve layers of the ASP.NET sample (.NET and the application itself), and at the very bottom is a snapshot named after the container, its writable layer. The sha256:… keys are not hashes of the compressed archives from the registry but chain identifiers: the hash of the unpacked layer together with all the layers below it. So the same layer in images with different histories is stored separately, while images with a common base share it: Nano Server is on disk once, even though both nanoserver:ltsc2025 and the ASP.NET sample are built on it.

The base layer

The base layer is easy to find: only it has a UtilityVM folder.

$base = (Get-ChildItem $s | Where-Object { Test-Path "$($_.FullName)\UtilityVM" }).FullName
Get-ChildItem "$base\Files"
Get-ChildItem "$base\Hives" | Format-Table Name, Length
Get-ChildItem $base -File | Format-Table Name, Length
Get-Content "$base\layout"
Mode   Name
----   ----
d-r--- Users
d----- Windows
-a---- License.txt

Name              Length
----              ------
DEFAULTUSER_BASE    8192
SAM_BASE            8192
SECURITY_BASE       8192
SOFTWARE_BASE    4190208
SYSTEM_BASE      2781184

Name              Length
----              ------
bcd.bak            16384
bcd.log.bak        16384
bcd.log1.bak           0
bcd.log2.bak           0
blank-base.vhdx 41943040
blank.vhdx       4194304
layout                15

vhd-with-hives
  • Files is the Nano Server file system: Windows and Users, 7048 files taking 687 MB. This is what the container will see as its C:\.
  • Hives is the registry: the SOFTWARE, SYSTEM, SAM and SECURITY hives and the default user profile. On regular Windows they are in C:\Windows\System32\config; a layer keeps them separately, because the container's registry is also assembled from layers.
  • blank-base.vhdx and blank.vhdx are virtual disk templates: the writable layer of every new container is made from them.
  • UtilityVM and the bcd* files are everything for Hyper-V isolation: they are used to build the utility VM inside which such a container runs. We will get to it in the sixth part.
  • layout is the format in which the layer is prepared: vhd-with-hives, that is, with disk templates and hives.

Unpacking the base layer is more than extracting an archive: the snapshotter prepares disk templates and registry hives. That is why Nano Server is 194 MB in the image registry and 1.28 GB on disk (you can see this in the nerdctl images output at the start of this part).

Layers above the base

Let's take the topmost image layer, the folder with Hives and the highest number:

$top = (Get-ChildItem $s | Where-Object { Test-Path "$($_.FullName)\Hives" } | Sort-Object { [int]$_.Name } | Select-Object -Last 1).FullName
Get-ChildItem "$top\Hives" | Format-Table Name, Length
Name               Length
----               ------
DefaultUser_Base    16384
DefaultUser_Delta    8192
Sam_Base            40960
Sam_Delta            8192
Security_Base       24576
Security_Delta       8192
Software_Base     4202496
Software_Delta       8192
System_Base       2789376
System_Delta         8192

In the layers above the base, Files contains only added and changed files, and Hives contains pairs of hives: *_Delta holds the registry changes made by this layer, and *_Base holds the already assembled state. There is neither UtilityVM nor disk templates here: they are needed only at the bottom of the chain.


The writable layer: sandbox.vhdx

The web container has its own snapshot, a folder with a single file:

$web = (Get-ChildItem $s | Where-Object { Test-Path "$($_.FullName)\sandbox.vhdx" }).FullName
Get-ChildItem $web | Format-Table Name, Length
Name           Length
----           ------
sandbox.vhdx 37748736

This is the writable layer: a 36 MB virtual disk created from blank.vhdx when the container starts. It becomes the container's C: drive. But it is not among the host's disks:

Get-Disk | Format-Table Number, FriendlyName, Size
Get-Volume | Format-Table DriveLetter, FileSystemType, DriveType, Size
fltmc volumes
Number FriendlyName      Size
------ ------------      ----
     0 Msft Virtual Disk 68719476736

DriveLetter FileSystemType DriveType        Size
----------- -------------- ---------        ----
            FAT32          Fixed       100663296
C           NTFS           Fixed     67885854720
D           Unknown        CD-ROM     7323224064
            NTFS           Fixed       708833280

Dos Name                        Volume Name                              FileSystem   Status
------------------------------  ---------------------------------------  ----------  --------
                                \Device\Mup                              Remote
C:                              \Device\HarddiskVolume3                  NTFS
                                \Device\NamedPipe                        NamedPipe
                                \Device\Mailslot                         Mailslot
                                \Device\HarddiskVolume1                  FAT
                                \Device\HarddiskVolume4                  NTFS
D:                              \Device\CdRom0                           UDFS
                                \Device\VhdHardDisk{efb5facc-e6d1-4301-b8b4-c04bf6ea9bda}  NTFS

Get-Disk and Get-Volume show only the host's own disk and volumes, while fltmc volumes, the list of volumes that file system filters can attach to, shows one more: \Device\VhdHardDisk{…} with NTFS. HCS attached sandbox.vhdx in such a way that the disk does not appear among the host's disks, but the volume with a file system exists. The volume also has a familiar name with the same GUID. mountvol shows it, and through it you can look at the container's disk from the host:

$guid = (fltmc volumes | Where-Object { $_ -notmatch 'Detached' } | Select-String 'VhdHardDisk\{(.+?)\}').Matches[0].Groups[1].Value
mountvol | Select-String $guid
$v = "\\?\Volume{$guid}\"
Get-ChildItem -LiteralPath $v -Force | Format-Table Mode, Name, Length
    \\?\Volume{efb5facc-e6d1-4301-b8b4-c04bf6ea9bda}\

Mode   Name           Length
----   ----           ------
d----- app
d----- Program Files
d--h-- ProgramData
d-r--- Users
d--hs- WcSandboxState
d----- Windows
-a---- License.txt    5647

This is the entire root of the container: Windows from the base layer, app and Program Files from the application layers. Windows\System32 has 454 files, although the disk itself is 36 MB, so the layers' files are not in it: the volume shows them but reads them from the layer folders. This is the job of the wcifs filter, covered below. The hidden WcSandboxState folder is for internal use:

Get-ChildItem -LiteralPath "${v}WcSandboxState" -Force -Recurse | Select-Object -First 8 FullName, Length
FullName                                                                                     Length
--------                                                                                     ------
\\?\Volume{efb5facc-e6d1-4301-b8b4-c04bf6ea9bda}\WcSandboxState\Hives
\\?\Volume{efb5facc-e6d1-4301-b8b4-c04bf6ea9bda}\WcSandboxState\initialized                  0
\\?\Volume{efb5facc-e6d1-4301-b8b4-c04bf6ea9bda}\WcSandboxState\Hives\DefaultUser_Delta      16384
\\?\Volume{efb5facc-e6d1-4301-b8b4-c04bf6ea9bda}\WcSandboxState\Hives\defaultuser_Delta.LOG1 0
\\?\Volume{efb5facc-e6d1-4301-b8b4-c04bf6ea9bda}\WcSandboxState\Hives\defaultuser_Delta.LOG2 0
\\?\Volume{efb5facc-e6d1-4301-b8b4-c04bf6ea9bda}\WcSandboxState\Hives\Sam_Delta              16384
\\?\Volume{efb5facc-e6d1-4301-b8b4-c04bf6ea9bda}\WcSandboxState\Hives\sam_Delta.LOG1         8192
\\?\Volume{efb5facc-e6d1-4301-b8b4-c04bf6ea9bda}\WcSandboxState\Hives\sam_Delta.LOG2         12288

This is the writable layer of the registry: its own *_Delta hives with .LOG1 and .LOG2 transaction logs, like real Windows hives. Everything the container changes in the registry goes here, on top of the layer hives.

Where writes go

Let's write a file from inside the container. The ASP.NET sample runs as ContainerUser, so it cannot write to the root of the disk, but it can write to its own profile:

nerdctl exec web cmd /c "echo %USERNAME% & echo marker> C:\marker.txt & echo marker> C:\Users\ContainerUser\marker.txt"
Test-Path -LiteralPath "${v}Users\ContainerUser\marker.txt"
@(Get-ChildItem $s -Recurse -Filter marker.txt -Force -ErrorAction SilentlyContinue).Count
ContainerUser
Access is denied.
True
0

The file appeared on the container's volume and did not appear in any layer folder: layers are read-only. And what if you change a file that came from a layer? Let's start a second container as the container administrator (ContainerAdministrator) and append a line to hosts:

nerdctl run -d --name probe --user ContainerAdministrator mcr.microsoft.com/windows/nanoserver:ltsc2025 ping -t localhost
nerdctl exec probe cmd /c "echo 10.0.0.5 test.example.com>> C:\Windows\System32\drivers\etc\hosts & type C:\Windows\System32\drivers\etc\hosts | findstr test"
(Get-Item "$base\Files\Windows\System32\drivers\etc\hosts").Length
Select-String -Path "$base\Files\Windows\System32\drivers\etc\hosts" -Pattern test.example.com -Quiet
10.0.0.5 test.example.com
824
False

The line is there inside the container, but the file in the base layer has not changed: the same 824 bytes, and the line is not in it. On the first write the file is copied to the writable layer and changed there (copy-on-write), while other containers based on the same image keep seeing the original.

Does sandbox.vhdx grow when this happens? Yes, but you need to look at it the right way. The file size that Get-Item sees is updated with a delay for an open file, so we will read it through an open handle. For this we need the NtObjectManager module, which opens kernel objects that standard PowerShell commands cannot see. On a fresh Server Core you first need the NuGet package provider:

Install-PackageProvider NuGet -MinimumVersion 2.8.5.201 -Force
Install-Module NtObjectManager -Force
Import-Module NtObjectManager

ctr snapshots mounts tells us the folder of the probe writable layer: it prints the command that would mount the snapshot, and the fourth word of that command is the folder path. The mount target in the command (C:\x) is needed only for this printout, and nothing gets created. Let's measure the disk before writing, after downloading a file and after deleting it:

$id = (nerdctl inspect probe | ConvertFrom-Json).Id
$vhd = ((ctr snapshots mounts C:\x $id) -split ' ')[3] + '\sandbox.vhdx'
function Get-SandboxSize { $f = Get-NtFile -Win32Path $vhd -Access ReadAttributes -ShareMode All; [int]($f.Length / 1MB); $f.Close() }
Get-SandboxSize
nerdctl exec probe curl.exe -sSL -o C:\dl.bin https://github.com/containerd/containerd/releases/download/v2.4.2/containerd-2.4.2-linux-amd64.tar.gz
Start-Sleep 15; Get-SandboxSize
nerdctl exec probe cmd /c "del C:\dl.bin"
Start-Sleep 15; Get-SandboxSize
nerdctl exec probe cmd /c "dir C:\ | findstr free"
36
68
68
               2 Dir(s)  21,292,843,008 bytes free

We downloaded 34 MB and the disk grew by 32 MB; we deleted the file and the disk did not shrink: a dynamic virtual disk grows as data is written and does not shrink by itself. And the last line shows the limit: the 19.8 GiB free is not space on the host's disk but the capacity of sandbox.vhdx. The writable layer is created at 20 GiB, and the container cannot write more than that, even if the host has room.

We no longer need the probe container; from here on we look only at web again:

nerdctl rm -f probe

File system filters: wcifs and bindflt

Neither containerd nor HCS merges the layers into one disk; a driver does. A file system filter is a driver that sits above a volume's file system and sees every file operation: it can pass the operation through, change it or carry it out itself. fltmc shows the list of filters:

fltmc filters
fltmc instances -f wcifs
fltmc instances -f bindflt
Filter Name                     Num Instances    Altitude    Frame
------------------------------  -------------  ------------  -----
bindflt                                 2       409800         0
FsDepends                               6       407000         0
WdFilter                                6       328010         0
storqosflt                              0       244000         0
wcifs                                   3       189900         0
UnionFS                                 0       130850         0
npsvctrig                               1        46000         0
Wof                                     1        40700         0

Instances for wcifs filter:

Volume Name                              Altitude        Instance Name       Frame  VlStatus
-------------------------------------  ------------  ----------------------  -----  --------
C:                                        189899     wcifs Outer Instance      0
C:                                        189900     wcifs Instance            0
                                          189900     wcifs Instance            0

Instances for bindflt filter:

Volume Name                              Altitude        Instance Name       Frame  VlStatus
-------------------------------------  ------------  ----------------------  -----  --------
C:                                        409800     bindflt Instance          0
                                          409800     bindflt Instance          0

wcifs (Windows Container Isolation File System) is the filter that makes one disk out of the layer folders and sandbox.vhdx. Its instances sit both on the host's C:, where the layers are, and on the container's volume (the line without a drive letter). When a container process opens a file, wcifs looks for it first in the writable layer, then in the image layers from top to bottom, and returns the first one it finds. On a write to a file from a layer, it copies the file to the writable layer: the copy-on-write we saw with hosts. The altitude is the filter's place in the stack: the larger the number, the higher the filter and the earlier it sees an operation.

bindflt is a path mapping filter. It mounts host folders and files inside the container: when a container process accesses C:\data, bindflt redirects the access to C:\share on the host. It sits above wcifs, so a mounted folder hides whatever might be at that path in the layers. Let's start a container with a mounted folder, and the new volume will get its own bindflt instance:

New-Item -ItemType Directory C:\share -Force | Out-Null
$id = nerdctl run -d -v C:\share:C:\data mcr.microsoft.com/windows/nanoserver:ltsc2025 ping -t localhost
fltmc instances -f bindflt
nerdctl rm -f $id
Instances for bindflt filter:

Volume Name                              Altitude        Instance Name       Frame  VlStatus
-------------------------------------  ------------  ----------------------  -----  --------
C:                                        409800     bindflt Instance          0
                                          409800     bindflt Instance          0
\Device\VhdHardDisk{a532b824-3801-493c-bb28-6bcceda8af5d}     409800     bindflt Instance          0    Detached
                                          409800     bindflt Instance          0

The line without a name at the end is the new container's volume. The Detached line is a volume that has already been disconnected: when a container starts, such a line stays for a few seconds and disappears by itself, and it likewise remains for a few seconds after nerdctl rm.

The mapping works by path, not by a specific file, which is why in the first part the container saw the new content of a mounted file that was deleted and recreated on the host. The list of filters also includes UnionFS, but it has no instances: these containers do without it.


Server silo

Now the main thing: what separates a container's processes from the rest of the system. We will look at it with the same NtObjectManager module.

The job object

Let's find which job object the application process belongs to:

$p = Get-NtProcess -ProcessId (Get-Process AspNetCoreRazorApp).Id
$j = Get-NtProcessJob -Process $p
$j | Format-List Name, Silo
$j.SiloBasicInformation | Format-List SiloId, SiloParentId, NumberOfProcesses, IsInServerSilo
($j.GetProcessIdList() | ForEach-Object { (Get-Process -Id $_).Name }) -join ', '
Name : Container_eaf37bc3911cbe5f1e11ad0c603f85fe3a3e4891011add150b7dd8cd605a5e51
Silo : True

SiloId            : 176
SiloParentId      : 0
NumberOfProcesses : 18
IsInServerSilo    : True

smss, csrss, wininit, services, lsass, svchost, svchost, svchost, svchost, CExecSvc, svchost, svchost, svchost, svchost, svchost, svchost, AspNetCoreRazorApp, svchost

The job object is named Container_ plus the container ID, and it is a silo (Silo : True). It is server silo number 176, with 18 processes. Among them is smss, the session manager, the first process of any Windows system. If you look for the container's processes by session number, it does not show up. The reason lies in the session:

Get-Process smss | Format-Table Name, Id, SessionId
Name   Id SessionId
----   -- ---------
smss  500         0
smss 5964         0

There are two smss processes in the system: the host's (PID 500) and the container's (PID 5964). The container keeps its own smss for as long as it runs, but in session 0 rather than in the container's session, so a selection by session misses it. It was smss that created the container's session itself, and csrss, wininit and the rest run in that session.

Its own object namespace

Everything in Windows (files, devices, events, pipes, registry hives) has a name in the shared kernel object namespace: \Device\HarddiskVolume3, \BaseNamedObjects\..., \GLOBAL??\C:. A server silo has its own, and on the host it is located in the \Silos\<silo number> directory:

$silo = $j.SiloBasicInformation.SiloId
Get-ChildItem NtObject:\Silos
Get-ChildItem NtObject:\Silos\$silo | Format-Table Name, TypeName
Name TypeName
---- --------
176  Directory

Name                       TypeName
----                       --------
PendingRenameMutex         Mutant
ObjectTypes                SymbolicLink
SystemRoot                 SymbolicLink
Sessions                   Directory
NLS                        Directory
Windows                    Directory
GLOBAL??                   Directory
RPC Control                Directory
PdcPort                    SymbolicLink
CExecSvcStarted            Event
Dfs                        SymbolicLink
CsrSbSyncEvent             Event
SeRmCommandPort            SymbolicLink
DosDevices                 SymbolicLink
KnownDlls32                Directory
Registry                   SymbolicLink
BaseNamedObjects           Directory
UMDFCommunicationPorts     Directory
KnownDlls                  Directory
FileSystem                 SymbolicLink
KernelObjects              Directory
SeLsaCommandPort           ALPC Port
Security                   Directory
ContainerMappedDirectories Directory
Device                     Directory
ConsoleInputServerPort     ALPC Port
DriverData                 SymbolicLink
SmApiPort                  ALPC Port
SAM_SERVICE_STARTED        Event
DriverStores               SymbolicLink
Driver                     SymbolicLink

This is the root of the object namespace as the container sees it: its own GLOBAL??, which holds the drive letters; its own BaseNamedObjects, so the named events and mutexes of applications do not overlap with the host's; its own KnownDlls, and its own LSA (SeLsaCommandPort) and session manager (SmApiPort) ports, which the container's lsass and smss use. The CExecSvcStarted event signals that the CExecSvc service, through which exec works, has started. Some objects are symbolic links to shared host objects: the container and the host have the same drivers, since there is only one kernel. Let's check where the links lead in the silo and on the host:

foreach ($n in 'GLOBAL??\C:', 'SystemRoot', 'Driver', 'FileSystem') { $l = Get-NtSymbolicLink "\Silos\$silo\$n"; "$n -> $($l.Target)"; $l.Close() }
(Get-NtSymbolicLink '\GLOBAL??\C:').Target
GLOBAL??\C: -> \Device\VhdHardDisk{efb5facc-e6d1-4301-b8b4-c04bf6ea9bda}
SystemRoot -> \GLOBAL??\C:\Windows
Driver -> \Driver
FileSystem -> \FileSystem
\Device\HarddiskVolume3

That is the whole secret of the container's C: drive: in its GLOBAL?? the letter C: is a link to the sandbox.vhdx volume, while on the host it is a link to \Device\HarddiskVolume3. SystemRoot, by which the system finds C:\Windows, goes through this letter, while \Driver and \FileSystem lead straight to the host's shared directories.

The container's registry

The kernel also keeps the container's registry hives on the host, in the \Registry\WC branch, which is not accessible from regedit:

Get-ChildItem NtKey:\WC | Select-Object -First 10 Name
Name
----
Silo1d381a65-06c4-5a2a-8ba6-842762e847b5defaultuser
Silo1d381a65-06c4-5a2a-8ba6-842762e847b5sam
Silo1d381a65-06c4-5a2a-8ba6-842762e847b5security
Silo1d381a65-06c4-5a2a-8ba6-842762e847b5software
Silo1d381a65-06c4-5a2a-8ba6-842762e847b5system
Silo2137054d-9b8a-5e07-8472-e1f06abc13fadefaultuser
Silo2137054d-9b8a-5e07-8472-e1f06abc13fasam
Silo2137054d-9b8a-5e07-8472-e1f06abc13fasecurity
Silo2137054d-9b8a-5e07-8472-e1f06abc13fasoftware
Silo2137054d-9b8a-5e07-8472-e1f06abc13fasystem

Five hives per container, the same ones as in the layers' Hives folders. Hives of removed containers also stay in this branch, so there are more names here than running containers. Each is assembled from the layer hives and the *_Delta hive from WcSandboxState in the writable layer: the kernel merges the registry in layers the same way wcifs merges files. Inside the container, HKLM\SOFTWARE points here, not to the host's registry.


Cleaning up

We no longer need the container, the images and the folder from this part:

nerdctl rm -f web
nerdctl rmi mcr.microsoft.com/dotnet/samples:aspnetapp-nanoserver-ltsc2025 mcr.microsoft.com/windows/nanoserver:ltsc2025
Remove-Item C:\share -Recurse -Force

What a container is made of

Let's put together what we found:

  • Processes: regular host processes in a job object that is also a server silo. The silo has its own smss, its own session and its own system processes.
  • Object namespace: the \Silos\<number> directory, with its own drive letters, named objects and system service ports.
  • The C: drive: a volume from sandbox.vhdx, on top of which the wcifs filter shows the files of the image layers. Writes go to sandbox.vhdx, and the layers are read-only. Host folders and files are mounted by the bindflt filter.
  • Registry: hives in \Registry\WC, assembled from the layer hives and the writable layer in WcSandboxState.
  • Network: an HNS endpoint in the container's network compartment; we saw it in the first part, and how it works is covered in part 37.

containerd is responsible only for files here: it unpacks the layers, copies blank.vhdx to sandbox.vhdx and calls the shim. The processes, the silo, the volume and the registry are created by HCS from the description the shim passes to it, and the network is set up by a containerd client: nerdctl starts the CNI plugin, which creates the endpoint through HNS. In the next parts we will first prepare the layers ourselves, and then remove containerd and pass this description to HCS ourselves.


Sources


Next — we go below containerd: we download an image from the registry by hand (the index, the manifest, a layer), lay it out with the wclayer utility from hcsshim, prepare the writable layer and look at the OCI specification that containerd passes to the shim.

Only registered users can leave a comment.

Sign in or register to leave a comment.