Vitaliy Leschenko’s blog
Виталий Лещенко

Hyper-V. Part 9: Building a failover cluster

All parts of the series “Hyper-V”

In the previous part we moved machines between standalone hosts — but every move was started by an administrator, and if a host simply burned out, the machines on it stayed down until someone stepped in manually. A failover cluster solves this problem: several hosts watch each other, and the machines of a failed node start on the remaining ones by themselves. Let's look at what a cluster gives and does not give you, what storage it needs, how shared volumes and quorum work, how to build a cluster from PowerShell and how to add machines to it.

What a cluster gives you

A standalone host is a single point of failure: a power supply burns out and all its machines go down. A failover cluster joins several servers, called nodes, which watch each other through heartbeat network messages. When a node disappears, its machines are started automatically on the remaining ones (Failover Clustering overview).

This is not live migration: the machines of the failed node are restarted, the downtime equals their boot time, and everything that was in memory is lost. Planned work, on the other hand — updates, hardware replacement — goes without downtime: the node is emptied by live migration and only then shut down.

A Hyper-V cluster on Windows Server 2025 has up to 64 nodes and up to 8,000 running machines (limits).

          clients                       management
             │                               │
───────┬─────┴───────────────┬─ VM network ──┴─────┬──────────
       │                     │                     │
   ┌───┴───┐ heartbeat   ┌───┴───┐ heartbeat   ┌───┴───┐
   │  hv1  ├─────────────┤  hv2  ├─────────────┤  hv3  │
   └───┬───┘             └───┬───┘             └───┬───┘
       │                     │                     │
───────┴── storage network ──┼─────────────────────┴──────────
                             │
                 ┌───────────┴───────────┐
                 │ shared storage:       │
                 │ CSV C:\ClusterStorage │
                 └───────────────────────┘

Shared storage

For a machine to start on another node, its files must be visible to all nodes. There are three options:

  • SAN — an external storage system to which all nodes are connected over iSCSI or Fibre Channel. Every node sees the same LUN.
  • Storage Spaces Direct (S2D) — software-defined storage: the local disks of 2 to 16 servers are combined into a shared pool, and data is mirrored between the nodes over SMB 3. No external storage array is needed, but you need the Datacenter edition, certified hardware and a fast network — 10 Gbit/s and above with RDMA is recommended (S2D overview).
  • SMB 3 file server — the machines live on a file share, usually on a clustered Scale-Out File Server.

CSV shared volumes

The ordinary NTFS file system is not designed to be mounted by several computers at once: each of them would consider itself the owner of the metadata, and the volume would fall apart. That is why the disks of a Hyper-V cluster are turned into CSV (Cluster Shared Volume). All nodes read and write machine files directly, while metadata changes — creating files, changing size — are carried out by one node, the volume coordinator. On every node a CSV is visible at the same path C:\ClusterStorage\VolumeN (CSV).

The file system for a CSV depends on the storage: volumes on a SAN are formatted with NTFS — then the nodes write to the disk directly (Direct I/O), while ReFS on a SAN switches the CSV into redirected mode, where all I/O goes through the coordinator over the network. S2D volumes, on the contrary, are formatted with ReFS (same source).

Quorum

The cluster has to decide which of the nodes have the right to keep working if the network between them breaks. Otherwise the two halves of the cluster would start the same machines on the same disks — this is split brain, and the data gets corrupted. The solution is quorum: every node has a vote, and only the part that has more than half of the votes keeps working. The smaller part stops the cluster service.

With an even number of nodes the votes can split equally, so a witness is added — one more vote, which goes to the half that reaches it first (quorum witness):

  • Disk witness — a small shared cluster disk with no drive letter and no other data.
  • File share witness — an SMB share on a server outside the cluster. Starting with Windows Server 2019 it can be a share on a machine that is not domain-joined. DFS and replicated storage are not supported: a replica can "split" the witness in two (file share witness).
  • Cloud witness — a blob in an Azure Storage account; the nodes need outbound HTTPS (TCP 443) (same source).

Microsoft recommends always configuring a witness: the cluster itself decides whether to count its vote with the current number of nodes. The witness must not live on a machine of the same cluster — otherwise, when it fails, the witness vote is lost as well.

Preparing the nodes

Every future node needs the Hyper-V role, the Failover Clustering feature, identical virtual switches with identical names (a machine looks up its switch by name when it moves) and access to the shared storage. The nodes are in the same domain. The account that creates the cluster needs the right to create computer objects in the nodes' OU: the cluster creates its own account in AD (CNO, Cluster Name Object), under which the cluster name works (creating a cluster).

$nodes = 'hv1', 'hv2'
Invoke-Command -ComputerName $nodes {
    Install-WindowsFeature -Name Failover-Clustering, Hyper-V -IncludeManagementTools -Restart
}
Invoke-Command -ComputerName $nodes { Get-VMSwitch | Select-Object Name, SwitchType }

Validation

Before creating the cluster — validation. Test-Cluster checks the hardware, network, storage and node settings and writes an HTML report; Microsoft supports a cluster only if its configuration has passed all tests (Test-Cluster):

Test-Cluster -Node $nodes -Include 'Inventory', 'Network', 'System Configuration', 'Storage', 'Hyper-V Configuration'

The command prints the path to the report at the end; all the warnings in it are worth reading, and the errors should be fixed before the cluster is created. The storage test temporarily takes the shared disks away from all nodes, so on a running cluster it is run only in a maintenance window.

Creating the cluster

The cluster is created with the name and address by which it will be reached, and without automatically grabbing disks, so that they are added deliberately (New-Cluster):

New-Cluster -Name hvclu -Node $nodes -StaticAddress 192.168.1.50 -NoStorage
Get-ClusterAvailableDisk -Cluster hvclu | Add-ClusterDisk
Get-ClusterResource -Cluster hvclu | Where-Object ResourceType -eq 'Physical Disk'
Add-ClusterSharedVolume -Cluster hvclu -Name 'Cluster Disk 1'
Set-ClusterQuorum -Cluster hvclu -FileShareWitness '\\fs1.corp.example.com\hvclu-witness$'

The parameters are in Add-ClusterSharedVolume and Set-ClusterQuorum. The cluster computer account hvclu$ needs write access to the witness folder — both in the share permissions and in NTFS.

For S2D, instead of adding SAN disks, the pool is enabled and the volumes are created right away as CSV:

Enable-ClusterStorageSpacesDirect -CimSession hvclu
New-Volume -CimSession hvclu -StoragePoolFriendlyName 'S2D*' -FriendlyName 'VMs' `
           -FileSystem CSVFS_ReFS -Size 2TB

Cluster networks

The cluster finds the nodes' networks by itself and assigns them a role: 0 — do not use, 1 — cluster traffic only, 3 — cluster and clients.

Get-ClusterNetwork -Cluster hvclu | Format-Table Name, Role, Address, Metric
(Get-ClusterNetwork -Cluster hvclu -Name 'Cluster Network 2').Role = 0    # iSCSI network
(Get-ClusterNetwork -Cluster hvclu -Name 'Cluster Network 3').Name = 'LiveMigration'

The iSCSI network is excluded from cluster communication so that heartbeat and migrations don't interfere with the disks. Live migration inside a cluster is configured not with Add-VMMigrationNetwork from the previous part, but through the cluster networks: by default it uses those that carry no client traffic. The networks that live migration must not use are listed in the MigrationExcludeNetworks parameter of the "Virtual Machine" resource type — as network IDs separated by semicolons, without spaces (Hyper-V team blog post):

Get-ClusterResourceType -Cluster hvclu -Name 'Virtual Machine' | Get-ClusterParameter -Name MigrationExcludeNetworks
Get-ClusterNetwork -Cluster hvclu | Format-Table Name, Id

Machines in the cluster

A machine becomes highly available only when it has been added to the cluster as a role. Its files must reside on a CSV (Add-ClusterVirtualMachineRole):

New-VM -Name app2 -Generation 2 -MemoryStartupBytes 4GB -SwitchName 'LAN' `
       -Path 'C:\ClusterStorage\Volume1' `
       -NewVHDPath 'C:\ClusterStorage\Volume1\app2\app2-os.vhdx' -NewVHDSizeBytes 80GB
Add-ClusterVirtualMachineRole -Cluster hvclu -VMName app2

A machine created on a node's local disk is first moved to a CSV with Move-VMStorage and only then added to the cluster. Moves within the cluster are managed by the cluster itself: Move-VM is not used on a clustered machine — it has Move-ClusterVirtualMachineRole instead (Move-ClusterVirtualMachineRole):

Move-ClusterVirtualMachineRole -Cluster hvclu -Name app2 -Node hv2 -MigrationType Live
Get-ClusterGroup -Cluster hvclu | Where-Object GroupType -eq 'VirtualMachine' |
    Format-Table Name, OwnerNode, State

-MigrationType Quick instead of Live saves the machine to disk and brings it up on the other node — a short downtime, but without copying memory over the network. Settings that change a clustered machine's configuration (disks, adapters) are better made through Failover Cluster Manager or followed by Update-ClusterVirtualMachineConfiguration, so that the cluster learns about the new configuration (Update-ClusterVirtualMachineConfiguration).

Looking at it by hand

The state of the cluster: nodes, CSV volumes with their coordinator and I/O mode, quorum.

Get-ClusterNode -Cluster hvclu | Format-Table Name, State, DrainStatus
Get-ClusterSharedVolumeState -Cluster hvclu | Format-Table Name, Node, StateInfo, FileSystemRedirectedIOReason
Get-ClusterSharedVolume -Cluster hvclu | Format-Table Name, OwnerNode, State
Get-ClusterQuorum -Cluster hvclu | Format-List

StateInfo equal to Direct means normal operation; FileSystemRedirected or BlockRedirected mean that the node writes to the volume through the coordinator over the network, and FileSystemRedirectedIOReason explains why. The OwnerNode of a CSV is its current coordinator.

Free space on the CSV volumes:

Get-ClusterSharedVolume -Cluster hvclu | ForEach-Object {
    $p = $_.SharedVolumeInfo.Partition
    [pscustomobject]@{ Name = $_.Name; Path = $_.SharedVolumeInfo.FriendlyVolumeName
                       FreeGB = [math]::Round($p.FreeSpace/1GB); SizeGB = [math]::Round($p.Size/1GB) }
}

Sources

A Hyper-V cluster is nodes with identical switches, shared storage in the form of CSV, quorum with a witness and machines added to the cluster as roles: when a node fails, its machines are restarted on its neighbours.

Next — part 10: maintaining a Hyper-V cluster. Pausing a node and draining its machines, Cluster-Aware Updating, automatic balancing and placement rules, a rolling upgrade of Windows Server on the nodes, a cluster without a domain and diagnostics from the logs.

Only registered users can leave a comment.

Sign in or register to leave a comment.

© 2006-2026 Vitaliy Leschenko