Hyper-V. Part 10: Maintaining a Hyper-V cluster
All parts of the series “Hyper-V”In the previous part we built a cluster and added machines to it. Now it has to be maintained: install updates on the nodes every month, make sure the load is spread evenly and copies of one service don't end up on the same node, move the cluster to a new Windows Server every few years, and figure out why machines suddenly moved. This is what the last part of the series is about.
Pausing a node
Before an update or a repair, a node is paused with its roles "drained": the cluster spreads its machines across the other nodes by live migration and does not place new ones on it (Suspend-ClusterNode):
Suspend-ClusterNode -Cluster hvclu -Name hv1 -Drain -Wait
Get-ClusterNode -Cluster hvclu | Format-Table Name, State, DrainStatus
# update, reboot
Resume-ClusterNode -Cluster hvclu -Name hv1 -Failback Immediate
DrainStatus shows the progress of the drain: InProgress, Completed or Failed — the last one means that some machine could not be moved, and the node must not be shut down until it has been moved off manually. -Failback Immediate returns to the node the machines that were on it before the pause; without it they stay where they were moved.
When the whole cluster is shut down rather than one node, use Stop-Cluster: it stops the roles and the cluster service on all nodes (Stop-Cluster).
Cluster-Aware Updating
There is no need to do this by hand for every node and every monthly update: Cluster-Aware Updating (CAU) goes through the nodes one by one — drains, installs updates, reboots, returns the machines. It runs either as a role inside the cluster on a schedule (self-updating) or is started from an external coordinator computer (remote-updating) (CAU):
Invoke-CauRun -ClusterName hvclu -CauPluginName Microsoft.WindowsUpdatePlugin -MaxFailedNodes 0 -RequireAllNodesOnline -Force
Add-CauClusterRole -ClusterName hvclu -DaysOfWeek Sunday -WeeksOfMonth 2 -EnableFirewallRules -Force
Get-CauReport -ClusterName hvclu -Last -Detailed
The first command is a one-off run from a workstation, the second is a role that updates the cluster by itself on the second Sunday of the month. -MaxFailedNodes 0 stops the run at the first node that could not be updated, and -RequireAllNodesOnline does not let it start if some node is already unavailable: updating a cluster that is already short of a node is dangerous. Get-CauReport shows what was installed and where (Invoke-CauRun).
Balancing
The cluster can even out the load by itself: it estimates the memory and processor load of the nodes and moves machines off overloaded ones by live migration. The mode is set by the AutoBalancerMode property: 0 — off, 1 — when a node joins the cluster (the default in Windows Server), 2 — when a node joins and every 30 minutes. The threshold is set by AutoBalancerLevel: 1 — move from a node loaded more than 80% (the default), 2 — more than 70%, 3 — when a node is loaded 5% above the cluster average (VM load balancing).
(Get-Cluster hvclu) | Format-List AutoBalancerMode, AutoBalancerLevel
(Get-Cluster hvclu).AutoBalancerLevel = 2
Placement rules
So that two copies of one service — two domain controllers, two nodes of a web farm — don't end up on the same node, the machines are given the same AntiAffinityClassNames value: the cluster will spread them across different nodes whenever possible.
$cls = New-Object System.Collections.Specialized.StringCollection; $cls.Add('DC') | Out-Null
(Get-ClusterGroup -Cluster hvclu -Name dc1).AntiAffinityClassNames = $cls
(Get-ClusterGroup -Cluster hvclu -Name dc2).AntiAffinityClassNames = $cls
Get-ClusterGroup -Cluster hvclu | Format-Table Name, OwnerNode, AntiAffinityClassNames
"Whenever possible" means that when there are not enough nodes, the cluster will still start both machines on one — the rule is soft. The nodes a machine is allowed to run on are restricted with a list of preferred and possible owners: Set-ClusterOwnerNode (Set-ClusterOwnerNode).
Set-ClusterOwnerNode -Cluster hvclu -Group dc1 -Owners hv1, hv2
The order in which machines start after a node failure is set by the group priority: (Get-ClusterGroup -Cluster hvclu -Name dc1).Priority = 3000 — high (3000), medium (2000, the default), low (1000), no auto start (0). Domain controllers and databases are brought up first, test machines last or not at all.
Upgrading the cluster OS
A cluster is moved to a new version of Windows Server without machine downtime by a rolling upgrade (Cluster OS Rolling Upgrade). The nodes are emptied one at a time, reinstalled with the new OS and returned to the cluster. While the cluster has nodes of different versions, it runs in mixed mode; it is recommended to complete this stage within four weeks and to manage the cluster from a new node. The step is only one version at a time: from 2022 to 2025, and from 2016 to 2025 through all the intermediate ones. The process is reversible until the last step — raising the cluster functional level (rolling upgrade):
Get-Cluster hvclu | Select-Object Name, ClusterFunctionalLevel
Get-ClusterNode -Cluster hvclu | Format-Table Name, MajorVersion, MinorVersion, BuildNumber
Update-ClusterFunctionalLevel -Cluster hvclu
Get-VM -ComputerName hv1, hv2 | Where-Object State -eq 'Off' | Update-VMVersion -Force
The machines' configuration version is raised after the functional level: it requires the machine to be shut down and is irreversible — see the part on moving machines. The last line updates only the machines that are off; running ones are updated one at a time in maintenance windows.
A cluster without a domain
A cluster can also be built from servers that are not domain members (a workgroup cluster): an identical local account is created on every node, the nodes add each other to TrustedHosts and get a common primary DNS suffix, and the cluster name is registered only in DNS (workgroup cluster):
New-Cluster -Name wgclu -Node hv7, hv8 -StaticAddress 192.168.1.60 -NoStorage -AdministrativeAccessPoint DNS
Hyper-V machines in such a cluster are supported starting with Windows Server 2025; file servers and SQL Server FCI are not. This is an option for a branch office without a domain controller, but everything that relies on Kerberos — delegation for live migration, access to file shares with domain accounts — will have to be replaced in it with certificates and local accounts.
Diagnostics
The cluster event log for the last hour is the first place to look after machines have suddenly moved:
Get-WinEvent -ComputerName hv1 -FilterHashtable @{
LogName = 'Microsoft-Windows-FailoverClustering/Operational'; StartTime = (Get-Date).AddHours(-1)
} | Format-Table TimeCreated, Id, Message -Wrap
Critical cluster events also go to the System log with the source Microsoft-Windows-FailoverClustering: for example, 1135 — a node was removed from the cluster because it stopped responding to heartbeat, and 1069 — a cluster resource went into a failed state (Failover Clustering events):
Get-WinEvent -ComputerName hv1 -FilterHashtable @{
LogName = 'System'; ProviderName = 'Microsoft-Windows-FailoverClustering'; StartTime = (Get-Date).AddDays(-1)
} | Format-Table TimeCreated, Id, Message -Wrap
If the event log is not enough, Get-ClusterLog collects the detailed log of the cluster service from all nodes into text files (Get-ClusterLog):
Get-ClusterLog -Cluster hvclu -TimeSpan 60 -Destination C:\Temp -UseLocalTime
-TimeSpan is how many minutes back to start, and -UseLocalTime writes the time by the node's clock rather than in UTC, so that it is easier to match with the event logs.
The state of roles that are not running right now — machines the cluster could not start:
Get-ClusterGroup -Cluster hvclu | Where-Object State -ne 'Online' | Format-Table Name, OwnerNode, State
Get-ClusterResource -Cluster hvclu | Where-Object State -ne 'Online' | Format-Table Name, ResourceType, OwnerGroup, State
Sources
- Suspend-ClusterNode — pausing a node and draining its roles;
- Cluster-Aware Updating overview and Invoke-CauRun — updating the cluster;
- Virtual machine load balancing — balancing modes and thresholds;
- Set-ClusterOwnerNode — group owners;
- Cluster operating system rolling upgrade — rolling upgrade of the cluster OS and the functional level;
- Create a workgroup cluster — a cluster without a domain;
- Troubleshoot cluster issue with Event ID 1135 — the node removal event;
- Get-ClusterLog — the detailed log of the cluster service;
- New-Cluster and Stop-Cluster — a DNS-only administrative access point, stopping the cluster.
This is where the series ends. We have gone from the first machine on a standalone host to a cluster that updates itself, balances the load and restarts the machines of a failed node: the hypervisor and the root partition, VMBus and integration services, VHDX disks and switches with VLANs, processor and memory, checkpoints, management from a workstation, live migration and CSV shared volumes. The full command reference is in the documentation of the Hyper-V and FailoverClusters modules.
Only registered users can leave a comment.
Sign in or register to leave a comment.