Hyper-V. Part 8: Moving virtual machines between hosts

All parts of the series “Hyper-V”

Until now, a machine lived on one host. But a host has to be updated, repaired and one day replaced, and the machines have to be moved off it. Let's go through all the methods: export and import, live migration of a running machine between hosts without a cluster, moving a machine's files to another volume, what prevents a move between different processors, when to raise the configuration version after moving to a new Windows Server, and how to keep a copy of a machine on another host in case of a disaster.


Export and import

The simplest way to move a machine is to export it to a folder and import it on another host (export and import). Export works with a running machine too: the folder receives the configuration, disks and checkpoints.

Export-VM -Name app1 -Path '\\fs1\export'

Import has three modes (Import-VM):

  • Register in place — the machine stays where its files are, with the same ID.
  • Restore (-Copy) — the files are copied to the specified folders, the ID is kept.
  • Copy (-Copy -GenerateNewId) — the machine gets a new ID; this is how several machines are made from one exported machine.
$vmcx = Get-ChildItem '\\fs1\export\app1\Virtual Machines\*.vmcx'
Import-VM -Path $vmcx.FullName -Copy `
          -VirtualMachinePath 'D:\Hyper-V' -VhdDestinationPath 'D:\Hyper-V\Virtual Hard Disks\app1'

If the new host has no switch with the same name or the configuration version isn't supported, the import fails. Compare-VM helps check this in advance and fix incompatibilities: it returns a report with incompatibility objects that can be fixed and passed to Import-VM -CompatibilityReport (Compare-VM).

$report = Compare-VM -Path $vmcx.FullName -Copy -GenerateNewId
$report.Incompatibilities | Format-Table Message, MessageId
$report.Incompatibilities[0].Source | Connect-VMNetworkAdapter -SwitchName 'LAN'
Import-VM -CompatibilityReport $report

The Source of an incompatibility is the object it relates to: here, a network adapter for which no switch was found, and it is connected to an existing one.

Live migration without a cluster

Live migration moves a running machine to another host without noticeable downtime (overview). Memory is copied to the new host while the machine keeps running; pages changed in the meantime are sent in repeated passes, and at the end the machine freezes for a fraction of a second to transfer the last remainder and the processor state. Since Windows Server 2016 this doesn't require a cluster: two standalone hosts of the same domain move a machine together with its disks — such a move is called shared nothing, as the hosts have no shared storage.

The setup is done on both hosts (setting up hosts):

Enable-VMMigration
Add-VMMigrationNetwork 192.168.40.0/24
Set-VMHost -VirtualMachineMigrationAuthenticationType Kerberos `
           -VirtualMachineMigrationPerformanceOption Compression `
           -MaximumVirtualMachineMigrations 2 -MaximumStorageMigrations 2
Get-VMHost | Format-List VirtualMachineMigrationEnabled, VirtualMachineMigrationAuthenticationType,
    VirtualMachineMigrationPerformanceOption, UseAnyNetworkForMigration
Get-VMMigrationNetwork

Migration network. Add-VMMigrationNetwork adds a network the host accepts migration on, Set-VMMigrationNetwork changes one already added, Remove-VMMigrationNetwork removes it (Add-VMMigrationNetwork). While UseAnyNetworkForMigration is off, the host accepts migration only on the networks in this list, and with an empty list it accepts none at all (Set-VMHost). Without a separate network gigabytes of memory will go through the same card as user traffic. Usually it is a separate host adapter in its own VLAN — the LiveMigration adapter we added on top of SET in the part about networking. Set-VMHost -UseAnyNetworkForMigration $true removes the restriction.

Memory transfer method (same source): TCPIP — as is, Compression — compressed (the default), SMB — over SMB 3 with SMB Multichannel and, on RDMA cards, SMB Direct. On RDMA cards the SMB option is the fastest; without RDMA compression usually wins. MaximumVirtualMachineMigrations and MaximumStorageMigrations set how many migrations the host performs simultaneously.

Authentication

Migration is started from the source host, and the source needs to act on the destination host on behalf of the administrator. There are two options:

  • CredSSP works without domain configuration, but only if the administrator is logged on to the source host interactively — at the console, over RDP or in a PowerShell session on it. Migration can't be started from a workstation this way.
  • Kerberos with constrained delegation: the source host is allowed to obtain tickets on behalf of the user to two services of the destination — Microsoft Virtual System Migration Service for the machine itself and cifs for its disks. Then migration can be started remotely, from an administrator's workstation.

On Windows Server 2025 the choice is effectively made for you: Credential Guard is enabled by default on all domain members except domain controllers, and live migration over CredSSP stops working — Kerberos is required (same source).

Setting up CredSSP

CredSSP needs no changes in Active Directory: on Windows Server 2022 and earlier it is the default method. If Kerberos was selected on the hosts before, CredSSP is set back explicitly on both:

Invoke-Command -ComputerName hv1, hv2 { Set-VMHost -VirtualMachineMigrationAuthenticationType CredSSP }

From there everything depends on where Move-VM runs. The administrator's credentials have to end up on the source host, so the migration is started in a session opened on that host itself (same source):

  • at the console or over RDP on the source host — Move-VM works right away, with no extra setup;
  • in a remote PowerShell session — only if the session is opened with CredSSP authentication. An ordinary Kerberos session doesn't pass the credentials to the host, and the migration fails just as it does without signing in.

CredSSP sessions to the hosts are enabled once (Enable-WSManCredSSP): on the workstation — for all hosts that machines will leave, by their FQDN; on the hosts — the server role:

Enable-WSManCredSSP -Role Client -DelegateComputer 'hv1.corp.example.com', 'hv2.corp.example.com' -Force   # on the workstation
Invoke-Command -ComputerName hv1, hv2 { Enable-WSManCredSSP -Role Server -Force }                          # on the hosts
Get-WSManCredSSP

And the move itself from the workstation:

$cred = Get-Credential 'CORP\hv-admin'
Invoke-Command -ComputerName hv1.corp.example.com -Authentication CredSSP -Credential $cred {
    Move-VM -Name app1 -DestinationHost hv2 -IncludeStorage -DestinationStoragePath 'D:\Hyper-V\app1'
}

The name in -ComputerName must match the one given in -DelegateComputer, hence the FQDN. How CredSSP is enabled for PowerShell Remoting and why it is dangerous is covered in the part about WinRM of the Windows Server series.

CredSSP has two limitations. The session is needed on the host the machine leaves: to bring it back from hv2 to hv1, the session is opened on hv2, otherwise the migration fails with "No credentials are available in the security package 0x8009030E" (same source). And the administrator's credentials are handed to the source host in full: CredSSP doesn't work for members of the Protected Users group, and on Windows Server 2025, as said above, Credential Guard blocks it. That is why Kerberos is chosen wherever possible.

Setting up constrained delegation

Delegation is configured in Active Directory, on the hosts' computer accounts, so you need an account allowed to change them: the documentation requires membership in Domain Admins. The hosts must be in the same domain or in domains that trust each other (same source).

There is one rule: every host that machines leave is allowed to delegate to the services of every host they arrive at. For the pair hv1 and hv2 that is two settings: the hv2 services on the hv1 account, and the hv1 services on the hv2 account. There are two services, and each is listed in two spellings — the short name and the FQDN, because a host may address its neighbor either way:

Service What for
Microsoft Virtual System Migration Service/<host> moving the machine itself — memory and state
cifs/<host> moving disks (Move-VM -IncludeStorage) and storage to another host

Hyper-V registers the migration service SPN on the host's account by itself when the role is installed: setspn -L hv2 shows Microsoft Virtual System Migration Service/hv2 and Microsoft Virtual System Migration Service/hv2.corp.example.com. There are no separate entries for cifs — it is served by the HOST/… entries every domain computer has.

In the "Active Directory Users and Computers" snap-in this is the Delegation tab in the properties of the source host's account: "Trust this computer for delegation to specified services only" → "Use any authentication protocol" → "Add" → the destination host → the two services from the list. In PowerShell — for all host pairs at once; the script runs on any machine with the ActiveDirectory module:

$hosts  = 'hv1', 'hv2', 'hv3'
$domain = 'corp.example.com'
foreach ($src in $hosts) {
    $spns = foreach ($dst in $hosts | Where-Object { $_ -ne $src }) {
        foreach ($svc in 'Microsoft Virtual System Migration Service', 'cifs') { "$svc/$dst"; "$svc/$dst.$domain" }
    }
    $have = (Get-ADComputer $src -Properties msDS-AllowedToDelegateTo).'msDS-AllowedToDelegateTo'
    $add  = $spns | Where-Object { $_ -notin $have }
    if ($add) { Set-ADComputer -Identity $src -Add @{ 'msDS-AllowedToDelegateTo' = [string[]]$add } }
    Set-ADAccountControl -Identity "$src$" -TrustedToAuthForDelegation $true
}

The msDS-AllowedToDelegateTo attribute is the list of services the account may delegate to — the very list on the Delegation tab. The TrustedToAuthForDelegation flag corresponds to the "Use any authentication protocol" option. The script only adds missing services, so it can be run again — for example, when a new host appears — and it leaves delegation to other services alone if there was any.

To check the result:

Get-ADComputer -Filter "Name -like 'hv*'" -Properties msDS-AllowedToDelegateTo, TrustedToAuthForDelegation |
    Select-Object Name, TrustedToAuthForDelegation, @{n='DelegateTo'; e={$_.'msDS-AllowedToDelegateTo' -join "`n"}} |
    Format-List

Then Kerberos is selected on every host — with the same Set-VMHost command as at the beginning of the section; on Windows Server 2022 and earlier the default there is CredSSP:

Invoke-Command -ComputerName $hosts { Set-VMHost -VirtualMachineMigrationAuthenticationType Kerberos }

The changes in AD take effect once they have replicated to the domain controllers the hosts talk to and the hosts have obtained new Kerberos tickets (same source). To avoid waiting, purge the computer account's ticket cache on every host:

Invoke-Command -ComputerName $hosts { klist purge -li 0x3e7 }

0x3e7 is the SYSTEM logon session the VMMS service runs in. After that, migration can be started right from the workstation. If delegation is set up only partly — in one direction or without cifs — a move in the other direction or a move with disks fails with an authentication error on the source host, and the details are in the Microsoft-Windows-Hyper-V-VMMS-Admin log.

The move

The move itself is one command (live migration without a cluster, Move-VM):

Move-VM -Name app1 -DestinationHost hv2 -IncludeStorage -DestinationStoragePath 'D:\Hyper-V\app1'

From a workstation, add -ComputerName hv1 — the host the machine is currently on. -IncludeStorage moves the disks too; without it, Move-VM moves only the memory and state, and the disks must be accessible to both hosts — for example, located on a shared folder. Migration progress is visible in Get-VM as Status, and errors in the Microsoft-Windows-Hyper-V-VMMS-Admin log.

With -IncludeStorage the disks on the destination don't land directly in -DestinationStoragePath but in its Virtual Hard Disks subfolder, next to Virtual Machines, Snapshots and the machine's other folders; Move-VMStorage lays the files out the same way. On the source an empty tree of the machine's folders remains after the move — it is deleted manually.

The move can also be started by an administrator from another domain that the hosts' domain trusts: delegation is configured on the hosts' accounts, and the user only obtains tickets for their own account. The machine's Uptime on the new host starts from zero — this is not a restart: the guest keeps running, the counter on the destination simply starts again. Whether the move really was live is shown by the log — more on that at the end of the part.

Storage migration

Sometimes you don't need to move a machine to another host, but to move its files to another volume of the same host — for example, to free up a disk that is running out of space. This is also done on a running machine (Move-VMStorage):

Move-VMStorage -VMName app1 -DestinationStoragePath 'E:\Hyper-V\app1'

This moves all of the machine's files at once — configuration, disks, checkpoints, the Smart Paging file. The -VirtualMachinePath, -SnapshotFilePath and -VHDs parameters let you spread them across different volumes.

Processor compatibility

A running machine sees the processor instruction set of its host. If the processor on the new host is older and lacks some instructions, migration is not allowed. Processor compatibility mode hides extended instructions from the machine, and it can move between different processor generations of the same vendor; live migration between Intel and AMD is impossible in any case (processor compatibility). It is enabled while the machine is off:

Stop-VM -Name app1
Set-VMProcessor -VMName app1 -CompatibilityForMigrationEnabled $true
Start-VM -Name app1

In Windows Server 2025 the mode became dynamic: in a cluster it calculates the set of capabilities common to all nodes instead of cutting the machine down to the minimum (same source).

Configuration version after a move

A machine moved or imported to a host with a newer Windows Server keeps its configuration version from part two and can go back to the old host. Updating requires a shutdown and is irreversible: the version can't be lowered afterwards, and a machine with the new version won't start or import on older hosts (Update-VMVersion).

Get-VM | Sort-Object Version | Format-Table Name, State, Version
Stop-VM -Name app1
Update-VMVersion -Name app1 -Force
Start-VM -Name app1

The order when moving to a new Windows Server: first all hosts, then the machine versions — in maintenance windows, one at a time. As long as there is still at least one old host in use that a machine might need to go back to, the version is left alone. In a cluster there is one more step between these — raising the cluster functional level, covered in the part about cluster maintenance.

Hyper-V Replica

Live migration is a planned move. For a disaster, when the source host is already unavailable, there is Hyper-V Replica: asynchronous replication of a machine to another host at an interval of 30 seconds, 5 or 15 minutes, with a history of up to 24 recovery points, with Kerberos authentication inside a domain or certificate authentication between independent hosts (Hyper-V Replica overview). The replica stays off until it is started manually; data for the last interval can be lost.

# on the destination
Set-VMReplicationServer -ReplicationEnabled $true -AllowedAuthenticationType Kerberos `
    -ReplicationAllowedFromAnyServer $true -DefaultStorageLocation 'D:\Replica'
# on the source
Enable-VMReplication -VMName app1 -ReplicaServerName hv2.corp.example.com `
    -ReplicaServerPort 80 -AuthenticationType Kerberos -ReplicationFrequencySec 300
Start-VMInitialReplication -VMName app1
Measure-VMReplication -VMName app1

The parameters are described in Enable-VMReplication. Port 80 here is the replication HTTP listener; with Kerberos the data is not encrypted; for encryption, certificate authentication and port 443 are used. On the destination you also need to open the inbound port in the firewall — the "Hyper-V Replica HTTP Listener (TCP-In)" rule.

Failover to the replica is done on the destination when the source is unavailable (Start-VMFailover):

Start-VMFailover -VMName app1          # on the destination: prepare the replica for starting
Start-VM -Name app1
Complete-VMFailover -VMName app1       # commit the choice of recovery point

A planned failover, when the source is alive, is started on the source with Start-VMFailover -Prepare on a shut-down machine: then the latest changes have time to reach the replica, and no data is lost.

Looking at it by hand

Replication state of all machines on the host:

Get-VMReplication | Format-Table VMName, State, Health, Mode, PrimaryServer, ReplicaServer, LastReplicationTime

How the recent live migrations went — events 20413 (the source started the move), 20415 (the source finished; the text gives the time the machine was frozen, the blackout time) and 20418 (the destination received the machine):

Get-WinEvent -FilterHashtable @{ LogName = 'Microsoft-Windows-Hyper-V-VMMS-Admin'; Id = 20413, 20415, 20418; StartTime = (Get-Date).AddDays(-1) } |
    Format-Table TimeCreated, Id, Message -Wrap

The source writes 20413 and 20415, the destination writes 20418, so the full picture comes from the logs of both hosts. For a small machine the pause is a fraction of a second; it grows with the amount of memory the machine manages to change during copying.

Recent migration errors:

Get-WinEvent -FilterHashtable @{ LogName = 'Microsoft-Windows-Hyper-V-VMMS-Admin'; Level = 2; StartTime = (Get-Date).AddDays(-1) } |
    Format-Table TimeCreated, Id, Message -Wrap

Sources


A machine can be moved while it is off — by export and import, or while it is running — by live migration together with its disks, and for a disaster you can keep its replica on another host. But all these moves are started by an administrator; for machines to restart on a healthy host by themselves, you need a cluster.

Next — part 9: building a failover cluster. What a cluster gives and what it doesn't, shared storage options, CSV shared volumes, quorum and witnesses, validating and building the cluster, its networks and adding machines to the cluster.

Only registered users can leave a comment.

Sign in or register to leave a comment.