We have a multi node Hyper V Cluster that has recently developed an issue with intermittent failure of live migrations.
We noticed this when one of our CAU runs failed because it could not place the Hosts into maintenance mode or successfully drain all the roles from them.
Scenario:
Place any node into Maintenance mode/drain roles.
Most VM's will drain and live migrate across onto other nodes. Randomly one or a few will refuse to move (it always varies in regards to the VM and which node it is moving to or from). The live migration ends with a failure generating event ID's 21502, 22038, 21111, 21024. If you run the process again (drain roles) it will migrate the VM's or if you manually live migrate them they will move just fine. Manually live migrating a VM can result in the same intermittent error but rerunning the process will succeed after one or two times or just waiting for a couple minutes.
This occurs on all Nodes in the cluster and can occur with seemingly any VM in the private cloud.
Pertinent content of the event ID's is:
Event 21502
Live migration of 'VM' failed.
Virtual machine migration operation for 'VM' failed at migration source 'NodeName'. (Virtual machine ID xxx)
Failed to send data for a Virtual Machine migration: The process cannot access the file because it is being used by another process. (0x80070020).
Event 22038
Failed to send data for a Virtual Machine migration: The process cannot access the file because it is being used by another process. (0x80070020).
According to this it would appear that something is locking the files or they are not transferring permissions properly, however all access to the back end SOFS is uniform across all the Nodes and the failure is intermittent rather than consistently happening on one Node.
Thanks in advance!