We have two Hyper V 2012 R2 (HVHOST 1 and 2) servers both with Server 2012 R2. These servers communicate via ISCI over a Dell switch to a MD3200i. All the VM's live on a single CSV virtual disk.
At first we thought the cluster was crashing just at points of high I/O - like during a backup. Then it crashed when IO was very low. Each crash we see this error message: 'Cluster Shared Volume X has entered a paused stated because of '(c00000)9d). All I/O will be temporarily be queued until a path to the volume is reestablished.' We lose connection to the CSV. Rebooting each HVHOST reestablishes this connection.
We see the below event on both HV Host servers occurring. During heavy I/O the frequency of this message increases. During low I/O this message is less frequent. 'Dell MD Series Device Specific Module for Multi-Path failed to return a path to \Device\MPIODisk1.' This MPIO Disk 1 is the CSV disk.
Windows Failover Cluster Networks
HMI - cluster and client traffic - LCAP 2GB throughput - connections to a switch
RMC - cluster only traffic - Dynamic 1GB throughput - point to point connection between HVHOST1 and HVHOST2
SAN/ISCI - Cluster traffic: none
Items Already Checked: - on both hosts
- Updates installed. https://support.microsoft.com/en-us/kb/2920151
- No AV
- Windows Firewall turned off
- Networks drivers failed to update on HVHOST2. Working with Dell to resolve this.
- BIOS are out dated
I realize the BIOS drivers and out dated drivers on HVHOST2 may be an issue. However should either of these cause a lack of communication with the CSV?
Any other ideas?
Thanks in advance.