I have an 8 blade cluster running Windows 2008R2 SP1. We are running about 350 virtual Windows 7 workstations that users RDP into via Wyse Win7e thin clients. The servers each are running dual 12 core Intel processors with about 96GB of memory. Each has dual active/active 8Gbps fiber channel paths each to a separate fiber channel switch which has 4 8Gbps paths each to an EMC CX4-120 SAN. I have 4 RAID5 disks groups on the EMC setup to give the cluster it's Clustered Storage Disks: Volume1 through Volume4. The VHDs and XML live on the same CSV and for the most part those virtual machine files are spread across these 4 CSVs pretty evenly and utilization is pretty low on the drives. The SAN also have 100GB of FAST cache SSD drives to also further improve performance.
The issue is occasionally I get an error on a specific blade where a virtual will blue screen and reboot. The event log "Microsoft-Windows-VHDMP/Operational" will log 4 events:
Error; Event ID 6 "Failed to write data to VHD c:\clusterstorage\VolumeX\Win7WorkstationXXX.VHD. Error status 0xC000009A"
Error; Event ID 4 "Failed to surface VHD c:\clusterstorage\VolumeX\Win7WorkstationXXX.VHD. Surface attempt was cancelled"
Error; Event ID 6 "Failed to write data to VHD c:\clusterstorage\VolumeX\Win7WorkstationXXX.VHD. Error status 0xC000009A"
Informational; Event ID 2 "The VHD c:\clusterstorage\VolumeX\Win7WorkstationXXX.VHD has been removed (unsurfaced) as disk number 0."
If I migrate the virtual machine with the issue to another blade the error stops and the machine runs fine until the next random virtual machine starts doing the same thing. There does not seem to be a rhyme or reason behind when the next VM will have issues; could be a an hour, could be days. This can happen on any virtual running on any of the 8 blades.
Any ideas what causes this?