I have a msa san with 4 hosts running server 2008 r2 core and hyper v in a cluster.
I recently expanded the cluster shared volume. Everything seemed ok as all nodes were able to see the now larger CSV. I then was able to increase the vhd of one of my virtual machines. After a few days random files on all my VM's started saying they were corrupted and inaccessible. Some VM's got stuck in a stopping state and some completely stopped, while 2 other VM's stayed running with a couple corrupted folders. I was also getting numerous errors in scvmm when trying to do anything, such as create a new VM:
Warning (2912)
An internal error has occurred trying to contact an agent on the FHNJHV8E03.pdc1.com server.
(The file or directory is corrupted and unreadable (0x80070570))
Recommended Action
Ensure the agent is installed and running. Ensure the WS-Management service is installed and running, then restart the agent.
I contacted microsoft for support. They kept escalating my issue, until i got a cluster expert who decided to remove the cluster during troubleshooting. The CSV volume immediately went into RAW state. I was then escalated to a disk specialist who attempted to recover the CSV volume by using diskprobe2. Microsoft was unable to recover the CSV volume and said it would have to be formatted.
Any help or ideas before i am stuck formatting the CSV volume and rebuilding each VM would be great. As i only have backups of the files on the VM's
or even ideas of third party recovery.