Hi Everybody,
We had a strange problem yesterday with a hypervisor running 2012 R2 where we lost the ability to manage it through the GUI. At first we noticed that SCVMM was flagging an alert for the hypervisor, we attempted a refresh and received the errors:
Error (2606)
Unable to perform the job because one or more of the selected objects are locked by another job.
Recommended Action
To find out which job is locking the object, in the Jobs view, group by Status, and find the running or canceling job for the object. When the job is complete, try again.
Error (2912)
An internal error has occurred trying to contact the capsoftashapp01.capsoft.local server: : .
WinRM: URL: [http://capsoftashapp01.capsoft.local:5985], Verb: [ENUMERATE], Resource: [http://schemas.microsoft.com/wbem/wsman/1/wmi/root/standardcimv2/MSFT_NetAdapter], Filter: []
Unknown error (0x80041013)
Recommended Action
Check that WS-Management service is installed and running on server capsoftashapp01.capsoft.local. For more information use the command "winrm helpmsg hresult". If capsoftashapp01.capsoft.local is a host/library/update server or a PXE server
role then ensure that VMM agent is installed and running. Refer to http://support.microsoft.com/kb/2742275 for more details.
Error (2911)
Insufficient resources are available to complete this operation on the capsoftashapp01.capsoft.local server.
Not enough storage is available to complete this operation (0x8007000E)
Recommended Action
Ensure that the virtual machine host has sufficient memory and disk space to perform this action. Try the operation again.
There were no resource issues on the hypervisor. Restarting the VMM agent and WinRM didn't resolve the issue.
The issue degraded further whereby we couldn't connect to the hypervisor through Hyper-V manager (remotely or locally) and could no longer connect to the cluster through Failover Cluster Manager (remotely, including the other cluster node or, locally).
When attempting to connect to the cluster, the following error occurred:
Log Name: Microsoft-Windows-FailoverClustering-Manager/Admin
Source: Microsoft-Windows-FailoverClustering-Manager
Date: 04/02/2015 16:16:59
Event ID: 4683
Task Category: MMC Snapin
Level: Error
Description:
Failover Cluster Manager failed while managing one or more cluster. The error was 'An error occurred trying to display the cluster information.'. For more information see the Failover Cluster Manager Diagnostic channel.
However, we were able to migrate the VMs from one hypervisor to the other using powershell then, reboot the hypervisor that had the issue. As soon as the reboot began, were able to connect to the cluster from the other node through Failover Cluster Manager. Once the reboot had been completed we were able to connect to the hypervisor through SCVMM & Hyper-V (remotely and locally).
At the moment I can't find any reason for the problem. The cluster and hypervisor were configured about 2 months ago and have been running fine. Although, it was the first time I'd used powershell to manage a cluster, it appeared as if the cluster was running fine - almost as if the issue was something that only affected the GUI. The hosted VMs were also running fine.
Does anyone have any idea what might have caused this? And how to avoid the problem re-occurring?
Any help would be greatly appreciated.
Steve