Hello Experts,
Our SQL Server with Failover Clustering has one node (let's call it SQLNODE2) is an unhealthy state. The network adapters for SQLNODE2 are marked as "Unavailable" but in the cluster.log and event log, it indicates an Event ID 1230 "Cluster resource 'Cluster Disk 3' (resource type '', DLL 'clusres.dll') either crashed or deadlocked. The Resource Hosting Subsystem (RHS) process will now attempt to terminate, and the resource will be marked to run in a separate monitor." Within the Failover Cluster Manager, this 'Cluster Disk 3' is labelled "Online" and I am able to access the files on it.
The other node (let's call it SQLNODE1) remains seemingly healthy and the SQL Server is running normally but this issue breaks our failover ability. I'm unable to apply windows updates to SQLNODE1 since bringing it offline for any amount of time will result in losing SQL Server connectivity and I'm afraid it may not come back up normally. Any help you can provide would be greatly appreciated. Please see the cluster.log below:
00000fa8.00000fb4::2015/04/17-16:27:24.068 ERR [RHS] RhsCall::DeadlockMonitor: Call OPENRESOURCE timed out for resource 'Cluster Disk 3'.00000fa8.00000fb4::2015/04/17-16:27:24.068 ERR [RHS] Resource Cluster Disk 3 handling deadlock. Cleaning current operation and terminaiting RHS process.
00000b70.00001148::2015/04/17-16:27:24.068 INFO [NM] Received request from client address SQLNODE2.
00000b70.00000e90::2015/04/17-16:27:24.068 INFO [RCM] HandleMonitorReply: FAILURENOTIFICATION for 'Cluster Disk 3', gen(0) result 4.
00000b70.00000e90::2015/04/17-16:27:24.068 WARN [RCM] rcm::RcmResource::HandleMonitorReply: Resource 'Cluster Disk 3' has crashed or deadlocked; marking it to run in a separate monitor.
00000b70.00000e90::2015/04/17-16:27:24.068 INFO [GUM] Node 2: Processing RequestLock 2:1
00000b70.00000db8::2015/04/17-16:27:24.068 ERR [RCM] rcm::RcmMonitor::RecoverProcess: Recovering monitor process 4008 / 0xfa8
00000b70.00000db8::2015/04/17-16:27:24.068 INFO [RCM] Created monitor process 2748 / 0xabc
00000abc.00001228::2015/04/17-16:27:24.083 INFO [RHS] Initializing.
00000b70.00000db8::2015/04/17-16:27:24.083 INFO [RCM] rcm::RcmResource::ReattachToMonitorProcess: (Cluster IP Address, Initializing)
00000b70.00000db8::2015/04/17-16:27:24.083 INFO [RCM] TransitionToState(Cluster IP Address) Initializing-->OpenCallIssued.
00000b70.00000db8::2015/04/17-16:27:24.083 INFO [RCM] rcm::RcmResource::ReattachToMonitorProcess: (SQL IP Address 1 (SqlClusterName), Initializing)
00000b70.00000db8::2015/04/17-16:27:24.083 INFO [RCM] TransitionToState(SQL IP Address 1 (SqlClusterName)) Initializing-->OpenCallIssued.
00000b70.00000db8::2015/04/17-16:27:24.083 INFO [RCM] rcm::RcmResource::ReattachToMonitorProcess: (GxClusPlugIn (sqlclustername.domain.com) (Instance001), Initializing)
00000b70.00000db8::2015/04/17-16:27:24.083 INFO [RCM] TransitionToState(GxClusPlugIn (sqlclustername.domain.com) (Instance001)) Initializing-->OpenCallIssued.
00000b70.00000db8::2015/04/17-16:27:24.083 INFO [RCM] rcm::RcmResource::ReattachToMonitorProcess: (Cluster IP Address 10.100.128.57, Initializing)
00000b70.00000db8::2015/04/17-16:27:24.083 INFO [RCM] TransitionToState(Cluster IP Address 10.100.128.57) Initializing-->OpenCallIssued.
00000b70.00000db8::2015/04/17-16:27:24.083 INFO [RCM] rcm::RcmResource::ReattachToMonitorProcess: (SQL Server Agent, Initializing)
00000b70.00000db8::2015/04/17-16:27:24.083 INFO [RCM] TransitionToState(SQL Server Agent) Initializing-->OpenCallIssued.
00000b70.00000db8::2015/04/17-16:27:24.083 INFO [RCM] rcm::RcmResource::ReattachToMonitorProcess: (FileServer-(SqlClusterName)(Cluster Disk 1), Initializing)
00000b70.00000db8::2015/04/17-16:27:24.083 INFO [RCM] TransitionToState(FileServer-(SqlClusterName)(Cluster Disk 1)) Initializing-->OpenCallIssued.
00000b70.00000db8::2015/04/17-16:27:24.083 INFO [RCM] rcm::RcmResource::ReattachToMonitorProcess: (Cluster Disk 3, OpenCallIssued)
00000b70.00000db8::2015/04/17-16:27:24.083 INFO [RCM] TransitionToState(Cluster Disk 3) Initializing-->OpenCallIssued.
00000b70.00000db8::2015/04/17-16:27:24.083 INFO [RCM] rcm::RcmResource::ReattachToMonitorProcess: (Cluster Disk 1, Initializing)
00000b70.00000db8::2015/04/17-16:27:24.083 INFO [RCM] TransitionToState(Cluster Disk 1) Initializing-->OpenCallIssued.
00000b70.00000db8::2015/04/17-16:27:24.083 INFO [RCM] rcm::RcmResource::ReattachToMonitorProcess: (Backup Agent Service for Cluster Group, Initializing)
00000b70.00000db8::2015/04/17-16:27:24.083 INFO [RCM] TransitionToState(Backup Agent Service for Cluster Group) Initializing-->OpenCallIssued.
00000b70.00000db8::2015/04/17-16:27:24.083 INFO [RCM] rcm::RcmResource::ReattachToMonitorProcess: (SQL IP Address 2 (SqlClusterName), Initializing)
00000b70.00000db8::2015/04/17-16:27:24.083 INFO [RCM] TransitionToState(SQL IP Address 2 (SqlClusterName)) Initializing-->OpenCallIssued.
00000b70.00000db8::2015/04/17-16:27:24.083 INFO [RCM] rcm::RcmResource::ReattachToMonitorProcess: (FileServer-(SqlClusterName)(Cluster Disk 2), Initializing)
00000b70.00000db8::2015/04/17-16:27:24.083 INFO [RCM] TransitionToState(FileServer-(SqlClusterName)(Cluster Disk 2)) Initializing-->OpenCallIssued.
00000b70.00000db8::2015/04/17-16:27:24.083 INFO [RCM] rcm::RcmResource::ReattachToMonitorProcess: (Backup Agent Service for SQL Server (MSSQLSERVER), Initializing)
00000b70.00000db8::2015/04/17-16:27:24.083 INFO [RCM] TransitionToState(Backup Agent Service for SQL Server (MSSQLSERVER)) Initializing-->OpenCallIssued.
00000b70.00000db8::2015/04/17-16:27:24.083 INFO [RCM] rcm::RcmResource::ReattachToMonitorProcess: (Cluster Name, Initializing)
00000b70.00000db8::2015/04/17-16:27:24.083 INFO [RCM] TransitionToState(Cluster Name) Initializing-->OpenCallIssued.
00000abc.00001220::2015/04/17-16:27:24.099 INFO [RES] Network Name <Cluster Name>: NetNameOpen Invoked
00000abc.00001250::2015/04/17-16:27:24.099 INFO [RES] Physical Disk: Enter EnumerateDevices: EnumDevice 2
00000abc.000011dc::2015/04/17-16:27:24.115 INFO [RES] IP Address <SQL IP Address 1 (SqlClusterName)>: NBT interface \Device\NetBt_If2 (instance 0x16FB7FFE) is no longer valid, status 2.
00000abc.000011dc::2015/04/17-16:27:24.115 WARN [RES] IP Address <SQL IP Address 1 (SqlClusterName)>: Failed to delete NBT interface information from database, status 70.
00000abc.000011dc::2015/04/17-16:27:24.115 INFO [RES] IP Address <SQL IP Address 1 (SqlClusterName)>: Resource open, resource ID = 38300880.
00000abc.0000120c::2015/04/17-16:27:24.115 INFO [RES] IP Address <SQL IP Address 2 (SqlClusterName)>: NBT interface \Device\NetBt_If1 (instance 0x16FB7FDF) is no longer valid, status 2.
00000abc.0000120c::2015/04/17-16:27:24.115 WARN [RES] IP Address <SQL IP Address 2 (SqlClusterName)>: Failed to delete NBT interface information from database, status 70.
00000abc.0000120c::2015/04/17-16:27:24.115 INFO [RES] IP Address <SQL IP Address 2 (SqlClusterName)>: Resource open, resource ID = 38307104.
00000abc.000011d8::2015/04/17-16:27:24.115 INFO [RES] IP Address <Cluster IP Address>: NBT interface \Device\NetBt_If3 (instance 0x1703D3F6) is no longer valid, status 2.
00000abc.000011d8::2015/04/17-16:27:24.115 WARN [RES] IP Address <Cluster IP Address>: Failed to delete NBT interface information from database, status 70.
00000abc.000011d8::2015/04/17-16:27:24.115 INFO [RES] IP Address <Cluster IP Address>: Resource open, resource ID = 38300416.
00000abc.000011e4::2015/04/17-16:27:24.115 INFO [RES] IP Address <Cluster IP Address 10.100.128.57>: NBT interface \Device\NetBt_If4 (instance 0x1703D3F6) is no longer valid, status 2.
00000abc.000011e4::2015/04/17-16:27:24.115 WARN [RES] IP Address <Cluster IP Address 10.100.128.57>: Failed to delete NBT interface information from database, status 70.
00000abc.000011e4::2015/04/17-16:27:24.115 INFO [RES] IP Address <Cluster IP Address 10.100.128.57>: Resource open, resource ID = 38306592.
00000abc.00001250::2015/04/17-16:27:24.115 INFO [RES] Physical Disk: PNP: Adding volume \\?\ide#cdromteac_dvd-rom_dv-28sw____________________r.2a____#5&ecf568&0&0.0.0#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b}
00000abc.00001250::2015/04/17-16:27:24.115 INFO [RES] Physical Disk: PNP: Ignoring removable drive \\?\ide#cdromteac_dvd-rom_dv-28sw____________________r.2a____#5&ecf568&0&0.0.0#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b}
00000abc.00001250::2015/04/17-16:27:24.115 INFO [RES] Physical Disk: PNP: Add Volume exit, status 0
00000abc.00001250::2015/04/17-16:27:24.115 INFO [RES] Physical Disk: PNP: Adding volume \\?\scsi#cdrom&ven_elby&prod_clonedrive&rev_1.4#1&1843ccbc&0&000000#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b}
00000abc.00001250::2015/04/17-16:27:24.115 INFO [RES] Physical Disk: PNP: Ignoring removable drive \\?\scsi#cdrom&ven_elby&prod_clonedrive&rev_1.4#1&1843ccbc&0&000000#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b}
00000abc.00001250::2015/04/17-16:27:24.115 INFO [RES] Physical Disk: PNP: Add Volume exit, status 0
00000abc.00001250::2015/04/17-16:27:24.115 INFO [RES] Physical Disk: PNP: Adding volume \\?\storage#volume#1&19f7e59c&0&signature80offset855a200length80730c00#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b}
00000abc.00001250::2015/04/17-16:27:24.115 INFO [RES] Physical Disk: PNP: Add Volume exit, status 0
00000abc.00001250::2015/04/17-16:27:24.115 INFO [RES] Physical Disk: PNP: Adding volume \\?\storage#volume#1&19f7e59c&0&signature80offset88d00000length217f200000#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b}
00000abc.00001250::2015/04/17-16:27:24.115 INFO [RES] Physical Disk: PNP: Add Volume exit, status 0
00000abc.00001250::2015/04/17-16:27:24.115 INFO [RES] Physical Disk: PNP: Adding volume \\?\storage#volume#1&19f7e59c&0&_??_usbstor#disk&ven_idrac&prod_lcdrive&rev_0323#7&2d5dfc65&0&20080519&0#{53f56307-b6bf-11d0-94f2-00a0c91efb8b}#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b}
00000abc.00001250::2015/04/17-16:27:24.115 INFO [RES] Physical Disk: PNP: Ignoring removable drive \\?\storage#volume#1&19f7e59c&0&_??_usbstor#disk&ven_idrac&prod_lcdrive&rev_0323#7&2d5dfc65&0&20080519&0#{53f56307-b6bf-11d0-94f2-00a0c91efb8b}#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b}
00000abc.00001250::2015/04/17-16:27:24.115 INFO [RES] Physical Disk: PNP: Add Volume exit, status 0
00000abc.00001250::2015/04/17-16:27:24.115 INFO [RES] Physical Disk: PNP: Adding volume \\?\storage#volume#1&19f7e59c&0&_??_usbstor#disk&ven_idrac&prod_virtual_floppy&rev_0323#7&379531cb&0&20080519&1#{53f56307-b6bf-11d0-94f2-00a0c91efb8b}#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b}
00000abc.00001250::2015/04/17-16:27:24.115 INFO [RES] Physical Disk: PNP: Ignoring removable drive \\?\storage#volume#1&19f7e59c&0&_??_usbstor#disk&ven_idrac&prod_virtual_floppy&rev_0323#7&379531cb&0&20080519&1#{53f56307-b6bf-11d0-94f2-00a0c91efb8b}#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b}
00000abc.00001250::2015/04/17-16:27:24.115 INFO [RES] Physical Disk: PNP: Add Volume exit, status 0
00000abc.00001250::2015/04/17-16:27:24.115 INFO [RES] Physical Disk: PNP: Adding volume \\?\usbstor#cdrom&ven_idrac&prod_virtual_cd&rev_0323#7&379531cb&0&20080519&0#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b}
00000abc.00001250::2015/04/17-16:27:24.115 INFO [RES] Physical Disk: PNP: Ignoring removable drive \\?\usbstor#cdrom&ven_idrac&prod_virtual_cd&rev_0323#7&379531cb&0&20080519&0#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b}
00000abc.00001250::2015/04/17-16:27:24.115 INFO [RES] Physical Disk: PNP: Add Volume exit, status 0
00000abc.00001250::2015/04/17-16:27:24.115 INFO [RES] Physical Disk: Exit EnumerateDevices: status 0
00000abc.00001220::2015/04/17-16:27:24.115 INFO [RES] Network Name <Cluster Name>: Successful open of resid 4591504