Hey all,
Having a Node that occasionally has its NIC disconnect on us (about once a month). We get nothing but the Event ID 1127 stating that the network interface fails. Its a NIC team and I've checked the driver/firmware versions of the NIC and the NIC team software, as well as the configuration on each and they are all identical on this Node and the working Node.
I've read someplace that the NIC Power Saving setting needs to be unchecked so the NIC doesn't go to sleep but in Windows 2008 (non-R2) I'm not finding that option anywhere on these NICs to check it.
I'm wondering if anyone could check out the Cluster log and see if they notice anything obvious and have any ideas that I can pursue for further troubleshooting:
00001770.00000b70::2014/08/05-16:56:06.358 WARN [RES] Physical Disk <52_F_Log01>: VerifyFS: Ignoring failure to open file \\?\GLOBALROOT\Device\Harddisk53\Partition1\BellDesk_Log2_BKUP.ldf Error: 5.
00001728.000008b8::2014/08/05-16:56:12.724 WARN [RES] Physical Disk <52_F_Data01>: VerifyFS: Ignoring failure to open file \\?\GLOBALROOT\Device\Harddisk11\Partition1\BellDeskEXC_1_BKUP.mdf Error: 5.
00001728.000008b8::2014/08/05-16:56:12.724 WARN [RES] Physical Disk <52_F_Data01>: VerifyFS: Ignoring failure to open file \\?\GLOBALROOT\Device\Harddisk11\Partition1\BellDeskEXC_BKUP.mdf Error: 5.
00001728.000008b8::2014/08/05-16:56:12.724 WARN [RES] Physical Disk <52_F_Data01>: VerifyFS: Ignoring failure to open file \\?\GLOBALROOT\Device\Harddisk11\Partition1\BellDeskEXC_log_BKUP.ldf Error: 5.
00001728.000008b8::2014/08/05-16:56:12.725 WARN [RES] Physical Disk <52_F_Data01>: VerifyFS: Ignoring failure to open file \\?\GLOBALROOT\Device\Harddisk11\Partition1\ePO4_EPO_1_old.ndf Error: 5.
00000e8c.00007390::2014/08/05-16:56:13.692 INFO [GUM] Node 4: Processing RequestLock 4:45497
00000e8c.000009b4::2014/08/05-16:56:13.694 INFO [GUM] Node 4: Processing GrantLock to 4 (sent by 3 gumid: 478989)
00000e8c.000009b4::2014/08/05-16:56:17.327 INFO [GUM] Node 4: Processing RequestLock 3:47054
00000e8c.000009b4::2014/08/05-16:56:17.327 INFO [GUM] Node 4: Processing GrantLock to 3 (sent by 4 gumid: 478990)
00000e8c.0000566c::2014/08/05-16:56:20.148 INFO [GUM] Node 4: Processing RequestLock 4:45498
00000e8c.000009b4::2014/08/05-16:56:20.150 INFO [GUM] Node 4: Processing GrantLock to 4 (sent by 3 gumid: 478991)
00000e8c.000009b4::2014/08/05-16:56:27.272 INFO [GUM] Node 4: Processing RequestLock 3:47055
00000e8c.000009b4::2014/08/05-16:56:27.272 INFO [GUM] Node 4: Processing GrantLock to 3 (sent by 4 gumid: 478993)
00000e8c.000009b0::2014/08/05-16:56:48.736 DBG [NETFTAPI] Signaled NetftRemoteUnreachable event, local address 10.199.17.48:003853 remote address 10.199.17.45:003853
00000e8c.000009b0::2014/08/05-16:56:48.736 DBG [NETFTAPI] Signaled NetftRemoteUnreachable event, local address 10.199.17.48:003853 remote address 10.199.17.45:003853
00000e8c.000009b0::2014/08/05-16:56:48.736 DBG [NETFTAPI] Signaled NetftRemoteUnreachable event, local address 10.199.17.48:003853 remote address 10.199.17.45:003853
00000e8c.000009b0::2014/08/05-16:56:48.736 DBG [NETFTAPI] Signaled NetftRemoteUnreachable event, local address 10.199.17.48:003853 remote address 10.199.17.47:003853
00000e8c.000009b0::2014/08/05-16:56:48.736 DBG [NETFTAPI] Signaled NetftRemoteUnreachable event, local address 10.199.17.48:003853 remote address 10.199.17.47:003853
00000e8c.000009b0::2014/08/05-16:56:48.736 DBG [NETFTAPI] Signaled NetftRemoteUnreachable event, local address 10.199.17.48:003853 remote address 10.199.17.47:003853
00000e8c.000009c4::2014/08/05-16:56:48.736 INFO [IM] got event: Remote endpoint 10.199.17.45:~3343~ unreachable from 10.199.17.48:~3343~
00000e8c.000009c4::2014/08/05-16:56:48.736 INFO [IM] Marking Route from 10.199.17.48:~3343~ to 10.199.17.45:~3343~ as down
00000e8c.000009c4::2014/08/05-16:56:48.736 INFO [NDP] Checking to see if all routes for route (virtual) local fe80::68fd:e989:2b47:f203:~0~ to remote fe80::a4fa:4845:1f98:c058:~0~ are down
00000e8c.000009c4::2014/08/05-16:56:48.736 INFO [NDP] Route local 192.168.17.48:~0~ to remote 192.168.17.45:~0~ is up
00000e8c.000009c4::2014/08/05-16:56:48.736 INFO [IM] Adding information for route Route from local 10.199.17.48:~3343~ to remote 10.199.17.47:~3343~, status: true, attributes: 0
00000e8c.000009c4::2014/08/05-16:56:48.736 INFO [IM] Adding information for route Route from local 10.199.17.48:~3343~ to remote 10.199.17.45:~3343~, status: false, attributes: 0
00000e8c.000009c4::2014/08/05-16:56:48.737 INFO [IM] Sending connectivity report to leader (node 1): <class mscs::InterfaceReport>
00000e8c.000009c4::2014/08/05-16:56:48.737 INFO <fromInterface>fe516577-6c30-4b96-a6b0-38adb0ccee3e</fromInterface>
00000e8c.000009c4::2014/08/05-16:56:48.737 INFO <upInterfaces><vector len='2'>
00000e8c.000009c4::2014/08/05-16:56:48.737 INFO <item>fe516577-6c30-4b96-a6b0-38adb0ccee3e</item>
00000e8c.000009c4::2014/08/05-16:56:48.737 INFO <item>073b1ff1-b43a-4a7b-9875-e0d6b8ac0b83</item>
00000e8c.000009c4::2014/08/05-16:56:48.737 INFO </vector>
00000e8c.000009c4::2014/08/05-16:56:48.737 INFO </upInterfaces>
00000e8c.000009c4::2014/08/05-16:56:48.737 INFO <downInterfaces><vector len='1'>
00000e8c.000009c4::2014/08/05-16:56:48.737 INFO <item>8a90dc45-ed7e-4f26-90c0-f36becfa3e0a</item>
00000e8c.000009c4::2014/08/05-16:56:48.737 INFO </vector>
00000e8c.000009c4::2014/08/05-16:56:48.737 INFO </downInterfaces>
00000e8c.000009c4::2014/08/05-16:56:48.737 INFO <viewId>1704</viewId>
00000e8c.000009c4::2014/08/05-16:56:48.737 INFO </class mscs::InterfaceReport>
00000e8c.000009c4::2014/08/05-16:56:48.737 INFO [IM] got event: Remote endpoint 10.199.17.47:~3343~ unreachable from 10.199.17.48:~3343~
00000e8c.000009c4::2014/08/05-16:56:48.737 INFO [IM] Marking Route from 10.199.17.48:~3343~ to 10.199.17.47:~3343~ as down
00000e8c.000009c4::2014/08/05-16:56:48.737 INFO [NDP] Checking to see if all routes for route (virtual) local fe80::68fd:e989:2b47:f203:~0~ to remote fe80::28da:1ec6:f11e:58da:~0~ are down
00000e8c.000009c4::2014/08/05-16:56:48.737 INFO [NDP] Route local 192.168.17.48:~0~ to remote 192.168.17.47:~0~ is up
00000e8c.000009c4::2014/08/05-16:56:48.737 INFO [IM] Adding information for route Route from local 10.199.17.48:~3343~ to remote 10.199.17.47:~3343~, status: false, attributes: 0
00000e8c.000009c4::2014/08/05-16:56:48.737 INFO [IM] Adding information for route Route from local 10.199.17.48:~3343~ to remote 10.199.17.45:~3343~, status: false, attributes: 0
00000e8c.000009c4::2014/08/05-16:56:48.737 INFO [IM] Sending connectivity report to leader (node 1): <class mscs::InterfaceReport>
00000e8c.000009c4::2014/08/05-16:56:48.737 INFO <fromInterface>fe516577-6c30-4b96-a6b0-38adb0ccee3e</fromInterface>
00000e8c.000009c4::2014/08/05-16:56:48.737 INFO <upInterfaces><vector len='1'>
00000e8c.000009c4::2014/08/05-16:56:48.737 INFO <item>fe516577-6c30-4b96-a6b0-38adb0ccee3e</item>
00000e8c.000009c4::2014/08/05-16:56:48.737 INFO </vector>
00000e8c.000009c4::2014/08/05-16:56:48.737 INFO </upInterfaces>
00000e8c.000009c4::2014/08/05-16:56:48.737 INFO <downInterfaces><vector len='2'>
00000e8c.000009c4::2014/08/05-16:56:48.737 INFO <item>073b1ff1-b43a-4a7b-9875-e0d6b8ac0b83</item>
00000e8c.000009c4::2014/08/05-16:56:48.737 INFO <item>8a90dc45-ed7e-4f26-90c0-f36becfa3e0a</item>
00000e8c.000009c4::2014/08/05-16:56:48.737 INFO </vector>
00000e8c.000009c4::2014/08/05-16:56:48.737 INFO </downInterfaces>
00000e8c.000009c4::2014/08/05-16:56:48.737 INFO <viewId>1704</viewId>
00000e8c.000009c4::2014/08/05-16:56:48.737 INFO </class mscs::InterfaceReport>
00000e8c.000011f0::2014/08/05-16:56:48.751 INFO [GUM] Node 4: Processing RequestLock 1:67
00000e8c.000009b4::2014/08/05-16:56:48.752 INFO [GUM] Node 4: Processing GrantLock to 1 (sent by 3 gumid: 478994)
00000e8c.00000fd4::2014/08/05-16:56:48.754 INFO [IM] Changing the state of adapters according to result: <class mscs::InterfaceResult>
00000e8c.00000fd4::2014/08/05-16:56:48.754 INFO <up><vector len='2'>
00000e8c.00000fd4::2014/08/05-16:56:48.754 INFO <item>8a90dc45-ed7e-4f26-90c0-f36becfa3e0a</item>
00000e8c.00000fd4::2014/08/05-16:56:48.754 INFO <item>073b1ff1-b43a-4a7b-9875-e0d6b8ac0b83</item>
00000e8c.00000fd4::2014/08/05-16:56:48.754 INFO </vector>
00000e8c.00000fd4::2014/08/05-16:56:48.754 INFO </up>
00000e8c.00000fd4::2014/08/05-16:56:48.754 INFO <down><vector len='1'>
00000e8c.00000fd4::2014/08/05-16:56:48.754 INFO <item>fe516577-6c30-4b96-a6b0-38adb0ccee3e</item>
00000e8c.00000fd4::2014/08/05-16:56:48.754 INFO </vector>
00000e8c.00000fd4::2014/08/05-16:56:48.754 INFO </down>
00000e8c.00000fd4::2014/08/05-16:56:48.754 INFO <unreachable><vector len='0'>
00000e8c.00000fd4::2014/08/05-16:56:48.754 INFO </vector>
00000e8c.00000fd4::2014/08/05-16:56:48.754 INFO </unreachable>
00000e8c.00000fd4::2014/08/05-16:56:48.754 INFO </class mscs::InterfaceResult>
0000143c.00000e1c::2014/08/05-16:56:48.755 WARN [RES] IP Address <SQL IP Address 1 (ENTSQL52)>: WorkerThread: NetInterface fe516577-6c30-4b96-a6b0-38adb0ccee3e has failed. Failing resource.
0000143c.00000e1c::2014/08/05-16:56:48.755 WARN [RES] IP Address <SQL IP Address 1 (ENTSQL54)>: WorkerThread: NetInterface fe516577-6c30-4b96-a6b0-38adb0ccee3e has failed. Failing resource.
0000143c.00000e1c::2014/08/05-16:56:48.755 WARN [RES] IP Address <SQL IP Address 1 (ENTSQL56)>: WorkerThread: NetInterface fe516577-6c30-4b96-a6b0-38adb0ccee3e has failed. Failing resource.
0000143c.00000e1c::2014/08/05-16:56:48.756 WARN [RES] IP Address <SQL IP Address 1 (ENTSQL58)>: WorkerThread: NetInterface fe516577-6c30-4b96-a6b0-38adb0ccee3e has failed. Failing resource.
0000143c.000068a0::2014/08/05-16:56:48.940 WARN [RES] IP Address <SQL IP Address 1 (ENTSQL56)>: IP Interface 3811C70A (address 10.199.17.56) failed LooksAlive check, status 1117.
0000143c.000068a0::2014/08/05-16:56:48.940 WARN [RES] IP Address <SQL IP Address 1 (ENTSQL56)>: IP Interface 3811C70A (address 10.199.17.56) failed IsAlive check, status 1117.
0000143c.000068a0::2014/08/05-16:56:48.940 WARN [RHS] Resource SQL IP Address 1 (ENTSQL56) IsAlive has indicated failure.
00000e8c.000076c4::2014/08/05-16:56:48.940 INFO [RCM] HandleMonitorReply: FAILURENOTIFICATION for 'SQL IP Address 1 (ENTSQL56)', gen(0) result 1.
00000e8c.000076c4::2014/08/05-16:56:48.940 INFO [RCM] TransitionToState(SQL IP Address 1 (ENTSQL56)) Online-->ProcessingFailure.
00000e8c.000076c4::2014/08/05-16:56:48.941 ERR [RCM] rcm::RcmResource::HandleFailure: (SQL IP Address 1 (ENTSQL56))
00000e8c.000076c4::2014/08/05-16:56:48.941 INFO [RCM] resource SQL IP Address 1 (ENTSQL56): failure count: 1, restartAction: 2.
00000e8c.000076c4::2014/08/05-16:56:48.941 INFO [RCM] Will restart resource in 500 milliseconds.
00000e8c.000076c4::2014/08/05-16:56:48.941 INFO [RCM] TransitionToState(SQL IP Address 1 (ENTSQL56)) ProcessingFailure-->[Terminating to DelayRestartingResource].
00000e8c.000076c4::2014/08/05-16:56:48.941 INFO [RCM] rcm::RcmGroup::ProcessStateChange: (ENTSQL56, Online --> Pending)
00000e8c.000076c4::2014/08/05-16:56:48.941 INFO [RCM] TransitionToState(SQL Network Name (ENTSQL56)) Online-->[Terminating to OnlineCallIssued].
00000e8c.000076c4::2014/08/05-16:56:48.941 INFO [RCM] TransitionToState(SQL Server (ENT56)) Online-->[Terminating to OnlineCallIssued].
00000e8c.000076c4::2014/08/05-16:56:48.941 INFO [RCM] TransitionToState(SQL Server Agent (ENT56)) Online-->[Terminating to OnlineCallIssued].
0000143c.000068a0::2014/08/05-16:56:48.942 INFO [RES] Network Name <SQL Network Name (ENTSQL56)>: Terminating resource...
0000143c.000068a0::2014/08/05-16:56:48.942 INFO [RES] Network Name <SQL Network Name (ENTSQL56)>: Offline of resource continuing...
0000143c.00005bb8::2014/08/05-16:56:48.942 INFO [RES] IP Address <SQL IP Address 1 (ENTSQL56)>: Terminating resource...
0000143c.00005bb8::2014/08/05-16:56:48.943 INFO [RES] IP Address <SQL IP Address 1 (ENTSQL56)>: Deleting IP interface 3811C70A.
00000e8c.00005d78::2014/08/05-16:56:48.944 INFO [GUM] Node 4: Processing RequestLock 4:45500
00000e8c.000076c4::2014/08/05-16:56:48.945 DBG [NETFTAPI] received NsiDeleteInstance for 10.199.17.56
00000e8c.000011f0::2014/08/05-16:56:48.945 INFO [GUM] Node 4: Processing GrantLock to 4 (sent by 1 gumid: 478995)
00000e8c.000076c4::2014/08/05-16:56:48.946 WARN [NETFTAPI] Failed to query parameters for 10.199.17.56 (status 80070490)
00000e8c.000076c4::2014/08/05-16:56:48.946 DBG [NETFTAPI] Signaled NetftLocalRemove event for 10.199.17.56
00000e8c.000076c4::2014/08/05-16:56:48.946 DBG [NETFTAPI] Signaled NetftLocalRemove event for 10.199.17.56
00000e8c.000076c4::2014/08/05-16:56:48.946 DBG [NETFTAPI] Signaled NetftLocalRemove event for 10.199.17.56
0000143c.00005bb8::2014/08/05-16:56:49.085 INFO [RES] IP Address <SQL IP Address 1 (ENTSQL56)>: Address 10.199.17.56 on adapter 10.199.17 Corp Team offline.
00000e8c.000056ac::2014/08/05-16:56:49.085 INFO [RCM] TransitionToState(SQL IP Address 1 (ENTSQL56)) [Terminating to DelayRestartingResource]-->DelayRestartingResource.
0000143c.00007414::2014/08/05-16:56:49.095 INFO [RES] Physical Disk: HardDiskpGetDiskInfo: Disk is of type MBR, signature 0x544006f
0000143c.00007414::2014/08/05-16:56:49.107 INFO [RES] Physical Disk: HardDiskpGetDiskInfo: Disk is of type GPT, guid {88f87390-fc69-43d2-991a-4b128bdfeaf6}
0000143c.00007414::2014/08/05-16:56:49.118 INFO [RES] Physical Disk: HardDiskpGetDiskInfo: Disk is of type GPT, guid {8293a54e-8426-4a7f-be9f-343327bebf28}
0000143c.00007414::2014/08/05-16:56:49.129 INFO [RES] Physical Disk: HardDiskpGetDiskInfo: Disk is of type MBR, signature 0x544001c
0000143c.00007414::2014/08/05-16:56:49.282 WARN [RES] IP Address <SQL IP Address 1 (ENTSQL58)>: IP Interface 3A11C70A (address 10.199.17.58) failed LooksAlive check, status 1117.
0000143c.00007414::2014/08/05-16:56:49.282 WARN [RES] IP Address <SQL IP Address 1 (ENTSQL58)>: IP Interface 3A11C70A (address 10.199.17.58) failed IsAlive check, status 1117.
0000143c.00007414::2014/08/05-16:56:49.282 WARN [RHS] Resource SQL IP Address 1 (ENTSQL58) IsAlive has indicated failure.
00000e8c.000076c4::2014/08/05-16:56:49.282 INFO [RCM] HandleMonitorReply: FAILURENOTIFICATION for 'SQL IP Address 1 (ENTSQL58)', gen(0) result 1.
00000e8c.000076c4::2014/08/05-16:56:49.282 INFO [RCM] TransitionToState(SQL IP Address 1 (ENTSQL58)) Online-->ProcessingFailure.
00000e8c.0000566c::2014/08/05-16:56:49.282 ERR [RCM] rcm::RcmResource::HandleFailure: (SQL IP Address 1 (ENTSQL58))
00000e8c.0000566c::2014/08/05-16:56:49.283 INFO [RCM] resource SQL IP Address 1 (ENTSQL58): failure count: 1, restartAction: 2.
00000e8c.0000566c::2014/08/05-16:56:49.283 INFO [RCM] Will restart resource in 500 milliseconds.
00000e8c.0000566c::2014/08/05-16:56:49.283 INFO [RCM] TransitionToState(SQL IP Address 1 (ENTSQL58)) ProcessingFailure-->[Terminating to DelayRestartingResource].
00000e8c.0000566c::2014/08/05-16:56:49.283 INFO [RCM] rcm::RcmGroup::ProcessStateChange: (ENTSQL58, Online --> Pending)
00000e8c.0000566c::2014/08/05-16:56:49.283 INFO [RCM] TransitionToState(SQL Network Name (ENTSQL58)) Online-->[Terminating to OnlineCallIssued].
00000e8c.0000566c::2014/08/05-16:56:49.283 INFO [RCM] TransitionToState(SQL Server (ENT58)) Online-->[Terminating to OnlineCallIssued].
00000e8c.0000566c::2014/08/05-16:56:49.283 INFO [RCM] TransitionToState(SQL Server Agent (ENT58)) Online-->[Terminating to OnlineCallIssued].
0000143c.00007414::2014/08/05-16:56:49.284 INFO [RES] Network Name <SQL Network Name (ENTSQL58)>: Terminating resource...
0000143c.00007414::2014/08/05-16:56:49.284 INFO [RES] Network Name <SQL Network Name (ENTSQL58)>: Offline of resource continuing...
0000143c.00005bb8::2014/08/05-16:56:49.284 INFO [RES] IP Address <SQL IP Address 1 (ENTSQL58)>: Terminating resource...
0000143c.00005bb8::2014/08/05-16:56:49.284 INFO [RES] IP Address <SQL IP Address 1 (ENTSQL58)>: Deleting IP interface 3A11C70A.
00000e8c.0000566c::2014/08/05-16:56:49.285 DBG [NETFTAPI] received NsiDeleteInstance for 10.199.17.58
00000e8c.0000566c::2014/08/05-16:56:49.286 WARN [NETFTAPI] Failed to query parameters for 10.199.17.58 (status 80070490)
00000e8c.0000566c::2014/08/05-16:56:49.286 DBG [NETFTAPI] Signaled NetftLocalRemove event for 10.199.17.58
00000e8c.0000566c::2014/08/05-16:56:49.286 DBG [NETFTAPI] Signaled NetftLocalRemove event for 10.199.17.58
00000e8c.0000566c::2014/08/05-16:56:49.286 DBG [NETFTAPI] Signaled NetftLocalRemove event for 10.199.17.58
0000143c.00005bb8::2014/08/05-16:56:49.291 INFO [RES] IP Address <SQL IP Address 1 (ENTSQL58)>: Address 10.199.17.58 on adapter 10.199.17 Corp Team offline.
00000e8c.0000566c::2014/08/05-16:56:49.292 INFO [RCM] TransitionToState(SQL IP Address 1 (ENTSQL58)) [Terminating to DelayRestartingResource]-->DelayRestartingResource.
0000143c.00006378::2014/08/05-16:56:49.304 INFO [RES] Physical Disk: HardDiskpGetDiskInfo: Disk is of type GPT, guid {20586068-4927-4492-9403-e826e0d60ad9}
0000143c.00006378::2014/08/05-16:56:49.320 INFO [RES] Physical Disk: HardDiskpGetDiskInfo: Disk is of type MBR, signature 0x544006c
00001518.000025e4::2014/08/05-16:56:49.325 INFO [RES] SQL Server <SQL Server (ENT56)>: [sqsrvres] OnlineThread: asked to terminate while waiting for QP.
0000143c.00006378::2014/08/05-16:56:49.339 INFO [RES] Physical Disk: HardDiskpGetDiskInfo: Disk is of type MBR, signature 0x544006d
0000143c.00006378::2014/08/05-16:56:49.350 INFO [RES] Physical Disk: HardDiskpGetDiskInfo: Disk is of type MBR, signature 0x54400b1
0000143c.00006378::2014/08/05-16:56:49.361 INFO [RES] Physical Disk: HardDiskpGetDiskInfo: Disk is of type MBR, signature 0x544001f
0000143c.00006378::2014/08/05-16:56:49.454 WARN [RES] IP Address <SQL IP Address 1 (ENTSQL52)>: IP Interface 3411C70A (address 10.199.17.52) failed LooksAlive check, status 1117.
0000143c.00006378::2014/08/05-16:56:49.454 WARN [RES] IP Address <SQL IP Address 1 (ENTSQL52)>: IP Interface 3411C70A (address 10.199.17.52) failed IsAlive check, status 1117.
0000143c.00006378::2014/08/05-16:56:49.454 WARN [RHS] Resource SQL IP Address 1 (ENTSQL52) IsAlive has indicated failure.
00000e8c.000056ac::2014/08/05-16:56:49.454 INFO [RCM] HandleMonitorReply: FAILURENOTIFICATION for 'SQL IP Address 1 (ENTSQL52)', gen(0) result 1.
00000e8c.000056ac::2014/08/05-16:56:49.454 INFO [RCM] TransitionToState(SQL IP Address 1 (ENTSQL52)) Online-->ProcessingFailure.
00000e8c.000076c4::2014/08/05-16:56:49.454 ERR [RCM] rcm::RcmResource::HandleFailure: (SQL IP Address 1 (ENTSQL52))
00000e8c.000076c4::2014/08/05-16:56:49.455 INFO [RCM] resource SQL IP Address 1 (ENTSQL52): failure count: 1, restartAction: 2.
00000e8c.000076c4::2014/08/05-16:56:49.455 INFO [RCM] Will restart resource in 500 milliseconds.
00000e8c.000076c4::2014/08/05-16:56:49.455 INFO [RCM] TransitionToState(SQL IP Address 1 (ENTSQL52)) ProcessingFailure-->[Terminating to DelayRestartingResource].
00000e8c.000076c4::2014/08/05-16:56:49.455 INFO [RCM] rcm::RcmGroup::ProcessStateChange: (ENTSQL52, Online --> Pending)
00000e8c.000076c4::2014/08/05-16:56:49.455 INFO [RCM] TransitionToState(SQL Network Name (ENTSQL52)) Online-->[Terminating to OnlineCallIssued].
00000e8c.000076c4::2014/08/05-16:56:49.455 INFO [RCM] TransitionToState(SQL Server (ENT52)) Online-->[Terminating to OnlineCallIssued].
00000e8c.000076c4::2014/08/05-16:56:49.455 INFO [RCM] TransitionToState(SQL Server Agent (ENT52)) Online-->[Terminating to OnlineCallIssued].
0000143c.00006378::2014/08/05-16:56:49.456 INFO [RES] Network Name <SQL Network Name (ENTSQL52)>: Terminating resource...
0000143c.00006378::2014/08/05-16:56:49.456 INFO [RES] Network Name <SQL Network Name (ENTSQL52)>: Offline of resource continuing...
0000143c.00005bb8::2014/08/05-16:56:49.456 INFO [RES] IP Address <SQL IP Address 1 (ENTSQL52)>: Terminating resource...
0000143c.00005bb8::2014/08/05-16:56:49.456 INFO [RES] IP Address <SQL IP Address 1 (ENTSQL52)>: Deleting IP interface 3411C70A.
00000e8c.000076c4::2014/08/05-16:56:49.458 DBG [NETFTAPI] received NsiDeleteInstance for 10.199.17.52
00000e8c.000076c4::2014/08/05-16:56:49.458 WARN [NETFTAPI] Failed to query parameters for 10.199.17.52 (status 80070490)
00000e8c.000076c4::2014/08/05-16:56:49.458 DBG [NETFTAPI] Signaled NetftLocalRemove event for 10.199.17.52
00000e8c.000076c4::2014/08/05-16:56:49.458 DBG [NETFTAPI] Signaled NetftLocalRemove event for 10.199.17.52
00000e8c.000076c4::2014/08/05-16:56:49.458 DBG [NETFTAPI] Signaled NetftLocalRemove event for 10.199.17.52
0000143c.00005bb8::2014/08/05-16:56:49.461 INFO [RES] IP Address <SQL IP Address 1 (ENTSQL52)>: Address 10.199.17.52 on adapter 10.199.17 Corp Team offline.
00000e8c.000076c4::2014/08/05-16:56:49.461 INFO [RCM] TransitionToState(SQL IP Address 1 (ENTSQL52)) [Terminating to DelayRestartingResource]-->DelayRestartingResource.
0000143c.00007108::2014/08/05-16:56:49.473 INFO [RES] Physical Disk: HardDiskpGetDiskInfo: Disk is of type MBR, signature 0x5440082
0000143c.00007108::2014/08/05-16:56:49.488 INFO [RES] Physical Disk: HardDiskpGetDiskInfo: Disk is of type MBR, signature 0x544007c
00001728.000008b8::2014/08/05-16:56:49.502 INFO [RES] Physical Disk: HardDiskpGetDiskInfo: Disk is of type MBR, signature 0x5440086
0000143c.00007108::2014/08/05-16:56:49.514 INFO [RES] Physical Disk: HardDiskpGetDiskInfo: Disk is of type MBR, signature 0x54400ba
0000143c.00007108::2014/08/05-16:56:49.526 INFO [RES] Physical Disk: HardDiskpGetDiskInfo: Disk is of type MBR, signature 0x54400bb
00000e8c.0000566c::2014/08/05-16:56:49.585 INFO [RCM] Delay-restarting SQL IP Address 1 (ENTSQL56) and any waiting dependents.
00000e8c.0000566c::2014/08/05-16:56:49.585 INFO [RCM] TransitionToState(SQL IP Address 1 (ENTSQL56)) DelayRestartingResource-->OnlineCallIssued.
0000143c.00007108::2014/08/05-16:56:49.585 INFO [RES] IP Address <SQL IP Address 1 (ENTSQL56)>: Bringing resource online...
00000e8c.000076c4::2014/08/05-16:56:49.586 INFO [RCM] HandleMonitorReply: ONLINERESOURCE for 'SQL IP Address 1 (ENTSQL56)', gen(1) result 997.
00000e8c.000076c4::2014/08/05-16:56:49.586 INFO [RCM] TransitionToState(SQL IP Address 1 (ENTSQL56)) OnlineCallIssued-->OnlinePending.
0000143c.00003878::2014/08/05-16:56:49.586 INFO [RES] IP Address <SQL IP Address 1 (ENTSQL56)>: Online thread running.
0000143c.00003878::2014/08/05-16:56:49.589 INFO [RES] IP Address <SQL IP Address 1 (ENTSQL56)>: Checking for network match: network masks 00FFFFFF=00FFFFFF and addresses 3811C70A^0011A8C0, role 1.
0000143c.00003878::2014/08/05-16:56:49.590 INFO [RES] IP Address <SQL IP Address 1 (ENTSQL56)>: Checking for network match: network masks 00FFFFFF=00FFFFFF and addresses 3811C70A^00BDD60A, role 0.
0000143c.00003878::2014/08/05-16:56:49.590 INFO [RES] IP Address <SQL IP Address 1 (ENTSQL56)>: Checking for network match: network masks 00FFFFFF=00FFFFFF and addresses 3811C70A^0011C70A, role 3.
0000143c.00003878::2014/08/05-16:56:49.596 INFO [RES] IP Address <SQL IP Address 1 (ENTSQL56)>: Online: Opened object handle for netinterface fe516577-6c30-4b96-a6b0-38adb0ccee3e.
0000143c.00003878::2014/08/05-16:56:49.597 INFO [RES] IP Address <SQL IP Address 1 (ENTSQL56)>: Online: Registered notification for netinterface fe516577-6c30-4b96-a6b0-38adb0ccee3e.
0000143c.00003878::2014/08/05-16:56:49.597 ERR [RES] IP Address <SQL IP Address 1 (ENTSQL56)>: NetInterface fe516577-6c30-4b96-a6b0-38adb0ccee3e has failed.
0000143c.00003878::2014/08/05-16:56:49.597 ERR [RHS] Online for resource SQL IP Address 1 (ENTSQL56) failed.
00000e8c.0000566c::2014/08/05-16:56:49.597 INFO [RCM] HandleMonitorReply: ONLINERESOURCE for 'SQL IP Address 1 (ENTSQL56)', gen(1) result 5018.
00000e8c.0000566c::2014/08/05-16:56:49.597 INFO [RCM] TransitionToState(SQL IP Address 1 (ENTSQL56)) OnlinePending-->ProcessingFailure.
00000e8c.000056ac::2014/08/05-16:56:49.597 ERR [RCM] rcm::RcmResource::HandleFailure: (SQL IP Address 1 (ENTSQL56))
00000e8c.000056ac::2014/08/05-16:56:49.597 INFO [RCM] resource SQL IP Address 1 (ENTSQL56): failure count: 2, restartAction: 2.
00000e8c.000056ac::2014/08/05-16:56:49.598 INFO [RCM] TransitionToState(SQL IP Address 1 (ENTSQL56)) ProcessingFailure-->[Terminating to Failed].
00000e8c.000056ac::2014/08/05-16:56:49.598 INFO [RCM] Resource SQL IP Address 1 (ENTSQL56) is causing group ENTSQL56 to failover. Posting worker thread.
0000143c.00007108::2014/08/05-16:56:49.598 INFO [RES] IP Address <SQL IP Address 1 (ENTSQL56)>: Terminating resource...
0000143c.00007108::2014/08/05-16:56:49.598 INFO [RES] IP Address <SQL IP Address 1 (ENTSQL56)>: Resource is already offline.
00000e8c.000056ac::2014/08/05-16:56:49.598 INFO [RCM] rcm::RcmGroup::Failover: (ENTSQL56)
00000e8c.00007480::2014/08/05-16:56:49.598 INFO [RCM] TransitionToState(SQL IP Address 1 (ENTSQL56)) [Terminating to Failed]-->Failed.
00001608.00002578::2014/08/05-16:56:49.614 INFO [RES] SQL Server <SQL Server (ENT58)>: [sqsrvres] OnlineThread: asked to terminate while waiting for QP.
00000e8c.000076c4::2014/08/05-16:56:49.792 INFO [RCM] Delay-restarting SQL IP Address 1 (ENTSQL58) and any waiting dependents.
00000e8c.000076c4::2014/08/05-16:56:49.792 INFO [RCM] TransitionToState(SQL IP Address 1 (ENTSQL58)) DelayRestartingResource-->OnlineCallIssued.
0000143c.00007108::2014/08/05-16:56:49.792 INFO [RES] IP Address <SQL IP Address 1 (ENTSQL58)>: Bringing resource online...
00000e8c.00002b98::2014/08/05-16:56:49.793 INFO [RCM] HandleMonitorReply: ONLINERESOURCE for 'SQL IP Address 1 (ENTSQL58)', gen(1) result 997.
00000e8c.00002b98::2014/08/05-16:56:49.793 INFO [RCM] TransitionToState(SQL IP Address 1 (ENTSQL58)) OnlineCallIssued-->OnlinePending.
0000143c.00006f80::2014/08/05-16:56:49.793 INFO [RES] IP Address <SQL IP Address 1 (ENTSQL58)>: Online thread running.
0000143c.00006f80::2014/08/05-16:56:49.796 INFO [RES] IP Address <SQL IP Address 1 (ENTSQL58)>: Checking for network match: network masks 00FFFFFF=00FFFFFF and addresses 3A11C70A^0011A8C0, role 1.
0000143c.00006f80::2014/08/05-16:56:49.797 INFO [RES] IP Address <SQL IP Address 1 (ENTSQL58)>: Checking for network match: network masks 00FFFFFF=00FFFFFF and addresses 3A11C70A^00BDD60A, role 0.
0000143c.00006f80::2014/08/05-16:56:49.798 INFO [RES] IP Address <SQL IP Address 1 (ENTSQL58)>: Checking for network match: network masks 00FFFFFF=00FFFFFF and addresses 3A11C70A^0011C70A, role 3.
0000143c.00006f80::2014/08/05-16:56:49.804 INFO [RES] IP Address <SQL IP Address 1 (ENTSQL58)>: Online: Opened object handle for netinterface fe516577-6c30-4b96-a6b0-38adb0ccee3e.
0000143c.00006f80::2014/08/05-16:56:49.804 INFO [RES] IP Address <SQL IP Address 1 (ENTSQL58)>: Online: Registered notification for netinterface fe516577-6c30-4b96-a6b0-38adb0ccee3e.
0000143c.00006f80::2014/08/05-16:56:49.804 ERR [RES] IP Address <SQL IP Address 1 (ENTSQL58)>: NetInterface fe516577-6c30-4b96-a6b0-38adb0ccee3e has failed.
0000143c.00006f80::2014/08/05-16:56:49.804 ERR [RHS] Online for resource SQL IP Address 1 (ENTSQL58) failed.
00000e8c.000076c4::2014/08/05-16:56:49.804 INFO [RCM] HandleMonitorReply: ONLINERESOURCE for 'SQL IP Address 1 (ENTSQL58)', gen(1) result 5018.
00000e8c.000076c4::2014/08/05-16:56:49.804 INFO [RCM] TransitionToState(SQL IP Address 1 (ENTSQL58)) OnlinePending-->ProcessingFailure.
00000e8c.0000660c::2014/08/05-16:56:49.805 ERR [RCM] rcm::RcmResource::HandleFailure: (SQL IP Address 1 (ENTSQL58))
00000e8c.0000660c::2014/08/05-16:56:49.805 INFO [RCM] resource SQL IP Address 1 (ENTSQL58): failure count: 2, restartAction: 2.
00000e8c.0000660c::2014/08/05-16:56:49.805 INFO [RCM] TransitionToState(SQL IP Address 1 (ENTSQL58)) ProcessingFailure-->[Terminating to Failed].
00000e8c.0000660c::2014/08/05-16:56:49.806 INFO [RCM] Resource SQL IP Address 1 (ENTSQL58) is causing group ENTSQL58 to failover. Posting worker thread.
0000143c.00007108::2014/08/05-16:56:49.806 INFO [RES] IP Address <SQL IP Address 1 (ENTSQL58)>: Terminating resource...
0000143c.00007108::2014/08/05-16:56:49.806 INFO [RES] IP Address <SQL IP Address 1 (ENTSQL58)>: Resource is already offline.
00000e8c.0000660c::2014/08/05-16:56:49.806 INFO [RCM] rcm::RcmGroup::Failover: (ENTSQL58)
00000e8c.00007480::2014/08/05-16:56:49.806 INFO [RCM] TransitionToState(SQL IP Address 1 (ENTSQL58)) [Terminating to Failed]-->Failed.
00000e8c.00007480::2014/08/05-16:56:49.961 INFO [RCM] Delay-restarting SQL IP Address 1 (ENTSQL52) and any waiting dependents.
00000e8c.00007480::2014/08/05-16:56:49.961 INFO [RCM] TransitionToState(SQL IP Address 1 (ENTSQL52)) DelayRestartingResource-->OnlineCallIssued.
0000143c.00007108::2014/08/05-16:56:49.961 INFO [RES] IP Address <SQL IP Address 1 (ENTSQL52)>: Bringing resource online...
00000e8c.00003170::2014/08/05-16:56:49.962 INFO [RCM] HandleMonitorReply: ONLINERESOURCE for 'SQL IP Address 1 (ENTSQL52)', gen(1) result 997.
00000e8c.00003170::2014/08/05-16:56:49.962 INFO [RCM] TransitionToState(SQL IP Address 1 (ENTSQL52)) OnlineCallIssued-->OnlinePending.
0000143c.00003720::2014/08/05-16:56:49.962 INFO [RES] IP Address <SQL IP Address 1 (ENTSQL52)>: Online thread running.
0000143c.00003720::2014/08/05-16:56:49.965 INFO [RES] IP Address <SQL IP Address 1 (ENTSQL52)>: Checking for network match: network masks 00FFFFFF=00FFFFFF and addresses 3411C70A^0011A8C0, role 1.
0000143c.00003720::2014/08/05-16:56:49.966 INFO [RES] IP Address <SQL IP Address 1 (ENTSQL52)>: Checking for network match: network masks 00FFFFFF=00FFFFFF and addresses 3411C70A^00BDD60A, role 0.
0000143c.00003720::2014/08/05-16:56:49.967 INFO [RES] IP Address <SQL IP Address 1 (ENTSQL52)>: Checking for network match: network masks 00FFFFFF=00FFFFFF and addresses 3411C70A^0011C70A, role 3.
0000143c.00003720::2014/08/05-16:56:49.973 INFO [RES] IP Address <SQL IP Address 1 (ENTSQL52)>: Online: Opened object handle for netinterface fe516577-6c30-4b96-a6b0-38adb0ccee3e.
0000143c.00003720::2014/08/05-16:56:49.973 INFO [RES] IP Address <SQL IP Address 1 (ENTSQL52)>: Online: Registered notification for netinterface fe516577-6c30-4b96-a6b0-38adb0ccee3e.
0000143c.00003720::2014/08/05-16:56:49.973 ERR [RES] IP Address <SQL IP Address 1 (ENTSQL52)>: NetInterface fe516577-6c30-4b96-a6b0-38adb0ccee3e has failed.
0000143c.00003720::2014/08/05-16:56:49.973 ERR [RHS] Online for resource SQL IP Address 1 (ENTSQL52) failed.
00000e8c.00007480::2014/08/05-16:56:49.973 INFO [RCM] HandleMonitorReply: ONLINERESOURCE for 'SQL IP Address 1 (ENTSQL52)', gen(1) result 5018.
00000e8c.00007480::2014/08/05-16:56:49.973 INFO [RCM] TransitionToState(SQL IP Address 1 (ENTSQL52)) OnlinePending-->ProcessingFailure.
00000e8c.00003098::2014/08/05-16:56:49.973 ERR [RCM] rcm::RcmResource::HandleFailure: (SQL IP Address 1 (ENTSQL52))
00000e8c.00003098::2014/08/05-16:56:49.974 INFO [RCM] resource SQL IP Address 1 (ENTSQL52): failure count: 2, restartAction: 2.
00000e8c.00003098::2014/08/05-16:56:49.974 INFO [RCM] TransitionToState(SQL IP Address 1 (ENTSQL52)) ProcessingFailure-->[Terminating to Failed].
00000e8c.00003098::2014/08/05-16:56:49.974 INFO [RCM] Resource SQL IP Address 1 (ENTSQL52) is causing group ENTSQL52 to failover. Posting worker thread.
0000143c.00007108::2014/08/05-16:56:49.974 INFO [RES] IP Address <SQL IP Address 1 (ENTSQL52)>: Terminating resource...
0000143c.00007108::2014/08/05-16:56:49.974 INFO [RES] IP Address <SQL IP Address 1 (ENTSQL52)>: Resource is already offline.
00000e8c.0000566c::2014/08/05-16:56:49.974 INFO [RCM] rcm::RcmGroup::Failover: (ENTSQL52)
00000e8c.00003170::2014/08/05-16:56:49.975 INFO [RCM] TransitionToState(SQL IP Address 1 (ENTSQL52)) [Terminating to Failed]-->Failed.
000017a0.00000c50::2014/08/05-16:56:50.352 INFO [RES] SQL Server <SQL Server (ENT52)>: [sqsrvres] OnlineThread: asked to terminate while waiting for QP.
This is just a cut out of the logs but if you need more info just let me know.
Thanks for any help!