We have a 2 node active/passive 2008 Sql Cluster that also has a fileshare on it that randomly fails over. We get events
We have updated the NIC drivers on each node, the Drivers and Bios have been updated on the HBA's. We have updated the srv.sys and the srv2.sys files thinking it might be an SMB issue. TCP offloading is disabled on the the Nics. We are running SP2 on both nodes and all the windows updates are current. In the cluster logs we are seeing what is listed below.
HYSQL02
========
00000cc8.00001364::2010/02/17-18:23:32.352 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
00000cc8.00001364::2010/02/17-18:24:32.353 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, ImportFiles), status 64. Tolerating...
00000cc8.00001364::2010/02/17-18:25:32.356 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, PreProd_HelpSystem), status 64. Tolerating...
00000cc8.00001364::2010/02/17-18:26:32.414 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, PreProd_ImportFiles), status 2114. Tolerating...
00000cc8.00001364::2010/02/17-18:29:32.369 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
00000cc8.00001364::2010/02/17-18:32:32.431 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 2114. Tolerating...
00000cc8.00001364::2010/02/17-18:35:32.387 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
00000cc8.00001364::2010/02/17-18:37:32.392 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, ImportFiles), status 64. Tolerating...
00000cc8.00001364::2010/02/17-18:42:32.408 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
00000cc8.00001364::2010/02/17-18:43:32.410 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, ImportFiles), status 64. Tolerating...
00000cc8.00001364::2010/02/17-18:44:32.425 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, PreProd_HelpSystem), status 64. Tolerating...
00000cc8.00001364::2010/02/17-18:48:32.798 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
00000cc8.00001364::2010/02/17-18:51:32.949 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
00000cc8.00001364::2010/02/17-18:54:33.045 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
00000cc8.00001364::2010/02/17-18:58:33.158 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 2114. Tolerating...
00000cc8.00001364::2010/02/17-19:01:33.192 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 2114. Tolerating...
00000cc8.00001364::2010/02/17-19:05:33.166 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
00000cc8.00001364::2010/02/17-19:10:33.182 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
00000cc8.00001364::2010/02/17-19:11:33.184 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, ImportFiles), status 64. Tolerating...
00000cc8.00001364::2010/02/17-19:13:33.190 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, PreProd_HelpSystem), status 64. Tolerating...
00000cc8.00001364::2010/02/17-19:22:33.218 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
00000cc8.00001364::2010/02/17-19:26:33.229 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
00000cc8.00001364::2010/02/17-19:27:33.232 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, ImportFiles), status 64. Tolerating...
00000cc8.00001364::2010/02/17-19:28:33.236 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, PreProd_HelpSystem), status 64. Tolerating...
00000cc8.00001364::2010/02/17-19:29:33.238 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, PreProd_ImportFiles), status 64. Tolerating...
00000cc8.00001364::2010/02/17-19:30:33.241 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, PreProd_ReportImages), status 64. Tolerating...
00000cc8.00000cd4::2010/02/17-19:30:34.000 ERR [RHS] RhsCall::DeadlockMonitor: Call ISALIVE timed out for resource 'FileServer-(MSCS3)(Cluster Disk 4- Database)'.
00000cc8.00000cd4::2010/02/17-19:30:34.000 ERR [RHS] Resource FileServer-(MSCS3)(Cluster Disk 4- Database) handling deadlock. Cleaning current operation and terminaiting RHS process.
000009ec.0000174c::2010/02/17-19:30:34.000 INFO [RCM] HandleMonitorReply: FAILURENOTIFICATION for 'FileServer-(MSCS3)(Cluster Disk 4- Database)', gen(0) result 4.
000009ec.0000174c::2010/02/17-19:30:34.000 INFO [RCM] rcm::RcmResource::HandleMonitorReply: Resource 'FileServer-(MSCS3)(Cluster Disk 4- Database)' consecutive failure count 1.
000009ec.0000174c::2010/02/17-19:30:34.002 ERR [RCM] rcm::RcmMonitor::RecoverProcess: Recovering monitor process 3272 / 0xcc8
000009ec.0000174c::2010/02/17-19:30:34.004 INFO [RCM] Created monitor process 2248 / 0x8c8
000008c8.000010c8::2010/02/17-19:30:34.019 INFO [RHS] Initializing.
000009ec.0000174c::2010/02/17-19:30:34.030 INFO [RCM] rcm::RcmResource::ReattachToMonitorProcess: (FileServer-(MSCS3)(Cluster Disk 4- Database), Online)
000009ec.0000174c::2010/02/17-19:30:34.030 INFO [RCM] TransitionToState(FileServer-(MSCS3)(Cluster Disk 4- Database)) Initializing-->OpenCallIssued.
000009ec.0000174c::2010/02/17-19:30:34.030 INFO [RCM] rcm::RcmGroup::ProcessStateChange: (SQL Server (SQLPRODA), Online --> PartialOnline)
000009ec.0000174c::2010/02/17-19:30:34.055 INFO [RCM] TransitionToState(FileServer-(MSCS3)(Cluster Disk 4- Database)) Online-->ProcessingFailure.
000009ec.0000174c::2010/02/17-19:30:34.055 INFO [RCM] rcm::RcmGroup::ProcessStateChange: (SQL Server (SQLPRODA), PartialOnline --> Failed)
000009ec.0000174c::2010/02/17-19:30:34.055 ERR [RCM] rcm::RcmResource::HandleFailure: (FileServer-(MSCS3)(Cluster Disk 4- Database))
000009ec.0000174c::2010/02/17-19:30:34.055 INFO [RCM] resource FileServer-(MSCS3)(Cluster Disk 4- Database): failure count: 1, restartAction: 2.
000009ec.0000174c::2010/02/17-19:30:34.055 INFO [RCM] Will restart resource in 500 milliseconds.
000009ec.0000174c::2010/02/17-19:30:34.055 INFO [RCM] TransitionToState(FileServer-(MSCS3)(Cluster Disk 4- Database)) ProcessingFailure-->[Terminating to DelayRestartingResource].
000009ec.0000174c::2010/02/17-19:30:34.055 INFO [RCM] rcm::RcmGroup::ProcessStateChange: (SQL Server (SQLPRODA), Failed --> Pending)
000008c8.00001784::2010/02/17-19:30:34.112 INFO [RES] File Server : FileServerDoTerminate: Terminate called... !!!
000009ec.0000126c::2010/02/17-19:30:34.119 INFO [RCM] TransitionToState(FileServer-(MSCS3)(Cluster Disk 4- Database)) [Terminating to DelayRestartingResource]-->DelayRestartingResource.
000009ec.0000174c::2010/02/17-19:30:34.619 INFO [RCM] Delay-restarting FileServer-(MSCS3)(Cluster Disk 4- Database) and any waiting dependents.
000009ec.0000174c::2010/02/17-19:30:34.619 INFO [RCM] TransitionToState(FileServer-(MSCS3)(Cluster Disk 4- Database)) DelayRestartingResource-->OnlineCallIssued.
000009ec.0000126c::2010/02/17-19:30:34.620 INFO [RCM] HandleMonitorReply: ONLINERESOURCE for 'FileServer-(MSCS3)(Cluster Disk 4- Database)', gen(1) result 997.
000009ec.0000126c::2010/02/17-19:30:34.620 INFO [RCM] TransitionToState(FileServer-(MSCS3)(Cluster Disk 4- Database)) OnlineCallIssued-->OnlinePending.
000008c8.000016cc::2010/02/17-19:30:34.657 INFO [RES] File Server : Shares 'are being scoped to virtual name MSCS3
HYSQL01
=========
000015ac.00001200::2010/02/17-21:42:54.976 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 2114. Tolerating...
000015ac.00001200::2010/02/17-21:47:51.082 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 2114. Tolerating...
000015ac.00001200::2010/02/17-21:51:51.094 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 2114. Tolerating...
000015ac.00001200::2010/02/17-21:56:51.056 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
000015ac.00001200::2010/02/17-22:06:51.139 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 2114. Tolerating...
000015ac.00001200::2010/02/17-22:09:51.148 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 2114. Tolerating...
000009e0.00001b08::2010/02/17-22:17:51.431 INFO [NM] Received request from client address 10.1.0.220.
000015ac.00001200::2010/02/17-22:21:51.184 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 2114. Tolerating...
000015ac.00001200::2010/02/17-22:25:31.804 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 2114. Tolerating...
000015ac.00001200::2010/02/17-22:30:34.959 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
000015ac.00001200::2010/02/17-22:31:36.518 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, ImportFiles), status 2114. Tolerating...
000015ac.00001200::2010/02/17-22:34:41.036 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 2114. Tolerating...
000015ac.00001200::2010/02/17-22:39:48.514 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
000015ac.00001200::2010/02/17-22:42:51.247 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 2114. Tolerating...
000009e0.0000132c::2010/02/17-22:44:16.801 INFO [NM] Received request from client address 10.1.0.220.
000015ac.00001200::2010/02/17-22:47:51.209 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
000015ac.00001200::2010/02/17-22:49:51.215 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, ImportFiles), status 64. Tolerating...
000009e0.000015f4::2010/02/17-22:51:27.511 INFO [NM] Received request from client address 10.1.0.220.
000015ac.00001200::2010/02/17-22:52:51.277 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 2114. Tolerating...
000015ac.00001200::2010/02/17-22:55:51.286 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 2114. Tolerating...
000015ac.00001200::2010/02/17-23:06:51.319 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 2114. Tolerating...
000015ac.00001200::2010/02/17-23:12:51.284 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
000015ac.00001200::2010/02/17-23:13:51.340 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, ImportFiles), status 2114. Tolerating...
000015ac.00001200::2010/02/17-23:16:51.349 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 2114. Tolerating...
2nd Issues
----------------
000018f0.0000137c::2010/02/16-18:03:23.988 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, ImportFiles), status 2114. Tolerating...
000018f0.0000137c::2010/02/16-18:07:23.947 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
000018f0.0000137c::2010/02/16-18:11:23.959 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
000018f0.0000137c::2010/02/16-18:13:23.965 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, ImportFiles), status 64. Tolerating...
000018f0.0000137c::2010/02/16-18:14:24.021 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, PreProd_HelpSystem), status 2114. Tolerating...
000018f0.0000137c::2010/02/16-18:20:23.986 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
000018f0.0000137c::2010/02/16-18:23:23.996 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
000018f0.0000137c::2010/02/16-18:26:24.005 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
000018f0.0000137c::2010/02/16-18:27:24.007 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, ImportFiles), status 64. Tolerating...
000018f0.0000137c::2010/02/16-18:28:24.063 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, PreProd_HelpSystem), status 2114. Tolerating...
000018f0.0000137c::2010/02/16-18:37:24.038 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
000018f0.0000137c::2010/02/16-18:38:24.094 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, ImportFiles), status 2114. Tolerating...
000018f0.0000137c::2010/02/16-18:41:24.102 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 2114. Tolerating...
000018f0.0000137c::2010/02/16-18:44:24.059 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
000018f0.0000137c::2010/02/16-18:50:24.129 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 2114. Tolerating...
000018f0.0000137c::2010/02/16-18:54:24.089 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
000018f0.0000137c::2010/02/16-18:55:24.091 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, ImportFiles), status 64. Tolerating...
000018f0.0000137c::2010/02/16-18:56:24.095 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, PreProd_HelpSystem), status 64. Tolerating...
000018f0.0000137c::2010/02/16-18:57:24.151 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, PreProd_ImportFiles), status 2114. Tolerating...
000009e0.00000d2c::2010/02/16-19:13:04.903 INFO [NM] Received request from client address 10.1.0.220.
000018f0.0000137c::2010/02/16-19:18:24.213 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 2114. Tolerating...
000018f0.0000137c::2010/02/16-19:22:24.172 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
000018f0.0000137c::2010/02/16-19:24:24.178 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, ImportFiles), status 64. Tolerating...
000018f0.000012dc::2010/02/16-19:25:25.000 ERR [RHS] RhsCall::DeadlockMonitor: Call ISALIVE timed out for resource 'FileServer-(MSCS3)(Cluster Disk 4- Database)'.
000018f0.000012dc::2010/02/16-19:25:25.000 ERR [RHS] Resource FileServer-(MSCS3)(Cluster Disk 4- Database) handling deadlock. Cleaning current operation and terminaiting RHS process.
000009e0.00000f48::2010/02/16-19:25:25.000 INFO [RCM] HandleMonitorReply: FAILURENOTIFICATION for 'FileServer-(MSCS3)(Cluster Disk 4- Database)', gen(1) result 4.
000009e0.00000f48::2010/02/16-19:25:25.000 INFO [RCM] rcm::RcmResource::HandleMonitorReply: Resource 'FileServer-(MSCS3)(Cluster Disk 4- Database)' consecutive failure count 1.
000009e0.00000f48::2010/02/16-19:25:25.002 ERR [RCM] rcm::RcmMonitor::RecoverProcess: Recovering monitor process 6384 / 0x18f0
000009e0.00000f48::2010/02/16-19:25:25.003 INFO [RCM] Created monitor process 6020 / 0x1784
00001784.00001b1c::2010/02/16-19:25:25.012 INFO [RHS] Initializing.
000009e0.00000f48::2010/02/16-19:25:25.023 INFO [RCM] rcm::RcmResource::ReattachToMonitorProcess: (FileServer-(MSCS3)(Cluster Disk 4- Database), Online)
000009e0.00000f48::2010/02/16-19:25:25.023 INFO [RCM] TransitionToState(FileServer-(MSCS3)(Cluster Disk 4- Database)) Initializing-->OpenCallIssued.
000009e0.00000f48::2010/02/16-19:25:25.023 INFO [RCM] rcm::RcmGroup::ProcessStateChange: (SQL Server (SQLPRODA), Online --> PartialOnline)
3)
00000d80.00000388::2010/02/16-12:15:13.281 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 2114. Tolerating...
00000d80.00000388::2010/02/16-12:19:19.253 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
00000d80.00000388::2010/02/16-12:24:22.132 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
00000d80.00000388::2010/02/16-12:25:22.187 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, ImportFiles), status 2114. Tolerating...
00000d80.00000388::2010/02/16-12:29:22.146 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
00000d80.00000388::2010/02/16-12:42:22.185 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
00000d80.00000388::2010/02/16-12:50:22.209 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
00000d80.00000388::2010/02/16-12:51:22.212 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, ImportFiles), status 64. Tolerating...
00000d80.00000388::2010/02/16-12:53:22.218 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, PreProd_HelpSystem), status 64. Tolerating...
00000d80.00000388::2010/02/16-12:54:22.274 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, PreProd_ImportFiles), status 2114. Tolerating...
00000d80.00000388::2010/02/16-13:01:31.308 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 2114. Tolerating...
00000d80.00000388::2010/02/16-13:10:22.322 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 2114. Tolerating...
00000d80.00000388::2010/02/16-13:13:22.279 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
00000d80.00000388::2010/02/16-13:17:22.291 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
00000d80.00000388::2010/02/16-13:20:22.300 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
00000d80.00000388::2010/02/16-13:22:22.305 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, ImportFiles), status 64. Tolerating...
00000d80.00000388::2010/02/16-13:24:22.311 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, PreProd_HelpSystem), status 64. Tolerating...
00000d80.00000d8c::2010/02/16-13:24:23.000 ERR [RHS] RhsCall::DeadlockMonitor: Call ISALIVE timed out for resource 'FileServer-(MSCS3)(Cluster Disk 4- Database)'.
00000d80.00000d8c::2010/02/16-13:24:23.000 ERR [RHS] Resource FileServer-(MSCS3)(Cluster Disk 4- Database) handling deadlock. Cleaning current operation and terminaiting RHS process.
000009e0.000015dc::2010/02/16-13:24:23.000 INFO [RCM] HandleMonitorReply: FAILURENOTIFICATION for 'FileServer-(MSCS3)(Cluster Disk 4- Database)', gen(0) result 4.
000009e0.000015dc::2010/02/16-13:24:23.000 INFO [RCM] rcm::RcmResource::HandleMonitorReply: Resource 'FileServer-(MSCS3)(Cluster Disk 4- Database)' consecutive failure count 1.
000009e0.000015dc::2010/02/16-13:24:23.002 ERR [RCM] rcm::RcmMonitor::RecoverProcess: Recovering monitor process 3456 / 0xd80
4)
00001770.00001594::2010/02/09-16:01:06.362 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, PreProd_ReportImages), status 2114. Tolerating...
00000aa4.0000183c::2010/02/09-16:01:15.630 INFO [RES] Physical Disk: HardDiskpGetDiskInfo: Disk is of type MBR, signature 0x3ba33338
00000aa4.0000183c::2010/02/09-16:01:19.036 INFO [RES] Physical Disk: HardDiskpGetDiskInfo: Disk is of type MBR, signature 0x3ba3333f
00000aa4.0000183c::2010/02/09-16:01:19.040 INFO [RES] Physical Disk: HardDiskpGetDiskInfo: Disk is of type MBR, signature 0x3ba3333a
00000aa4.0000183c::2010/02/09-16:01:19.044 INFO [RES] Physical Disk: HardDiskpGetDiskInfo: Disk is of type MBR, signature 0x3ba33339
00001770.00001910::2010/02/09-16:05:06.311 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
00001770.00001910::2010/02/09-16:06:06.314 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, ImportFiles), status 64. Tolerating...
00001770.00001910::2010/02/09-16:07:06.317 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, PreProd_HelpSystem), status 64. Tolerating...
00001770.00001910::2010/02/09-16:08:06.320 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, PreProd_ImportFiles), status 64. Tolerating...
00001770.00000d14::2010/02/09-16:08:07.000 ERR [RHS] RhsCall::DeadlockMonitor: Call ISALIVE timed out for resource 'FileServer-(MSCS3)(Cluster Disk 4- Database)'.
00001770.00000d14::2010/02/09-16:08:07.000 ERR [RHS] Resource FileServer-(MSCS3)(Cluster Disk 4- Database) handling deadlock. Cleaning current operation and terminaiting RHS process.
000009f0.00001324::2010/02/09-16:08:07.000 INFO [RCM] HandleMonitorReply: FAILURENOTIFICATION for 'FileServer-(MSCS3)(Cluster Disk 4- Database)', gen(4) result 4.
000009f0.00001324::2010/02/09-16:08:07.000 INFO [RCM] rcm::RcmResource::HandleMonitorReply: Resource 'FileServer-(MSCS3)(Cluster Disk 4- Database)' consecutive failure count 1.
000009f0.00001324::2010/02/09-16:08:07.002 ERR [RCM] rcm::RcmMonitor::RecoverProcess: Recovering monitor process 6000 / 0x1770
000009f0.00001324::2010/02/09-16:08:07.003 INFO [RCM] Created monitor process 4748 / 0x128c
Analysis
----------------
We are getting Error 64 and 2114 and the File share is failing with a Deadlock Error
Status 64 = the specified network name is no longer available.
Status 2114 = The Server service is not started.
We setup Netmon and ran traces yesterday when the issue happened and they did not show anything. The Server service does not seem to get any errors.
We have also engaged EMC into the issue and MS has escalated the case but wanted to see if anyone else has experienced this issue or found any resolution. We have run out of options.
Events from Cluster Admin
Event ID 1230
cluster resource 'FileServer-(MSCS3)(Cluster Disk 4- Database)' (resource type '', DLL 'clusres.dll') either crashed or deadlocked. The Resource Hosting Subsystem (RHS) process will now attempt to terminate, and the resource will be marked to run in a separate monitor.
Event2
EventID 1146
the cluster resource host subsystem (RHS) stopped unexpectedly. An attempt will be made to restart it. This is usually due to a problem in a resource DLL. Please determine which resource DLL is causing the issue and report the problem to the resource vendor.
Event 3
EventID 1069
Cluster resource 'FileServer-(MSCS3)(Cluster Disk 4- Database)' in clustered service or application 'SQL Server (SQLPRODA)' failed.
Event 4
Event ID 1205
The Cluster service failed to bring clustered service or application 'SQL Server (SQLPRODA)' completely online or offline. One or more resources may be in a failed state. This may impact the availability of the clustered service or application.
We have updated the NIC drivers on each node, the Drivers and Bios have been updated on the HBA's. We have updated the srv.sys and the srv2.sys files thinking it might be an SMB issue. TCP offloading is disabled on the the Nics. We are running SP2 on both nodes and all the windows updates are current. In the cluster logs we are seeing what is listed below.
HYSQL02
========
00000cc8.00001364::2010/02/17-18:23:32.352 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
00000cc8.00001364::2010/02/17-18:24:32.353 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, ImportFiles), status 64. Tolerating...
00000cc8.00001364::2010/02/17-18:25:32.356 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, PreProd_HelpSystem), status 64. Tolerating...
00000cc8.00001364::2010/02/17-18:26:32.414 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, PreProd_ImportFiles), status 2114. Tolerating...
00000cc8.00001364::2010/02/17-18:29:32.369 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
00000cc8.00001364::2010/02/17-18:32:32.431 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 2114. Tolerating...
00000cc8.00001364::2010/02/17-18:35:32.387 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
00000cc8.00001364::2010/02/17-18:37:32.392 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, ImportFiles), status 64. Tolerating...
00000cc8.00001364::2010/02/17-18:42:32.408 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
00000cc8.00001364::2010/02/17-18:43:32.410 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, ImportFiles), status 64. Tolerating...
00000cc8.00001364::2010/02/17-18:44:32.425 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, PreProd_HelpSystem), status 64. Tolerating...
00000cc8.00001364::2010/02/17-18:48:32.798 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
00000cc8.00001364::2010/02/17-18:51:32.949 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
00000cc8.00001364::2010/02/17-18:54:33.045 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
00000cc8.00001364::2010/02/17-18:58:33.158 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 2114. Tolerating...
00000cc8.00001364::2010/02/17-19:01:33.192 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 2114. Tolerating...
00000cc8.00001364::2010/02/17-19:05:33.166 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
00000cc8.00001364::2010/02/17-19:10:33.182 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
00000cc8.00001364::2010/02/17-19:11:33.184 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, ImportFiles), status 64. Tolerating...
00000cc8.00001364::2010/02/17-19:13:33.190 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, PreProd_HelpSystem), status 64. Tolerating...
00000cc8.00001364::2010/02/17-19:22:33.218 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
00000cc8.00001364::2010/02/17-19:26:33.229 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
00000cc8.00001364::2010/02/17-19:27:33.232 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, ImportFiles), status 64. Tolerating...
00000cc8.00001364::2010/02/17-19:28:33.236 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, PreProd_HelpSystem), status 64. Tolerating...
00000cc8.00001364::2010/02/17-19:29:33.238 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, PreProd_ImportFiles), status 64. Tolerating...
00000cc8.00001364::2010/02/17-19:30:33.241 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, PreProd_ReportImages), status 64. Tolerating...
00000cc8.00000cd4::2010/02/17-19:30:34.000 ERR [RHS] RhsCall::DeadlockMonitor: Call ISALIVE timed out for resource 'FileServer-(MSCS3)(Cluster Disk 4- Database)'.
00000cc8.00000cd4::2010/02/17-19:30:34.000 ERR [RHS] Resource FileServer-(MSCS3)(Cluster Disk 4- Database) handling deadlock. Cleaning current operation and terminaiting RHS process.
000009ec.0000174c::2010/02/17-19:30:34.000 INFO [RCM] HandleMonitorReply: FAILURENOTIFICATION for 'FileServer-(MSCS3)(Cluster Disk 4- Database)', gen(0) result 4.
000009ec.0000174c::2010/02/17-19:30:34.000 INFO [RCM] rcm::RcmResource::HandleMonitorReply: Resource 'FileServer-(MSCS3)(Cluster Disk 4- Database)' consecutive failure count 1.
000009ec.0000174c::2010/02/17-19:30:34.002 ERR [RCM] rcm::RcmMonitor::RecoverProcess: Recovering monitor process 3272 / 0xcc8
000009ec.0000174c::2010/02/17-19:30:34.004 INFO [RCM] Created monitor process 2248 / 0x8c8
000008c8.000010c8::2010/02/17-19:30:34.019 INFO [RHS] Initializing.
000009ec.0000174c::2010/02/17-19:30:34.030 INFO [RCM] rcm::RcmResource::ReattachToMonitorProcess: (FileServer-(MSCS3)(Cluster Disk 4- Database), Online)
000009ec.0000174c::2010/02/17-19:30:34.030 INFO [RCM] TransitionToState(FileServer-(MSCS3)(Cluster Disk 4- Database)) Initializing-->OpenCallIssued.
000009ec.0000174c::2010/02/17-19:30:34.030 INFO [RCM] rcm::RcmGroup::ProcessStateChange: (SQL Server (SQLPRODA), Online --> PartialOnline)
000009ec.0000174c::2010/02/17-19:30:34.055 INFO [RCM] TransitionToState(FileServer-(MSCS3)(Cluster Disk 4- Database)) Online-->ProcessingFailure.
000009ec.0000174c::2010/02/17-19:30:34.055 INFO [RCM] rcm::RcmGroup::ProcessStateChange: (SQL Server (SQLPRODA), PartialOnline --> Failed)
000009ec.0000174c::2010/02/17-19:30:34.055 ERR [RCM] rcm::RcmResource::HandleFailure: (FileServer-(MSCS3)(Cluster Disk 4- Database))
000009ec.0000174c::2010/02/17-19:30:34.055 INFO [RCM] resource FileServer-(MSCS3)(Cluster Disk 4- Database): failure count: 1, restartAction: 2.
000009ec.0000174c::2010/02/17-19:30:34.055 INFO [RCM] Will restart resource in 500 milliseconds.
000009ec.0000174c::2010/02/17-19:30:34.055 INFO [RCM] TransitionToState(FileServer-(MSCS3)(Cluster Disk 4- Database)) ProcessingFailure-->[Terminating to DelayRestartingResource].
000009ec.0000174c::2010/02/17-19:30:34.055 INFO [RCM] rcm::RcmGroup::ProcessStateChange: (SQL Server (SQLPRODA), Failed --> Pending)
000008c8.00001784::2010/02/17-19:30:34.112 INFO [RES] File Server : FileServerDoTerminate: Terminate called... !!!
000009ec.0000126c::2010/02/17-19:30:34.119 INFO [RCM] TransitionToState(FileServer-(MSCS3)(Cluster Disk 4- Database)) [Terminating to DelayRestartingResource]-->DelayRestartingResource.
000009ec.0000174c::2010/02/17-19:30:34.619 INFO [RCM] Delay-restarting FileServer-(MSCS3)(Cluster Disk 4- Database) and any waiting dependents.
000009ec.0000174c::2010/02/17-19:30:34.619 INFO [RCM] TransitionToState(FileServer-(MSCS3)(Cluster Disk 4- Database)) DelayRestartingResource-->OnlineCallIssued.
000009ec.0000126c::2010/02/17-19:30:34.620 INFO [RCM] HandleMonitorReply: ONLINERESOURCE for 'FileServer-(MSCS3)(Cluster Disk 4- Database)', gen(1) result 997.
000009ec.0000126c::2010/02/17-19:30:34.620 INFO [RCM] TransitionToState(FileServer-(MSCS3)(Cluster Disk 4- Database)) OnlineCallIssued-->OnlinePending.
000008c8.000016cc::2010/02/17-19:30:34.657 INFO [RES] File Server : Shares 'are being scoped to virtual name MSCS3
HYSQL01
=========
000015ac.00001200::2010/02/17-21:42:54.976 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 2114. Tolerating...
000015ac.00001200::2010/02/17-21:47:51.082 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 2114. Tolerating...
000015ac.00001200::2010/02/17-21:51:51.094 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 2114. Tolerating...
000015ac.00001200::2010/02/17-21:56:51.056 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
000015ac.00001200::2010/02/17-22:06:51.139 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 2114. Tolerating...
000015ac.00001200::2010/02/17-22:09:51.148 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 2114. Tolerating...
000009e0.00001b08::2010/02/17-22:17:51.431 INFO [NM] Received request from client address 10.1.0.220.
000015ac.00001200::2010/02/17-22:21:51.184 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 2114. Tolerating...
000015ac.00001200::2010/02/17-22:25:31.804 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 2114. Tolerating...
000015ac.00001200::2010/02/17-22:30:34.959 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
000015ac.00001200::2010/02/17-22:31:36.518 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, ImportFiles), status 2114. Tolerating...
000015ac.00001200::2010/02/17-22:34:41.036 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 2114. Tolerating...
000015ac.00001200::2010/02/17-22:39:48.514 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
000015ac.00001200::2010/02/17-22:42:51.247 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 2114. Tolerating...
000009e0.0000132c::2010/02/17-22:44:16.801 INFO [NM] Received request from client address 10.1.0.220.
000015ac.00001200::2010/02/17-22:47:51.209 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
000015ac.00001200::2010/02/17-22:49:51.215 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, ImportFiles), status 64. Tolerating...
000009e0.000015f4::2010/02/17-22:51:27.511 INFO [NM] Received request from client address 10.1.0.220.
000015ac.00001200::2010/02/17-22:52:51.277 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 2114. Tolerating...
000015ac.00001200::2010/02/17-22:55:51.286 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 2114. Tolerating...
000015ac.00001200::2010/02/17-23:06:51.319 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 2114. Tolerating...
000015ac.00001200::2010/02/17-23:12:51.284 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
000015ac.00001200::2010/02/17-23:13:51.340 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, ImportFiles), status 2114. Tolerating...
000015ac.00001200::2010/02/17-23:16:51.349 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 2114. Tolerating...
2nd Issues
----------------
000018f0.0000137c::2010/02/16-18:03:23.988 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, ImportFiles), status 2114. Tolerating...
000018f0.0000137c::2010/02/16-18:07:23.947 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
000018f0.0000137c::2010/02/16-18:11:23.959 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
000018f0.0000137c::2010/02/16-18:13:23.965 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, ImportFiles), status 64. Tolerating...
000018f0.0000137c::2010/02/16-18:14:24.021 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, PreProd_HelpSystem), status 2114. Tolerating...
000018f0.0000137c::2010/02/16-18:20:23.986 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
000018f0.0000137c::2010/02/16-18:23:23.996 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
000018f0.0000137c::2010/02/16-18:26:24.005 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
000018f0.0000137c::2010/02/16-18:27:24.007 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, ImportFiles), status 64. Tolerating...
000018f0.0000137c::2010/02/16-18:28:24.063 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, PreProd_HelpSystem), status 2114. Tolerating...
000018f0.0000137c::2010/02/16-18:37:24.038 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
000018f0.0000137c::2010/02/16-18:38:24.094 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, ImportFiles), status 2114. Tolerating...
000018f0.0000137c::2010/02/16-18:41:24.102 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 2114. Tolerating...
000018f0.0000137c::2010/02/16-18:44:24.059 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
000018f0.0000137c::2010/02/16-18:50:24.129 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 2114. Tolerating...
000018f0.0000137c::2010/02/16-18:54:24.089 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
000018f0.0000137c::2010/02/16-18:55:24.091 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, ImportFiles), status 64. Tolerating...
000018f0.0000137c::2010/02/16-18:56:24.095 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, PreProd_HelpSystem), status 64. Tolerating...
000018f0.0000137c::2010/02/16-18:57:24.151 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, PreProd_ImportFiles), status 2114. Tolerating...
000009e0.00000d2c::2010/02/16-19:13:04.903 INFO [NM] Received request from client address 10.1.0.220.
000018f0.0000137c::2010/02/16-19:18:24.213 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 2114. Tolerating...
000018f0.0000137c::2010/02/16-19:22:24.172 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
000018f0.0000137c::2010/02/16-19:24:24.178 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, ImportFiles), status 64. Tolerating...
000018f0.000012dc::2010/02/16-19:25:25.000 ERR [RHS] RhsCall::DeadlockMonitor: Call ISALIVE timed out for resource 'FileServer-(MSCS3)(Cluster Disk 4- Database)'.
000018f0.000012dc::2010/02/16-19:25:25.000 ERR [RHS] Resource FileServer-(MSCS3)(Cluster Disk 4- Database) handling deadlock. Cleaning current operation and terminaiting RHS process.
000009e0.00000f48::2010/02/16-19:25:25.000 INFO [RCM] HandleMonitorReply: FAILURENOTIFICATION for 'FileServer-(MSCS3)(Cluster Disk 4- Database)', gen(1) result 4.
000009e0.00000f48::2010/02/16-19:25:25.000 INFO [RCM] rcm::RcmResource::HandleMonitorReply: Resource 'FileServer-(MSCS3)(Cluster Disk 4- Database)' consecutive failure count 1.
000009e0.00000f48::2010/02/16-19:25:25.002 ERR [RCM] rcm::RcmMonitor::RecoverProcess: Recovering monitor process 6384 / 0x18f0
000009e0.00000f48::2010/02/16-19:25:25.003 INFO [RCM] Created monitor process 6020 / 0x1784
00001784.00001b1c::2010/02/16-19:25:25.012 INFO [RHS] Initializing.
000009e0.00000f48::2010/02/16-19:25:25.023 INFO [RCM] rcm::RcmResource::ReattachToMonitorProcess: (FileServer-(MSCS3)(Cluster Disk 4- Database), Online)
000009e0.00000f48::2010/02/16-19:25:25.023 INFO [RCM] TransitionToState(FileServer-(MSCS3)(Cluster Disk 4- Database)) Initializing-->OpenCallIssued.
000009e0.00000f48::2010/02/16-19:25:25.023 INFO [RCM] rcm::RcmGroup::ProcessStateChange: (SQL Server (SQLPRODA), Online --> PartialOnline)
3)
00000d80.00000388::2010/02/16-12:15:13.281 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 2114. Tolerating...
00000d80.00000388::2010/02/16-12:19:19.253 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
00000d80.00000388::2010/02/16-12:24:22.132 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
00000d80.00000388::2010/02/16-12:25:22.187 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, ImportFiles), status 2114. Tolerating...
00000d80.00000388::2010/02/16-12:29:22.146 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
00000d80.00000388::2010/02/16-12:42:22.185 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
00000d80.00000388::2010/02/16-12:50:22.209 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
00000d80.00000388::2010/02/16-12:51:22.212 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, ImportFiles), status 64. Tolerating...
00000d80.00000388::2010/02/16-12:53:22.218 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, PreProd_HelpSystem), status 64. Tolerating...
00000d80.00000388::2010/02/16-12:54:22.274 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, PreProd_ImportFiles), status 2114. Tolerating...
00000d80.00000388::2010/02/16-13:01:31.308 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 2114. Tolerating...
00000d80.00000388::2010/02/16-13:10:22.322 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 2114. Tolerating...
00000d80.00000388::2010/02/16-13:13:22.279 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
00000d80.00000388::2010/02/16-13:17:22.291 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
00000d80.00000388::2010/02/16-13:20:22.300 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
00000d80.00000388::2010/02/16-13:22:22.305 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, ImportFiles), status 64. Tolerating...
00000d80.00000388::2010/02/16-13:24:22.311 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, PreProd_HelpSystem), status 64. Tolerating...
00000d80.00000d8c::2010/02/16-13:24:23.000 ERR [RHS] RhsCall::DeadlockMonitor: Call ISALIVE timed out for resource 'FileServer-(MSCS3)(Cluster Disk 4- Database)'.
00000d80.00000d8c::2010/02/16-13:24:23.000 ERR [RHS] Resource FileServer-(MSCS3)(Cluster Disk 4- Database) handling deadlock. Cleaning current operation and terminaiting RHS process.
000009e0.000015dc::2010/02/16-13:24:23.000 INFO [RCM] HandleMonitorReply: FAILURENOTIFICATION for 'FileServer-(MSCS3)(Cluster Disk 4- Database)', gen(0) result 4.
000009e0.000015dc::2010/02/16-13:24:23.000 INFO [RCM] rcm::RcmResource::HandleMonitorReply: Resource 'FileServer-(MSCS3)(Cluster Disk 4- Database)' consecutive failure count 1.
000009e0.000015dc::2010/02/16-13:24:23.002 ERR [RCM] rcm::RcmMonitor::RecoverProcess: Recovering monitor process 3456 / 0xd80
4)
00001770.00001594::2010/02/09-16:01:06.362 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, PreProd_ReportImages), status 2114. Tolerating...
00000aa4.0000183c::2010/02/09-16:01:15.630 INFO [RES] Physical Disk: HardDiskpGetDiskInfo: Disk is of type MBR, signature 0x3ba33338
00000aa4.0000183c::2010/02/09-16:01:19.036 INFO [RES] Physical Disk: HardDiskpGetDiskInfo: Disk is of type MBR, signature 0x3ba3333f
00000aa4.0000183c::2010/02/09-16:01:19.040 INFO [RES] Physical Disk: HardDiskpGetDiskInfo: Disk is of type MBR, signature 0x3ba3333a
00000aa4.0000183c::2010/02/09-16:01:19.044 INFO [RES] Physical Disk: HardDiskpGetDiskInfo: Disk is of type MBR, signature 0x3ba33339
00001770.00001910::2010/02/09-16:05:06.311 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, HelpSystem), status 64. Tolerating...
00001770.00001910::2010/02/09-16:06:06.314 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, ImportFiles), status 64. Tolerating...
00001770.00001910::2010/02/09-16:07:06.317 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, PreProd_HelpSystem), status 64. Tolerating...
00001770.00001910::2010/02/09-16:08:06.320 WARN [RES] File Server : Failed in NetShareGetInfo(MSCS3, PreProd_ImportFiles), status 64. Tolerating...
00001770.00000d14::2010/02/09-16:08:07.000 ERR [RHS] RhsCall::DeadlockMonitor: Call ISALIVE timed out for resource 'FileServer-(MSCS3)(Cluster Disk 4- Database)'.
00001770.00000d14::2010/02/09-16:08:07.000 ERR [RHS] Resource FileServer-(MSCS3)(Cluster Disk 4- Database) handling deadlock. Cleaning current operation and terminaiting RHS process.
000009f0.00001324::2010/02/09-16:08:07.000 INFO [RCM] HandleMonitorReply: FAILURENOTIFICATION for 'FileServer-(MSCS3)(Cluster Disk 4- Database)', gen(4) result 4.
000009f0.00001324::2010/02/09-16:08:07.000 INFO [RCM] rcm::RcmResource::HandleMonitorReply: Resource 'FileServer-(MSCS3)(Cluster Disk 4- Database)' consecutive failure count 1.
000009f0.00001324::2010/02/09-16:08:07.002 ERR [RCM] rcm::RcmMonitor::RecoverProcess: Recovering monitor process 6000 / 0x1770
000009f0.00001324::2010/02/09-16:08:07.003 INFO [RCM] Created monitor process 4748 / 0x128c
Analysis
----------------
We are getting Error 64 and 2114 and the File share is failing with a Deadlock Error
Status 64 = the specified network name is no longer available.
Status 2114 = The Server service is not started.
We setup Netmon and ran traces yesterday when the issue happened and they did not show anything. The Server service does not seem to get any errors.
We have also engaged EMC into the issue and MS has escalated the case but wanted to see if anyone else has experienced this issue or found any resolution. We have run out of options.