Hi Everyone
I have built a Windows 2008 R2 Failover Cluster. It is scheduled to be a HyperV cluster, but I have removed HyperV from the equation as I am having problems.
Its a 2 node cluster connecting to a PS4000 via iscsi.
Network config is
Nic 1 - Host and Cluster Management - 10.160.19.x/24
Nic 2 & 3 - icsci 10.160.46..x/24, MPIO,
Nic 4 - CSV - 10.160.47/24
Nic 5 - LiveMigration - 10.160.49.x/24
Nic6 - VMSwitch - HyperV Virtual Machine "External" nic.
The cluster ip is on 10.160.19.x.
The Cluster validation passes sucessfully and so does creating the cluster and performing post creating validations. When I simulate failures of the Quorum disk, it fails over to the second node after requisites retrys on the first node...all good.
However when I simulate failures on the second cluster disk, it just fails on the primary node. I can bring it online on same node or if I stop the cluster service on that node, everything successfully fails over to the second node. But automatic/simulating failover isn't working.
When I dump out the cluster log(below)this is all the entries pertaining to the failover attempt. Sorry for the log dump.
My analysis of the log, leads me in two directions, Is my nic assignment and config correct? Can I confirm the Cluster disk guids are identical between nodes?
I appreciate any assistance. Apologies on the large log.
Thanks
[NM] Received request from client address 10.160.19.22.
00000b18.00000f50::2011/01/13-16:10:32.005 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
00000730.00000d3c::2011/01/13-16:10:32.005 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00000730.00000d3c::2011/01/13-16:10:32.005 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node. Attempting to find a good node...
00000b18.00000f50::2011/01/13-16:10:32.005 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
00000730.00000d40::2011/01/13-16:10:32.005 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00000b18.00000f50::2011/01/13-16:10:32.317 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
00000730.00000d3c::2011/01/13-16:10:32.317 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000730.00000d3c::2011/01/13-16:10:32.317 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node. Attempting to find a good node...
00000b18.00000f50::2011/01/13-16:10:32.317 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
00000730.000005e0::2011/01/13-16:10:32.317 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000b18.00000f50::2011/01/13-16:10:32.317 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
00000730.000005e0::2011/01/13-16:10:32.317 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00000730.000005e0::2011/01/13-16:10:32.317 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node. Attempting to find a good node...
00000b18.00000c1c::2011/01/13-16:10:32.317 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
00000730.00000d3c::2011/01/13-16:10:32.317 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00000b18.00000c1c::2011/01/13-16:10:32.332 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
00000730.00000d3c::2011/01/13-16:10:32.332 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000730.00000d3c::2011/01/13-16:10:32.332 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node. Attempting to find a good node...
00000b18.00000c1c::2011/01/13-16:10:32.332 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
00000730.00000d40::2011/01/13-16:10:32.332 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000b18.00000c1c::2011/01/13-16:10:32.332 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
00000730.000005e0::2011/01/13-16:10:32.332 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00000730.000005e0::2011/01/13-16:10:32.332 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node. Attempting to find a good node...
00000b18.00000c1c::2011/01/13-16:10:32.332 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
00000730.00000d40::2011/01/13-16:10:32.332 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00000b18.00000c1c::2011/01/13-16:10:32.332 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
00000730.00000d3c::2011/01/13-16:10:32.332 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000730.00000d3c::2011/01/13-16:10:32.332 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node. Attempting to find a good node...
00000b18.00000c1c::2011/01/13-16:10:32.332 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
00000730.000005e0::2011/01/13-16:10:32.332 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000730.00000d3c::2011/01/13-16:10:35.296 INFO [RCM] rcm::RcmApi::FailResource: (VM1)
00000730.00000d40::2011/01/13-16:10:35.296 INFO [RCM] HandleMonitorReply: FAILURENOTIFICATION for 'VM1', gen(1) result 0.
00000730.00000d40::2011/01/13-16:10:35.296 INFO [RCM] TransitionToState(VM1) Online-->ProcessingFailure.
00000730.00000d40::2011/01/13-16:10:35.296 INFO [RCM] rcm::RcmGroup::UpdateStateIfChanged: (Available Storage, Online --> Failed)
00000730.00000d40::2011/01/13-16:10:35.296 ERR [RCM] rcm::RcmResource::HandleFailure: (VM1)
00000730.00000d40::2011/01/13-16:10:35.296 INFO [RCM] resource VM1: failure count: 1, restartAction: 2.
00000730.00000d40::2011/01/13-16:10:35.296 INFO [RCM] Will restart resource in 500 milliseconds.
00000730.00000d40::2011/01/13-16:10:35.296 INFO [RCM] TransitionToState(VM1) ProcessingFailure-->[WaitingToTerminate to DelayRestartingResource].
00000730.00000d40::2011/01/13-16:10:35.296 INFO [RCM] rcm::RcmGroup::UpdateStateIfChanged: (Available Storage, Failed --> Pending)
00000730.00000d40::2011/01/13-16:10:35.296 INFO [RCM] TransitionToState(VM1) [WaitingToTerminate to DelayRestartingResource]-->[Terminating to DelayRestartingResource].
00000bd4.00000eb0::2011/01/13-16:10:35.296 INFO [RES] Physical Disk <VM1>: Terminate request.
00000bd4.00000eb0::2011/01/13-16:10:35.327 WARN [RES] Physical Disk <VM1>: Terminate: Failed to offline volume \Device\Harddisk3\Partition1, Error 5
00000b18.00000c1c::2011/01/13-16:10:35.343 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
00000730.00000d3c::2011/01/13-16:10:35.343 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00000730.00000d3c::2011/01/13-16:10:35.343 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node. Attempting to find a good node...
00000b18.00000c1c::2011/01/13-16:10:35.343 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
00000730.00000d30::2011/01/13-16:10:35.343 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00000bd4.00000eb0::2011/01/13-16:10:35.343 INFO [RES] Physical Disk <VM1>: HardDiskpCloseSVIHandles: Exit
00000b18.00000c1c::2011/01/13-16:10:35.343 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
00000730.00000d3c::2011/01/13-16:10:35.343 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000730.00000d3c::2011/01/13-16:10:35.343 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node. Attempting to find a good node...
00000b18.00000c1c::2011/01/13-16:10:35.343 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
00000730.00000d30::2011/01/13-16:10:35.343 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000b18.00000c1c::2011/01/13-16:10:35.359 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
00000730.00000d3c::2011/01/13-16:10:35.359 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00000730.00000d3c::2011/01/13-16:10:35.359 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node. Attempting to find a good node...
00000b18.00000c1c::2011/01/13-16:10:35.359 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
00000730.00000d30::2011/01/13-16:10:35.359 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00000b18.00000c1c::2011/01/13-16:10:35.359 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
00000730.00000d3c::2011/01/13-16:10:35.359 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000730.00000d3c::2011/01/13-16:10:35.359 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node. Attempting to find a good node...
00000b18.00000c1c::2011/01/13-16:10:35.359 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
00000730.00000d30::2011/01/13-16:10:35.359 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000b18.00000c1c::2011/01/13-16:10:35.437 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
00000730.00000d3c::2011/01/13-16:10:35.437 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00000730.00000d3c::2011/01/13-16:10:35.437 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node. Attempting to find a good node...
00000b18.00000c1c::2011/01/13-16:10:35.437 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
00000730.00000d30::2011/01/13-16:10:35.437 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00000b18.00000c1c::2011/01/13-16:10:35.452 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
00000730.00000d3c::2011/01/13-16:10:35.452 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000730.00000d3c::2011/01/13-16:10:35.452 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node. Attempting to find a good node...
00000b18.00000c1c::2011/01/13-16:10:35.452 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
00000730.00000d30::2011/01/13-16:10:35.452 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000b18.00000c1c::2011/01/13-16:10:35.452 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
00000730.00000d3c::2011/01/13-16:10:35.452 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00000730.00000d3c::2011/01/13-16:10:35.452 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node. Attempting to find a good node...
00000b18.00000c1c::2011/01/13-16:10:35.452 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
00000730.00000d30::2011/01/13-16:10:35.452 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00000b18.00000c1c::2011/01/13-16:10:35.452 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
00000730.00000d3c::2011/01/13-16:10:35.452 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000730.00000d3c::2011/01/13-16:10:35.452 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node. Attempting to find a good node...
00000b18.00000c1c::2011/01/13-16:10:35.452 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
00000730.00000d30::2011/01/13-16:10:35.452 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000bd4.00000eb0::2011/01/13-16:10:36.263 INFO [RES] Physical Disk: ReleaseDisk: stop reserve succeeded on device 3 (sig 44423655)
00000730.00000d30::2011/01/13-16:10:36.279 INFO [RCM] HandleMonitorReply: TERMINATERESOURCE for 'VM1', gen(2) result 0.
00000730.00000d30::2011/01/13-16:10:36.279 INFO [RCM] TransitionToState(VM1) [Terminating to DelayRestartingResource]-->DelayRestartingResource.
00000730.00000d3c::2011/01/13-16:10:36.794 INFO [RCM] Delay-restarting VM1 and any waiting dependents.
00000730.00000d3c::2011/01/13-16:10:36.794 INFO [RCM] TransitionToState(VM1) DelayRestartingResource-->OnlineCallIssued.
00000bd4.00000eb0::2011/01/13-16:10:36.794 INFO [RES] Physical Disk <VM1>: Online request.
00000730.00000298::2011/01/13-16:10:36.794 INFO [RCM] HandleMonitorReply: ONLINERESOURCE for 'VM1', gen(2) result 997.
00000730.00000298::2011/01/13-16:10:36.794 INFO [RCM] TransitionToState(VM1) OnlineCallIssued-->OnlinePending.
00000bd4.00000c04::2011/01/13-16:10:36.794 INFO [RES] Physical Disk <VM1>: Arbitrate request: FastPath 0
00000bd4.00000c04::2011/01/13-16:10:36.794 INFO [RES] Physical Disk: Enter EnumerateDevices: EnumDevice 0
00000bd4.00000c04::2011/01/13-16:10:36.809 INFO [RES] Physical Disk: Exit EnumerateDevices: status 0
00000bd4.00000c04::2011/01/13-16:10:36.841 INFO [RES] Physical Disk: Successful reserve, disk is unowned, key 4daf7466734d
00000bd4.00000c04::2011/01/13-16:10:36.841 INFO [RES] Physical Disk: Update disk props returns 0
00000bd4.00000c04::2011/01/13-16:10:36.841 INFO [RES] Physical Disk <VM1>: Disk is offline
00000730.00000298::2011/01/13-16:10:36.841 INFO [GUM] Node 1: Processing RequestLock 1:42
00000730.00000314::2011/01/13-16:10:36.841 INFO [GUM] Node 1: Processing GrantLock to 1 (sent by 2 gumid: 566)
00000bd4.00000c04::2011/01/13-16:10:36.841 WARN [RES] Physical Disk <VM1>: OnlineThread: Failed to get volume guid for device\\?\GLOBALROOT\Device\Harddisk3\Partition1\. Error 3
00000b18.00000c1c::2011/01/13-16:10:36.856 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
00000730.00000d3c::2011/01/13-16:10:36.856 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00000730.00000d3c::2011/01/13-16:10:36.856 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node. Attempting to find a good node...
00000bd4.00000bf8::2011/01/13-16:10:36.856 INFO [RES] Physical Disk: PNP: Adding volume
\\?\STORAGE#Volume#{73432ff7-18ee-11e0-aded-0026b93b46a0}#0000000000100000#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b}
00000bd4.00000f30::2011/01/13-16:10:36.856 INFO [RES] Physical Disk <VM1>: Completed HardDiskpUpdateRegistryWorker (start time 16:10:36.841)
00000bd4.00000bf8::2011/01/13-16:10:36.856 INFO [RES] Physical Disk: PNP: Add Volume exit, status 0
00000b18.00000c1c::2011/01/13-16:10:36.856 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
00000730.00000d40::2011/01/13-16:10:36.856 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00000b18.00000c1c::2011/01/13-16:10:36.856 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
00000730.00000d3c::2011/01/13-16:10:36.856 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000730.00000d3c::2011/01/13-16:10:36.856 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node. Attempting to find a good node...
00000b18.00000c1c::2011/01/13-16:10:36.856 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
00000730.00000d40::2011/01/13-16:10:36.856 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000b18.00000c1c::2011/01/13-16:10:36.872 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
00000730.00000d3c::2011/01/13-16:10:36.872 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00000730.00000d3c::2011/01/13-16:10:36.872 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node. Attempting to find a good node...
00000b18.00000c1c::2011/01/13-16:10:36.872 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
00000730.00000d40::2011/01/13-16:10:36.872 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00000b18.00000c1c::2011/01/13-16:10:36.872 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
00000730.00000d3c::2011/01/13-16:10:36.872 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000730.00000d3c::2011/01/13-16:10:36.872 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node. Attempting to find a good node...
00000b18.00000c1c::2011/01/13-16:10:36.872 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
00000730.00000298::2011/01/13-16:10:36.872 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000bd4.00000c04::2011/01/13-16:10:40.850 WARN [RES] Physical Disk <VM1>: OnlineThread: Failed to set volguid \??\Volume{7343300a-18ee-11e0-aded-0026b93b46a0}. Error: 183.
00000bd4.00000c04::2011/01/13-16:10:40.850 INFO [RES] Physical Disk <VM1>: Found 2 mount points for device \Device\Harddisk3\Partition1
00000bd4.00000c04::2011/01/13-16:10:40.850 INFO [RES] Physical Disk <VM1>: VolumeIsNtfs: Volume\\?\GLOBALROOT\Device\Harddisk3\Partition1\ has FS type NTFS
00000bd4.00000c04::2011/01/13-16:10:40.850 INFO [RES] Physical Disk: Volume\\?\GLOBALROOT\Device\Harddisk3\Partition1\ has FS type NTFS
00000bd4.00000c04::2011/01/13-16:10:40.850 INFO [RES] Physical Disk: MountPoint V:\ points to volume\\?\Volume{7343300a-18ee-11e0-aded-0026b93b46a0}\
00000bd4.00000c04::2011/01/13-16:10:40.850 DBG [ClRtl] CaptureShareInfo: Share 0000000000000050, Server 0000000000000000, Path 0000000000000056 SD 0000000000000000 size 242
00000bd4.00000c04::2011/01/13-16:10:40.850 WARN [ClRtl] SsCoreShareAdd(): status = 2118 share = V$ server = (null)
00000bd4.00000c04::2011/01/13-16:10:40.850 DBG [ClRtl] CaptureShareInfo: Share 0000000000000050, Server 0000000000000000, Path 0000000000000056 SD 0000000000000000 size 242
00000bd4.00000c04::2011/01/13-16:10:40.850 INFO [RHS] Resource VM1 has come online. RHS is about to report status change to RCM
00000730.00000d3c::2011/01/13-16:10:40.850 INFO [RCM] HandleMonitorReply: ONLINERESOURCE for 'VM1', gen(2) result 0.
00000730.00000d3c::2011/01/13-16:10:40.850 INFO [RCM] TransitionToState(VM1) OnlinePending-->Online.
00000730.00000d3c::2011/01/13-16:10:40.850 INFO [RCM] rcm::RcmGroup::UpdateStateIfChanged: (Available Storage, Pending --> Online)
00000bd4.00000c04::2011/01/13-16:10:40.850 INFO [RES] Physical Disk: DriveLetter mask: 0x200000
00000b18.00000c1c::2011/01/13-16:10:40.896 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
00000730.00000d30::2011/01/13-16:10:40.896 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00000730.00000d30::2011/01/13-16:10:40.896 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node. Attempting to find a good node...
00000b18.00000c1c::2011/01/13-16:10:40.896 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
00000730.000006d8::2011/01/13-16:10:40.896 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00000b18.00000c1c::2011/01/13-16:10:40.912 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
00000730.00000d30::2011/01/13-16:10:40.912 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000730.00000d30::2011/01/13-16:10:40.912 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node. Attempting to find a good node...
00000b18.00000c1c::2011/01/13-16:10:40.912 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
00000730.000006d8::2011/01/13-16:10:40.912 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000b18.00000c1c::2011/01/13-16:10:40.912 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
00000730.00000d30::2011/01/13-16:10:40.912 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00000730.00000d30::2011/01/13-16:10:40.912 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node. Attempting to find a good node...
00000b18.00000c1c::2011/01/13-16:10:40.912 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
00000730.000006d8::2011/01/13-16:10:40.912 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00000b18.00000c1c::2011/01/13-16:10:40.912 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
00000730.00000d30::2011/01/13-16:10:40.912 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000730.00000d30::2011/01/13-16:10:40.912 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node. Attempting to find a good node...
00000b18.00000c1c::2011/01/13-16:10:40.912 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
00000730.000006d8::2011/01/13-16:10:40.912 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000b18.00000c1c::2011/01/13-16:10:40.928 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
00000730.00000d30::2011/01/13-16:10:40.928 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00000730.00000d30::2011/01/13-16:10:40.928 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node. Attempting to find a good node...
00000b18.00000c1c::2011/01/13-16:10:40.928 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
00000730.000006d8::2011/01/13-16:10:40.928 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00000b18.00000c1c::2011/01/13-16:10:40.928 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
00000730.00000d30::2011/01/13-16:10:40.928 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000730.00000d30::2011/01/13-16:10:40.928 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node. Attempting to find a good node...
00000b18.00000c1c::2011/01/13-16:10:40.928 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
00000730.000006d8::2011/01/13-16:10:40.928 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000b18.00000c1c::2011/01/13-16:10:40.943 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
00000730.00000d30::2011/01/13-16:10:40.943 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00000730.00000d30::2011/01/13-16:10:40.943 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node. Attempting to find a good node...
00000b18.00000c1c::2011/01/13-16:10:40.943 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
00000730.000006d8::2011/01/13-16:10:40.943 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00000b18.00000c1c::2011/01/13-16:10:40.943 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
00000730.00000d30::2011/01/13-16:10:40.943 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000730.00000d30::2011/01/13-16:10:40.943 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node. Attempting to find a good node...
00000b18.00000c1c::2011/01/13-16:10:40.943 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
00000730.000006d8::2011/01/13-16:10:40.943 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000730.00000d30::2011/01/13-16:10:47.354 INFO [RCM] rcm::RcmApi::FailResource: (VM1)
00000730.00000d40::2011/01/13-16:10:47.354 INFO [RCM] HandleMonitorReply: FAILURENOTIFICATION for 'VM1', gen(3) result 0.
00000730.00000d40::2011/01/13-16:10:47.354 INFO [RCM] TransitionToState(VM1) Online-->ProcessingFailure.
00000730.00000d40::2011/01/13-16:10:47.354 INFO [RCM] rcm::RcmGroup::UpdateStateIfChanged: (Available Storage, Online --> Failed)
00000730.00000d40::2011/01/13-16:10:47.354 ERR [RCM] rcm::RcmResource::HandleFailure: (VM1)
00000730.00000d40::2011/01/13-16:10:47.354 INFO [RCM] resource VM1: failure count: 2, restartAction: 2.
00000730.00000d40::2011/01/13-16:10:47.354 INFO [RCM] TransitionToState(VM1) ProcessingFailure-->[WaitingToTerminate to Failed].
00000730.00000d40::2011/01/13-16:10:47.354 INFO [RCM] rcm::RcmGroup::UpdateStateIfChanged: (Available Storage, Failed --> Pending)
00000730.00000d40::2011/01/13-16:10:47.354 INFO [RCM] TransitionToState(VM1) [WaitingToTerminate to Failed]-->[Terminating to Failed].
00000bd4.00000eb0::2011/01/13-16:10:47.354 INFO [RES] Physical Disk <VM1>: Terminate request.
00000730.00000d40::2011/01/13-16:10:47.354 INFO [RCM] Resource VM1 is causing group Available Storage to failover. Posting worker thread.
00000730.00000d40::2011/01/13-16:10:47.354 INFO [RCM] rcm::RcmGroup::Failover: (Available Storage)
00000bd4.00000eb0::2011/01/13-16:10:47.417 WARN [RES] Physical Disk <VM1>: Terminate: Failed to offline volume \Device\Harddisk3\Partition1, Error 5
00000b18.00000c1c::2011/01/13-16:10:47.432 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
00000730.0000076c::2011/01/13-16:10:47.432 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00000730.0000076c::2011/01/13-16:10:47.432 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node. Attempting to find a good node...
00000b18.00000c1c::2011/01/13-16:10:47.432 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
00000730.00000d30::2011/01/13-16:10:47.432 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00000b18.00000c1c::2011/01/13-16:10:47.432 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
00000730.0000076c::2011/01/13-16:10:47.432 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000730.0000076c::2011/01/13-16:10:47.432 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node. Attempting to find a good node...
00000b18.00000c1c::2011/01/13-16:10:47.432 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
00000730.00000d30::2011/01/13-16:10:47.432 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000b18.00000c1c::2011/01/13-16:10:47.432 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
00000730.0000076c::2011/01/13-16:10:47.432 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00000730.0000076c::2011/01/13-16:10:47.432 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node. Attempting to find a good node...
00000b18.00000c1c::2011/01/13-16:10:47.448 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
00000730.00000d30::2011/01/13-16:10:47.448 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00000b18.00000c1c::2011/01/13-16:10:47.448 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
00000730.0000076c::2011/01/13-16:10:47.448 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000730.0000076c::2011/01/13-16:10:47.448 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node. Attempting to find a good node...
00000b18.00000c1c::2011/01/13-16:10:47.448 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
00000730.00000d30::2011/01/13-16:10:47.448 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000bd4.00000eb0::2011/01/13-16:10:47.666 INFO [RES] Physical Disk <VM1>: HardDiskpCloseSVIHandles: Exit
00000bd4.00000eb0::2011/01/13-16:10:48.259 INFO [RES] Physical Disk: ReleaseDisk: stop reserve succeeded on device 3 (sig 44423655)
00000730.0000076c::2011/01/13-16:10:48.368 INFO [RCM] HandleMonitorReply: TERMINATERESOURCE for 'VM1', gen(4) result 0.
00000730.0000076c::2011/01/13-16:10:48.368 INFO [RCM] TransitionToState(VM1) [Terminating to Failed]-->Failed.
00000730.0000076c::2011/01/13-16:10:48.368 INFO [RCM] rcm::RcmGroup::UpdateStateIfChanged: (Available Storage, Pending --> Failed)
00000730.00000d40::2011/01/13-16:10:48.384 WARN [RCM] Not failing over group Available Storage, failoverCount 1, failover threshold 0, nodeAvailCount 0.
00000730.00000d40::2011/01/13-16:10:48.384 INFO [RCM] Will retry online of VM1 in 3600000 milliseconds.
00000b18.00000c1c::2011/01/13-16:10:48.384 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
00000730.00000d40::2011/01/13-16:10:48.384 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00000730.00000d40::2011/01/13-16:10:48.384 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node. Attempting to find a good node...
00000b18.00000c1c::2011/01/13-16:10:48.384 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
00000730.0000076c::2011/01/13-16:10:48.384 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00000b18.00000c1c::2011/01/13-16:10:48.400 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
00000730.00000d40::2011/01/13-16:10:48.400 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000730.00000d40::2011/01/13-16:10:48.400 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node. Attempting to find a good node...
00000b18.00000c1c::2011/01/13-16:10:48.400 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
00000730.0000076c::2011/01/13-16:10:48.400 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000b18.00000c1c::2011/01/13-16:10:48.400 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
00000730.00000d40::2011/01/13-16:10:48.400 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00000730.00000d40::2011/01/13-16:10:48.400 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node. Attempting to find a good node...
00000b18.00000c1c::2011/01/13-16:10:48.400 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
00000730.0000076c::2011/01/13-16:10:48.400 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00000b18.00000c1c::2011/01/13-16:10:48.400 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
00000730.00000d40::2011/01/13-16:10:48.400 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000730.00000d40::2011/01/13-16:10:48.400 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node. Attempting to find a good node...
00000b18.00000c1c::2011/01/13-16:10:48.400 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
00000730.0000076c::2011/01/13-16:10:48.400 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.