Hello,
We have a 3 node Hyper-V 2008 R2 failover cluster. Yesterday we had problems and all VM's on node 3 went down. At 17:00 the back-up starts of the VM's which are on CSV(with DPM 2010 and hardware providers for Dell) but then one nodes crashed. It look like node 3 tries to take CSV04 ownership but node 1 doesn't accept that. See the cluster log below. Can someone tell me what triggered this? And how to fix this? Many thanks!
It start with a message that some Volume manager disk group and Geocluster disks are not found. This says nothing to me.
00000af0.00000c24::2014/02/23-17:00:27.738 WARN Resource type Volume Manager Disk Group not found.
00000af0.00000c24::2014/02/23-17:00:27.738 WARN Resource type GeoCluster Replicated Disk not found.
00000af0.000017fc::2014/02/23-17:00:35.117 INFO [RCM] rcm::RcmApi::MoveGroup: (6d07505c-cd56-4354-bb78-f0d452eb7350, 1)
00000af0.000017fc::2014/02/23-17:00:35.117 INFO [RCM] rcm::RcmGroup::Move: (6d07505c-cd56-4354-bb78-f0d452eb7350, 1)
00000af0.000017fc::2014/02/23-17:00:35.117 INFO [RCM] rcm::RcmGroup::Move: Bringing group '6d07505c-cd56-4354-bb78-f0d452eb7350' offline first...
00000af0.000017fc::2014/02/23-17:00:35.117 INFO [RCM] TransitionToState(CSV04) Online-->OfflineCallIssued.
00000af0.000017fc::2014/02/23-17:00:35.117 INFO [RCM] rcm::RcmGroup::UpdateStateIfChanged: (6d07505c-cd56-4354-bb78-f0d452eb7350, Online --> Pending)
00000af0.000017fc::2014/02/23-17:00:35.117 INFO [DCM] PreOffline for CSV resource CSV04
00000af0.000017fc::2014/02/23-17:00:35.117 INFO [DCM] Unmapping volumes for cfs resource CSV04
00000af0.00002330::2014/02/23-17:00:35.117 INFO [NM] Received request from client address 172.16.0.1.
00000af0.00000d04::2014/02/23-17:00:35.179 INFO [DCM] Processing message dcm/pause
00000af0.00000d04::2014/02/23-17:00:35.179 INFO [DCM] Push.AsyncPauseDisk for 7177671d-4b75-4c6e-ad3d-4ff3671ce779
00000af0.00001594::2014/02/23-17:00:35.179 INFO [DCM] SyncHandler for 7177671d-4b75-4c6e-ad3d-4ff3671ce779
00000af0.00001594::2014/02/23-17:00:35.179 INFO [DCM] enter_AllGood(7177671d-4b75-4c6e-ad3d-4ff3671ce779) P0..75 P0..150
00000af0.00001594::2014/02/23-17:00:35.179 INFO [DCM] MappingManager::PauseVolume 'Volume4'
00000af0.00001594::2014/02/23-17:00:35.179 INFO [DCM] Filter.ChangeState (ctx=2, state=CfsVolumeStatePaused)
00000af0.00002130::2014/02/23-17:00:35.273 INFO [NM] Received request from client address HV03.
00000af0.00002130::2014/02/23-17:00:35.288 ERR [RCM] rcm::RcmResControl::DoResourceControl: ERROR_RESOURCE_NOT_ONLINE(5004)' because of 'PreprocessControl(16777765) failed for resource 'CSV04'.'
00000af0.00002130::2014/02/23-17:00:35.288 WARN [RCM] ResourceControl(STORAGE_GET_SHARED_VOLUME_INFO) to CSV04 returned 5004.
00000af0.00002130::2014/02/23-17:00:35.288 INFO [NM] Received request from client address HV03.
00000af0.00000c24::2014/02/23-17:00:35.320 ERR [RCM] rcm::RcmResControl::DoResourceControl: ERROR_RESOURCE_NOT_ONLINE(5004)' because of 'PreprocessControl(16777765) failed for resource 'CSV04'.'
00000af0.00000c24::2014/02/23-17:00:35.320 WARN [RCM] ResourceControl(STORAGE_GET_SHARED_VOLUME_INFO) to CSV04 returned 5004.
00000f30.00001888::2014/02/23-17:00:35.663 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
00000af0.00002960::2014/02/23-17:00:35.663 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00000af0.00002960::2014/02/23-17:00:35.663 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node. Attempting to find a good node...
00000f30.00001888::2014/02/23-17:00:35.663 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
00000af0.00000c24::2014/02/23-17:00:35.663 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00000f30.00001888::2014/02/23-17:00:35.710 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
00000af0.00000c24::2014/02/23-17:00:35.710 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00000f30.00001888::2014/02/23-17:00:35.912 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
00000af0.00002960::2014/02/23-17:00:35.912 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000af0.00002960::2014/02/23-17:00:35.912 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node. Attempting to find a good node...
00000f30.00001888::2014/02/23-17:00:35.912 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
00000af0.00000c24::2014/02/23-17:00:35.912 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000f30.00001888::2014/02/23-17:00:35.912 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
00000af0.00000c24::2014/02/23-17:00:35.912 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000af0.00000c24::2014/02/23-17:00:35.928 WARN Resource type Volume Manager Disk Group not found.
00000af0.00000c24::2014/02/23-17:00:35.928 WARN Resource type GeoCluster Replicated Disk not found.
00000af0.00002960::2014/02/23-17:00:36.209 INFO [NM] Received request from client address HV03.
00000f30.000029d4::2014/02/23-17:00:36.599 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
00000af0.00000c24::2014/02/23-17:00:36.599 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00000af0.00000c24::2014/02/23-17:00:36.599 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node. Attempting to find a good node...
00000f30.000029d4::2014/02/23-17:00:36.599 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
00000af0.00002960::2014/02/23-17:00:36.599 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00000f30.000029d4::2014/02/23-17:00:36.599 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
00000af0.00000c24::2014/02/23-17:00:36.599 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000af0.00000c24::2014/02/23-17:00:36.599 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node. Attempting to find a good node...
00000f30.000029d4::2014/02/23-17:00:36.599 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
00000af0.00002960::2014/02/23-17:00:36.599 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000f30.000029d4::2014/02/23-17:00:36.614 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
00000af0.00002960::2014/02/23-17:00:36.614 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00000f30.000029d4::2014/02/23-17:00:36.630 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
00000af0.00002960::2014/02/23-17:00:36.630 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000af0.00000c24::2014/02/23-17:00:36.911 ERR [RCM] rcm::RcmResControl::DoResourceControl: ERROR_RESOURCE_NOT_ONLINE(5004)' because of 'PreprocessControl(16777765) failed for resource 'CSV04'.'
00000af0.00000c24::2014/02/23-17:00:36.911 WARN [RCM] ResourceControl(STORAGE_GET_SHARED_VOLUME_INFO) to CSV04 returned 5004.
00000af0.00002960::2014/02/23-17:00:36.989 ERR [RCM] rcm::RcmResControl::DoResourceControl: ERROR_RESOURCE_NOT_ONLINE(5004)' because of 'PreprocessControl(16777765) failed for resource 'CSV04'.'
00000af0.00002960::2014/02/23-17:00:36.989 WARN [RCM] ResourceControl(STORAGE_GET_SHARED_VOLUME_INFO) to CSV04 returned 5004.
00000af0.000014c8::2014/02/23-17:00:39.251 INFO [DCM] filter.Event ->CfsVolumeStatePaused FromPause for ctx=2 status 00000000
00000af0.00001594::2014/02/23-17:00:39.251 INFO [DCM] volume paused 'Volume4'
00000af0.000017fc::2014/02/23-17:00:39.391 INFO [DCM] dcm/pause successfully completed on all nodes
00000af0.000017fc::2014/02/23-17:00:39.391 INFO [DCM] removing share 7177671d-4b75-4c6e-ad3d-4ff3671ce779-135266304$, status 0
00000af0.00002960::2014/02/23-17:00:39.391 INFO [NM] Received request from client address 172.16.0.1.
00000af0.000017fc::2014/02/23-17:00:39.438 INFO [DCM] ClearVolumeStates: resource 'CSV04' states <vector len='2'>
00000af0.000017fc::2014/02/23-17:00:39.438 INFO <item>1</item>
00000af0.000017fc::2014/02/23-17:00:39.438 INFO <item>135266304 0</item>
00000af0.000017fc::2014/02/23-17:00:39.438 INFO </vector>
00001680.000023b4::2014/02/23-17:00:39.438 INFO [RES] Physical Disk <CSV04>: Offline request.
00001680.00001e04::2014/02/23-17:00:39.438 INFO [RES] Physical Disk: DriveLetter mask: 0x0
00000af0.000017fc::2014/02/23-17:00:39.438 INFO [RCM] HandleMonitorReply: OFFLINERESOURCE for 'CSV04', gen(0) result 997.
00000af0.000017fc::2014/02/23-17:00:39.438 INFO [RCM] TransitionToState(CSV04) OfflineCallIssued-->OfflinePending.
00001680.00001e04::2014/02/23-17:00:39.454 INFO [RES] Physical Disk <CSV04>: HardDiskpCloseSVIHandles: Exit
00001680.00001e04::2014/02/23-17:00:39.454 INFO [RES] Physical Disk <CSV04>: VolumeIsNtfs: Volume\\?\GLOBALROOT\Device\Harddisk2\Partition2\ has FS type NTFS
00001680.00001e04::2014/02/23-17:00:39.454 INFO [RES] Physical Disk <CSV04>: OfflineThread: partition 2 offset 135266304 is a CSV volume, skipping lock volume
00001680.00001e04::2014/02/23-17:00:40.421 INFO [RES] Physical Disk: ReleaseDisk: stop reserve succeeded on device 2 (sig ceba743a)
00001680.00001e04::2014/02/23-17:00:40.452 INFO [RHS] Resource CSV04 has come offline. RHS is about to report resource status to RCM.
00000af0.00002960::2014/02/23-17:00:40.452 INFO [RCM] HandleMonitorReply: OFFLINERESOURCE for 'CSV04', gen(0) result 0.
00000af0.00002960::2014/02/23-17:00:40.452 INFO [RCM] TransitionToState(CSV04) OfflinePending-->OfflineSavingCheckpoints.
00000af0.00002960::2014/02/23-17:00:40.452 INFO [RCM] TransitionToState(CSV04) OfflineSavingCheckpoints-->Offline.
00000af0.00002960::2014/02/23-17:00:40.452 INFO [RCM] rcm::RcmGroup::UpdateStateIfChanged: (6d07505c-cd56-4354-bb78-f0d452eb7350, Pending --> Offline)
00000af0.00000c24::2014/02/23-17:00:40.452 INFO [RCM] rcm::RcmGum::GroupMoveOperation(6d07505c-cd56-4354-bb78-f0d452eb7350,1)
00000af0.000017fc::2014/02/23-17:00:40.452 WARN [RCM] rcm::RcmApi::ResourceControl: forwarded, no retry on error 5908
00000af0.000017fc::2014/02/23-17:00:40.452 WARN [RCM] ResourceControl(GET_CLASS_INFO) to CSV04 returned 5908.
00000af0.00002960::2014/02/23-17:00:40.452 WARN [RCM] rcm::RcmApi::GetResourceState: retrying: 6d07505c-cd56-4354-bb78-f0d452eb7350, 5908.
00000af0.000017fc::2014/02/23-17:00:40.452 ERR [RCM] s_RcmRpcGetResourceState: ERROR_CLUSTER_GROUP_MOVING(5908)' because of ''CSV04' is owned by node 1, not 3.'