2-node Windows 2012 Hyper-V Cluster (running for about 2 years - NOT a new install). The nodes are connected to a IBM SVC via F/C. Recently we noticed problems with accessing machines on a certain CSV - but only from node-2. We did several checks but nothing obivous seems to be wrong. Around the timeframe where we recognized the problem, an IBM tech had to replace serveral (damaged?/broken?) SFPs. After a restart of the problematic-node, diskpart seems to be "hanging". After serveral minutes I am able to "list disks" ... but then it does not show the DISK with the CSV envolved.
Cluster.log does not leave me with something "google-able" oder "bing-able ;-). Anybody with a clue?
Especially the line with, but I might be wrong with my focus:
00000aa0.00000dec::2014/11/13-14:28:32.534 ERR mscs::MappingManager::ActivateVolume: ERROR_NOT_FOUND(1168)' because of 'VBUS Failed to activate network path'
Best Regards and thank you very much in advance!
Walter
00000aa0.000015ec::2014/11/13-14:28:32.498
INFO [DCM] MapDisk: new target address fe80::784e:9680:c298:8dcf%34 for
Clusterdatenträger 3
00000aa0.000015ec::2014/11/13-14:28:32.498
INFO [DCM] Reservation.SetPrKey(Clusterdatenträger 3,1)
00000aa0.0000186c::2014/11/13-14:28:32.498
INFO [DCM] Populate Rdr handle - target node ip
fe80::784e:9680:c298:8dcf%34, node id 2
00001498.00001580::2014/11/13-14:28:32.503
INFO [RES] Physical Disk <Clusterdatenträger 3>:
HardDiskpQueryDiskFromStm: ClusterStmFindDisk returned
device='\\?\mpio#disk&ven_ibm&prod_2145&rev_0000#1&7f6ac24&0&3630303530373638303138303837313530383030303030303030303030303139#{53f56307-b6bf-11d0-94f2-00a0c91efb8b}'
00001498.00001580::2014/11/13-14:28:32.517
INFO [RES] Physical Disk <Clusterdatenträger 3>: SetSharedPRKey:
registered shared PR key high 4CADAE17 low 0001734D for device 1
00001498.00001580::2014/11/13-14:28:32.517
INFO [RES] Physical Disk <Clusterdatenträger 3>: HardDiskpSetUnsetDiskFlags(mask=0x00000002,
SetCluster=0, SetCsv=1, SetMaintenanceMode=0, Notify=1, Update=0) for device=1
00001498.00001580::2014/11/13-14:28:32.523
INFO [RES] Physical Disk <Clusterdatenträger 3>:
HardDiskpQueryDiskFromStm: ClusterStmFindDisk returned
device='\\?\mpio#disk&ven_ibm&prod_2145&rev_0000#1&7f6ac24&0&3630303530373638303138303837313530383030303030303030303030303139#{53f56307-b6bf-11d0-94f2-00a0c91efb8b}'
00001498.00001580::2014/11/13-14:28:32.524
INFO [RES] Physical Disk <Clusterdatenträger 3>: GET_DISK_NUMBER:
Disk number is 1
00000aa0.000015ec::2014/11/13-14:28:32.532
INFO [DCM] GetDriveLayout: device \??\PhysicalDrive1, partition count 4
00000aa0.000015ec::2014/11/13-14:28:32.532
INFO [DCM] MapDisk mapping volume 'Volume2:8582b5ab-942f-4d75-b8b9-c337aef51406'
00000aa0.000015ec::2014/11/13-14:28:32.532
INFO [DCM] MapDisk: calling MapVolume with target address
fe80::784e:9680:c298:8dcf%34
00000aa0.00000dec::2014/11/13-14:28:32.532
INFO [DCM] volume 'Volume2' has been already mapped
00000aa0.000015ec::2014/11/13-14:28:32.532
INFO [DCM] MapDisk completed for resource 'Clusterdatenträger 3'
00000aa0.00000dec::2014/11/13-14:28:32.532
INFO [DCM] ActivateNetworkPath 8582b5ab-942f-4d75-b8b9-c337aef51406
00000aa0.00000dec::2014/11/13-14:28:32.533
ERR [DCM] VBUS Failed to activate network path status 1168
00000aa0.00000dec::2014/11/13-14:28:32.534
INFO [DCM] CsvFs event CsvFsVolumeStateChangeFromActivate for volume
Volume2:8582b5ab-942f-4d75-b8b9-c337aef51406, status (490)
00000aa0.00000dec::2014/11/13-14:28:32.534
INFO [DCM] Push.AsyncRefreshDisk for d6f35c7b-06d7-4139-b015-b606e5d362c9
00000aa0.00000dec::2014/11/13-14:28:32.534
ERR mscs::MappingManager::ActivateVolume: ERROR_NOT_FOUND(1168)'
because of 'VBUS Failed to activate network path'
00000aa0.000015ec::2014/11/13-14:28:32.534
INFO [DCM] ActivateDisk completed for resource 'Clusterdatenträger 3'
00000aa0.000015ec::2014/11/13-14:28:32.534
INFO [DCM] RefreshDisk: complete for resource Clusterdatenträger 3
00000aa0.000015ec::2014/11/13-14:28:32.534
INFO [DCM] SyncHandler for Clusterdatenträger 3
00000aa0.000015ec::2014/11/13-14:28:37.541
INFO [DCM] RefreshDisk: resource Clusterdatenträger 3, completed false,
desired state 2
00000aa0.000015ec::2014/11/13-14:28:37.541
INFO [DCM] RefreshDisk: resource Clusterdatenträger 3 doing local repair
00000aa0.000015ec::2014/11/13-14:28:37.544
INFO [DCM] DcmComputeRouteToTarget: returning netft address
fe80::784e:9680:c298:8dcf%34
00000aa0.000015ec::2014/11/13-14:28:37.544
INFO [DCM] MapDisk: new target address fe80::784e:9680:c298:8dcf%34 for
Clusterdatenträger 3
00000aa0.000015ec::2014/11/13-14:28:37.544
INFO [DCM] Reservation.SetPrKey(Clusterdatenträger 3,1)
00000aa0.00000e84::2014/11/13-14:28:37.544
INFO [DCM] Populate Rdr handle - target node ip
fe80::784e:9680:c298:8dcf%34, node id 2
00001498.00001580::2014/11/13-14:28:37.548
INFO [RES] Physical Disk <Clusterdatenträger 3>:
HardDiskpQueryDiskFromStm: ClusterStmFindDisk returned
device='\\?\mpio#disk&ven_ibm&prod_2145&rev_0000#1&7f6ac24&0&3630303530373638303138303837313530383030303030303030303030303139#{53f56307-b6bf-11d0-94f2-00a0c91efb8b}'
00001498.00001580::2014/11/13-14:28:37.563
INFO [RES] Physical Disk <Clusterdatenträger 3>: SetSharedPRKey:
registered shared PR key high 4CADAE17 low 0001734D for device 1
00001498.00001580::2014/11/13-14:28:37.563
INFO [RES] Physical Disk <Clusterdatenträger 3>:
HardDiskpSetUnsetDiskFlags(mask=0x00000002, SetCluster=0, SetCsv=1,
SetMaintenanceMode=0, Notify=1, Update=0) for device=1
00001498.00001580::2014/11/13-14:28:37.569
INFO [RES] Physical Disk <Clusterdatenträger 3>:
HardDiskpQueryDiskFromStm: ClusterStmFindDisk returned
device='\\?\mpio#disk&ven_ibm&prod_2145&rev_0000#1&7f6ac24&0&3630303530373638303138303837313530383030303030303030303030303139#{53f56307-b6bf-11d0-94f2-00a0c91efb8b}'
00001498.00001580::2014/11/13-14:28:37.570
INFO [RES] Physical Disk <Clusterdatenträger 3>: GET_DISK_NUMBER:
Disk number is 1
00000aa0.000015ec::2014/11/13-14:28:37.579
INFO [DCM] GetDriveLayout: device \??\PhysicalDrive1, partition count 4
00000aa0.000015ec::2014/11/13-14:28:37.579
INFO [DCM] MapDisk mapping volume
'Volume2:8582b5ab-942f-4d75-b8b9-c337aef51406'
00000aa0.000015ec::2014/11/13-14:28:37.579
INFO [DCM] MapDisk: calling MapVolume with target address
fe80::784e:9680:c298:8dcf%34
00000aa0.00000dec::2014/11/13-14:28:37.579
INFO [DCM] volume 'Volume2' has been already mapped
00000aa0.000015ec::2014/11/13-14:28:37.579
INFO [DCM] MapDisk completed for resource 'Clusterdatenträger 3'
00000aa0.00000dec::2014/11/13-14:28:37.579
INFO [DCM] ActivateNetworkPath 8582b5ab-942f-4d75-b8b9-c337aef51406
00000aa0.00000dec::2014/11/13-14:28:37.581
ERR [DCM] VBUS Failed to activate network path status 1168
00000aa0.00000dec::2014/11/13-14:28:37.581
INFO [DCM] CsvFs event CsvFsVolumeStateChangeFromActivate for volume
Volume2:8582b5ab-942f-4d75-b8b9-c337aef51406, status (490)
00000aa0.00000dec::2014/11/13-14:28:37.581
INFO [DCM] Push.AsyncRefreshDisk for d6f35c7b-06d7-4139-b015-b606e5d362c9
00000aa0.00000dec::2014/11/13-14:28:37.581
ERR mscs::MappingManager::ActivateVolume: ERROR_NOT_FOUND(1168)'
because of 'VBUS Failed to activate network path'
00000aa0.000015ec::2014/11/13-14:28:37.581
INFO [DCM] ActivateDisk completed for resource 'Clusterdatenträger 3'
00000aa0.000015ec::2014/11/13-14:28:37.581
INFO [DCM] RefreshDisk: complete for resource Clusterdatenträger 3
00000aa0.000015ec::2014/11/13-14:28:37.581
INFO [DCM] SyncHandler for Clusterdatenträger 3
00000aa0.000015ec::2014/11/13-14:28:42.588
INFO [DCM] RefreshDisk: resource Clusterdatenträger 3, completed false,
desired state 2
00000aa0.000015ec::2014/11/13-14:28:42.588
INFO [DCM] RefreshDisk: resource Clusterdatenträger 3 doing local repair
00000aa0.000015ec::2014/11/13-14:28:42.590
INFO [DCM] DcmComputeRouteToTarget: returning netft address
fe80::784e:9680:c298:8dcf%34
00000aa0.000015ec::2014/11/13-14:28:42.590
INFO [DCM] MapDisk: new target address fe80::784e:9680:c298:8dcf%34 for
Clusterdatenträger 3
00000aa0.000015ec::2014/11/13-14:28:42.590
INFO [DCM] Reservation.SetPrKey(Clusterdatenträger 3,1)
00000aa0.00000e84::2014/11/13-14:28:42.590
INFO [DCM] Populate Rdr handle - target node ip
fe80::784e:9680:c298:8dcf%34, node id 2
00001498.00001580::2014/11/13-14:28:42.595
INFO [RES] Physical Disk <Clusterdatenträger 3>:
HardDiskpQueryDiskFromStm: ClusterStmFindDisk returned
device='\\?\mpio#disk&ven_ibm&prod_2145&rev_0000#1&7f6ac24&0&3630303530373638303138303837313530383030303030303030303030303139#{53f56307-b6bf-11d0-94f2-00a0c91efb8b}'
00001498.00001580::2014/11/13-14:28:42.616
INFO [RES] Physical Disk <Clusterdatenträger 3>: SetSharedPRKey:
registered shared PR key high 4CADAE17 low 0001734D for device 1
00001498.00001528::2014/11/13-14:28:42.616
INFO [RES] Physical Disk <Clusterdatenträger 3>:
HardDiskpSetUnsetDiskFlags(mask=0x00000002, SetCluster=0, SetCsv=1,
SetMaintenanceMode=0, Notify=1, Update=0) for device=1
00001498.00001528::2014/11/13-14:28:42.621
INFO [RES] Physical Disk <Clusterdatenträger 3>:
HardDiskpQueryDiskFromStm: ClusterStmFindDisk returned
device='\\?\mpio#disk&ven_ibm&prod_2145&rev_0000#1&7f6ac24&0&3630303530373638303138303837313530383030303030303030303030303139#{53f56307-b6bf-11d0-94f2-00a0c91efb8b}'
00001498.00001528::2014/11/13-14:28:42.623
INFO [RES] Physical Disk <Clusterdatenträger 3>: GET_DISK_NUMBER:
Disk number is 1
00000aa0.000015ec::2014/11/13-14:28:42.630
INFO [DCM] GetDriveLayout: device \??\PhysicalDrive1, partition count 4
00000aa0.000015ec::2014/11/13-14:28:42.630
INFO [DCM] MapDisk mapping volume
'Volume2:8582b5ab-942f-4d75-b8b9-c337aef51406'
00000aa0.000015ec::2014/11/13-14:28:42.630
INFO [DCM] MapDisk: calling MapVolume with target address
fe80::784e:9680:c298:8dcf%34
00000aa0.00000dec::2014/11/13-14:28:42.630
INFO [DCM] volume 'Volume2' has been already mapped
00000aa0.000015ec::2014/11/13-14:28:42.630
INFO [DCM] MapDisk completed for resource 'Clusterdatenträger 3'
00000aa0.00000dec::2014/11/13-14:28:42.630
INFO [DCM] ActivateNetworkPath 8582b5ab-942f-4d75-b8b9-c337aef51406
00000aa0.00000dec::2014/11/13-14:28:42.632
ERR [DCM] VBUS Failed to activate network path status 1168
00000aa0.00000dec::2014/11/13-14:28:42.633
INFO [DCM] CsvFs event CsvFsVolumeStateChangeFromActivate for volume
Volume2:8582b5ab-942f-4d75-b8b9-c337aef51406, status (490)
00000aa0.00000dec::2014/11/13-14:28:42.633
INFO [DCM] Push.AsyncRefreshDisk for d6f35c7b-06d7-4139-b015-b606e5d362c9
00000aa0.00000dec::2014/11/13-14:28:42.633
ERR mscs::MappingManager::ActivateVolume: ERROR_NOT_FOUND(1168)'
because of 'VBUS Failed to activate network path'
00000aa0.000015ec::2014/11/13-14:28:42.633
INFO [DCM] ActivateDisk completed for resource 'Clusterdatenträger 3'
00000aa0.000015ec::2014/11/13-14:28:42.633
INFO [DCM] RefreshDisk: complete for resource Clusterdatenträger 3
00000aa0.000015ec::2014/11/13-14:28:42.633
INFO [DCM] SyncHandler for Clusterdatenträger 3
000017d4.00001228::2014/11/13-14:28:44.763
ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of
'Startup routine for ResType MSMQ returned 21.'
00000aa0.0000186c::2014/11/13-14:28:44.763
INFO [RCM] result of first load attempt for type MSMQ: 21
00000aa0.0000186c::2014/11/13-14:28:44.763
WARN [RCM] Failed to load restype 'MSMQ': error 21.
000017d4.00001228::2014/11/13-14:28:44.766
ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of
'Startup routine for ResType MSMQTriggers returned 21.'
00000aa0.0000186c::2014/11/13-14:28:44.766
INFO [RCM] result of first load attempt for type MSMQTriggers: 21
00000aa0.0000186c::2014/11/13-14:28:44.766
WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000aa0.0000186c::2014/11/13-14:28:44.775
INFO [RCM] [GIM] Scheduling Local Node Crawler to run in 300000 millisec.
00000aa0.000015ec::2014/11/13-14:28:47.640
INFO [DCM] RefreshDisk: resource Clusterdatenträger 3, completed false,
desired state 2
00000aa0.000015ec::2014/11/13-14:28:47.640
INFO [DCM] RefreshDisk: resource Clusterdatenträger 3 doing local repair
00000aa0.000015ec::2014/11/13-14:28:47.642
INFO [DCM] DcmComputeRouteToTarget: returning netft address
fe80::784e:9680:c298:8dcf%34
00000aa0.000015ec::2014/11/13-14:28:47.642
INFO [DCM] MapDisk: new target address fe80::784e:9680:c298:8dcf%34 for
Clusterdatenträger 3
00000aa0.000015ec::2014/11/13-14:28:47.642
INFO [DCM] Reservation.SetPrKey(Clusterdatenträger 3,1)
00000aa0.0000186c::2014/11/13-14:28:47.642
INFO [DCM] Populate Rdr handle - target node ip
fe80::784e:9680:c298:8dcf%34, node id 2
00001498.00001528::2014/11/13-14:28:47.647
INFO [RES] Physical Disk <Clusterdatenträger 3>:
HardDiskpQueryDiskFromStm: ClusterStmFindDisk returned
device='\\?\mpio#disk&ven_ibm&prod_2145&rev_0000#1&7f6ac24&0&3630303530373638303138303837313530383030303030303030303030303139#{53f56307-b6bf-11d0-94f2-00a0c91efb8b}'
00001498.00001528::2014/11/13-14:28:47.661
INFO [RES] Physical Disk <Clusterdatenträger 3>: SetSharedPRKey:
registered shared PR key high 4CADAE17 low 0001734D for device 1
00001498.00001528::2014/11/13-14:28:47.662
INFO [RES] Physical Disk <Clusterdatenträger 3>:
HardDiskpSetUnsetDiskFlags(mask=0x00000002, SetCluster=0, SetCsv=1,
SetMaintenanceMode=0, Notify=1, Update=0) for device=1
00001498.00001528::2014/11/13-14:28:47.667
INFO [RES] Physical Disk <Clusterdatenträger 3>: HardDiskpQueryDiskFromStm:
ClusterStmFindDisk returned
device='\\?\mpio#disk&ven_ibm&prod_2145&rev_0000#1&7f6ac24&0&3630303530373638303138303837313530383030303030303030303030303139#{53f56307-b6bf-11d0-94f2-00a0c91efb8b}'
00001498.00001528::2014/11/13-14:28:47.668
INFO [RES] Physical Disk <Clusterdatenträger 3>: GET_DISK_NUMBER:
Disk number is 1
00000aa0.000015ec::2014/11/13-14:28:47.677
INFO [DCM] GetDriveLayout: device \??\PhysicalDrive1, partition count 4
00000aa0.000015ec::2014/11/13-14:28:47.677
INFO [DCM] MapDisk mapping volume
'Volume2:8582b5ab-942f-4d75-b8b9-c337aef51406'
00000aa0.000015ec::2014/11/13-14:28:47.677
INFO [DCM] MapDisk: calling MapVolume with target address
fe80::784e:9680:c298:8dcf%34
00000aa0.00000dec::2014/11/13-14:28:47.677
INFO [DCM] volume 'Volume2' has been already mapped
00000aa0.000015ec::2014/11/13-14:28:47.677
INFO [DCM] MapDisk completed for resource 'Clusterdatenträger 3'
00000aa0.00000dec::2014/11/13-14:28:47.677
INFO [DCM] ActivateNetworkPath 8582b5ab-942f-4d75-b8b9-c337aef51406
00000aa0.00000dec::2014/11/13-14:28:47.680
ERR [DCM] VBUS Failed to activate network path status 1168
00000aa0.00000dec::2014/11/13-14:28:47.680
INFO [DCM] CsvFs event CsvFsVolumeStateChangeFromActivate for volume
Volume2:8582b5ab-942f-4d75-b8b9-c337aef51406, status (490)
00000aa0.00000dec::2014/11/13-14:28:47.680
INFO [DCM] Push.AsyncRefreshDisk for d6f35c7b-06d7-4139-b015-b606e5d362c9
00000aa0.00000dec::2014/11/13-14:28:47.680
ERR mscs::MappingManager::ActivateVolume: ERROR_NOT_FOUND(1168)'
because of 'VBUS Failed to activate network path'
00000aa0.000015ec::2014/11/13-14:28:47.680
INFO [DCM] ActivateDisk completed for resource 'Clusterdatenträger 3'
00000aa0.000015ec::2014/11/13-14:28:47.680
INFO [DCM] RefreshDisk: complete for resource Clusterdatenträger 3
00000aa0.000015ec::2014/11/13-14:28:47.680
INFO [DCM] SyncHandler for Clusterdatenträger 3