I'm really struggling with this one and am hoping someone may be able to assist or point me in the right direction.
I am trying to build a three node cluster that will be used for a later SQL installation. All validation tests pass except for some storage warnings, but that's because the storage is not being shared. I realize this isn't typical, but I've built a handful of clusters exactly like this for stand-alone SQL instances that will be configured for AlwaysOn replication.
I do have permissions on the OU where this is getting created. There are no other objects in the domain with the same name. 2008 R2 domain. No AV on these nodes.
I get this generic error when the cluster creation fails:
An error occurred while creating the cluster and the nodes will be cleaned up. Please wait...
An error occurred while creating the cluster and the nodes will be cleaned up. Please wait...
There was an error cleaning up the cluster nodes.
Use Clear-ClusterNode to manually clean up the nodes.
An error occurred while creating the cluster. An error occurred creating cluster 'PARCLUST'.
This operation returned because the timeout period expired
Within the Event Viewer FailoverCluster-Client Diagnostics, I get the following errors (for reach node):
OpenClusterImpl (1119): Could not connect to cluster. sc = 1753
ConnectCluster (999): Failed to open remote cluster 'auditsql.customer.com', status == 1753.
ConnectRemoteCluster (856): Couldn't resolve RPC binding to cluster 'auditsql.customer.com', Status = 1753
Here is the Cluster.log. One odd thing is that the time appears to be off. It seems to be seven hours ahead of the Windows time...not sure why that is. But according to this particular log snip it's approx 12:53pm, but in reality it was 5:53am. Where would it be pulling the 12:53 time from, I wonder? Even in the log there's a spot where it shows the current time.
I've chased the warnings and errors found in here, and am still unable to figure out exactly what is causing the cluster creation to fail.
My apologies for so much of the log, I just didn't want to leave too much out
00000c58.00000de4::2016/04/08-12:51:52.965 INFO -----------------------------+ LOG BEGIN +----------------------------- 00000c58.00000de4::2016/04/08-12:51:52.965 INFO [CS] Starting clussvc as a service 00000c58.00001318::2016/04/08-... 00000c58.00001318::2016/04/08-12:51:53.012 INFO [QUORUM] Node 1: Initializing. 00000c58.00001318::2016/04/08-12:51:53.012 INFO [DCM] Bitlocker is not installed.
00000c58.00001318::2016/04/08-12:51:53.012 INFO [DCM] Load Bitlocker lib (exit), status 126 00000c58.00001318::2016/04/08-12:51:53.012 ERR [DCM] DiskControlManager bitlocker load status 126 00000c58.00001318::2016/04/08-12:51:53.012 INFO [STARTUP] ClusterNode 1: startup config 00000c58.00001318::2016/04/08-12:51:53.012 INFO <StartupConfig localNodeId='1' testMode='false' newCluster='true' 00000c58.00001318::2016/04/08-12:51:53.012 INFO <ClusterName> PARCLUST </ClusterName>
00000c58.00001318::2016/04/08-12:51:53.012 INFO <Node name='auditsql' id='3'/> 00000c58.00001318::2016/04/08-12:51:53.012 INFO <Node name='parsql' id='1'/> 00000c58.00001318::2016/04/08-12:51:53.012 INFO <Node name='treport' id='2'/> 00000c58.00001318::2016/04/08-12:51:53.012 INFO </Nodes>
00000c58.00001318::2016/04/08-12:51:53.012 INFO <Virtual IPv6> fe80::d003:5d7c:4ce:9425%30:~3343~ </Virtual IPv6>
00000c58.00001318::2016/04/08-12:51:53.012 INFO <Virtual IPv4> 169.254.1.253:~3343~ </Virtual IPv4> 00000c58.00001318::2016/04/08-12:51:53.012 INFO <NoVersionCheck> false </NoVersionCheck>
00000c58.00000ca8::2016/04/08-12:51:53.012 INFO [CS] ClusterNode object creation completed.
00000c58.00000ca8::2016/04/08-12:51:53.012 INFO [CS] The current time is 2016/04/08-05:51:53.01200000c58.00000ca8::2016/04/08-12:51:53.012 INFO [CS] Using virtual IPv6 Address fe80::d003:5d7c:4ce:9425%30:~3343~
00000c58.00000ca8::2016/04/08-12:51:53.012 INFO [CS] Using virtual IPv4 Address 169.254.1.253:~3343~ 00000c58.00000ca8::2016/04/08-12:51:53.012 INFO [CS] Max Licensed nodes is set 64
00000c58.00000ca8::2016/04/08-12:51:53.137 INFO <StartupConfig localNodeId='1' testMode='false' newCluster='true' 00000c58.00000ca8::2016/04/08-12:51:53.137
INFO <ClusterName> PARCLUST </ClusterName> 00000c58.00000ca8::2016/04/08-12:51:53.137 INFO <PaxosTag> 0:0:1 </PaxosTag>
00000c58.00000ca8::2016/04/08-12:51:53.137 INFO <Node name='parsql' id='1'/> 00000c58.00000ca8::2016/04/08-12:51:53.137 INFO <Node name='treport' id='2'/> 00000c58.00000ca8::2016/04/08-12:51:53.137 INFO </Nodes>
00000c58.00000ca8::2016/04/08-12:51:53.137 INFO <LocalHiveFile> C:\Windows\Cluster\CLUSDB </LocalHiveFile> 00000c58.00000ca8::2016/04/08-12:51:53.137 INFO <LocalHiveKey> Cluster </LocalHiveKey> 00000c58.00000ca8::2016/04/08-12:51:53.137 INFO <ParametersKey> System\\CurrentControlSet\\Services\ClusSvc\Parameters</ParametersKey>
00000c58.00000ca8::2016/04/08-12:51:53.137 INFO <Virtual IPv6> fe80::d003:5d7c:4ce:9425%30:~3343~ </Virtual IPv6>
00000c58.00000ca8::2016/04/08-12:51:53.137 INFO <Virtual IPv4> 169.254.1.253:~3343~ </Virtual IPv4> 2016/04/08-12:51:53.137 INFO <RetainNetFTConnSecurity> false </RetainNetFTConnSecurity>
00000c58.00000ca8::2016/04/08-12:51:53.137 INFO [RGP] Parameters set: stage 5 s, tick 300 ms 00000c58.000012e8::2016/04/08-12:51:53.137 INFO [CS] Disabling connection security.
00000c58.000012e8::2016/04/08-12:51:53.152 DBG [CS] Done with connection security 00000c58.00000ca8::2016/04/08-12:51:53.152 INFO [API] Successfully added Mgmt API SPNs to node's Computer Object 00000c58.00000ca8::2016/04/08-12:51:53.152 ERR [API] DmQueryString failed to retrieve the security descriptor status 2, default security descriptor will be used for authorizing client connections
00000c58.00000ca8::2016/04/08-12:51:53.152 INFO [API] Successfully built the default cluster SD.
00000c58.00000ca8::2016/04/08-12:51:53.152 INFO [MQ-treport] Starting
00000c58.00000ca8::2016/04/08-12:51:53.152 INFO [MQ-treport] Clearing 0 unsent and 0 unacknowledged messages. 00000c58.00000ca8::2016/04/08-12:51:53.152 INFO <ObjectManager>
00000c58.00000ca8::2016/04/08-12:51:53.152 INFO <ROOT id='.root' name='.root'>
00000c58.00000ca8::2016/04/08-12:51:53.152 INFO <CHMA id='ConnectivityMonitorAgent' name='ConnectivityMonitorAgent'/>
00000c58.00000ca8::2016/04/08-12:51:53.152 INFO <VSSR id='Cluster VSS Writer' name='Cluster VSS Writer'/> 00000c58.00000ca8::2016/04/08-12:51:53.152 INFO <CLUS id='PARCLUST' name='PARCLUST'/> 00000c58.00000ca8::2016/04/08-12:51:53.152 INFO <NMRR id='[NetworkMRR-1] ' name='[NetworkMRR-1] '/> 00000c58.00000ca8::2016/04/08-12:51:53.152 INFO <TP2P id='[RouteUnicast-1] ' name='[RouteUnicast-1] '/> 00000c58.00000ca8::2016/04/08-12:51:53.152 INFO </ROOT> 00000c58.00000ca8::2016/04/08-12:51:53.152 INFO </ObjectManager>
00000c58.00000ca8::2016/04/08-12:51:53.152 INFO [MM] Node 1: got a network stream to 1. Creating new NodeObject 00000c58.00000ca8::2016/04/08-12:51:53.152 INFO [QUORUM] unique node id for new node 1 is 00000000-0000-0000-0000-000000000001
00000c58.00000ca8::2016/04/08-12:51:53.152 INFO [MQ-parsql] Starting
00000c58.00000ca8::2016/04/08-12:51:53.152 INFO [MQ-parsql] Clearing 0 unsent and 0 unacknowledged messages. 0000c58.00000ca8::2016/04/08-12:51:53.152 INFO [NODE] Node 1: Done processing new stream to n1. 00000c58.00000f24::2016/04/08-12:51:53.152 INFO [PULLER parsql] Just about to start reading from <LoopbackStream name='parsql'/>
00000c58.00000ca8::2016/04/08-12:51:53.152 INFO [CS] Starting GEMV2
00000c58.00000ca8::2016/04/08-12:51:53.152 INFO [CS] GEM message buffer length is 50. 00000c58.00000ca8::2016/04/08-12:51:53.152 INFO [GEM] Node 1: Message buffer length changed to 50. It will take effect during the next regroup
00000c58.00000ca8::2016/04/08-12:51:53.152 INFO [CS] Starting HM Server
00000c58.00000f1c::2016/04/08-12:51:53.152 INFO [GEM] Node 1: Pause (proposedView = 001(1), currentView = 001(1), joiners = (), InSteadyState = true)
00000c58.00000f1c::2016/04/08-12:51:53.152 INFO [MRR] Node 1: Process view 001(1)
00000c58.00000f1c::2016/04/08-12:51:53.152 DBG [RCM] rcm::RcmAgent::ProcessView(<ViewChanged joiners=(1) downers=() newView=001(1) oldView=000() joiner=true form=true/>)
00000c58.00000f1c::2016/04/08-12:51:53.152 DBG [GEM] Node 1: message added to send queue 00000c58.00000f1c::2016/04/08-12:51:53.152 INFO [QUORUM] Node 1: I am the form/join coordinator (offline and lowest number node). Launching worker to request quorum config from other nodes
00000c58.00000ca8::2016/04/08-12:51:53.152 INFO [NODE] Node 1: New join with n3: stage: 'Listen for Connection' 00000c58.00000ca8::2016/04/08-12:51:53.152 INFO [NODE] Node 1: New join with n2: stage: 'Listen for Connection' 00000c58.00000ca8::2016/04/08-12:51:53.152 INFO [CS] Starting JPM (attempting connections to other cluster nodes) 00000c58.00000ca8::2016/04/08-12:51:53.152 DBG [JPM] Node 1: Dead node mentioned auditsql 00000c58.00000ca8::2016/04/08-12:51:53.152 DBG [JPM] Node 1: Dead node mentioned parsql 00000c58.00000ca8::2016/04/08-12:51:53.152 DBG [JPM] Node 1: Dead node mentioned treport 00000c58.00000ca8::2016/04/08-12:51:53.152 INFO [NETFT] Starting NetFT eventing for TM 00000c58.00000f1c::2016/04/08-12:51:53.152 DBG [DCM] ProcessView: current node set (1) 00000c58.00000ca8::2016/04/08-12:51:53.152 INFO [NETFT] TM NetFT event handler ready for events. 00000c58.00000ca8::2016/04/08-12:51:53.152 INFO [CS] Subscribing for VSS requests
00000c58.00000ca8::2016/04/08-12:51:53.152 INFO [VSS] Initializing Cluster VSS Writer at the command of the VSS framework.
00000c58.00000f1c::2016/04/08-12:51:53.152 INFO [GEM] Node 1: Resume (newView = 001(1), previousView = 000()) 00000c58.00000f1c::2016/04/08-12:51:53.152 DBG [GEM] Node 1: Deleting cached messages upon completion of repair. 00000c58.00000eb0::2016/04/08-12:51:53.152 DBG [DCM] HandleMessage: dcm/viewChanged 00000c58.00000f1c::2016/04/08-12:51:53.152 INFO [GEM] Node 1: Adjusting the size of message buffer because it was changed to 50
00000c58.00000eb0::2016/04/08-12:51:53.152 INFO [DCM] UpdateClusDiskMembership (enter): ctl 300228 nodeSet (1) 00000c58.00000514::2016/04/08-12:51:53.152 DBG [JPM] Node 1: Building contact data for node auditsql. 00000c58.00000eb0::2016/04/08-12:51:53.152 INFO [DCM] UpdateClusDiskMembership: ctl 300228 nodeSet (1), status 0 00000c58.00000f1c::2016/04/08-12:51:53.152 DBG [GEM] Node 1: SendNextMessage
00000c58.00000f1c::2016/04/08-12:51:53.152 INFO [CORE] EmitReadyForState for node 1 00000c58.00000f1c::2016/04/08-12:51:53.152 INFO [CORE] Issuing NeedStateViewChanged((1)) 00000c58.00000514::2016/04/08-12:51:53.152 INFO [JPM] Node 1: Scheduling DNS query for node name auditsql 00000c58.00000514::2016/04/08-12:51:53.152 DBG [JPM] Node 1: no valid contact endpoints yet for node auditsql. 00000c58.00000514::2016/04/08-12:51:53.152 DBG [JPM] Node 1: Building contact data for node treport. 00000c58.00000514::2016/04/08-12:51:53.152 INFO [JPM] Node 1: Scheduling DNS query for node name treport 00000c58.00000514::2016/04/08-12:51:53.152 DBG [JPM] Node 1: no valid contact endpoints yet for node treport. 00000c58.000013b4::2016/04/08-12:51:53.152 INFO [QUORUM] Node 1: received request for quorum info. Replying with quorum state <QuorumConfig tag='0:0:1' set='(1 2 3)' weights='(1 2 3)'/>
00000c58.000007d8::2016/04/08-12:51:53.152 INFO [QUORUM] Node 1: setting next best epoch to 0 00000c58.000007d8::2016/04/08-12:51:53.152 INFO [QUORUM] Node 1: The best quorum config so far is from node 1: <QuorumConfig tag='0:0:1' set='(1 2 3)' weights='(1 2 3)'/>
00000c58.000007d8::2016/04/08-12:51:53.152 INFO [QUORUM] Node 1: Coordinator: no quorum yet. do nothing 00000c58.00000514::2016/04/08-12:51:53.152 INFO [JPM] Node 1: Found possible contact address 192.168.33.159:~0~ for node auditsql via DNS.
00000c58.00000514::2016/04/08-12:51:53.152 INFO [JPM] Node 1: Found possible contact address 192.168.33.142:~0~ for node treport via DNS.
00000c58.00000ca8::2016/04/08-12:51:53.168 DBG [NETFT] Enabling heartbeats
00000c58.00000ca8::2016/04/08-12:51:53.168 INFO [CORE] Updating user mode deadlock recovery values. Action: UserModeActionType::TerminateProcess, Timeout: 60.
00000c58.00000ca8::2016/04/08-12:51:53.168 INFO [NETFT] Route history length is now: 10 00000c58.00000ca8::2016/04/08-12:51:53.168 INFO [CS] Setting route history length to 10. 00000c58.00000ca8::2016/04/08-12:51:53.168 DBG SetPreshutdownOrder()
00000c58.00000ca8::2016/04/08-12:51:53.168 INFO [CS] Reporting to SCM that cluster service has started. 00000c58.000007d8::2016/04/08-12:51:53.262 INFO [NM] Received request from client address ::1. 00000c58.000007d8::2016/04/08-12:51:53.262 DBG [API] Authenticated client--Client: PEPPERS\cluster_startup Interface: b97db8b2-4c63-11cf-bff6-08002be23f2f Server: parsql$@customer.com Level: RPC_C_AUTHN_LEVEL_PKT_PRIVACY Service: RPC_C_AUTHN_WINNT Protocol Sequence: ncacn_ip_tcp Client Address: ::1 Network Option: . 00000c58.000007d8::2016/04/08-12:51:53.262 WARN [VER] Could not read version data from database for node auditsql (id 3).
00000c58.000007d8::2016/04/08-12:51:53.262 WARN [VER] Could not read version data from database for node treport (id 2).
00000c58.000007d8::2016/04/08-12:51:53.262 DBG [VER] Calculated cluster versions: highest [Major 8 Minor 9600 Upgrade 3 ClusterVersion 0x00082580], lowest [Major 8 Minor 9600 Upgrade 3 ClusterVersion 0x00082580] with exclude node list: ()
00000c58.00000514::2016/04/08-12:51:54.152 DBG [JPM] Node 1: contacts size for node auditsql is 1, current index 0 00000c58.00000514::2016/04/08-12:51:54.152 DBG [JPM] Node 1: Trying to connect to node auditsql (IP: 192.168.33.159:~0~)
00000c58.00000514::2016/04/08-12:51:54.152 DBG [HM] Trying to connect to auditsql at 192.168.33.159:~3343~ 00000c58.00000514::2016/04/08-12:51:54.152 DBG [JPM] Node 1: contacts size for node treport is 1, current index 0 00000c58.00000514::2016/04/08-12:51:54.152 DBG [JPM] Node 1: Trying to connect to node treport (IP: 192.168.33.142:~0~)
00000c58.00000514::2016/04/08-12:51:54.152 DBG [HM] Trying to connect to treport at 192.168.33.142:~3343~ 00000c58.000013b4::2016/04/08-12:51:54.152 INFO [CONNECT] 192.168.33.159:~3343~: Established connection to remote endpoint 192.168.33.159:~3343~.
00000c58.00000514::2016/04/08-12:51:54.152 INFO [CONNECT] 192.168.33.142:~3343~: Established connection to remote endpoint 192.168.33.142:~3343~.
00000c58.000013b4::2016/04/08-12:51:54.152 INFO [NODE] Node 1: New join with n3: stage: 'Attempt Initial Connection' 00000c58.000013b4::2016/04/08-12:51:54.152 INFO [SV] New real route: local (192.168.33.149:~51004~) to remote auditsql (192.168.33.159:~3343~).
00000c58.00000514::2016/04/08-12:51:54.152 INFO [NODE] Node 1: New join with n2: stage: 'Attempt Initial Connection' 00000c58.000013b4::2016/04/08-12:51:54.152 INFO [SV] Got a new outgoing stream to auditsql at 192.168.33.159:~3343~
00000c58.00000514::2016/04/08-12:51:54.152 INFO [SV] New real route: local (192.168.33.149:~51005~) to remote treport (192.168.33.142:~3343~).
00000c58.00000514::2016/04/08-12:51:54.152 INFO [SV] Got a new outgoing stream to treport at 192.168.33.142:~3343~
00000c58.00000514::2016/04/08-12:51:54.152 DBG [SM] Joiner: Initialized with SPN = treport, RequiredCtxAttrib = 0, HandShakeTimeout = 40000
00000c58.000013b4::2016/04/08-12:51:54.152 DBG [SM] Joiner: Initialized with SPN = auditsql, RequiredCtxAttrib = 0, HandShakeTimeout = 40000
00000c58.00000f30::2016/04/08-12:51:54.152 DBG [SM] Handling auth handshake posted by thread id 1300 00000c58.000011e4::2016/04/08-12:51:54.152 DBG [SM] Handling auth handshake posted by thread id 5044 00000c58.00000f30::2016/04/08-12:51:54.152 DBG [SM] Joiner: ISC returned status = 590610 output Blob size 1526, service principal name HOST/treport, auth type MSG_AUTH_PACKAGE::KerberosAuth
00000c58.000011e4::2016/04/08-12:51:54.152 DBG [SM] Joiner: Sending SSPI blob of size 1527 to Sponsor 00000c58.00000f30::2016/04/08-12:51:54.152 DBG [SM] Joiner: Received SSPI blob from the Sponsor of size 48000, auth type: MSG_AUTH_PACKAGE::CurrentAuth
00000c58.000011e4::2016/04/08-12:51:54.152 DBG [SM] Joiner: Received SSPI blob from the Sponsor of size 48000, auth type: MSG_AUTH_PACKAGE::CurrentAuth
00000c58.00000f30::2016/04/08-12:51:54.152 DBG [SM] Joiner: ISC returned status = -2146893048 output Blob size 48000, service principal name HOST/treport, auth type MSG_AUTH_PACKAGE::KerberosAuth 00000c58.000011e4::2016/04/08-12:51:54.152 DBG [SM] Joiner: ISC returned status = -2146893048 output Blob size 48000, service principal name HOST/auditsql, auth type MSG_AUTH_PACKAGE::KerberosAuth 0000c58.00000514::2016/04/08-12:51:54.152 INFO [NODE] Node 1: New join with n2: stage: 'Authenticate Initial Connection' status HrError(0x80090308) reason: '[SV] Authentication failed'
00000c58.00000514::2016/04/08-12:51:54.152 DBG [CHANNEL 192.168.33.142:~3343~] Close(). 00000c58.000013b4::2016/04/08-12:51:54.152 INFO [NODE] Node 1: New join with n3: stage: 'Authenticate Initial Connection' status HrError(0x80090308) reason: '[SV] Authentication failed'
00000c58.000013b4::2016/04/08-12:51:54.152 DBG [CHANNEL 192.168.33.159:~3343~] Close(). 00000c58.000013b4::2016/04/08-12:51:54.152 WARN cxl::ConnectWorker::operator (): HrError(0x80090308)' because of '[SV] Authentication or Authorization Failed'
00000c58.00000514::2016/04/08-12:51:54.152 WARN cxl::ConnectWorker::operator (): HrError(0x80090308)' because of '[SV] Authentication or Authorization Failed'
00000c58.000013b4::2016/04/08-12:51:54.152 DBG [HM] Connection attempt to auditsql failed with error HrError(0x80090308): Failed to post-process connection to remote endpoint 192.168.33.159:~3343~. 00000c58.00000514::2016/04/08-12:51:54.152 DBG [HM] Connection attempt to treport failed with error HrError(0x80090308): Failed to post-process connection to remote endpoint 192.168.33.142:~3343~. 00000c58.0000115c::2016/04/08-12:51:54.152 INFO [ACCEPT] 0.0.0.0:~3343~: Accepted inbound connection from remote endpoint 192.168.33.142:~49916~.
00000c58.00000514::2016/04/08-12:51:54.152 INFO [SV] New real route: local (192.168.33.149:~3343~) to remote (192.168.33.142:~49916~).
00000c58.00000514::2016/04/08-12:51:54.152 INFO [SV] Got a new incoming stream from 192.168.33.142:~49916~ 00000c58.00000514::2016/04/08-12:51:54.152 DBG [SM] SrvCtxt initialized with MaxTokenSize = 48000, RequiredCtxAttrib = 0, HandShakeTimeout = 40000, ServerName = parsql
00000c58.00001308::2016/04/08-12:51:54.152 DBG [SM] Handling auth handshake posted by thread id 1300 00000c58.00001308::2016/04/08-12:51:54.152 DBG [SM] Sponsor: Received SSPI blob from the Joiner of size 1527, auth type: MSG_AUTH_PACKAGE::CurrentAuth
00000c58.00000514::2016/04/08-12:51:54.152 DBG [CHANNEL 192.168.33.142:~49916~] Close(). 00000c58.00000514::2016/04/08-12:51:54.152 WARN mscs::ListenerWorker::operator (): HrError(0x8009030c)' because of '[SV] Authentication or Authorization Failed'
00000c58.0000115c::2016/04/08-12:51:54.168 INFO [ACCEPT] 0.0.0.0:~3343~: Accepted inbound connection from remote endpoint 192.168.33.159:~50542~.
00000c58.00000514::2016/04/08-12:51:54.168 INFO [SV] New real route: local (192.168.33.149:~3343~) to remote (192.168.33.159:~50542~).
00000c58.00000514::2016/04/08-12:51:54.168 INFO [SV] Got a new incoming stream from 192.168.33.159:~50542~ 00000c58.00000514::2016/04/08-12:51:54.168 DBG [SM] SrvCtxt initialized with MaxTokenSize = 48000, RequiredCtxAttrib = 0, HandShakeTimeout = 40000, ServerName = parsql
00000c58.00001308::2016/04/08-12:51:54.168 DBG [SM] Handling auth handshake posted by thread id 1300 00000c58.00001308::2016/04/08-12:51:54.168 DBG [SM] Sponsor: Received SSPI blob from the Joiner of size 1537, auth type: MSG_AUTH_PACKAGE::CurrentAuth
00000c58.00000514::2016/04/08-12:51:54.168 DBG [CHANNEL 192.168.33.159:~50542~] Close(). 00000c58.00000514::2016/04/08-12:51:54.168 WARN mscs::ListenerWorker::operator (): HrError(0x8009030c)' because of '[SV] Authentication or Authorization Failed'
00000c58.00000514::2016/04/08-12:52:54.153 DBG [JPM] Node 1: contacts size for node auditsql is 1, current index 0 00000c58.00000514::2016/04/08-12:52:54.153 DBG [JPM] Node 1: Trying to connect to node auditsql (IP: 192.168.33.159:~0~)
00000c58.00000514::2016/04/08-12:52:54.153 DBG [HM] Trying to connect to auditsql at 192.168.33.159:~3343~ 00000c58.00000514::2016/04/08-12:52:54.153 DBG [JPM] Node 1: contacts size for node treport is 1, current index 0 00000c58.00000514::2016/04/08-12:52:54.153 DBG [JPM] Node 1: Trying to connect to node treport (IP: 192.168.33.142:~0~)
00000c58.00000514::2016/04/08-12:52:54.153 DBG [HM] Trying to connect to treport at 192.168.33.142:~3343~ 00000c58.000007d8::2016/04/08-12:52:54.153 INFO [CONNECT] 192.168.33.159:~3343~: Established connection to remote endpoint 192.168.33.159:~3343~.
00000c58.00000514::2016/04/08-12:52:54.153 INFO [CONNECT] 192.168.33.142:~3343~: Established connection to remote endpoint 192.168.33.142:~3343~.
00000c58.000007d8::2016/04/08-12:52:54.153 INFO [SV] New real route: local (192.168.33.149:~51008~) to remote auditsql (192.168.33.159:~3343~).
00000c58.00000514::2016/04/08-12:52:54.153 INFO [SV] New real route: local (192.168.33.149:~51009~) to remote treport (192.168.33.142:~3343~).
00000c58.000007d8::2016/04/08-12:52:54.153 INFO [SV] Got a new outgoing stream to auditsql at 192.168.33.159:~3343~
00000c58.00000514::2016/04/08-12:52:54.153 INFO [SV] Got a new outgoing stream to treport at 192.168.33.142:~3343~
00000c58.000007d8::2016/04/08-12:52:54.153 DBG [SM] Joiner: Initialized with SPN = auditsql, RequiredCtxAttrib = 0, HandShakeTimeout = 40000
00000c58.00000514::2016/04/08-12:52:54.153 DBG [SM] Joiner: Initialized with SPN = treport, RequiredCtxAttrib = 0, HandShakeTimeout = 40000
00000c58.00000f30::2016/04/08-12:52:54.153 DBG [SM] Handling auth handshake posted by thread id 1300 00000c58.000011e4::2016/04/08-12:52:54.153 DBG [SM] Handling auth handshake posted by thread id 2008 00000c58.00000f30::2016/04/08-12:52:54.153 DBG [SM] Joiner: ISC returned status = 590610 output Blob size 1526, service principal name HOST/treport, auth type MSG_AUTH_PACKAGE::KerberosAuth
00000c58.000011e4::2016/04/08-12:52:54.153 DBG [SM] Joiner: ISC returned status = 590610 output Blob size 1527, service principal name HOST/auditsql, auth type MSG_AUTH_PACKAGE::KerberosAuth
00000c58.00000f30::2016/04/08-12:52:54.153 DBG [SM] Joiner: Received SSPI blob from the Sponsor of size 48000, auth type: MSG_AUTH_PACKAGE::CurrentAuth
00000c58.000011e4::2016/04/08-12:52:54.153 DBG [SM] Joiner: ISC returned status = -2146893048 output Blob size 48000, service principal name HOST/auditsql, auth type MSG_AUTH_PACKAGE::KerberosAuth 00000c58.00000f30::2016/04/08-12:52:54.153 DBG [SM] Joiner: ISC returned status = -2146893048 output Blob size 48000, service principal name HOST/treport, auth type MSG_AUTH_PACKAGE::KerberosAuth 00000c58.00000f30::2016/04/08-12:52:54.153 DBG [SM] Joiner: Sending SSPI blob of size 48000 to Sponsor 00000c58.000011e4::2016/04/08-12:52:54.153 DBG [SM] Joiner: Sending SSPI blob of size 48000 to Sponsor 00000c58.00000514::2016/04/08-12:52:54.153 INFO [NODE] Node 1: New join with n2: stage: 'Authenticate Initial Connection' status HrError(0x80090308) reason: '[SV] Authentication failed'
00000c58.00000514::2016/04/08-12:52:54.153 DBG [CHANNEL 192.168.33.142:~3343~] Close(). 00000c58.000007d8::2016/04/08-12:52:54.153 INFO [NODE] Node 1: New join with n3: stage: 'Authenticate Initial Connection' status HrError(0x80090308) reason: '[SV] Authentication failed'
00000c58.000007d8::2016/04/08-12:52:54.153 DBG [CHANNEL 192.168.33.159:~3343~] Close(). 00000c58.00000514::2016/04/08-12:52:54.153 WARN cxl::ConnectWorker::operator (): HrError(0x80090308)' because of '[SV] Authentication or Authorization Failed'
00000c58.00000514::2016/04/08-12:52:54.153 DBG [HM] Connection attempt to treport failed with error HrError(0x80090308): Failed to post-process connection to remote endpoint 192.168.33.142:~3343~. 00000c58.000007d8::2016/04/08-12:52:54.153 WARN cxl::ConnectWorker::operator (): HrError(0x80090308)' because of '[SV] Authentication or Authorization Failed'
00000c58.000007d8::2016/04/08-12:52:54.153 DBG [HM] Connection attempt to auditsql failed with error HrError(0x80090308): Failed to post-process connection to remote endpoint 192.168.33.159:~3343~. 00000c58.0000115c::2016/04/08-12:52:54.153 INFO [ACCEPT] 0.0.0.0:~3343~: Accepted inbound connection from remote endpoint 192.168.33.142:~49920~.
00000c58.000007d8::2016/04/08-12:52:54.153 INFO [SV] New real route: local (192.168.33.149:~3343~) to remote (192.168.33.142:~49920~).
00000c58.000007d8::2016/04/08-12:52:54.153 INFO [SV] Got a new incoming stream from 192.168.33.142:~49920~ 00000c58.000007d8::2016/04/08-12:52:54.153 DBG [SM] SrvCtxt initialized with MaxTokenSize = 48000, RequiredCtxAttrib = 0, HandShakeTimeout = 40000, ServerName = parsql
00000c58.00001308::2016/04/08-12:52:54.153 DBG [SM] Handling auth handshake posted by thread id 2008 00000c58.00001308::2016/04/08-12:52:54.153 DBG [SM] Sponsor: Received SSPI blob from the Joiner of size 1527, auth type: MSG_AUTH_PACKAGE::CurrentAuth
00000c58.00001308::2016/04/08-12:52:54.153 DBG [SM] Sponsor: Authentication handshake final Status -2146893044 00000c58.000007d8::2016/04/08-12:52:54.153 DBG [CHANNEL 192.168.33.142:~49920~] Close(). 00000c58.000007d8::2016/04/08-12:52:54.153 WARN mscs::ListenerWorker::operator (): HrError(0x8009030c)' because of '[SV] Authentication or Authorization Failed'
00000c58.0000115c::2016/04/08-12:52:54.169 INFO [ACCEPT] 0.0.0.0:~3343~: Accepted inbound connection from remote endpoint 192.168.33.159:~50545~.
00000c58.000007d8::2016/04/08-12:52:54.169 INFO [SV] New real route: local (192.168.33.149:~3343~) to remote (192.168.33.159:~50545~).
00000c58.000007d8::2016/04/08-12:52:54.169 INFO [SV] Got a new incoming stream from 192.168.33.159:~50545~ 00000c58.000007d8::2016/04/08-12:52:54.169 DBG [SM] SrvCtxt initialized with MaxTokenSize = 48000, RequiredCtxAttrib = 0, HandShakeTimeout = 40000, ServerName = parsql
00000c58.000007d8::2016/04/08-12:52:54.169 WARN mscs::ListenerWorker::operator (): HrError(0x8009030c)' because of '[SV] Authentication or Authorization Failed'
00000c58.00000514::2016/04/08-12:53:54.154 DBG [JPM] Node 1: contacts size for node auditsql is 1, current index 0 00000c58.00000514::2016/04/08-12:53:54.154 DBG [JPM] Node 1: Trying to connect to node auditsql (IP: 192.168.33.159:~0~)
00000c58.00000514::2016/04/08-12:53:54.154 DBG [HM] Trying to connect to auditsql at 192.168.33.159:~3343~ 00000c58.00000514::2016/04/08-12:53:54.154 DBG [JPM] Node 1: contacts size for node treport is 1, current index 0 00000c58.00000514::2016/04/08-12:53:54.154 DBG [JPM] Node 1: Trying to connect to node treport (IP: 192.168.33.142:~0~)
00000c58.00000514::2016/04/08-12:53:54.154 DBG [HM] Trying to connect to treport at 192.168.33.142:~3343~ 00000c58.0000117c::2016/04/08-12:53:54.154 INFO [CONNECT] 192.168.33.159:~3343~: Established connection to remote endpoint 192.168.33.159:~3343~.
00000c58.00000514::2016/04/08-12:53:54.154 INFO [CONNECT] 192.168.33.142:~3343~: Established connection to remote endpoint 192.168.33.142:~3343~.
00000c58.0000117c::2016/04/08-12:53:54.154 INFO [SV] New real route: local (192.168.33.149:~51012~) to remote auditsql (192.168.33.159:~3343~).
00000c58.00000514::2016/04/08-12:53:54.154 INFO [SV] New real route: local (192.168.33.149:~51013~) to remote treport (192.168.33.142:~3343~).
00000c58.0000117c::2016/04/08-12:53:54.154 INFO [SV] Got a new outgoing stream to auditsql at 192.168.33.159:~3343~
00000c58.00000514::2016/04/08-12:53:54.154 INFO [SV] Got a new outgoing stream to treport at 192.168.33.142:~3343~
00000c58.0000117c::2016/04/08-12:53:54.154 DBG [SM] Joiner: Initialized with SPN = auditsql, RequiredCtxAttrib = 0, HandShakeTimeout = 40000
00000c58.00000514::2016/04/08-12:53:54.154 DBG [SM] Joiner: Initialized with SPN = treport, RequiredCtxAttrib = 0, HandShakeTimeout = 40000
00000c58.000011e4::2016/04/08-12:53:54.154 DBG [SM] Joiner: ISC returned status = 590610 output Blob size 1527, service
principal name HOST/auditsql, auth type MSG_AUTH_PACKAGE::KerberosAuth
00000c58.00000f30::2016/04/08-12:53:54.154 DBG [SM] Joiner: Sending SSPI blob of size 1526 to Sponsor 00000c58.000011e4::2016/04/08-12:53:54.154 DBG [SM] Joiner: Sending SSPI blob of size 1527 to Sponsor 00000c58.000011e4::2016/04/08-12:53:54.154 DBG [SM] Joiner: Received SSPI blob from the Sponsor of size 48000, auth type: MSG_AUTH_PACKAGE::CurrentAuth
00000c58.00000f30::2016/04/08-12:53:54.154 DBG [SM] Joiner: Received SSPI blob from the Sponsor of size 48000, auth type: MSG_AUTH_PACKAGE::CurrentAuth
00000c58.000011e4::2016/04/08-12:53:54.154 DBG [SM] Joiner: ISC returned status = -2146893048 output Blob size 48000, service principal name HOST/auditsql, auth type MSG_AUTH_PACKAGE::KerberosAuth 00000c58.000011e4::2016/04/08-12:53:54.154 DBG [SM] Joiner: Sending SSPI blob of size 48000 to Sponsor 00000c58.0000117c::2016/04/08-12:53:54.154 INFO [NODE] Node 1: New join with n3: stage: 'Authenticate Initial Connection' status HrError(0x80090308) reason: '[SV] Authentication failed'
00000c58.00000f30::2016/04/08-12:53:54.154 DBG [SM] Joiner: Sending SSPI blob of size 48000 to Sponsor 00000c58.0000117c::2016/04/08-12:53:54.154 WARN cxl::ConnectWorker::operator (): HrError(0x80090308)' because of '[SV] Authentication or Authorization Failed'
00000c58.00000514::2016/04/08-12:53:54.154 INFO [NODE] Node 1: New join with n2: stage: 'Authenticate Initial Connection' status HrError(0x80090308) reason: '[SV] Authentication failed'
00000c58.00000514::2016/04/08-12:53:54.154 DBG [CHANNEL 192.168.33.142:~3343~] Close(). 00000c58.0000117c::2016/04/08-12:53:54.154 DBG [HM] Connection attempt to auditsql failed with error HrError(0x80090308): Failed to post-process connection to remote endpoint 192.168.33.159:~3343~. 00000c58.00000514::2016/04/08-12:53:54.154 WARN cxl::ConnectWorker::operator (): HrError(0x80090308)' because of '[SV] Authentication or Authorization Failed'
00000c58.00000514::2016/04/08-12:53:54.154 DBG [HM] Connection attempt to treport failed with error HrError(0x80090308): Failed to post-process connection to remote endpoint 192.168.33.142:~3343~. 00000c58.0000115c::2016/04/08-12:53:54.154 INFO [ACCEPT] 0.0.0.0:~3343~: Accepted inbound connection from remote endpoint 192.168.33.142:~49922~.
00000c58.00000514::2016/04/08-12:53:54.154 INFO [SV] New real route: local (192.168.33.149:~3343~) to remote (192.168.33.142:~49922~).
00000c58.00000514::2016/04/08-12:53:54.154 INFO [SV] Got a new incoming stream from 192.168.33.142:~49922~ 00000c58.00000514::2016/04/08-12:53:54.154 DBG [SM] SrvCtxt initialized with MaxTokenSize = 48000, RequiredCtxAttrib = 0, HandShakeTimeout = 40000, ServerName = parsql
00000c58.00001308::2016/04/08-12:53:54.154 DBG [SM] Sponsor: Authentication handshake final Status -2146893044 00000c58.00000514::2016/04/08-12:53:54.154 DBG [CHANNEL 192.168.33.142:~49922~] Close(). 00000c58.00000514::2016/04/08-12:53:54.154 WARN mscs::ListenerWorker::operator (): HrError(0x8009030c)' because of '[SV] Authentication or Authorization Failed'
00000c58.0000115c::2016/04/08-12:53:54.170 INFO [ACCEPT] 0.0.0.0:~3343~: Accepted inbound connection from remote endpoint 192.168.33.159:~50547~.
00000c58.00000514::2016/04/08-12:53:54.170 INFO [SV] New real route: local (192.168.33.149:~3343~) to remote (192.168.33.159:~50547~).
00000c58.00000514::2016/04/08-12:53:54.170 INFO [SV] Got a new incoming stream from 192.168.33.159:~50547~ 00000c58.00000514::2016/04/08-12:53:54.170 DBG [SM] SrvCtxt initialized with MaxTokenSize = 48000, RequiredCtxAttrib = 0, HandShakeTimeout = 40000, ServerName = parsql
00000c58.00000514::2016/04/08-12:53:54.186 DBG [CHANNEL 192.168.33.159:~50547~] Close(). 00000c58.00000514::2016/04/08-12:53:54.186
WARN mscs::ListenerWorker::operator (): HrError(0x8009030c)' because of '[SV] Authentication or Authorization Failed'
00000c58.000002e0::2016/04/08-12:54:53.093 DBG [NODE] Node 1: eating message sent to the dead node 2 00000c58.000002e0::2016/04/08-12:54:53.093
DBG [NODE] Node 1: eating message sent to the dead node 3 00000c58.000002e0::2016/04/08-12:54:53.093 INFO [CS] Service Stopping...
00000c58.000002e0::2016/04/08-12:54:53.093 INFO [CORE] Node quorum state is 'Not yet formed or joined a cluster'. Form/join status with other nodes is as follows:
00000c58.000002e0::2016/04/08-12:54:53.093 INFO [NODE] Node 1: Farthest reported progress joining with node auditsql (id 3) is: Authenticate Initial Connection at time 2016/04/08-12:53:54.154: status 776 [SV] Authentication failed 00000c58.000002e0::2016/04/08-12:54:53.093 INFO [NODE] Node 1: Farthest reported progress joining with node treport (id 2) is: Authenticate Initial Connection at time 2016/04/08-12:53:54.154: status 776 [SV] Authentication failed 00000c58.000002e0::2016/04/08-12:54:53.093 INFO [DCM] UnregisterSwProvider(): CSV providers are not registered 00000c58.000002e0::2016/04/08-12:54:53.093 INFO [RGP] node 1: MergeAndRestart +() -(1) 00000c58.000002e0::2016/04/08-12:54:53.093 INFO [RGP] sending to 64 nodes 1: 001(1) => 101() +() -(1) [()] 00000c58.00000f1c::2016/04/08-12:54:53.093 INFO [CORE] Node 1: Proposed View is <ViewChanged joiners=() downers=(1) newView=101() oldView=001(1) joiner=false form=false/>
00000c58.000002e0::2016/04/08-12:54:53.093 DBG [NODE] Node 1: eating message sent to the dead node 2 00000c58.000002e0::2016/04/08-12:54:53.093 DBG [NODE] Node 1: eating message sent to the dead node 3 00000c58.000002e0::2016/04/08-12:54:53.202 INFO [DM]: Shutting down, so unloading the cluster database. 00000c58.000002e0::2016/04/08-12:54:53.202 INFO [DM] Shutting down, so unloading the cluster database (waitForLock: true).
00000c58.000002e0::2016/04/08-12:54:53.202 DBG [DM] Unloading Hive, Key \Registry\Machine\Cluster, discardCurrentChanges true
00000c58.000002e0::2016/04/08-12:54:53.202 INFO [CS] Service Stopped...
00000c58.000002e0::2016/04/08-12:54:53.202 INFO [CS] About to exit service...
00000c58.000002e0::2016/04/08-12:54:53.202 INFO [NETFT] Cluster Service preterminate succeeded.
I've been struggling with this site for a while now and I'm really at a loss. Any ideas or insight would be most welcome.