An all-hardware Exchange 2010 SP3 UR4 DAG cluster is having an issue when the Microsoft Loopback adapter is installed (from Device Manager...Add Legacy Hardware) to support DSR operations with hardware load balancer (HLB).

• The HLB provides HA endpoint for RPC Client Access, SMTP, etc. DSR is required to preserve source IP--on which      Exchange receive connectors that filter on source IP for security depend.
• It is server DAG, with 3 x production severs at the datacenter and 2 x DAG DR servers located in a DR site.
• Only the 3 x production servers at the main site have the loopback adapter installed.
• The loopback-DSR-specific settings like 'weakhostrecive, etc' are in effect.

The problem only involves the 3 servers in the DAG with loopback adapters.

The issue is that when a DAG member restarts, sometimes it will cause the online production cluster node which isnot the Cluster Host Server to fail. Consider:

• DAGNode1, Loopback enabled, Healthy, Is Cluster Host Server
• DAGNode2, Loopback enabled, Healthy
• DAGNode3, Loopback enabled, is Restarted

In this scenario, the cluster service on DAGNode2 will experience a loss of network connectivity when DAGNode3 rejoins the cluster (DAGNode2 reports cluster failure on all other nodes) and shortly afterwards the Cluster Service on DAGNode2 will terminate. FailoverClustering 1572 is seen on DAGNode2:

Node 'DAGNode2' failed to join the cluster because it could not send and receive failure detection network messages with other cluster nodes. Please run the Validate a Configuration wizard to ensure network settings. Also verify the Windows Firewall 'Failover Clusters' rules.

Interestingly, if you disable the Loopback on DAGNode3, DAGNode2 will immediately rejoin the cluster! Re-enable the Loopback on DAGNode3 and DAGNode2 immediately fails again! With some more server restarts possibly, you get a stable cluster again with Loopback enabled on all production nodes. The status of the loopback (enabled or not) on the Cluster Host does not impact this issue.

As I mentioned, it is only some restarts that this occurs, usually there is no problem. Also note the Loopback network/adapters do not appear in Cluster Manager and are not listed as cluster networks with cluster.exe. Cluster Validation Wizard passes everything except noting that every node has a duplicate IP on an installed adapter.

Looking for others with experience that have combined DSR-based HLB with CAS/Hub/MBX DAG Cluster on same Exchange computers and were able to use reliably.

There is an unanswered thread from 2010 on this topic:

http://social.technet.microsoft.com/Forums/windowsserver/en-US/7616b0e5-6fb6-4be7-a859-14baa2e9b925/cluster-network-is-partitioned-due-to-loopback-adapter?forum=winserverClustering

Some questions / any answers are very welcome!

• Can I add the Loopback adapter to the cluster configuration so that I can use Cluster.exe to ignore the loopback adapter?
• Can I prevent other cluster nodes from seeing the loopback adapters in the other nodes? Is there an ‘ignore partner adapter’ setting?

Thank you!

John Joyner MVP-SC-CDM

P.S. I add this information 3/1/2014:

This link suggests that if you allow the cluster network to partition it will discover the loopback adapters and they will appear in cluster manager: (Did this by enabled IPV6 on Loopback, when done this made Loopback Network appear in Cluster Manager. Then used Cluster.exe to set IgnoreNetwork=$true on the Loopback network.) Result: No change, still caused cluster communication outage when Loopback enabled on third production node that is not the Cluster Group Host.

http://social.technet.microsoft.com/Forums/windowsserver/en-US/311f7763-9f72-4dfe-bb35-3fd1a1dc567c/adding-additional-network-adapters-to-a-cluster?forum=winserverClustering

Developed: A workaround!

1. Just before restarting a node, after drain stop in NS, and after running StartDAGServerMaintenance.PS1 (which pauses the node in Cluster Manager) disable the Loopback Adapter so that when the computer restarts, Loopback is disabled.

2. After node restarts and rejoins cluster in Paused status, and after running StopDAGServerMaintenance.PS1, issue this command to move the Cluster Group host to the computer that was restarted and has the Loopback disabled.

cluster <clustername> group "Cluster Group" /moveto:<nodename>

3. Then safely enable the Loopback on the computer that was restarted and is now the Cluster Group host.

4. Then take the computer out of drain stop in the NS.

This of course only applies to controlled restarts.

In the event of unexpected server crash and recoveries, there is nothing stopping this from happening when the crashed server restarts. Still need a real fix! With knowledge of how to defuse the situation when it happens (disable loopback on the production node that is not the Cluster Group host), it clears the condition immediately. You than then fix it by steps 2 and 3 in the workaround.

One of my customers has a Windows Server 2008 R2 cluster for an Exchange 2010 Mailbox Database Availability Group.  Lately, they've been having problems with one of their nodes (the one node that is on a different subnet in a different datacenter) where their Exchange databases aren't replicating.  While looking into this issue it seems that the problem is the Network Manager isn't started because the cluster service is failing.  Since the issue seems to be with the cluster service, and not Exchange, I'm asking here. 

When the cluster service starts, it appears to start working, but within a few minutes the following is logged in the system event log.

FailoverClustering

1572

Critical

Cluster Virtual Adapter

Node 'nodename' failed to join the cluster because it could not send and receive failure detection network messages with other cluster nodes. ...

It seems that the problem is with the 169.254 address on the cluster virtual adapter.  An entry in the cluster.log file says: Aborting connection because NetFT route to node nodename on virtual IP 169.254.1.44:~3343~ has failed to come up. 

In my experience, you never have to mess with the cluster virtual adapter.  I'm not sure what happened here, but I doubt it has been modified.  I need the cluster to communicate with its other nodes on our routed 10. network.  I've never experienced this before and found little in my searches on the subject.  Any idea how I can fix this?

Thanks,

Joe

Joseph M. Durnal MCM: Exchange 2010 MCITP: Enterprise Messaging Administrator, Exchange 2010 MCITP: Enterprise Messaging Administrator, MCITP: Enterprise Administrator

Hi. 

I would like to configure a DCHP server failover on Windows 7.

On a part I have a windows 2008 R2 server which currently leases adresses. I recently had a crash of this server.

In order to insure avaibility of dhcp on the next crash, I whish install a DHCP server freeware and configure a script powershell to start dhcp on failover of the first one.

An other tip is to split the pool of leasing. Is it possible to do it with a windows 7 software?

Thanks a lot for your help.

Hello,

Anyone have thoughts on our experiment to build a simple 'Beowulf Cluster' under a Windows environment ?  Presently we have four older Dell single core machines, nearly identical, that would fit well into this test. I have been reviewing the book from MIT entitled "Beowulf Cluster Computing with Linux" published in 2003 but that uses Ubuntu as the basis for the cluster.

I have also been looking at articles from the Ubuntu wiki entitled: "Ubuntu Kerrighed Cluster Guide" at  https://wiki.ubuntu.com/EasyUbuntuClustering/UbuntuKerrighedClusterGuide with a step by step guide on how to set up a Kerrighed Cluster.

In 2006, Microsoft introduced Windows Compute Cluster Server however this appears to be for high-performance computing clusters and I think was meant to replace Linux. It's probably not applicable to this project.  Windows CCS is also fairly expensive for just an experiment.

I am learning my way through this experimental project and therefore am starting with no knowledge on the subject. 

Any help or direction would be greatly appreciated.

Thanks,

Dan

Hello All,

We have a Server 2008 R2 two-node Hyper-V cluster running on an IBM DS3400 SAN.
We would like to upgrade the nodes to Server 2012 R2. The node hardware supports 2012 R2. But IBM hasn't listed support of 2012 R2 for the DS3400 SAN. The latest supported OS is 2008 R2.

My question is, despite this lack of support, what is the likelihood that 2012 R2 will work with the DS3400 SAN? Is it one of those things where it is likely to work, or is it something that is a risky undertaking due to the lack of listed compatibility, and I should leave it alone? A lot of the general documentation seems to suggest that, in general, if a SAN worked for 2008 R2, then it should be fine with 2012 R2 .. but, as mentioned, it is quite general information.

Thanks!

I have a 3 node failover cluster that I think needs an IP address change because the current cluster IP address is on my Node Management network.

Here is my network config:

FC Internal Communication
10.24.24.0/24, Cluster Use: Internal

FC Live Migration
10.25.25.0/24, Cluster Use: Internal

iSCSI Traffic
10.23.23.0/24 Cluster Use: Disabled

Node Management
10.20.20.0/24 Cluster Use: Enabled (also allows clients to connect

The cluster IP Address is 10.20.20.200, which of course means it sits on the Node Management network.

I want to disallow cluster use of the Node Management network, but that network hold the IP Address of my failover cluster, so as soon as stop allowing cluster network communication on that network, my cluster IP Address fails

Is it best practice to have the cluster IP address set on a network dedicated to cluster communication?  If so, how can I change the address of a running cluster?

Thanks for any help you can offer,

hi

i have created fail over cluster with windows server 2012

i have a DHCP vm 

when i choose roles>> dhcp role to configure that DHCP vm it says no dhcp server found!!

so my question is isnt it possibel to make a DHCP vm as DHCP fail over role ??

but if i install DHCP role in physical machine then it shows availabel in cluster role

istiaq

Hi.

I am clustering two HYPER-V 2012 servers, i connected to two LUNs coming from SAN using ISCSI initiator. one of the LUN is online and assigned a drive letter for quorum. another LUN is not coming online and when initializing it says "incorrect function"?

I can see that drive is offline on both hyper-v 2012 servers, and i can't make them online. SAN administrator says its not an issue with SAN as both LUNs are comming from the same SAN drive and hyper-v servers can connect to both successfully.

but i am not able to figure out where is the issue? please help.

regards..

Himanshu Rana

MCTS|MCSE|MCSA:Messaging|CCNA

I have a two-node Hyper-V 2012 Core Server cluster. Initially, the cluster was working fine and I was able to do live migrations with no problem. I first realized that something had gone wrong when I was attempting to move a VM's storage to the Cluster Disk. When I right click on the VM and select Move > Virtual Machine Storage, the bottom left area of the "move storage" screen says "Loading..." forever and never displays the cluster storage. This happened once before and I was able to correct the issue by taking the Server Name resource offline and doing More Actions > Repair. However, if I try a repair now I receive the following error messages:

"There was an error repairing the active directory object for 'Cluster Name'. Error Code: 0x800713b8. The cluster request is not valid for this object". 

I have verified that the cluster name Active Directory object (CCSDCluster) still exists and each node as well as itself has full permissions to the object. What I am not certain of is whether the object is still in the same Organizational Unit as it was originally. After the failed repair attempt, I am able to bring the Server Name resource back online successfully.

In addition, I am continually receiving cluster event errors indicating the following:

"Cluster network name resource 'Cluster Name' failed registration of one or more associated DNS name(s) for the following reason: The handle is invalid. Ensure that the network adapters associated with dependent IP address resources are configured with at least one accessible DNS server." 

Each of the two nodes has multiple NICs configured. One for management, one for guest OSs (Hyper-V virtual switch is configured to use this NIC and not allow management OS access), and two for iSCSI (for redundancy and multi-path). Live migration is set to only use the management NICs. The management NIC on both nodes is configured with valid DNS servers. The CCSDCluster DNS entry has full permissions for both nodes and itself.

Any suggestions on what to try would be greatly appreciated. Thank you!

Cal Miyatake

I can't find internet help on creating Windows 2008 cluster log via C#.

can some help with samples?

I'm in the process of setting up a new cluster using Windows Server 2012 with a view to migrate from our existing 2008 R2 cluster.  I am having problems validating the storage and the wizard keeps failing on the List All Potential Cluster Disks test (which causes all the other storage tests to be cancelled).

I'm getting the following error:

Failed while verifying removal of any Persistent Reservation on physical disk 19ba99a4 at node RC-HYPERV-3.riddlesdown.local.

When I look in validation report, disk 19ba99a4 is the management LUN (Access LUN 31 as termed by the storage software).

Looking on my 2008 R2 cluster this doesn't appear in Disk Management.

Any ideas how I can get rid of it or the error that it is causing?

Trying to setup a brand new cluster (first node) on Server 2012. Hardware passes cluster validation tests and consists of a dell 2950 with an MD1000 JBOD enclosure configured with a bunch of 7.2K RPM SAS and 15k SAS Drives. There is no RAID card or any other storage fabric, just a SAS adapter and an external enclosure.

I can create a regular storage pool just fine and access it with no issues on the same box when I don't add it to the cluster. However when I try to add it to the cluster I keep getting these errors on adding a disk:

Error Code: 0x8007139F if I try to add a disk (The group or resource is not in the correct state to perform the requested operation)

When adding the Pool I get this error:

Error Code 0x80070016 The Device Does not recognize the command

Full Error on adding the pool

----------------------------------------------------------------

Cluster resource 'Cluster Pool 1' of type 'Storage Pool' in clustered role 'b645f6ed-38e4-11e2-93f4-001517b8960b' failed. The error code was '0x16' ('The device does not recognize the command.').

Based on the failure policies for the resource and role, the cluster service may try to bring the resource online on this node or move the group to another node of the cluster and then restart it.  Check the resource and group state using Failover Cluster Manager or the Get-ClusterResource Windows PowerShell cmdlet.

----------------------------------------------------------------

if I try to just add the raw disks to the storage -- without using a pool or anything - almost every one of them but one fails with incorrect function except for one (a 7.2K RPM SAS drive). I cannot see any difference between it and the other disks. Any ideas? The error codes aren't anything helpful. I would imagine there's something in the drive configuration or hardware I am missing here I just don't know what considering the validation is passing and I am meeting the listed prerequisites.

If I can provide any more details that would assist please let me know. Kind of at a loss here.

I am configuring Cluster-Aware Updating on a 5-node cluster. At this time I'm not enabling Self-Updating mode. Instead I'm going to use Remote-Updating mode. In the Cluster Aware Updating console I ran Analyze cluster updating readiness. I have two warnings that I can safely ignore (about local machine proxy and CAU clustered role not being enabled). But there is an error that I'm stuck on. Rule ID 13 gives me an error saying, "The configured CAU plug-in must be registered on all failover cluster nodes."

The resolution says to ensure that the configured CAU plug-in is inatlled on the all cluster nodes. I ran Get-CauPlugin | fl -Property * on each node and each of them returned the expected Microsoft.WindowsUpdatePlugin and Microsoft.Hotfix.Plugin listings.

Any ideas on how I can troubleshoot and get the error cleared?

I have a H-V 2012 cluster with 3 nodes.  I use a CSV volume to store the VMs.  I have all the latest patches installed. I have an Equallogic SAN (PS4000) with the latest firmware on it providing the LUN for the CSV.  Everything in my environment is supposed to support re-thinning (or unmapping, or whatever the right term is) the LUN.  I have about 500 GB of unused space on the 2TB volume, and the volume was thin provisioned.  A restore of some very large vhd files from backup caused the thin provisioned volume to grow and to use almost the entire volume at one point but the corrupt VHDs have since been deleted.  Now I have "dirty" blocks in the LUN that I want to reclaim into free space on the SAN. This all happens, apparently, when Server 2012 performs an "Optimize" on the disks.  In my environment this is scheduled to happen once a week.  It did apparently do something this last week, because my volume utilization on the SAN went from 96% to 91%.  Not even close to reclaiming all dirty blocks, but it's a start I guess.  So now I went in to the "Defragment and Optimize Drives" utility and told it to commence a manual optimization.  Nothing happens and event viewer give me this error:

So my questions are these:

Shouldn't it put the CSV in redirect mode if it needs to do this in order to optimize the drive automatically?

If it can't do this automatically, how did it return 5% of the CSV SAN volume to free space last week?

Can I put the volume in redirect mode manually and do the optimize manually? Redirect mode is not supposed to be necessary in 2012 CSV any more- at least not for backup.  Why here?

Will my environment re-thin, Unmap, whatever?  It appears it MIGHT.  Does it take several iterations (ie weeks)?

Can anyone explain this incredibly vague and cloaked process from a Windows server 2012 perspective?

Thank you for any help!

DML

DLovitt

the disks show offline and the server: file server 

if i try to get it online it says it took more than usual and still will fail

Hi,

We experience some strange problem with our SQL Cluster. First I can't connect to the SQL Instance using SSMS.

Using wireshark showing a lot of TCP Retransmission. I using iperf to test the speed, and notice that with the ip address of the Cluster host, the connection is good. But using the Ip address of the Named Instance of SQL, the connection speed is very bad.

Hope somebody can help me figure this out.

Hi all,

windows 2003 cluster with one node

what's the proper way to reboot it? pause and reboot?

thank you!

Hi All,

We are having a cluster with two nodes and I found one node down also cluster service is in stopped. While trying to start the service we are getting errors. Server has been rebooted. I found some errors in cluster log file I have attached below:

This is Windows 2003 cluster. If anyone can find some solution will be very helpful. 

====Cluster LOG==========================================

000015c8.0000127c::2014/03/06-17:20:43.436 INFO [CS] Cluster Service started - Cluster Node Version 4.3790
000015c8.0000127c::2014/03/06-17:20:43.436 INFO                                OS Version 5.2.3790 - Service Pack 2 (ADS 03000112L)
000015c8.0000127c::2014/03/06-17:20:43.436 INFO                                Local Time is  2014/03/06-17:20:43.436
000015c8.00001764::2014/03/06-17:20:43.436 INFO [CS] Service Starting...
000015c8.00001764::2014/03/06-17:20:43.436 INFO [INIT] ClusterInitialize called to start cluster.
000015c8.00001764::2014/03/06-17:20:43.451 INFO [EP] Initialization...
000015c8.00001764::2014/03/06-17:20:43.451 INFO [DM] Initialization
000015c8.00001764::2014/03/06-17:20:43.451 ERR  [DM] DmInitialize: The hive was loaded- rollback, unload and reload again
000015c8.00001764::2014/03/06-17:20:43.451 INFO [DM] DmpRestartFlusher: Entry
000015c8.00001764::2014/03/06-17:20:43.451 INFO [DM] DmpUnloadHive: unloading the hive
000015c8.00001764::2014/03/06-17:20:43.498 INFO [Qfs] QfsSetFileAttributes C:\WINDOWS\Cluster\CLUSDB.BKP$ 80, status 2
000015c8.00001764::2014/03/06-17:20:43.498 INFO [Qfs] QfsDeleteFile C:\WINDOWS\Cluster\CLUSDB.BKP$, status 2
000015c8.00001764::2014/03/06-17:20:43.498 INFO [DM] Loading cluster database from C:\WINDOWS\Cluster\CLUSDB
000015c8.00001764::2014/03/06-17:20:43.530 INFO [DM] DmpStartFlusher: Entry
000015c8.00001764::2014/03/06-17:20:43.530 INFO [DM] DmpStartFlusher: thread created
000015c8.00001764::2014/03/06-17:20:43.530 INFO [NM] Initializing...
000015c8.00001764::2014/03/06-17:20:43.530 INFO [NM] Local node name = GB02QDB101RBB27.
000015c8.00001764::2014/03/06-17:20:43.530 INFO [NM] Local node ID = 1.
000015c8.00001764::2014/03/06-17:20:43.530 INFO [NM] Creating object for node 1 (GB02QDB101RBB27)
000015c8.00001764::2014/03/06-17:20:43.530 INFO [NM] Initializing networks.
000015c8.00001764::2014/03/06-17:20:43.530 INFO [NM] Initializing network interfaces.
000015c8.00001764::2014/03/06-17:22:43.544 ERR  [NM] Couldn't establish connection point with Net Connection Manager, status 80080005.
000015c8.00001764::2014/03/06-17:22:43.544 WARN [NM] Couldn't initialize Net Connection Manager advise sink, status 80080005
000015c8.00001764::2014/03/06-17:22:43.544 ERR  [NM] Initialization failed -2146959355
000015c8.00000940::2014/03/06-17:22:43.544 INFO [NM] Pnp worker thread terminating.
000015c8.00001764::2014/03/06-17:22:43.544 INFO [NM] Shutdown starting...
000015c8.00001764::2014/03/06-17:22:43.544 INFO [NM] Interface cleanup starting...
000015c8.00001764::2014/03/06-17:22:43.544 INFO [NM] Network interface cleanup complete
000015c8.00001764::2014/03/06-17:22:43.544 INFO [NM] Network cleanup starting...
000015c8.00001764::2014/03/06-17:22:43.544 INFO [NM] Network cleanup complete
000015c8.00001764::2014/03/06-17:22:43.544 INFO [NM] Node cleanup starting...
000015c8.00001764::2014/03/06-17:22:43.544 INFO [NM] Node cleanup complete
000015c8.00001764::2014/03/06-17:22:43.544 WARN [NM] Shutdown of the Cluster Network driver failed, status 21
000015c8.00001764::2014/03/06-17:22:43.544 INFO [NM] Deleting object for node 1.
000015c8.00001764::2014/03/06-17:22:43.544 INFO [NM] destroying node 1
000015c8.00001764::2014/03/06-17:22:43.544 INFO [OM] Deleting object GB02QDB101RBB27 (1)
000015c8.00001764::2014/03/06-17:22:43.544 INFO [NM] Shutdown complete.
000015c8.00001764::2014/03/06-17:22:43.544 ERR  [CS] ClusterInitialize failed -2146959355
000015c8.00001764::2014/03/06-17:22:43.544 WARN [INIT] The cluster service is shutting down.
000015c8.00001764::2014/03/06-17:22:43.544 WARN [FM] Shutdown: Failover Manager requested to shutdown groups.
000015c8.00001764::2014/03/06-17:22:43.544 INFO [Dm] DmShutdown
000015c8.00001764::2014/03/06-17:22:43.544 INFO [DM] DmpShutdownFlusher: Entry
000015c8.00001764::2014/03/06-17:22:43.544 INFO [DM] DmpShutdownFlusher: Setting event
000015c8.00000ae0::2014/03/06-17:22:43.544 INFO [DM] DmpRegistryFlusher: got 0
000015c8.00000ae0::2014/03/06-17:22:43.544 INFO [DM] DmpRegistryFlusher: exiting
000015c8.00001764::2014/03/06-17:22:43.544 WARN [MM] MMLeave is called when rgp=NULL.
000015c8.00001764::2014/03/06-17:22:43.544 ERR  [CS] Service Stopped. exit code = 2148007941

Best Wishes, San3d

I recently implemented Cluster Aware Updating on my Windows 2012 cluster.  During the setup of the role I selected the option to "Add the CAU clustered role with self-updating mode enabled, to this cluster."  The wizzard created a computer object "CAUHyper4gf" in AD for this role.  Since that time, roughly every 15 mintues Event ID 1196 with the following error is logged:

"Cluster network name resource 'CAUHyper4gf' failed registration of one or more associated DNS name(s) for the following reason:
This operation returned because the timeout period expired.
.

Ensure that the network adapters associated with dependent IP address resources are configured with at least one accessible DNS server."

The Computer Object is listed in DNS and responds to pings.  I haven't been able to find any articles addressing this specific issue.  I'm not sure how to correct this issue.  What can be done to eliminate these errors?