I have run into a networking problem on my cluster, and I cannot figure out what changed.
I had a cluster configured and working. Its access network is 192.168.10.0/24. I have other networks, including a node management network on 10.29.130.0/24. The access network is private to my lab; it's here I have my AD defined. The management network is a 'lab' network that has a gateway available so we can access the lab from our corporate network with no issues. In other words, the access network is only routed within my private lab, but the management network can be routed to corporate access.
But, as you know, Windows simply doesn't like creating two different networks with default gateways. So before building the cluster, I removed the gateway from the management network, ensuring there was only a single gateway configured on each host. Ran the validation and it came through fine (typical network warnings about non-routed networks not able to reach other networks, but that is expected and presents no problems). Built the cluster.
I wanted to try to create an environment that would allow me to access the physical hosts through the management network, so I tried issuing a route add command specific to the management network.
route add 10.29.130.0 mask 255.255.255.0 10.29.130.1 if 3
It didn't work as expected (I am by no means a networking expert, but I figured I would try it.) Since it didn't work, I deleted it.
route delete 10.29.130.0 mask 255.255.255.0 10.29.130.1 if 3
Came back and ran another validation wizard on the cluster and now the validation fails with the following error (to each of the other nodes in the cluster):
Network interfaces FT4-Infra01.VSPEX.COM - Mgmt and FT4-Infra03.VSPEX.COM - Mgmt are on the same cluster network, yet address 10.29.130.37 is not reachable from 10.29.130.35 using UDP on port 3343.
I check my firewall, and those ports are open on all node for all firewall profiles. From FT4-Infra01 (the machine I was messing with), I can ping the other nodes of the cluster. From the other nodes in the cluster, I cannot ping FT4-Infra01. Yes, I know ping is a different rule, but I always go back to basics. And the fact that it is not responding to pings after playing with the route is strange.
Does anybody have any ideas about what playing with the route command could have changed? I am assuming that is the cause because everything worked fine before issuing the command and now these errors are there. No other changes were made to the cluster or the nodes.
.:|:.:|:. tim