Hello Bob,

"Was the node that lost its NICs the 'preferred' Fault Domain?"

Yes.

"Using vDS or standard switches?"

Standard vSwitches. One for Management and VM. The second vSwitch is used for vMotion and vSAN (each on an active/passive setup but it's the same dual-port PCIe NIC)

vmk0 is configured for witness traffic.

"vCenter running on this cluster or somewhere else?"

Somewhere else (in the main environment which is a vBlock)

"Did the healthy node lose its proverbial marbles when the other host came back up, as opposed to when it went down?"

When the problem-node came back up. It came back with a vSwitch without any nic-ports whatsoever (so not half-dead like before the reboot).

According to the logs, we never lost access to the witness appliance (which runs in a 3rd datacenter).

Again, the question that races in my mind is: why does the "good" node even care about the other "broken" node? It has a mirror copy, it has access to the witness so 2/3rd of the components are available. Why does it care that the other guy is foaming at the mouth?

Could it be that, as they could still communicate over their management interfaces, they exchanged information leading to the "good" node getting confused or something? The physical back2back NIC ports where gone on the "broken" node so the vSAN Network between the nodes was down. But that is the same situation when you reboot a node for regular maintenance.

Could we be talking about a split-brain / partition? Both nodes had perfect connectivity to the outside world, incl. the Witness appliance. It was only the inter-node vSAN communication that was down.

So essentially, both nodes are in an identical situation. Neither can communicate vSAN traffic to the other guy, but both can talk to everything else. It's like pulling out both back2back cables isnt't it. That raises the question: how will the Wittness appliance decide which node should win? Will it select the node in the preferred fault-domain? Both nodes have the same issue of both NIC ports being down (the reason why is not that relevant). Is this a "double failure" to whom there is no protection?

It's the regular default gateway (we only use the default IP-stack).

According to your own blog, it's irrelevant in a 2-node back2back cluster  (or am I a donkey by interpreting your writings incorrectly).

"If both nodes still have access to the witness then the preferred fault domain always wins."

Exactly. And it was the preferred FD node that lost the NIC. But that should be irrelevant, as effectively, both nodes lost all back2back connections to each other, so the preferred FD node should still have won after it came back, instead the datastore died as soon as the pref.FD node came back up after the reboot and all files and folders optically disappeared (but came back after the NIC was replaced in the pref.FD and was powered back on).

Discussion seems to have died. Pity. Maybe we can chat about it at the VMUG on March 8th in Switzerland. See you there Duncan.

In the meanwhile i'm building a 2-node in our Lab to see if I can re-produce the issue. I'll simply yank out both back2back cables at the same time, simulating a total vsan-network failure while both nodes can talk to the witness appliance via their vmk0 vmkernel-interfaces.

I'll share the results when i'm done. Will take a couple of days though as i'm very busy.

I am able to reproduce this in the lab with 100% repeatability.

I've talked to Duncan about it when he was in Switzerland last week. The root issue is, simply put:

- The preferred node being away (reboot) during the split brain. Nothing bad happens at this point.

- The secondary node lives on, data is of course updated only there. The data on the downed preferred node get's older every second.

- The witness appliance now only sees the secondary which has valid data so the datastore remains intact. That is why vm's keep running as long as the preferred node does not come back.

- The preferred node is coming back after it's reboot (both are still in split brain at this point)

- The preferred node reports in with the witness appliance and says "i'm the boss", the witness goes "wft ??" and invalidates the data on the secondary, causing all VM's losing access to their disks

- As soon as the vSAN network (direct-connect) is restored, the split-brain is gone, everybody now understands the bigger picture and the (good) data on the secondary is regarded as the valid data (because it is)

- At this point, VM's are automatically started by HA and all is good again.

All of this is exactly what I see happening in my Lab.

Also remember that I, or the customer, never lost data.

Duncan told me this situation will be resolved in vSAN 6.7  (remember, vSAN version has little to do with the ESXi version).

In the meanwhile, he asked me to try the following:  when, from a vSAN network perspective, both nodes lose sight to one another and the split-brain effectively happens, before bringing the preferred node back up, change the cluster Fault Domain Config and swap the nodes in the preferred Fault-domain with the secondary.

Pref. FD  ESXi-A  ->  Second. FD

Second. FD ESXi-B -> Pref. FD

Then bring the primary back up. When it comes up, and reports in with the witness and with vCenter, it gets pressed in the role of secondary. The old secondary was promoted to preferred just before, thus it's data is regarded as "the winner" and thus the datastore is not brought down by the self-protect mechanism which shuts everything down to avoid data-corruption.

Mind you, an EMC vPLEX and many other streched-data-cluster technologies, do exactly the same thing. If the vPLEX witness cannot determine, with 100% certainty, which side should win, it halts all IO and let's the human make the decision where IO should be resumed. Being 99% sure is not enough. It's your data.

This concept of temporarily swapping the hosts in the two fault domains is what will be automated in some way in 6.7. The exact details he could not disclose, which I respect.

I will try this in my lab. As said, I can reproduce "what happened" at this customer with 100% repeatability and will write the results of the suggested workflow here.