I'm getting ready for a real vSAN deployment so I wanted to build it out in a lab first. I set up three ESXi hosts on an R840. Configured everything with vmks, hostnames, DNS, etc. Everything worked great. All the hosts could ping each other and they could all ping out.
I go through the quick config for vSAN, it migrates the vmks to the distributed switch and bam. All host to host communication immediately dies. All networking involved has the security settings disabled and is set to promiscuous, there are no VLANs anywhere, the hosts can still ping out of the virtual environment and reach the internet, but they cannot talk to each other. I set up another vmk on a regular vSwitch and that still works. Problem is you have to use a distributed switch for vSAN evidently and I'm at a complete loss as to why this isn't working.
On a laptop outside the virtual environment I can see the hosts ARPing for each other but a tcpdump on the individual hosts shows that they never receive that arp. The only logical explanation is that the distributed switch is failing to forward it but I have no idea why it isn't.
