Fully isolated vSAN network

[TIRJO]

Dear friends,

I have a problem. I recently reconfigured the vSAN I have (ESXi host in 6.0 and vCenter in 6.7) at the company to have an isolated network. We bought a 10GBE SFP+ switch and I created a completely separate network from the management network, the vMotion network and the VM network. It is physically an exclusive switch for the vSAN network, which has no connection from other networks (isolated network).

However, at the end of the configuration, I started receiving vSAN alerts about connectivity between hosts. However, the vSAN datastore continues to work normally. I did a ping test (vmkping), defining the exclusive vSAN network interface (vmkping -d -I vmk2 IP_HOST), to assess whether the hosts "saw" each other. Everything is OK! But when I test the ping, without setting the network interface, it fails. Clearly this occurs because the network interface used is the default one (vmk0), which is not physically connected to the vSAN network you create. So the ping fails and the non-connectivity alert does not go away.

What solutions can I try, without having to physically connect the two networks together? I cannot set a different default gateway for the vSAN network. I wouldn't even know which one to use, as this network has no GW. It is a network only for communication between hosts for the use of vSAN.

Live long and prosper,
Marcelo Magalhães

[kastlr]

Hi,

when I remember correctly vSAN 6.0 relies on a DvS which did use/allow IPv6 Multicast and Snooping, so you should check if those problems are caused by IPv6 

[TIRJO]

IPV6 disable... and I dont use DVS.... only vSwitch.

[Tibmeister]

Are vmk2 and vmk0 on the same subnet?

[TIRJO]

Good point.... yes both network interfaces are on the same subnet, vmk0 = 192.168.1.200/22 and vmk2 = 192.168.1.202/22. But both are connected to different physical switches AND these switches are not connected to each other.

Can be it? The fact that both interfaces, vmk0 and vmk2, are in the same subnet and the GW (192.168.1.1/22) is associated with vmk0, is making the vmkping go out through vmk0 and not through vmk2.

Live long and prosper,
Marcelo Magalhães

[Tibmeister]

Being on the same subnet ESXi, well any OS, will use the lowest number interface first to send the data, in this case vmk0. Put vmk2 on a separate subnet and you’re golden. 

[TIRJO]

Okay... got it. Now that you've raised that question, I'm pretty sure that's it. The doubt (and concern) now is changing the vmk2 IP in the 3 hosts the VMs will be down, right? Is there any risk of "breaking the vSAN"?

Would I be better off shutting down the VMs and putting the hosts in maintenance mode?

Live long and prosper,
Marcelo Magalhães

[Tibmeister]

You will definitely interrupt vSAN networking which can cause bad things to happen. What I would do is create new vmkernel interfaces on the vSwitch with the new IP config. Then, enable vSAN on the new vmk’s and let things sit. Then you should be able to disable vSAN on the old vmk’s and as long as there’s IP connectivity on the new vmk’s you should be golden without issues. 
Now I’ve never done this but in theory it should work. If this is production, open a ticket with support and run it by then to ensure no oddities. At the very least, they can be ready invade something goes sideways and can help get things going again.