VSAN networking design

[ManivelR]

Hi All,

I have a question about VSAN networking.

Our setup:-

All the 3 ESXi servers OS is running on SD card.There is only one VSAN datastore(in a 3 node cluster)

ESXi management/vmotion--->seperate subnet---> two physical NICs(active/active with IP hash load balancing)

VSAN network---> two physical NICs(active/active with IP hash load balancing)

VM network---> ---> two physical NICs(active/active with IP hash load balancing)

We are currently conducting  VSAN testing.We disabled one by one physical NIC from Cisco Nexus switch(tagged to DVS VSAN switch) and the traffic is going out through another NIC after few seconds.

When we disable both the VSAN NICs(tagged on one ESXi server as an example) from the respective physical Cisco switch, VMs are getting restarted because of HA.

Is this expected behavior as there is no heartbeat ?

When configuring a VSAN cluster, it is recommended to disable heartbeat datastores in your cluster ? https://bmanone.com/2016/09/15/__trashed/

Should be need to configure Host Isolation Addresses Recommendations ?

VMware Virtual SAN & vSphere HA Recommendations - VMware vSphere Blog

Note:- I have only one default gateway.VSAN network uses ESXi managment network gatweway.Is this recommended one ?

Thank you,

Manivel R

[sk84]

When we disable both the VSAN NICs(tagged on one ESXi server as an example) from the respective physical Cisco switch, VMs are getting restarted because of HA.

Is this expected behavior as there is no heartbeat ?

Yes, because in a vSAN cluster the HA communication is handled via the vSAN network and not the management network.

I have only one default gateway.VSAN network uses ESXi managment network gatweway.Is this recommended one ?

That depends on your network setup. If the ESXi host can reach the default gateway via the vSAN network, everything is fine. But in many setups, there is another vmkernel port and VLANs used for vSAN and management traffic, so usually the ESXi host can't reach the default gateway via the vSAN network.

But you can check it, if you try to do a vmkping to the default gateway ip address via the vSAN enabled vmkernel interface:

vmkping -I vmkX x.x.x.x

(where vmkX is the vmkernel interface for vSAN and x.x.x.x is the default gateway ip address)

If this command is not successful, you should configure an IP address in the vSAN network and implement the isolation address recommendations. Maybe this article will also help you to understand why it's necessary: vSphere HA heartbeat datastores, the isolation address and vSAN - Yellow Bricks

Btw, we had the same problem and have configured these isolation ip addresses directly on our nexus switches for the vSAN VLANs.

[ManivelR]

Thanks Sebastian for your valuable inputs.

Yes.management network and VSAN network are different and we are using 2 vmkernel ports.

management network -->vmk0--> 172.16.254.0/255.255.255.0/172.16.254.1

VSAN network -->vmk1--> 172.16.255.0/255.255.255.0/172.16.254.1(it uses management network GW).

We cannot reach management gateway through VSAN network via vmk1.

In this case,we ask network team to configure isolation IP address on two Cisco Nexus switches.Once done,we will configure isolation IP address on VSAN cluster.

Am i right in this case?

Thank you,

Manivel R

[sk84 · Accepted solution]

Yes. That's right. As long as one of the isolation addresses is reachable from a vSAN host, HA should not be triggered.

[ManivelR]

Thanks Sebastian for you great help.

I got your point.(If any one of the isolation IP address(lets say 172.16.255.1) become unreachable,then HA should not be triggered(because it will go via 172.16.255.2).

If both the isolation IP address are not reachable via VSAN network(172.16.255.1 and 172.16.255.2),then HA will come in to action(i mean VM will be restarted on some other ESXi host).

I guess,Iam right.

Thanks,

Manivel R