Looking for Advice on New 4-Node VSAN Implementation - NETWORKING specifically

[WDGNet]

I have a 4-node VSAN project I'm working on.  I am looking for opinions on the best use of my networking equipment to simplify this design as much as possible.

Each node has four 10G network ports, and two 1G ports.  I will be using a single top of rack 48 port 10G switch (could alter this if needed), single data rack.

My initial idea was to run VSAN traffic and VMotion traffic on the same 10G NIC?  Then I could have one 10G network port for VSAN/VMotion teamed with another 10G network port acting as standby.  Then have my third 10G network port used for production VM network traffic teamed with my fourth 10G network port as standby.

1G NIC1 - Management

1G NIC2 - Redundant Management

10G NIC1/10G NIC2 - VSAN/VMotion (Active/Standby)

10G NIC3/10G NIC4 - Production VM Network (Active/Standby)

Looking for guidance and opinions on what others are seeing out there with a similar setup like mine.  Thanks!

[Lord_Klaud]

You should also factor in backup network and replication network (if required). Can you confirm if this is going to be a All Flash set-up or hybrid? And the version of ESXi/vSAN?

-cA

[jameseydoyle]

Hi,

I would avoid running vMotion and vSAN on the same 10G NIC unless absolutely necessary. You could still do something similar but separate the vMotion and vSAN.

10G NIC1 Active for vSAN (with 10G NIC2 as standby)

10G NIC2 Active for vMotion (with 10G NIC1 as standby)

This would be best configured on a distributed switch, as then you can configure Network IO Control to guarantee bandwidth to vSAN traffic in the case that it fails over to its standby NIC and has to share bandwidth with vMotion.

For the Production traffic, I would make both NICs active and use a load-balancing option to ensure that both NICs are being utilized.

Ideally, I would recommend redundant switches as the 10G switch will be a single-point-of-failure. Also, make sure you split the connections out between the different ASICs on the switch to ensure that you can sustain the full 10G throughput on all ports. This becomes particularly important if data needs to be moved during a resync operation on the vSAN cluster.

[WDGNet]

This is an All-Flash setup.  vCenter will be 6.5, ESX hosts will be on 6.6.1

[WDGNet]

A distributed switch would be new to me, but hey, so is all this VSAN stuff

I like the swapping of NIC1 and NIC2 for vSAN and vMotion.  Makes sense.  I do understand the use of NIOC, just never implemented it.

As for the production VM network, I can utilize the vSwitch for load balancing then?

I agree about the redundant 10G physical switches being important.  Currently I only have one at this point, but I will see about getting another and avoid that single point of failure.  Makes sense.

[depping]

looked at this yet? https://storagehub.vmware.com/#!/vmware-vsan/vmware-r-vsan-tm-network-design/