VMware Cloud Community
FLSTFB
Contributor
Contributor

VSAN and TOR switch links

Hello community; I have two questions at the end based on the planned config below. Implementing 2 new SN2010M TOR switches in the near future. VSAN is currently connected to a single 10GB switch; bad design I know, hence the upgrade.


• 3 host VSAN 6.7U3 cluster (adding 4th node in the near future)
• Dual port 25GB SFP28 NICs for each host (VSAN and vMotion)
• Management traffic will be single ethernet port to a third switch (Cisco)
• VM traffic will be LAG across 3 ethernet ports to a third switch (Cisco)
• Incorporating two new HPE SN2010M TOR switches (Planning to isolate with VSAN and vMotion only)
• No LAG/LACP configured on any ports for VSAN or vMotion links connected to the TOR switches (for simplicity)
• Create one vmnic0 for vMotion (25gb SFP28 port)
• Create one vmnic1 for VSAN (25gb SFP28 port)
• At the VDS port group for vMotion
o Set vmnic0 to active uplink
o Set vmnic1 to standby uplink
o Load balancing: use explicit failover order
o Network failure detection: link status only
o Notify switches: Yes
o Failback: Yes
• At the VDS port group for VSAN
o Set vmnic1 to active uplink
o Set vmnic0 to standby uplink
o Load balancing: use explicit failover order
o Network failure detection: link status only
o Notify switches: Yes
o Failback: Yes

In review, two 25gb SFP28 links from each host would connect (1 link to each physical switch). This way VSAN and vMotion get their own dedicated 25gb links. In the event of a link failure, both services would share their respective 25gb standby uplinks, until the link down issue is resolved.


The first question I need answered is; for example, hosts A-B-C active adapters are up and running and connected to TOR switch A. If a link goes down on host A and it switches to its standby adapter, host A is now sending traffic to TOR switch B but, hosts B-C didn't fail and are still using their active adapters, which are connected to TOR switch A. Doesn't that mean there has to be a link between the two TOR switches in order for host A to re-establish a connection with hosts B-C? This seems like simple networking to me but, I've been told there doesn't need to be a link between the TOR switches for VSAN to still work in this failure scenario. I also don't have a complete technical understanding of what happens when adapters switch during a failure; (maybe that contains the pieces I'm missing).

Secondly, if there does need to be a link, could the IPL link feature of the TOR switches be used for this (which allows all vlans to pass by default), or maybe a simple LAG trunk to allow all vlans?

I appreciate your help.

Reply
0 Kudos
4 Replies
depping
Leadership
Leadership

you are saying that the TOR switches are not able to reach each other in your current plan of record?

Reply
0 Kudos
depping
Leadership
Leadership

I think this article describes what I would expect an environment to look like: 

https://www.virtuallanger.com/2017/06/27/notes-from-the-field-vsan-virtual-network-design/

Reply
0 Kudos
FLSTFB
Contributor
Contributor

I think I have learned that when you use mlag, lacp is configured on the switch ports. Therefore I don’t think i can use the simple active/standby with explicit failover.  I think you must use lacp on the vds, if using mlag.

 

Reply
0 Kudos
dhawkinson
Contributor
Contributor

We're using the same SN2010M switches in our VSAN design. We have the 2 SN2010M connected together using 2 100G 1M DAC cables and these 2 100G ports are setup in a LACP trunk. This allows cross traffic between both switches in case of a single host port failure as you've described.  

Reply
0 Kudos