Find the right WTS net config for LOCAL 2 node cluster / confusing 2 node guide

[alexanderleutz]

Hi everybody,

The customer has 2 vSAN data nodes and 1 witness all LOCAL. All in one physical datacenter, but in 3 different rooms. Connected with the 10 Gb LAN Switches.

We are a little bit confused because the guides are not as clear as say should be, when it is recommanded and when not to use layer 3.

My customer has „Option 2“ on this side https://storagehub.vmware.com/t/vmware-vsan/vsan-2-node-guide/vsan-witness-host-networking-examples/

Witness management and vsan data nodes management localy and in the same subnet.

What was your preferred solution of these two solutions under „Option 2“?

In my opinion the pictures and the text are wrong: they are switched.

I prefere picture 2 / text 1

The customer needs a valid and supported configuration and the offical „okay“ from VMware.

What do you think? In which way do you read the guide?

Best regards, and thank your for your answers

[sk84]

I don't quite understand your question. The document states that both setups in option 2 are a supported configuration and that the traffic is handled via L2 and therefore no routing (L3) is needed.

In this configuration, static routing is not required because Layer 2 networking is use.

This is a supported configuration.

[alexanderleutz]

Hi Sebastian you are right.

The original question was, do we need layer 3 in this local (including witness) 2 node cluster?

I think layer 2 would be fine and supported by VMware

But in my opinion the text or the pictures for Option 2 A and Option 2 B are switched.

I would say text 2 A is for picture 2 B and vice versa.

Or do I have a misunderstanding here?

Best regards,

Alexander

[Jasemccarty]

alexanderleutz​ - I maintain the 2 Node Guide.

I'll try to address your question and then look at the doc to see if any updates are required.

Static routes are required when:

A vSAN interface needs to connect somewhere that a vSAN VMkernel interface would need to use vmk0's default gateway.

This is because vSAN uses the same TCP stack as the Management VMkernel interface (vmk0) and vSAN does not honor specifying a default gateway on vSAN VMkernel interfaces today.

Example 1 Static Routing Required)

vSAN 2 Node using Direct Connect, with Data Node vmk1 setup for "witness" traffic and the vSAN Witness Host interface ('vsan' traffic) is not on the same segment.

NODE1 -

• Management vmk0 - 192.168.10.11 (netmask 255.255.255.0) - Tagged management
• Witness vmk1 - 192.168.20.11 (netmask 255.255.255.0) - Tagged witness
• vSAN vmk2 - 192.168.30.11 (netmask 255.255.255.0) - Tagged vsan
• vMotion vmk3 - 192.168.30.11 (netmask 255.255.255.0) - Tagged vmotion

NODE2 -

• Management vmk0 - 192.168.10.12 (netmask 255.255.255.0) - Tagged management
• Witness vmk1 - 192.168.20.12 (netmask 255.255.255.0) - Tagged witness
• vSAN vmk2 - 192.168.30.12 (netmask 255.255.255.0) - Tagged vsan
• vMotion vmk3 - 192.168.30.12 (netmask 255.255.255.0) - Tagged vmotion

vSAN Witness Host -

• Management vmk0 - 192.168.110.13 (netmask 255.255.255.0) - Tagged management
• WitnessPg vmk1 - 192.168.120.13 (netmask 255.255.255.0) - Tagged vsan

*Static Routes are required from

• 192.168.20.11 & 12 to 192.168.120.13
• 192.168.120.13 to 192.168.20.11 & 12

Example 2 Static Routing Not Required)

vSAN 2 Node using Direct Connect, with Data Node vmk1 setup for "witness" traffic and the vSAN Host interface ('vsan' traffic) is on the the same segment

NODE1 -

• Management vmk0 - 192.168.10.11 (netmask 255.255.255.0) - Tagged management
• Witness vmk1 - 192.168.20.11 (netmask 255.255.255.0) - Tagged witness
• vSAN vmk2 - 192.168.30.11 (netmask 255.255.255.0) - Tagged vsan
• vMotion vmk3 - 192.168.30.11 (netmask 255.255.255.0) - Tagged vmotion

NODE2 -

• Management vmk0 - 192.168.10.12 (netmask 255.255.255.0) - Tagged management
• Witness vmk1 - 192.168.20.12 (netmask 255.255.255.0) - Tagged witness
• vSAN vmk2 - 192.168.30.12 (netmask 255.255.255.0) - Tagged vsan
• vMotion vmk3 - 192.168.30.12 (netmask 255.255.255.0) - Tagged vmotion

vSAN Witness Host -

• Management vmk0 - 192.168.110.13 (netmask 255.255.255.0) - Tagged management
• WitnessPg vmk1 - 192.168.20.13 (netmask 255.255.255.0) - Tagged vsan

Static Routes are not required from 192.168.20.11/12 to 192.168.20.12 (and vice versa) because they are on a Layer 2 segment.

Example 3 Static Routing Not Required)

vSAN 2 Node using Direct Connect, with Data Node vmk0 (yes this is management) setup for 'witness' traffic and Witness host has 'vsan' tagged on Management

Expectations: Data Nodes vmk0 have route to Witness Host vmk0 & vice versa, regardless of where the Host resides.

NODE1 -

• Management vmk0 - 192.168.10.11 (netmask 255.255.255.0) - Tagged management & witness
• vSAN vmk2 - 192.168.30.11 (netmask 255.255.255.0) - Tagged vsan
• vMotion vmk3 - 192.168.30.11 (netmask 255.255.255.0) - Tagged vmotion

NODE2 -

• Management vmk0 - 192.168.10.12 (netmask 255.255.255.0) - Tagged management & witness
• vSAN vmk2 - 192.168.30.12 (netmask 255.255.255.0) - Tagged vsan
• vMotion vmk3 - 192.168.30.12 (netmask 255.255.255.0) - Tagged vmotion

vSAN Witness Host -

• Management vmk0 - 192.168.110.13 (netmask 255.255.255.0) - Tagged management & vsan
  • This could be Layer 3 in this example if there is a route between the vmk0 of NODES/Witness
  • The Witness's vmk0 could be even on the same segment as NODE1/2 (Layer 2)
• WitnessPg vmk1 - 192.168.120.13 (netmask 255.255.255.0) - Not used because nothing is tagged.

*Note: The vSAN Witness Host cannot have vmk0 and vmk1 on the same segment if vmk1 is used for 'vsan' traffic. This results in a multi-homing issue and causes an error. KB 2010877 details this, and it is not a vSAN specific issue.

The TL/DR, is that 2 Node vSAN using Direct Connect (Witness Traffic Separation) can use a L3 connection to a vSAN Witness Host if that vSAN Witness Host resides on another network somewhere else. That somewhere else could be in a different segment in a local datacenter, remote datacenter, or in another hosting facility. If the vSAN Witness Host is on the same segment, like in a remote site, then Layer 2 is supported.

[Jasemccarty · Accepted solution]

Also, the images somehow were reversed in the publishing platform.

They have been corrected.
vSAN Witness Host Networking Examples