vSAN direct connect 2 nodes

[ChrisFD2]

Hi all,

Current network setup at home for learning purposes:

VLAN 1 - 192.168.1.0/24 - Home network

VLAN 2 - 10.0.0.0/24 - Homelab

90% of home things are on VLAN 1 as you would expect.

I have two Supermicro SYS-E200-8D servers on VLAN 2, connected via 2 1Gbit NICs, tagged at the switch, untagged on the management interface. Routing between the subnets works and is done on a stick using my Ubiquiti USG. I have vSAN enabled with a third witness node, also on VLAN 2. The witness appliance is running on an Intel NUC and this host does nothing else.

Currently vSAN works fine, although is limited to the two 1 Gbit interfaces on each server.

The servers have a two more NICs, both 10 Gbit. I have seen online that I can use direct connect using these two interfaces, so I have them connected via two cross over cables.

I tagged the new VMkernel with vSAN traffic, but this resulted in the witness not being able to see the other nodes, I presume due to Multi-homing.

Is there a blog or step by step instruction on how to achieve this?

Should I put the witness on VLAN 1 and tag the interfaces? I've tried a few things, but nothing seems to have cleared the vSAN alarms.

All hosts running 6.7 build 9484548.

vCSA is 6.7 build 9451876.

[TheBobkin · Accepted solution]

Hello ChrisFD2​,

Just to clarify - are you trying to set-up WTS (Witness traffic Seperation)?

This needs to be configured via the CLI with the data-nodes set for type witness traffic on the alternate interface and the Witness set as vsan traffic:

Configure Network Interface for Witness Traffic

More info:

Understanding the vSAN Witness Host - Traffic Tagging - Virtual Blocks

https://storagehub.vmware.com/t/vmware-vsan/vsan-stretched-cluster-2-node-guide/

Bob

[ChrisFD2]

Thanks, I did try to seperate the traffic but I was getting errors.

The VMkernel interfaces on the direct connect 'network', should they have IPs in the VLAN 2 range or a private range?

[ChrisFD2]

Okay, I have configured the VMkernel adaptors as follows:

[root@esxi2:~] esxcli vsan network list

Interface

   VmkNic Name: vmk1

   IP Protocol: IP

   Interface UUID: e13d845b-18ce-a853-1acc-ac1f6b6a3e28

   Agent Group Multicast Address: 224.2.3.4

   Agent Group IPv6 Multicast Address: ff19::2:3:4

   Agent Group Multicast Port: 23451

   Master Group Multicast Address: 224.1.2.3

   Master Group IPv6 Multicast Address: ff19::1:2:3

   Master Group Multicast Port: 12345

   Host Unicast Channel Bound Port: 12321

   Multicast TTL: 5

   Traffic Type: vsan

Interface

   VmkNic Name: vmk0

   IP Protocol: IP

   Interface UUID: 97e57f5b-5d67-021d-4edc-ac1f6b6a3e28

   Agent Group Multicast Address: 224.2.3.4

   Agent Group IPv6 Multicast Address: ff19::2:3:4

   Agent Group Multicast Port: 23451

   Master Group Multicast Address: 224.1.2.3

   Master Group IPv6 Multicast Address: ff19::1:2:3

   Master Group Multicast Port: 12345

   Host Unicast Channel Bound Port: 12321

   Multicast TTL: 5

   Traffic Type: witness

This is on both vSAN hosts, where vmk1 is the Management/vMotion interface with Witness tagged and vmk2 is the two directly connected 10 Gbit interfaces.

On the witness, which is not in the vSAN cluster, it shows the following:

Host with vSAN service enabled is not in the vCenter cluster

Host cannot communicate with one or more other nodes in the vSAN enabled cluster

First one is as expected, but why am I seeing the error message for not able to communicate with nodes in the cluster? What logs can I check?

Thanks again.

[BerndtSchumann]

I did not have any positive experience with vSAN, as it was performing terribly. If I were to choose a virtualized storage solution, these two would be the ones to consider.