VSAN 2 node direct connect tribulations

[MyMikro]

Hi, i know there are plenty of newb questions like mine around but one more cant hurt

Im looking for a 2 node hyperconverged solution, till now i did use Starwind  but the 6.5 direct connect made vsphere interesting so i did set up a lab to try it out:

2 x Intel / EMC Server System R2224WTTYS with Intel RMS3CC080 RAID card (JBOD mode on) , 2x samsung EVOs and 3x Intel S4600  ssds on each, server has two spf+ and i did use those for direct connect

HP gen 8 Microserver with 12gb of ram for VCenter and Witness with an ssd and a couple of hdds

I installed Hosts and vCenter without any issues, made an VDS for vmotion and vsan, enabled witness traffic on management netwok on each and deployed Witness (enabled vsan on its second network). Enabled the Cluster, used the recommended all flash settings but when i did try to add an VM i noticed  Witness Appliance is missing and cannot be added.

-all hosts have the same build

-i did not do any network changes to the Witness cept  static ip address and vsan, I dont use vlans (yet) so same subnet as the  management network.

Could it be that the lack of ram (12gb for vcenter and witness) cosing me problems with the witness deployment or did I mess the config ?

[TheBobkin]

Hello MyMikro​,

"when i did try to add an VM i noticed  Witness Appliance is missing and cannot be added."

Can you elaborate on what you mean by "missing"?

As in missing from the vSphere inventory or not communicating with the other nodes in the cluster?(# esxcli vsan cluster get)

"-all hosts have the same build"

What build and is the vCenter on equivalent or higher build? Assuming by "hosts" you are including the Witness.

Bob

[MyMikro]

After i turned on the Cluster, i did try to make a VM and got the VM Policy warrning  that i need three failsafe nodes, after esxcli vsan cluster get i noticed it does not communicate with other nodes. Trying to add it (in the stretched cluster option) resulted in an error. Witness is in the vsphere inventory  and running. Had no issues deploying the ova

"What build "

Build number of all three esix host, i assumed i might be using an older esix version for Witness hence the issue but thy are all recent.

All the Licensing works also

[IT_pilot]

Cluster\Monitor\vSAN\Health what shows? Can a screenshot

[TheBobkin]

Hello MyMikro​,

"Enabled the Cluster, used the recommended all flash settings but when i did try to add an VM i noticed  Witness Appliance is missing and cannot be added."

Specifically how did you do this? Via the option to create stretched cluster and select a Witness node?

If you are on 6.6 (6.5 U1) it should have a basic pre-check of vSAN network configuration when doing this - did this show all green?

"enabled witness traffic on management netwok on each and deployed Witness (enabled vsan on its second network)."

Just vsan traffic type enabled on the Witness?

'Do not configure the witness traffic type on the witness host.'

docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.virtualsan.doc/GUID-03204C22-C069-4A18-AD96-26E1E1155D21.html

Can you share the output of these on each of the three nodes?:

# esxcli network ip interface list

# esxcli vsan network list

# esxcli vsan cluster get

# esxcli vsan cluster unicastagent list

# esxcli vsan storage list

Bob

[MyMikro]

[MyMikro]

node1

esxcli network ip interface list

vmk0

   Name: vmk0

   MAC Address: a4:bf:01:02:67:18

   Enabled: true

   Portset: vSwitch0

   Portgroup: Management Network

   Netstack Instance: defaultTcpipStack

   VDS Name: N/A

   VDS UUID: N/A

   VDS Port: N/A

   VDS Connection: -1

   Opaque Network ID: N/A

   Opaque Network Type: N/A

   External ID: N/A

   MTU: 1500

   TSO MSS: 65535

   Port ID: 33554436

vmk1

   Name: vmk1

   MAC Address: 00:50:56:61:46:7f

   Enabled: true

   Portset: DvsPortset-0

   Portgroup: N/A

   Netstack Instance: defaultTcpipStack

   VDS Name: DSwitch

   VDS UUID: 50 37 3c 7a 13 50 b2 1d-1c 66 42 94 72 05 6c ef

   VDS Port: 14

   VDS Connection: 176776213

   Opaque Network ID: N/A

   Opaque Network Type: N/A

   External ID: N/A

   MTU: 1500

   TSO MSS: 65535

   Port ID: 50331654

vmk2

   Name: vmk2

   MAC Address: 00:50:56:6b:74:42

   Enabled: true

   Portset: DvsPortset-0

   Portgroup: N/A

   Netstack Instance: defaultTcpipStack

   VDS Name: DSwitch

   VDS UUID: 50 37 3c 7a 13 50 b2 1d-1c 66 42 94 72 05 6c ef

   VDS Port: 23

   VDS Connection: 314105563

   Opaque Network ID: N/A

   Opaque Network Type: N/A

   External ID: N/A

   MTU: 1500

   TSO MSS: 65535

   Port ID: 50331655

esxcli vsan network list

Interface

   VmkNic Name: vmk1

   IP Protocol: IP

   Interface UUID: f561c25a-bbb3-a5cb-c035-001e67f83f04

   Agent Group Multicast Address: 224.2.3.4

   Agent Group IPv6 Multicast Address: ff19::2:3:4

   Agent Group Multicast Port: 23451

   Master Group Multicast Address: 224.1.2.3

   Master Group IPv6 Multicast Address: ff19::1:2:3

   Master Group Multicast Port: 12345

   Host Unicast Channel Bound Port: 12321

   Multicast TTL: 5

   Traffic Type: vsan

Interface

   VmkNic Name: vmk0

   IP Protocol: IP

   Interface UUID: 9f1bc05a-a43c-8cb8-4ae4-001e67f83f04

   Agent Group Multicast Address: 224.2.3.4

   Agent Group IPv6 Multicast Address: ff19::2:3:4

   Agent Group Multicast Port: 23451

   Master Group Multicast Address: 224.1.2.3

   Master Group IPv6 Multicast Address: ff19::1:2:3

   Master Group Multicast Port: 12345

   Host Unicast Channel Bound Port: 12321

   Multicast TTL: 5

   Traffic Type: witness

esxcli vsan cluster get

Cluster Information

   Enabled: true

   Current Local Time: 2018-04-03T14:19:07Z

   Local Node UUID: 5ab7c056-d6a7-eb76-24f0-001e67f83f04

   Local Node Type: NORMAL

   Local Node State: MASTER

   Local Node Health State: HEALTHY

   Sub-Cluster Master UUID: 5ab7c056-d6a7-eb76-24f0-001e67f83f04

   Sub-Cluster Backup UUID: 5ab7cbd1-dfb3-c263-417a-001e67fdfeb0

   Sub-Cluster UUID: 52f1e4f6-a725-96b6-7e40-6953e234208d

   Sub-Cluster Membership Entry Revision: 3

   Sub-Cluster Member Count: 3

   Sub-Cluster Member UUIDs: 5ab7cbd1-dfb3-c263-417a-001e67fdfeb0, 5ab7c056-d6a7-eb76-24f0-001e67f83f04, 5abf9347-b586-ab8e-5a68-a0369f039414

   Sub-Cluster Membership UUID: 606ec25a-3413-ad7f-5a35-001e67f83f04

   Unicast Mode Enabled: true

   Maintenance Mode State: OFF

   Config Generation: 0f55a3a2-4ed9-4d93-ac45-5c83244dd325 2 2018-04-02T17:53:58.830

NodeUuid

IsWitness  Supports Unicast  IP Address

Port  Iface Name

------------------------------------  ---------  ----------------  ------------  -----  ----------

5ab7cbd1-dfb3-c263-417a-001e67fdfeb0	0	true  10.10.10.12   12321
00000000-0000-0000-0000-000000000000	1	true  192.168.1.33

esxcli vsan storage list

naa.55cd2e404c007306

   Device: naa.55cd2e404c007306

   Display Name: naa.55cd2e404c007306

   Is SSD: true

   VSAN UUID: 520001f5-dc2d-35ed-9f38-55b3bdc1d417

   VSAN Disk Group UUID: 529761dc-f845-8313-cc3b-6c749a9b4f76

   VSAN Disk Group Name: naa.5002538d41ad6e6e

   Used by this host: true

   In CMMDS: true

   On-disk format version: 5

   Deduplication: false

   Compression: false

   Checksum: 18007400109580189469

   Checksum OK: true

   Is Capacity Tier: true

   Encryption: false

   DiskKeyLoaded: false

naa.55cd2e404c007d0b

   Device: naa.55cd2e404c007d0b

   Display Name: naa.55cd2e404c007d0b

   Is SSD: true

   VSAN UUID: 520934e6-8e31-a615-2018-33ddf507f0f2

   VSAN Disk Group UUID: 529761dc-f845-8313-cc3b-6c749a9b4f76

   VSAN Disk Group Name: naa.5002538d41ad6e6e

   Used by this host: true

   In CMMDS: true

   On-disk format version: 5

   Deduplication: false

   Compression: false

   Checksum: 13537571910612716818

   Checksum OK: true

   Is Capacity Tier: true

   Encryption: false

   DiskKeyLoaded: false

naa.5002538d41ad6e6e

   Device: naa.5002538d41ad6e6e

   Display Name: naa.5002538d41ad6e6e

   Is SSD: true

   VSAN UUID: 529761dc-f845-8313-cc3b-6c749a9b4f76

   VSAN Disk Group UUID: 529761dc-f845-8313-cc3b-6c749a9b4f76

   VSAN Disk Group Name: naa.5002538d41ad6e6e

   Used by this host: true

   In CMMDS: true

   On-disk format version: 5

   Deduplication: false

   Compression: false

   Checksum: 7037256140502740033

   Checksum OK: true

   Is Capacity Tier: false

   Encryption: false

   DiskKeyLoaded: false

naa.500a075113478863

   Device: naa.500a075113478863

   Display Name: naa.500a075113478863

   Is SSD: true

   VSAN UUID: 52c97e24-5ae4-91a3-0fa0-ce213cca1f12

   VSAN Disk Group UUID: 529761dc-f845-8313-cc3b-6c749a9b4f76

   VSAN Disk Group Name: naa.5002538d41ad6e6e

   Used by this host: true

   In CMMDS: true

   On-disk format version: 5

   Deduplication: false

   Compression: false

   Checksum: 1221719537490088665

   Checksum OK: true

   Is Capacity Tier: true

   Encryption: false

   DiskKeyLoaded: false

Witness App.

esxcli network ip interface list

vmk0

   Name: vmk0

   MAC Address: 00:50:56:b7:6c:13

   Enabled: true

   Portset: vSwitch0

   Portgroup: Management Network

   Netstack Instance: defaultTcpipStack

   VDS Name: N/A

   VDS UUID: N/A

   VDS Port: N/A

   VDS Connection: -1

   Opaque Network ID: N/A

   Opaque Network Type: N/A

   External ID: N/A

   MTU: 1500

   TSO MSS: 65535

   Port ID: 33554436

vmk1

   Name: vmk1

   MAC Address: 00:50:56:b7:5c:e2

   Enabled: true

   Portset: witnessSwitch

   Portgroup: witnessPg

   Netstack Instance: defaultTcpipStack

   VDS Name: N/A

   VDS UUID: N/A

   VDS Port: N/A

   VDS Connection: -1

   Opaque Network ID: N/A

   Opaque Network Type: N/A

   External ID: N/A

   MTU: 1500

   TSO MSS: 65535

   Port ID: 50331652

esxcli vsan network list

Interface

   VmkNic Name: vmk1

   IP Protocol: IP

   Interface UUID: 9394c25a-9401-7fe7-731f-005056b76c13

   Agent Group Multicast Address: 224.2.3.4

   Agent Group IPv6 Multicast Address: ff19::2:3:4

   Agent Group Multicast Port: 23451

   Master Group Multicast Address: 224.1.2.3

   Master Group IPv6 Multicast Address: ff19::1:2:3

   Master Group Multicast Port: 12345

   Host Unicast Channel Bound Port: 12321

   Multicast TTL: 5

   Traffic Type: vsan

esxcli network ip interface list

vmk0

   Name: vmk0

   MAC Address: 00:50:56:b7:6c:13

   Enabled: true

   Portset: vSwitch0

   Portgroup: Management Network

   Netstack Instance: defaultTcpipStack

   VDS Name: N/A

   VDS UUID: N/A

   VDS Port: N/A

   VDS Connection: -1

   Opaque Network ID: N/A

   Opaque Network Type: N/A

   External ID: N/A

   MTU: 1500

   TSO MSS: 65535

   Port ID: 33554436

vmk1

   Name: vmk1

   MAC Address: 00:50:56:b7:5c:e2

   Enabled: true

   Portset: witnessSwitch

   Portgroup: witnessPg

   Netstack Instance: defaultTcpipStack

   VDS Name: N/A

   VDS UUID: N/A

   VDS Port: N/A

   VDS Connection: -1

   Opaque Network ID: N/A

   Opaque Network Type: N/A

   External ID: N/A

   MTU: 1500

   TSO MSS: 65535

   Port ID: 50331652

[IT_pilot]

vSAN 6.6 allows changing witness host.

Remove and connect it. But before the final stage where the disks are selected make a screenshot.

[MyMikro]

[IT_pilot · Accepted solution]

Sorry, point 3. Claim disks for witness host

[MyMikro]

i have an ssd and couple of hdds ready

[MyMikro]

Eh, I've realized my mistake, silly really, this isnt a normal esix deployment and you cant build your own appliance, it comes out of the box nested with virtual drives

Thank you for your help lads, cluster is up and running now

[IT_pilot]

it's really easy to make a mistake and skip the installation of checkmarks by clicking "next"

[adgate]

Thanks for sharing. I was looking for SDS alternatives to VMware VSAN, and since you had mentioned StarWind I checked it. And I found that it is free and ships as Virtual Storage Appliance based on Linux, which is free too. I have just started the configuration. From what I see, StarWind's VSAN performs great on top on flash storage. What was your reason switching from it to VMware VSAN?

[Dav1]

Testing, learning purposes due to high demand. Starwind vsan doesnt need an external witness, it mirrors disks between nodes, direct connect and ram cache was a given years ago, you can run it on Win2016 Standard and avoid the need for Datacenter license. No HCL or HBA needed (can make use of  that expensive server raid ) , S2D doesnt even allow raid passthrough

but it has its own issues, Free vesion comes without UI (CLI, powershell at least is on the fritz atm since its work in progress) , and only 2 nodes max  but you can get a NFR license without any fuss..  Support / community is nowhere great as here and as with all vsan options what to pick is highly situational.

Its great to see vmware adding a ROBO option for a smaller deployment i just hope thy will remove the need for an external witness and move it to each node. I believe that is already possible but not yet supported

[IT_pilot]

I used the NFR license. I thought the reason in the absence of license when I set up the cluster. But even with the license, the cluster did not work. And without fault tolerance and a common mount point, this solution is not interesting. It was a year ago. Then I switched to vSAN. And with each new release it makes me more and more happy. vSAN is a developing and scalable solution. The documentation is constantly updated, there is community support. New certified servers are added. It seems to me very promising.