2-node vsan stretched cluster with physical ESXi Witness host issues

[Balteck71]

Hello.

I'm testing on my lab a 2-node Vsan cluster with 3 host:

First host is connected with a cross cable to second host for vsan trafic

Third host has vcenter server VM and acts as witness host.

Every hosts have SSD for cache tier and SAS disks for data tier, only third host have some other SAS disk as Datastore for VCSA

I wish to use the physical third ESXi host as witness, because the plan is to buy a essential plus license, that permits at VCSA to manage only 3 hosts.

I configurated vsan and verified any connection (ping, tagged the witness trafic with esxcli command, etc), create a cluster with first and second host and put the third outside the cluster: vcenter gives me no errors.

But when I try to create a new VM, I got this error when I choose the vsan datastore:

Datastore does not match current VM policy.

This storage policy requires at least 3 sites but only 2 were found.

If I deploy a Virtual Witness Appliance and I change the witness physical host with the virtual one, VM policy doesn't show me any error and I can create a new VM.

But in this way I have 4 hosts (3 grey and one blue) managed on VCSA and this breaks Essential plus license for vServer (I'm on trial license now, so I don't know what could happen when I will apply essential plus license)

And in the documentation I read that I can use Virtual or physical witness host. So what is wrong?

[TheBobkin]

Hello Balteck,

"This storage policy requires at least 3 sites but only 2 were found."

This implies that a node (likely Witness) has not been configured correctly - how did you add the Witness to the vSAN cluster and did it show as present and configured as Witness when running esxcli vsan cluster get?

If you want to use 3rd host as a physical Witness Appliance then this is the only thing that can be running on this, thus using a host with SSD/HDD disk-group(s) and same specs as data-nodes is a bit wasteful.

If all hosts have the same configuration, any reason you are not just configuring this as a standard 3-node cluster? (assuming 2-node/ROBO license may be the reason).

When deploying this as a Witness VM, it should be running on the 3rd host which is not configured for vSAN nor residing on vsandatastore as running it on these is unsupported.

If you go with the first option and only have Witness Appliance running on 3rd node, then you can run vCenter on vsandatstore which will allow more flexibility but also may result in less visibility if vSAN is impacted and vC becomes unavailable. Also of course, the full disk-group on this node wouldn't be required.

Witness Appliances do not consume a license so potentially they don't count towards license limits in vC, I will check this.

Bob

[Balteck71 · Accepted solution]

Hello Bob, I did a esxcli vsan cluster get and I discovered that the third host was in MM for VSAN, but not for vcenter (infact there was 2 vms that was running on it)

So, I stopped any VMs, I put and exit the host in system MM and I resolved the issue.

The third host is a old server with only one SSD and 3 SAS disks (one for vSAN and 2 in RAID1 for local datastore) that not have the power for a real vSan cluster.

Anyway I have a budget only for 2 vSAN licenses and essential kit license and not for 3 big hosts with a good 10GB switch and another vSAN license.

If you can discover if witness appliance consumes a VC essential license (surely it doesn't consume a host essential kit license), it will very appreciated

Thank you very much

[TheBobkin]

Hello Balteck,

Okay great, glad looking at that resolved your first issue, that command is much improved in 6.5 onwards due to previously the only way of telling this was using cmmds-tool (#cmmds-tool find -t NODE_DECOM_STATE -f json), though this method still has use as can identify 'Entering vSAN MM' states.

If a node is in MM for vSAN but not vC then enter and leave MM from vC or alternatively exit vSAN MM from CLI will fix .

Hate to be the bearer of bad news, but I found the reference I recalled where it was specified and it does count as a host here:

"vSphere Essentials Kit or vSphere Essentials Plus Kit licensing limits the number of hosts

managed by vCenter Server Essentials to three. The vSAN witness host—virtual appliance or

physical—is considered a host in these Essentials licensing bundles."

vmware.com/content/dam/digitalmarketing/vmware/en/pdf/products/vsan/vmware-vsan-66-licensing-guide.pdf

How many VMs are you planning on running on vSAN? If less than 25 then ROBO license might be cheaper here.

Either way you, should be able to make this work by having Physical Witness on host 3 and run any VMs and vC that you had considered running on host 3 on the vSAN nodes instead.

It should also be possible to use Witness Appliance VM and whatever other VMs running on local storage on host 3 and have this as a standalone host not managed by vC using free license, though I wouldn't run vC on this as if this host went down you would lose vC and Witness.

Bob

[Balteck71]

Thank you very much for the good tips.

Putting VC on vSan cluster gives me the possibility to expand vSan storage putting the SAS disks, that I use for local datastore on the third host, to first and second host.

vSphere 6 ROBO license is cheaper and it is a good idea.

Can I mix 25 vSphere 6 ROBO with 2 license vSAN standard (one for first host and one for second host, both they have one socket populated), instead of using 25 vSan ROBO License standard that it is very expensive?

vSphere 6 ROBO VC still have 3 hosts limit?

[TheBobkin]

Hello Balteck,

I had a look at it again and curiously I can find no basic list price for how much ROBO vSAN costs, other than estimates online of $500 per VM sold in packs of 25 so ~$12500, thus 2-node direct-connect single-CPU nodes with standard vSAN license and Essentials Plus is likely going to be the cheapest set-up you will get here.

(note that the 'ROBO' vSphere product that is listed at store.vmware.com includes neither vSAN nor vCenter).

Bob