What about a vmware vsan 3 or 4 node stretched cluster direct connection ?

[KurtDePauw1]

Hello,

I've read that a vmware vsan 2 node stretched cluster with direct connection is supported ...

What about a vmware vsan 3 or 4 node stretched cluster with direct connection ?

If you should use 1 link in standby en 1 link as active, then you would need 6 x 10GBit ports on each node

and you could eliminate the need of 10Gbit switches

Something like the below simple picture

Or maybe a 3 node cluster ( Essentials Plus ) stretched over 3 DataCenters ? --> that would be the cheap solution !!

Is that even possible ?

NOTE : Downtime on one of the Datacenters happens frequently !

[KurtDePauw1]

Just got a whatsapp message from a colleague that this is not possible without 10GBit hardware switches.

And that you need min. of 2 nodes per datacenter for a stretched cluster...

Damn ..... no cheap solution available so it looks

[KurtDePauw1]

On the other hand ...

a 3 node clusters with VSAN ( essentials plus ) in 1 Datacenter works fantastic.

If I use 1 node per datacenter spread over 3 datacenters with 10 gbit switches .... that should work not ?

For VMware this looks like the nodes are in 1 datacenter , or am I wrong ?

[TheBobkin]

Hello Kurt,

"you need min. of 2 nodes per datacenter for a stretched cluster."

This is not true, you can run a stretched cluster in a 1+1+1 configuration (2 data/compute sites + 1 Witness site), you can even run a semi-pointless lop-sided 2+1+1 if you want (basically the 2nd node at Site A would only be used for failover of the other node at that site). 1+1+1 requires a Standard edition ESXi licence or higher, any more nodes than this requires Enterprise edition.

"For VMware this looks like the nodes are in 1 datacenter"

Not necessarily, stretched cluster still has minimum network requirements:

- Between Witness and hosts = 500 milliseconds latency RTT

- Between node to node:

"Latency or RTT (Round Trip Time) between sites hosting virtual

machine objects should not be greater than 5msec (< 2.5msec one-way)."

More useful information:

https://blogs.vmware.com/virtualblocks/2015/09/11/vmware-virtual-san-robo-edition/

http://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/products/vsan/vmware-virtual-san-6....

Bob

-o- If you found this comment useful or answer please click the 'Helpful' button and/or select as 'Answer', please ask follow-up questions if you have any -o-

[KurtDePauw1]

Thank you for your reply.

We have now a 3 node cluster with VSAN in 1 Datacenter.

every node is a witness.

What if I split this 3 node cluster and move the nodes each to another datacenter ( same building )

Then I would have in theory the same as before , and each node is again a witness.

The only difference would be that the nodes are in different rooms.

I Just would like to split the nodes from each other in case of a power outage in one of the datacenters.

So What I should do is

Datacenter 1	Datacenter 2	Datacenter 3
1 x ESX Node 1 x 10GBit Switch ( Storage Switch ) 1 x 1 GBit Switch ( Management Switch )	1 x ESX Node 1 x 10GBit Switch ( Storage Switch ) 1 x 1 GBit Switch ( Management Switch )	1 x ESX Node 1 x 10GBit Switch ( Storage Switch ) 1 x 1 GBit Switch ( Management Switch )

So all 3 Data Centers will be connected with each other over the same switches as they are in 1 datacenter.

So Latency is not a problem

Regards,

Kurt

[TheBobkin]

Hello Kurt,

Just to clarify something here:

In a normal 3-node cluster with FTT=1 Storage Policy (e.g. vSAN default policy) applied to all VMs/Objects, each Object (e.g. a vmdk) will have 2 data components (essentially a RAID1 configuration) + 1 Witness component (small metadata componet used to decide quorum), these are split across all 3 nodes, one on each.

In a 1+1+1 ROBO configuration, the Components per object are the same, however how they are distributed is a bit different:

One duplicate set of R1 data will go on each data-node, and ALL the Witness components will go on the Witness Appliance.

You mentioned that outages are common, is this just for a single site?

Bob

-o- If you found this comment useful or answer please click the 'Helpful' button and/or select as 'Answer', please ask follow-up questions if you have any -o-

[KurtDePauw1]

Hello again

as you mentioned below, this is what we have now, the default standard VSAN 3 node setup with the witness on each node

Quote
In a normal 3-node cluster with FTT=1 Storage Policy (e.g. vSAN default policy) applied to all VMs/Objects, each Object (e.g. a vmdk) will have 2 data components (essentially a RAID1 configuration) + 1 Witness component (small metadata component used to decide quorum), these are split across all 3 nodes, one on each.

So we keep this setup but we just physically move each node to another room in the same building.

In theory this would be the same as if the nodes are in 1 datacenter. correct ?

Then we do not need a separate witness server, because the witness is still spread over the 3 nodes.

Sorry for the stupid questions but .... it looks like I'm missing something or not seeing the difference.

Just searching for a cheap solution with the same hardware 🙂

We just have 1 building with lot's of rooms

The floor with the servers are heaving power outages

so moving a node to floor 1, another to floor 2 and another to floor 3 would be a solution

Regards,

Kurt

[TheBobkin]

Hello Kurt,

"Sorry for the stupid questions"

There is no such thing as a stupid question :smileygrin:

How you would switch from current 3-node to a 1+1+1 set-up depends on what you have available:

Option 1:

- Back-up/migrate all data from vSAN to somewhere/something else

- Wipe vsandatastore

- Move the boxes and set-up a 1+1+1 stretched cluster

- Restore/migrate data back on to the vSAN cluster.

Option 2:

- Make back-ups of all VMs in case something goes wrong that could compromise data integrity(e.g. physical disk failure).

- Apply a FTT=0 Storage Policy to ALL Objects on vsandatastore.

- Evacuate the disk-groups of 2/3 nodes so that all the data is on a single node.

- Move the boxes and set-up a 1+1+1 stretched cluster.

- Apply a FTT=1 Storage Policy back to ALL Objects on vsandatastore. (which will basically recreate the R1 mirror data on the other empty data node and the Witness components on the Witness Appliance).

Option 1 is preferable as Option 2 could put you ina  precarious position if anything failed while all the data was on a single node.

As discussed you wil require either Standard vSAN licensing or ROBO Standard licensing to achieve this type of set-up.

Pg 5-9 of this guide explain better how this is implemented:

https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/products/vsan/vmware-vsan-62-licen...

Here is a typical example of how this is set-up:

http://www.yellow-bricks.com/2017/01/24/two-host-stretched-vsan-cluster-with-standard-license/

Bob

-o- If you found this comment useful or answer please click the 'Helpful' button and/or select as 'Answer', please ask follow-up questions if you have any -o-

[TomHowarth]

I can see what you are getting at, but lets look at this from the perspective of data-protection, a 3 node VSAN can only utilize Mirroring as its data-protection method, 4 nodes are greater is required for erasure coding.  this mean a guaranteed loss of resiliency on loss of a site.  effectively 3 single points of failure in your design. 

this may be acceptable for a test and development environment, but never above.

I would consider separate VSAN nodes on each site and use VDP to manage off-site replicas.  rather than run a high risk of such data loss.

This should be possible however there may be a license limitation if using Essential Plus.

[TomHowarth]

Paul, from my understanding the OP is asking if they can have 3 active nodes in a 1x1x1 cluster, what you appear to be describing as a ROBO deployment of 2 active nodes and a virtual witness.

[TheBobkin]

Hello Tom,

Erasure coding doesn't really come into this at all, if by "loss of resiliency" you mean to say that running off one copy or mirror of data in the event that one site/node fails - yes...but this is also true for the same reasons in a normal 3-node cluster (which is currently in use and being negatively affected by power issues).

"this may be acceptable for a test and development environment, but never above."

Most Production vSAN clusters run FTT=1 configurations (sure, stretched here is likely adding more site-local infrastructure that could be failure points but this fact remains)

"high risk of such data loss" - with FTT=1 Storage Policies applied to all Objects, a single vSAN node/site becoming unavailable due to power-outage (this is the ask here) is not going to cause data loss. One of the sites could catch on fire and the VMs data could still be rebuilt from the remaining nodes and all VMs that were on the host that went on fire could be restarted on the remaining data-node immediately.

"3 active nodes in a 1x1x1 cluster, what you appear to be describing as a ROBO deployment of 2 active nodes and a virtual witness."

Short answer - no, can't have more than 2 stretched data-nodes unless have an Enterprise vSAN licence, thus the choice of ROBO setup(typically called a 1+1+1).

If there is other ESXi-infrastructure available to put the Witness Appliance VM on then great, if not then could run it on the 3rd host (directly, while possible would be a bit of a waste, so could even just set-up a small vmfs on it with the existing disks).

I concur that this is far from an ideal set-up, feel free to suggest alternatives using what the OP specified though.

@Kurt De Pauw:

Are you only getting power-outages on one floor of your building?

If so then why not just moving the existing vSAN cluster to a floor that is not or less-affected?

Bob