What happens when a rack fails in a vSAN stretched cluster

[sevenlogic]

Hi all, I have spent quite a lot of time thinking about how vSAN handles certain failure scenarios. I have one scenario that I have had difficulty figuring out. In the example below, I have a stretched cluster vSAN setup, each AZ located in a dedicated physical site.

But in this setup, I am also doing a multi-rack architecture. Where the workload domains are spanned across physically separated racks. The design goal is to have a setup that could withstand a single rack failure (entire rack), or a complete site failure (2 entire racks within a site).

But what happens exact when we loose a single rack? If lets say, I am using a RAID-6 SPBM Policy (Dual Site Mirroring). If I was to loose an entire rack, I suspect that VMs with LOCAL affinity (no site mirroring) would HA restart on the surviving rack in that site. But VMs with SITE MIRRORING would restart on the other site.

The confusion comes from the fact that vSAN Stretched clusters define the datacenters as the fault domain, and are not "rack aware". But I am trying to figure out what can happen in this scenario. Its my understanding that you can use a multi-rack design, unless this is not fully supported.

Is this correct? / any thoughts?

[depping · Accepted solution]

Actually, if you do not have ANY VM-Host rules defined for those VMs then the VMs will be restarted where ever HA feels it should be restarted. Location of storage components is not taken into account by HA in any shape or form, UNLESS the other hosts cannot access the storage components. However, with vSAN being a distributed solution where every host in the cluster will automatically be able to read/write to and from components on every host in the cluster, HA will also be able to restart the VMs anywhere!

[sevenlogic]

Thank you for the information, Duncan. Another quick question, when it comes to SPBM policies for a stretched cluster setup. In the past we used RAID-6, Dual Site Mirroring with Dedup & Compression for the vSAN Default Storage Policy. We would like to change this to RAID-5, Dual Site Mirroring. The main risks I see with this is that you can only loose 1 node, so if you have a host that is down or in Maintenance Mode, you basically cannot loose another node as this would cause a component rebuild.

So when it comes to a stretched cluster deployment running a dual site mirroring policy, lets say I am running RAID-5. My understanding is that you can loose 1 host in that group in both sites, as well as an entire datacenter site as the data is replicated on the other site. Is this correct?

[depping · Accepted solution]

yes, that is correct, but do note, if you lose 1 host in a site, and there's hosts left for a rebuild than vSAN will try to rebuild the data locally if it can.

But indeed, per "datacenter" you can tolerate a failure, and as you are replicating across sites you can also tolerate a full site failure.

[sevenlogic]

Thanks Duncan! Much appreciated.