Scryden
Contributor
Contributor

vSAN cluster on RAID1 disks

Jump to solution

Hi all,

We have two individual ESXi hosts with about 20TB internal storage running on version 5.5. We want to upgrade these servers to version 6.5. The servers are in the HCL compatability list and support version 6.5. After the upgrade we want to deploy vSAN/HA/DRS on these two servers as a ROBO cluster.

I have read on many websites that vSAN only supports physical RAID0 sets and pass-through disks. Our hosts are set up with RAID1 disks (12*3.5TB so effectively 6*3.5TB). Before I came across the best practice designs, it was my understanding that I could just add the RAID1 drives to vSAN and then apply a RAID1 VM Storage policy on my virtual machines. Why is RAID1 not supported for vSAN?

My biggest concern with RAID0 is that whenever a drive fails, the entire RAID0 set fails. We will then have to replace the failed disk, recreate the RAID0 set and re-add it to the vSAN cluster. I'd rather have RAID1 disks which I can hot swappable replace whenever a disk fails.

If there is no other option then we are kind of forced to wipe the disks and reconfigure them as RAID0 (RAID controller doesn't support pass-through) but my question is to "Why?".

Also with pass-through disks, doesn't this give me the same issue as with RAID0? Whenever a disk fails in pass-through mode, what happens to my vSAN node? Can I simply replace that disk?

Thank you all in advance.

1 Solution

Accepted Solutions
TheBobkin
VMware Employee
VMware Employee

Hello Scryden,

"The servers are in the HCL compatability list and support version 6.5"

Okay, so the servers may be able to run 6.5 but the important part is that the trifecta are supported for vSAN: I/O Controller, Cache SSDs, Capacity SSDs/HDDs.

In comparison to other products (VMware or otherwise), vSAN is QUITE specific about what will work or not, if it is not on the vSAN HCL it is likely for a good enough reason and I can testify that I have seen the potentially gruesome outcome of using unsupported hardware (YMMV, I am sure some things work for some, but maybe just up until they don't...)

http://www.vmware.com/resources/compatibility/search.php?deviceCategory=vsan

(annoyingly this link goes to the ready-node page, you have to click 'Build Your Own based on Certified Components' then select one of the trifecta and specify as needed)

"Why is RAID1 not supported for vSAN?"

Because in a typical scenario you are passing the management of this configuration onto vSAN e.g. vSAN Default Storage Policy has FTT=1 which is essentially R1 but handled at the Object-storage level as opposed to the controller/disk level.

"My biggest concern with RAID0 is that whenever a drive fails, the entire RAID0 set fails. We will then have to replace the failed disk, recreate the RAID0 set and re-add it to the vSAN cluster. I'd rather have RAID1 disks which I can hot swappable replace whenever a disk fails."

This is where Storage Policies come in - you can specify on a per-Object basis (vmdk/namespace/.vswp/snapshot) whether you want it as FTT=0 (single copy of data) or FTT=1 (Essentially R1 + a tie-breaker component) and then multiple other configurations such as stripe-width, space-reservations, cache-reservations, IO limits etc.

If you run FTT=1 (as most seem to do) If a single copy of this R1 becomes unavailable the VM keeps running and the missing mirror is rebuilt if its absence is not transient (e.g. a host going down for a short time).

If a disk fails in a vSAN host (with R0 not pass-through), you can hot-replace it if your controller supports hot-swap, if not, it is not a huge deal as if you are running ROBO with all VMs FTT=1 you can run all VMs off one node, shut down the affected node replace the failed disk and bring it back up.

This reference is going back a long time but the architecture-design remains the same and might help elaborate on this more:

Storage Controller RAID0 mode

For storage controller RAID adapters that do not support pass-through mode, Virtual SAN fully supports RAID0 mode. RAID0 mode is utilized by creating a single drive RAID0 set via the storage controller software, and then presenting this to Virtual SAN. You will then have to mark which of these single drive RAID0 sets are SSDs within vSphere. When RAID0 mode is utilized, the the following occurs

Virtual SAN will not manage hot-plug capabilities of drives

Hot-plug will be managed by the storage controller software

https://blogs.vmware.com/vsphere/2014/01/virtual-san-hardware-guidance-part-ii-storage-controllers.h...

Another consideration since you are thinking of going ROBO:

Do you have another site/cluster somewhere relatively near that you can run the Witness and/or vCenter from?

If you feel that I did not answer anything completely feel free to PM me or ask more questions here.

Bob

-o- If you found this comment useful or answer please select as 'Answer' and/or click the 'Helpful' button, please ask follow-up questions if you have any -o-

View solution in original post

0 Kudos
3 Replies
ChrisKuhns
Enthusiast
Enthusiast

Because the VSAN is handling the RAID for you. That is why pass through cards are preferred. Then you're not having to deal with the RAID 0 configuration. I have been there on my original vSAN. It is a pain. If you're looking to do a ROBO you're probably not dealing with an enormous system. I'd spring for new cards that will do pass-through for you. It is well worth the investment.

TheBobkin
VMware Employee
VMware Employee

Hello Scryden,

"The servers are in the HCL compatability list and support version 6.5"

Okay, so the servers may be able to run 6.5 but the important part is that the trifecta are supported for vSAN: I/O Controller, Cache SSDs, Capacity SSDs/HDDs.

In comparison to other products (VMware or otherwise), vSAN is QUITE specific about what will work or not, if it is not on the vSAN HCL it is likely for a good enough reason and I can testify that I have seen the potentially gruesome outcome of using unsupported hardware (YMMV, I am sure some things work for some, but maybe just up until they don't...)

http://www.vmware.com/resources/compatibility/search.php?deviceCategory=vsan

(annoyingly this link goes to the ready-node page, you have to click 'Build Your Own based on Certified Components' then select one of the trifecta and specify as needed)

"Why is RAID1 not supported for vSAN?"

Because in a typical scenario you are passing the management of this configuration onto vSAN e.g. vSAN Default Storage Policy has FTT=1 which is essentially R1 but handled at the Object-storage level as opposed to the controller/disk level.

"My biggest concern with RAID0 is that whenever a drive fails, the entire RAID0 set fails. We will then have to replace the failed disk, recreate the RAID0 set and re-add it to the vSAN cluster. I'd rather have RAID1 disks which I can hot swappable replace whenever a disk fails."

This is where Storage Policies come in - you can specify on a per-Object basis (vmdk/namespace/.vswp/snapshot) whether you want it as FTT=0 (single copy of data) or FTT=1 (Essentially R1 + a tie-breaker component) and then multiple other configurations such as stripe-width, space-reservations, cache-reservations, IO limits etc.

If you run FTT=1 (as most seem to do) If a single copy of this R1 becomes unavailable the VM keeps running and the missing mirror is rebuilt if its absence is not transient (e.g. a host going down for a short time).

If a disk fails in a vSAN host (with R0 not pass-through), you can hot-replace it if your controller supports hot-swap, if not, it is not a huge deal as if you are running ROBO with all VMs FTT=1 you can run all VMs off one node, shut down the affected node replace the failed disk and bring it back up.

This reference is going back a long time but the architecture-design remains the same and might help elaborate on this more:

Storage Controller RAID0 mode

For storage controller RAID adapters that do not support pass-through mode, Virtual SAN fully supports RAID0 mode. RAID0 mode is utilized by creating a single drive RAID0 set via the storage controller software, and then presenting this to Virtual SAN. You will then have to mark which of these single drive RAID0 sets are SSDs within vSphere. When RAID0 mode is utilized, the the following occurs

Virtual SAN will not manage hot-plug capabilities of drives

Hot-plug will be managed by the storage controller software

https://blogs.vmware.com/vsphere/2014/01/virtual-san-hardware-guidance-part-ii-storage-controllers.h...

Another consideration since you are thinking of going ROBO:

Do you have another site/cluster somewhere relatively near that you can run the Witness and/or vCenter from?

If you feel that I did not answer anything completely feel free to PM me or ask more questions here.

Bob

-o- If you found this comment useful or answer please select as 'Answer' and/or click the 'Helpful' button, please ask follow-up questions if you have any -o-

0 Kudos
TheBobkin
VMware Employee
VMware Employee

Hehe spent too long writing an answer and ChrisKuhnshas a good point, but depends on the requirement - sure you can get controllers that are on the vSAN HCL for under $200 but if you want a decent queue depth you will likely have to spend 2-3 times that (unless getting refurb).

Thus the ask of the set-up really determines what can or cannot be used or what will work best for it at least.

0 Kudos