VSAN cache tier help needed for two node cluster

[7007VM7007]

I am looking to setup my home lab with VSAN 6.6. Currently my setup comprises of the following:

Two Supermicro 5028D-TN4T hosts with 128GB RAM each and an 8 core Xeon CPU which has dual 10Gb and dual 1Gb RJ45 connectors

I then have a Starwinds Virtual SAN for shared iSCSI storage which has 4 Samsung SM863 Enterprise SSDs

What I'd like to do it move from iSCSI storage to a two node VSAN 6.6 cluster as follows:

I was planning on using two of the Samsung SM863 drives in each host for the storage tier (each drive is 480GB)

Run a crossover cable between the two 10Gb RJ45 connectors for VSAN/vMotion traffic

Rebuild my current Starwinds Virtual SAN server and use it for the Witness Appliance

The problem is I am unsure what PCIe/M2 SSD drive to get for each host for the cache tier. I was considering the Intel DC P3600 Add-in Card PCIe 400GB.

So I would have two Samsun SM863 480GB SATA SSD drives in each host for the storage tier and one Intel DC P3600 Add-in Card PCIe 400GB for the cache tier.

I think all of these drives are on the HCL for VSAN 6.6.

Can someone confirm if this setup would work correctly with a two node VSAN 6.6 setup? I'll be running 25-30 VMs and do plan on adding more SSDs to the storage tier.

Thanks!

[TheBobkin]

Hello,

Your proposed set-up looks okay, the disks you mentioned are all on HCL and more specifically they are certified for the purposes you intend to use them (All-Flash capacity-tier and All-Flash caching-tier respectively) .

More RAM is always better but whether you need it depends on the VMs in use here.

You have one issue and one potential issue here:

No controllers! You will need one of these on each host to manage IO to the disk-groups.

And the potential issue is connecting the Witness to each data-node, you will need a switch, not sure if you implied that but thought would mention it.

Here are some resources relating to this implementation that you might find useful:

https://storagehub.vmware.com/export_to_pdf/vmware-vsan/vsan-stretched-cluster-2-node-guide

https://storagehub.vmware.com/export_to_pdf/vsan-6-5-2-node-direct-connect-setup-checklist

Bob

-o- If you found this comment useful please click the 'Helpful' button and/or select as 'Answer' if you consider it so, please ask follow-up questions if you have any -o-

[7007VM7007]

Hi Bob

Thanks for the reply!

Currently I am using 100GB of RAM out of the 128GB available (taking HA into account) so I think I am ok here.

Can you expand on what you mean by "no controllers"? The Supermicro servers I have can take up to 6 SATA drives, one M2 drive and there is a PCIe slot for another drive. I think you can have a maximum of 7 drives in total in these machines. Why would I need a controller at all? Can't I just plug the capacity drives into the hot swapable bays and plug the cache tier PCIe drive into the PCIe slot? Why do I need a controller?

Regarding the Witness, yes, I have a switch so the two nodes and Witness appliance can communicate with each other.

[TheBobkin]

Hello,

Note there is also memory consumption for ESXi (already accounted for if you are running on this) and additional memory overhead per vSAN disk-group and varying depending if Hybrid or AF, this outlines these:

kb.vmware.com/kb/2113954

Sure, you *could* likely set-up a vSAN using an onboard controller (I tried to find specs of what comes on a 5028D-TN4T but none found easily) but this would likely not function very well due to the low Queue Depth typically capable on these types of devices, pretty much nullifying the benefit of All-Flash and even vSAN at all really.

It would likely end up being a bad bottleneck, depping covered this topic long ago:

yellow-bricks.com/2014/06/09/queue-depth-matters

For a comparison of what should be expected minimum Queue Depth you can refer to even the lowest-capacity/specs ready-node require minimum >256 Queue Depth, which is the minimum requirement for vSAN:

partnerweb.vmware.com/programs/vsan/Virtual_SAN_Hardware_Quick_Start_Guide.pdf

vmware.com/resources/compatibility/vsan_profile.html

Bob

-o- If you found this comment useful please click the 'Helpful' button and/or select as 'Answer' if you consider it so, please ask follow-up questions if you have any -o-