High-Capacity VSAN 6 Nodes

[jwhitehv]

In looking at the current VSAN 6 HCL, there are no high capacity SATA drives. Actually, no SATA drives at all for VSAN 6. Just wondering how others are putting high-capacity VSAN nodes together?

Is this a data error? Or a problem with SATA and VSAN 6? Or just a backlog in the qualification lab?

Without 4TB or 6TB drive options, the possible density per node plummets. Standard 2U servers generally come in either 12 x 3.5" or 24 x 2.5" drives. 12 x 4TB drives in VSAN 5.5 configs would yield 48TB of raw storage. 12 x 6TB drives gets 72TB raw. At least the 4TB drives are available in NL-SAS configs from some vendors and OEMs, but not all. Anyone doing this at all? Dell FX2 could get a full 35 drives, but the 2TB 2.5" SAS drives aren't on the HCL, so no 70TB 2.5" configuration.

Also, there's no FusionIO products on the list.

Maybe they're just on there as OEM parts? Actually, I double-checked and there are no PCIe cards on the VSAN 6 HCL.

How are other people doing high-capacity nodes?

Is 12 x 6TB per 2U server even possible with parts off the HCL?

[jonretting]

I could have sworn there were some SATA Dell disks listed on their, or I am probably thinking Seagate Constellation SAS.  From my experience as long as you have an HBA with decent QD (min LSI 2308), and over the top Enterprise PCIe SSD NVME, 6TB SATA 7200 disk have worked out just fine. Had flying success with high-end PCIe and 4x 6TB SATA per host. Works great. However 4TB SAS really is the way togo. IMHO

Clarification would great pertaining to you finding.

Cheers

[zdickinson]

I have previously asked in forums about the lack of any PCIe cards on the HCL, I believe the answer was (as you suggested) a back log in QA.

As for the lack of large capacity SATA drives...  I would agree with retting, SAS is the way to go.  And I'm curious about your use case.  If you take 12 x 4 TB drives = 48 TB per node x 64 nodes (Max in a vSAN 6 cluster) = 3,072 TB.  I mean wow!  What are you scaling to 3 Petabytes?

Thank you, Zach.

[Bleeder]

The funny thing is that VMware already published a whitepaper (Virtual SAN 6.0 Performance: Scalability and Best Practices) several months ago where they used Intel P3700 PCIe SSDs.. but they're still not HCL'd.  Customers have been waiting over a year for those things to show up on the HCL..

[jwhitehv]

I'm curious about your use case.  If you take 12 x 4 TB drives = 48 TB per node x 64 nodes (Max in a vSAN 6 cluster) = 3,072 TB.  I mean wow!  What are you scaling to 3 Petabytes?

One doesn't need to scale to 64 nodes to want high storage density.Any time someone asks about VSAN, storage density per socket becomes an issue. That's $2.5K per socket and some customers aren't willing to drop down to a single socket per host, meaning $5K/host in VSAN licensing. Since external storage expansion doesn't seem to be supported despite GA announcements, VSAN starts looking like a capacity license.

2U 24 disk SFF host:
3 disk groups of 7 HDD + 1 SDD

The largest SFF HDD on the HCL is 1.2TB, so

24 * 1.2TB = 28.8TB/host

Using 6TB LFF drives:

12 * 6TB = 72TB/host

That's 2.5x the storage per VSAN socket license.

With Dell's FX2 architecture, you can get three FD322 storage modules with 5 x 7 SFF drive disk groups.
35 * 1.2TB = 42TB

6TB drives are still 1.7x the storage per VSAN socket license.

If you're willing to go larger than 2U, the HP SL4540 will scale to 60 LFF drives, so you can reach the 35 drive/host VSAN limit. The Cisco C160 scales to 60 LFF drives, but it's controller isn't on the VSAN HCL.

HP seems to have the best capability to scale VSAN, with the SL4540, 35 LFF drives, and a controller and 6TB SAS drives on the HCL. And even then, the largest HP SDD on the VSAN HCL is 1.6TB, when you should have 2.1TB per 42TB disk group.

I smell a blog post coming on.

[jwhitehv]

The funny thing is that VMware already published a whitepaper (Virtual SAN 6.0 Performance: Scalability and Best Practices) several months ago where they used Intel P3700 PCIe SSDs.. but they're still not HCL'd.  Customers have been waiting over a year for those things to show up on the HCL.

That is really frustrating. Scalability needs to be supported.

[jwhitehv]

as long as you have an HBA with decent QD (min LSI 2308), and over the top Enterprise PCIe SSD NVME, 6TB SATA 7200 disk have worked out just fine

Except if the components aren't on the HCL, the user can be denied support. That's not a great place to be.

[Bleeder]

I think you hit the nail on the head with the comment about VSAN socket licenses.. Think about it.

[jonretting]

Blog post indeed.

[zdickinson]

Let's take this example:

Using 6TB LFF drives:

12 * 6TB = 72TB/host

Let's say it's a minimum 3 node cluster, that's 216 TB before overhead.  It work load can consume that much space, but still perform well with only 6 sockets in the cluster?  It seems like with disk that dense, you would run out of compute power before you could fill it up.  Is it a few VMs that are massive?  A lot of little VMs that don't consume much CPU?  Something else I'm missing?  Thank you, Zach.

[Bleeder]

Does NexentaConnect, SoftNAS, etc answer that question?

[jonretting]

Correct me if im wrong but Netextenda just utalizes a VM with VSAN attached vmdks, and then offered up as SAN/iSCSI/NFS/CIFS. Along the same notion VVols would be a separate solution from VSAN, more akin to DAS with the benefits of VSAN like Virtual Volumes, performance tiers, etc. Or so is my current understanding  Best, -Jon

[Bleeder]

Correct, using 3rd-party solutions that turn VSAN into a NAS.  Not much compute needed.  Just lots of space and IO

[jonretting]

Correct -- It all depends on the risk assessment. The cutting edge is always not without risk. However vSphere by definition removes a great deal of risk. Somethings not working shift things around and rebuild it. Being locked down to physical hardware and networking is the real risk. In the case where I deployed that non HCL 6TB cluster, it was a very established high capacity replicated vSphere environment. In the end it was their acknowledged risk, as I do not condone SATA in any form. This statement does not however make your assessment of the current HCL list any less important. Best, -Jon

[jonretting]

It does in my minds eye add significant complexity, and once again locked into hardware types. Also how many Vvol system failures can you tolerate... In essence it seems like inverted convergence, where we took what we learned from VSAN convergence, inverted the principals and applied it to the physical SAN. Thanks, -Jon

[jwhitehv]

Let's say it's a minimum 3 node cluster, that's 216 TB before overhead.  It work load can consume that much space, but still perform well with only 6 sockets in the cluster?  It seems like with disk that dense, you would run out of compute power before you could fill it up.  Is it a few VMs that are massive?  A lot of little VMs that don't consume much CPU?  Something else I'm missing?

Not sure what you mean. Environments don't have a uniform compute to storage ratio and adjusting the core count per socket from 4-12 would be my first choice of scaling up compute requirements. 6 sockets can represent 72 cores. This client has a ton of data to store, but no analytics to run against it. One needs to weigh the cost of high-end CPUs against the cost of additional VSAN socket costs.    

[jwhitehv]

Does NexentaConnect, SoftNAS, etc answer that question?

Answer what question? Definitely not, "How are other users scaling VSAN?"

[elerium]

The VSAN HCL is pretty slow to update. Since release of 6.0 i don't think any NVME cards or SATA disks have made it into HCL yet. Of all the parts on the HCL, raid controller is most important followed by SSD. I'm not sure how important magnetic disks are to the HCL so far. The generalized thing I keep reading is that if you don't use enterprise/quality parts or try to cheap out with consumer level parts, things will suck and data will be lost. In particular, SSD drives will need power loss protection capability which isn't found on many consumer level drives.

I'm willing to take some risks since my cluster is used for development purposes, so I've built my VSAN6 cluster with magnetic drives and SSD not on HCL. I'm using WD RE4 4TB SAS, and Intel P3700 1.6TB for SSD on Dell R730xd which has 16 3.5" bays. I end up with 56TB raw or 28TB usable (ftt=1) storage for 2U of rackspace. Seeing that other users have reported great success with the P3700 and both Intel/VMWare have released a performance whitepaper with the P3700 in use, I decided to go with this SSD. The only grief I've had is actually with the Dell H730 raid controller and that was when it was on the HCL firmware version and on HBA mode (both recommended on HCL). I told the VMWare support engineer that I flashed to a newer non-HCL firmware from Dell to fix PSOD/controller resets and change from HBA to RAID0 and it's been golden since. Support didn't give me any kind of grief at all for not following HCL.

In regards to VMWare not supporting you when you aren't using HCL parts, I have not found this to be the case. I had 2 separate tickets open, support checked if everything was on HCL and warned me that data loss could occur from not following HCL recommendations and asked if I understood. Then proceeded to assist with my tickets.

In regards to licensing, I don't like how nodes that don't contribute to storage but are in the same cluster still require VSAN licensing, otherwise even with socket license pricing, it's an amazing performance for value product.

[jwhitehv]

There are some 6TB and 4TB SAS drives on the HCL:

HGST drives and only HP as an OEM. So if building on anything other than HP, users can't get OEM support on their drives.

Better, as users can get 4TB drives from four OEMs (I'm not counting IBM since the sale of System X to Lenovo). Cisco is noticeably absent from both lists.

[jonretting]

This is kinda nit picky. You would have the only listed options of HGST or OEM HP. That isn't to say your cluster wouldn't be supported by vmware, if you went with the Dell edition. I think the issue comes down to a frequent mantra of limited HCL testing capabilities, or something to that tune. Granted the discussion would no longer be technical, but might it be wiser a subject. As previous users mention the lack of NVME PCIe and many other things. Is the HCL a lofty goal, but under serviced? Thinking aloud. -Jon