VMware Cloud Community
benjamin000
Enthusiast
Enthusiast
Jump to solution

vSAN 2 x Cache HDD or 1 x Cache HDD

Hi

I have a simple question and need advice.

The time has come to upgrade our existing vSAN and Hosts.

We have the option of purchasing the following configurations and want to know which would suit for best performance.

1st Setup

1 x 480GB SSD

1 x 6 TB SAS HDD

2nd Setup

1 x 960GB SSD

2 x 6TB HDD

3rd Setup

2 x 480GB SSD

2 x 6 TB HDD

4th Setup

1 x 960GB SSD

3 x 4TB HDD

Can one SSD cache HDD be used in the same group with 2 x HDD ?

For performance I am not sure in vSAN if having multiple HDD's is best or multiple SSD's....This is an important question

Would 2 x SSD drives be better than 1 in vSAN and would more capacity for the SSD Cache be used as we have the option of either 480GB SSD drives or 960GB SSD but if we purchase the larger SSD's would the storage be used or would it just be a waste ?

Any thoughts Sreec​ or marinod73

Regards Ben McGuire
Reply
0 Kudos
1 Solution

Accepted Solutions
jameseydoyle
VMware Employee
VMware Employee
Jump to solution

Hi Ben,

In hybrid diskgroups (as yours would be given the mix of flash and non-flash devices) the cache tier is used for both read caching and write caching, with the split being 70:30 Read:Write. Write cache has an upper limit of 600GB, so you could technically use a 2TB flash device and still be able to use it completely. However, for your use case, if performance is the top priority, having more cache cannot hurt. It will mean that you have more read cache available to service reads, so the likelihood of you experiencing read cache misses will be greatly reduced. However, if your workloads are always the reading the same data, maybe having too much read cache would be of limited benefit. It really depends on your workloads whether your would really see a major improvement. However, if the cost is not too prohibitive, go for the bigger SSDs, as they will ultimately give you better performance, and will allow you to add further capacity drives later, while still maintaining a suitable cache:capacity ratio.

Check out these links for more information:

Storage and Availability Technical Documents

Storage and Availability Technical Documents

Regards,

James

View solution in original post

Reply
0 Kudos
8 Replies
admin
Immortal
Immortal
Jump to solution

Hi Benjamin,

unfortunately, I am not a an expert in VSAN. Why don't you post this in vcenter forum?

As far as I know, one disk group should contain one SSD disk one, per one maximum of 7disks per disk group.

Cheers,

Domenico

Reply
0 Kudos
benjamin000
Enthusiast
Enthusiast
Jump to solution

Hi Dom

I thought asking a question about vSAN would be applicable to the vSAN group but ill review the vsphere group.

I am upgrading all 3 compute clusters so I want to get the storage right with vSAN. Thanks for you reply it is highly appreciated.

Regards Ben McGuire
Reply
0 Kudos
admin
Immortal
Immortal
Jump to solution

Hi Benjamin,

so sorry... my bad!

I replied this morning, thinking that it was related to VIO! I did not check you already wrote on VSAN forum!

Cheers,

Domenico

Reply
0 Kudos
jameseydoyle
VMware Employee
VMware Employee
Jump to solution

Hi Benjamin,

If your primary concern is performance, over availability, any of the options would suffice, assuming that the SSDs are of a similar type and the capacity is the only major difference. Each of the disk groups you would be building with your options have the same HDD/SSD capacity ratio of ~12:1, which for most workloads would be adequate. You performance will be primarily dictated by the speed and latency of your SSD devices. The HDD's performance will only really come into play when you are experiencing read cache misses. The 4th setup would give you the best read performance in the case of cache misses, therefore, as you have specified 3 HDDs and one SSD, resulting in a single diskgroup with 3 spindles.

However, I would argue that another configuration would be even better, if the costs, and the server hardware allows for it. It looks like you want 12TB of capacity and 960GB SSD per node. Could you go for a configuration like the following?

2 x 480GB SSD

4 x 3TB HDD (or 6 x 2TB HDD)

That means you have the same ratio of SSD:HDD capacity, but greater redundancy and more failure domains. i.e EAch host will now create 2 diskgroups, with one SSD for cache and multiple HDDs for capacity. This way, a disk failure will less impact on your cluster, while still providing roughly the same performance characteristics.

Each disk will also have their own queue depth, so more disks allows for a greater overall queue depth at the disk level, allowing for potentially better performance.

Before making any decisions though, you have to consider future growth also. If you fill all the drive bays with smaller capacity disks, you may not have the option to add disks later to scale up. You may be forced to scale out by purchasing more servers.

Lots to consider!

Reply
0 Kudos
benjamin000
Enthusiast
Enthusiast
Jump to solution

Hi James,

Thanks for your detailed reply.

I do not need huge storage capacity as i will be purchasing 3 servers with the same storage configuration so even with 2 x 4TB on one server ill have 24TB in total on 3 hosts,

The problem lies with HDD capacity for vSAN as we can only allocated 4 HDD on each host for vSAN which is why I did allow for more than 4 HDD in the post.

I think what you are saying is more SSDs are best for performance however I would also want some redundancy so having 2 HDD and 2 SSD's may be a viable option would you agree or could I get away with having 3 x SSD's and 1 x 8TB HDD as I will have redundancy because Ill also have 2 other 8TB HDD in the other 2 hosts.

I hope all that makes sense.

Regards Ben McGuire
Reply
0 Kudos
jameseydoyle
VMware Employee
VMware Employee
Jump to solution

Hi Ben,

Yes I understand what you are saying. If you are limited to 4 HDDs per host, I would recommend using 2 SSDs per host if possible. I understand the redundancy is provided by other hosts, but you don't want a single disk failure to cause an entire host to be useless, which would be the case if you had only 1 x SSD and 1 x HDD per host. If the SSD fails, the entire diskgroup fails. If you only have one HDD, if that fails, your entire host cannot host components. You are basically throwing all your eggs in one basket. With no redundancy within the host (i.e. by configuring multiple disk groups), there is nowhere left to rebuild components in the case of a failure and you are left with no redundancy after a failure.

However, if you have multiple diskgroups on your host, a single disk failure does not necessarily mean a permanent reduction in redundancy. With a 2nd disk group on your hosts, components from the failed diskgroup can be rebuilt on the 2nd diskgroup provided there is sufficient capacity, meaning you can regain full redundancy even after a disk failure.

Reply
0 Kudos
benjamin000
Enthusiast
Enthusiast
Jump to solution

Hi James

You make complete sense. So I am best to have 2 disk groups.  1 x SSD and 1 x HDD in 2 disk groups.

Once last thing before I award you. As I have an option of either 480GB or 960GB SSD would vSAN make FULL use of more SSD capacity ?

I understand the 10% rule as a minimum but I am not overly sure if vSAN takes full advantage of a large SSD.

FYI I would be looking at the 6TB HDDs.

Regards Ben McGuire
Reply
0 Kudos
jameseydoyle
VMware Employee
VMware Employee
Jump to solution

Hi Ben,

In hybrid diskgroups (as yours would be given the mix of flash and non-flash devices) the cache tier is used for both read caching and write caching, with the split being 70:30 Read:Write. Write cache has an upper limit of 600GB, so you could technically use a 2TB flash device and still be able to use it completely. However, for your use case, if performance is the top priority, having more cache cannot hurt. It will mean that you have more read cache available to service reads, so the likelihood of you experiencing read cache misses will be greatly reduced. However, if your workloads are always the reading the same data, maybe having too much read cache would be of limited benefit. It really depends on your workloads whether your would really see a major improvement. However, if the cost is not too prohibitive, go for the bigger SSDs, as they will ultimately give you better performance, and will allow you to add further capacity drives later, while still maintaining a suitable cache:capacity ratio.

Check out these links for more information:

Storage and Availability Technical Documents

Storage and Availability Technical Documents

Regards,

James

Reply
0 Kudos