VMware Cloud Community
chaptech
Contributor
Contributor
Jump to solution

two flash disks in one disk group vsan

Morning all.

We have had our second flash ssd fail bringing down the entire disk group in vsan, not an issue all resynced and recovered ok.

I know we can make multiple disk groups per host with its own flash ssd per group, my question, can we put two flash ssds into one disk group? If so if one of the two fails will the entire disk group go offline? or will it continue to function ok until both of the flash disks fail?

thanks in advance

Nathan

Tags (2)
Reply
0 Kudos
1 Solution

Accepted Solutions
JohnNicholsonVM
Enthusiast
Enthusiast
Jump to solution

Stripe 2 will improve performance, FTT2 will lower back end write IOPS (possibly an issue, possibly not depending on back pressure and cache fill). In theory it could improve read cache miss performance, but between ARC, the read ahead caching system for cache misses and everything else I wouldn't expect much.

Smaller disk groups, more flash etc. I'd recommend you fire up VSAN observer and look at your Read Cache hit rate and see if another disk groups (and more flash) would be a good idea.  You tend to get diminishing returns trying to push it past 90% cache hit rate Smiley Happy

So keep in mind that unless you have a really bad object balance issue (It would require a lot of effort to be honest, of careful VM creation and deletion and then setting SPBM policies to create it) you should see a heavy many to many rebuild (IE when a disk or Disk group is lost the mirror pairs for those objects will be scattered across the cluster, and they will rebuild into the free space of all other disks until the mirroring policy is re-enforced).  Keep in mind that objects is already chunked into new components ever 256GB (by default). Because of this design as the cluster gets larger and ads more failure domains (hosts, disk groups, drives) you should actually expect rebuild performance to get better!

Did you experience performance issues on your rebuild, and if so what are your flash/disk types and make model of your server HBA etc?

View solution in original post

Reply
0 Kudos
8 Replies
JohnNicholsonVM
Enthusiast
Enthusiast
Jump to solution

RAID 1 mirroring the SSD is NOT supported Smiley Happy

Now you could keep a spare SSD in the host and flag it as usable after the failure (I'd have to lab/test this), but I'm still not certain this would trigger a re-sync instead of a rebuild.

Lets talk about the real concern here.  Rebuild overhead, and rebuild times.  There are a few ways to mitigate it.

1. Deploy more SSD's and break up your disk groups into smaller failure domains (Possibly even dedicating SAS HBA's to each group).

2. configure FTT=2 for more important workloads that your are concerned about (not ideal).

3. Make sure you have deployed either 10Gbps or 40Gbps and set NIOC properly.

4. Make sure your using reliable SSD's that are on the HCL and flashed to the recommended firmware version.

Reply
0 Kudos
chaptech
Contributor
Contributor
Jump to solution

Thanks John. Yeah same I wasn't sure how it would react but just adding a second flash ssd into the disk group, if It still went offline completely or stayed online, did a recync etc.

I was already toying with the idea of making smaller disk groups on each host with their own flash just for the performance gains.

Currently our storage policy is FTT1 and stripe 1, we have 5 hosts on 3 10G switches, I was considering making FTT2 and stripe 2 also, would this improve the io load and time to resync? or make it worse than it already is when things get out of wack?

cheers

Reply
0 Kudos
JohnNicholsonVM
Enthusiast
Enthusiast
Jump to solution

Stripe 2 will improve performance, FTT2 will lower back end write IOPS (possibly an issue, possibly not depending on back pressure and cache fill). In theory it could improve read cache miss performance, but between ARC, the read ahead caching system for cache misses and everything else I wouldn't expect much.

Smaller disk groups, more flash etc. I'd recommend you fire up VSAN observer and look at your Read Cache hit rate and see if another disk groups (and more flash) would be a good idea.  You tend to get diminishing returns trying to push it past 90% cache hit rate Smiley Happy

So keep in mind that unless you have a really bad object balance issue (It would require a lot of effort to be honest, of careful VM creation and deletion and then setting SPBM policies to create it) you should see a heavy many to many rebuild (IE when a disk or Disk group is lost the mirror pairs for those objects will be scattered across the cluster, and they will rebuild into the free space of all other disks until the mirroring policy is re-enforced).  Keep in mind that objects is already chunked into new components ever 256GB (by default). Because of this design as the cluster gets larger and ads more failure domains (hosts, disk groups, drives) you should actually expect rebuild performance to get better!

Did you experience performance issues on your rebuild, and if so what are your flash/disk types and make model of your server HBA etc?

Reply
0 Kudos
chaptech
Contributor
Contributor
Jump to solution

Hi John,

The main issue was, currently our 5 hosts have 7200RPM sata disks with intel SSDs. We have only had 2 SSD fail the whole time so not too stressed about the fact they are failing, its more the struggle the system has resyncing. Murphys law it happens early in the morning just before the peak load when everyone is logging into the vms, the resyncing is still happening and the vms is extremely slow, to almost unusable. We have 10G to each host but it is on 3750 Cisco switches so I know the latency isn't brilliant and we are looking at infiniband for our next cluster build and will go PCIe for flash and all SSD for storage, but that's a ways off yet.

So my original query with adding a second flash SSD into the disk group is purely to avoid the disk group going offline and the system needing to resync causing these times of heavy lot. I know the Sata 7200 disks aren't great but I wouldn't have expected its a heavy io load process resyncing, more sequential reading and writing of data, so wasn't convinced the Sata disks were solely responsible for the bottleneck when resyncing.

So if adding a second flash SSD to each disk group isn't really an option to keeping them online in the event of an SSD flash disk failing then looking at the additional striping and setting FTT to 2 as a means of keeping the system more stable and usable when this happens??

Cheers

Reply
0 Kudos
yobhoohboy
Contributor
Contributor
Jump to solution

"I know we can make multiple disk groups per host with its own flash ssd per group, my question, can we put two flash ssds into one disk group? If so if one of the two fails will the entire disk group go offline? or will it continue to function ok until both of the flash disks fail?"

I also have the same question, but from the answers posted I don't believe that the original question was answered.


I.e. any benefit of using 2 x 400GB SSDs in a disk group rather than have 1 x 800GS SSD. Are there any issues with this, does it make a difference in performance and will the entire disk group go offline if one of the 400GB SSDs fail. It's noted that R1 isn't supported (and it's not being asked).


If someone could comment on this I would really appreciate it. More spindles = better performance, but as the SSDs are used as caching I'm not sure if that would apply and actually may add latency.


Thanks in advance!

Reply
0 Kudos
srodenburg
Expert
Expert
Jump to solution

are you using non-enterprise sata disks?   If so, read this:

Buy HCL-Listed SATA or SAS SSD's ? Why consumer grade SATA SSD's make no sense.

(scroll to the end of the thread)

About 1 Gbps networks:  expect massive latencies during rebuilds/node evacuations. The advice to use 10GB has a VERY good reason...

Reply
0 Kudos
srodenburg
Expert
Expert
Jump to solution

"I.e. any benefit of using 2 x 400GB SSDs in a disk group rather than have 1 x 800GS SSD"

A diskgroup can technically only have one SSD so the question is invalid.

Instead of using 1 SDD with 4 disks one can use 1 SSD with 2 disks and that times two. It enhances performance.  Please mind that using 1 Gbit networking will totally negate the speed-benefits of having two diskgroups per node (each with it's single SSD) so adjust your expectations.

Reply
0 Kudos
Thayal
Contributor
Contributor
Jump to solution

I agree, you can not use two flash disk in single group, better option is to split the HDDs in multiple disk group and add a flash disk in each group for better performance.

! !@@@@@@@@

Reply
0 Kudos