1 diskgroup per HBA ?

[therealhostmanu]

Hi,

Looking for some clarification on something pls folks!

I'm trying to fully understand disk groups and their interaction with HBAs.  I understand a diskgroup can contain up to 7 SSDs for capacity and 1 cache device (nvme device most likely).

Can a single HBA have multiple diskgroups?  I want to use a 24 bay chassis, with a sas expander, initially 7 bays populated for the capacity tier, 2 bays populated with nvme ssds for caching.  For future storage growth I want to be able to drop in additional capacity drives and cache drives, will this work with a single HBA?  Or is it a case of it will work, but advisable to have another HBA, as it creates another fault domain and presumably could impact performance if all drives are on one HBA?  The nvme devices will be directly connected to the mobo from the sas expander.

From what I can tell the sas expander can be connected to two HBAs, splitting the drives between controllers.

[TheBobkin]

Hello therealhostmanuk​

Welcome to Communities.

"Can a single HBA have multiple diskgroups?"

Yes.

"Or is it a case of it will work, but advisable to have another HBA, "

I would advise splitting it over 2 HBAs as these are relatively cheap components, this will ensure that issues with 1 controller won't render this node unavailable for component placement and will also spread the load.

Yes, you can use SAS-expanders to increase the amount of devices that can be attached to a single controller but do note that some features such as blink LED will not function in this configuration.

Bob

[therealhostmanu]

Thanks for response.  I do wonder about the logic of having multiple HBAs however, as they are not hot swapable, the system has to be taken offline to replace a failed HBA anyway.

hmmm, blink LED not working with a sas expander is quite an issue when needing to replace failed drives.  Is there anyway round this?  Short of making a note of which drive serial number is in which drive slot before anything fails.

[TheBobkin]

Hello therealhostmanuk​,

"I do wonder about the logic of having multiple HBAs however, as they are not hot swapable, the system has to be taken offline to replace a failed HBA anyway."

Sure, but you might be waiting some time to get replacement etc. and in some situations (e.g. 2/3-node cluster or 4-node with RAID5) and a single HBA with 3 Disk-Groups attached means this data is FTT=0 until you get that replaced; whereas if you had 1 HBA for each of the 3 Disk-Groups (and enough space) then the data could be rebuilt to FTT=1 - adding resiliency to clusters is imperative in my opinion as this allows healing following multiple failures (Murphy's Law and all that).

Decent HBAs are cheaper than even very low capacity, basic Enterprise capacity-tier drives so I don't understand why anyone would skimp on these (other than limited connectors).

"hmmm, blink LED not working with a sas expander is quite an issue when needing to replace failed drives."

Sure, I agree this can lead to severe headaches if the appropriates steps are not taken e.g. being aware of this when you build your cluster and taking the appropriate measures.

"Is there anyway round this?  Short of making a note of which drive serial number is in which drive slot before anything fails."

I think everyone should be doing something similar to this anyway - documentation is key and this shouldn't take long (e.g. map serial to slot to naa).

I am actually not entirely sure about Dell/other SAS Expanders functionality with regard to this but I know HPE controllers have this limitation:

https://kb.vmware.com/s/article/2151284

Probably should have mentioned this to begin with but SAS Expanders are only supported on ReadyNodes, not Build-Your-Own clusters:

https://blogs.vmware.com/virtualblocks/2016/05/17/vmware-virtual-san-sas-expanders/

https://kb.vmware.com/s/article/52084

Bob