vSAN Memory Consumption

[Knowledgable]

Hi All,

I've deployed a new 6.7 vsan with the below specs across all hosts.

Hosts = 3

Memory per Host = 128 GB

Disk Groups = 2

Cache per Disk Group = 200 GB

Capacity per Disk Group = 5 x 1.2 TB

All health checks have passed inclusive of firmware and drivers. However, I'm noticing an extremely large consumption of memory 52 GB (image vsan01) but the only deployed VM is the vCenter appliance on the cluster. On the host with the VM compute resource, its at 24 GB while the other two are 13 GB. I have moved the appliance between servers but usage is the same.

Is this normal behavior ? Any tips on how to improve/balance the consumption would be greatly appreciated.

[TheBobkin · Accepted solution]

Hello Knowledgable​,

I am assuming on of those screenshots is the cluster resources and other on the host with the vCenter running on it (which accounts for ~10GB usage).

The other 13GB is being consumed by the overhead from the Disk-Groups - how much this requires is based on whether is it All-Flash or Hybrid, size of cache, number of capacity-drives per Disk-Group and then multiply this by the number of Disk-Groups:

BaseConsumption +

(NumDiskGroups * (DiskGroupBaseConsumption + (SSDMemOverheadPerGB * SSDSize))) +

(NumCapacityDisks * CapacityDiskBaseConsumption)

VMware Knowledge Base

The amount of Memory overhead these require is mandatory and cannot be avoided or reduced (without changing any of the set-up).

You have very small cache-tier devices and low cache:capacity ratio here, I do hope the caches are good performing models and/or you aren't expecting the workload on this cluster to be too taxing.

Bob

[Knowledgable]

Thanks for explaining. Is it correct to say that I should reduce my capacity tier, maybe 1:3 per disk group to make it more in line with the current cache ?

[TheBobkin]

Hello Knowledgable​

"Thanks for explaining."

No problem - I do wish vSAN was a little bit more transparent about this (e.g. informational alert when creating disk-groups or break-down under consumption), maybe this should be implemented if not already in the works (or displayed somewhere that I have not noticed :smileygrin: ).

"Is it correct to say that I should reduce my capacity tier, maybe 1:3 per disk group to make it more in line with the current cache ?"

As the disk-group base consumption is what consumes the bulk of the memory, this wouldn't be the most ideal solution here - e.g. the same storage split over 3 Disk-groups instead of 2 will use more memory in total. If feasible, using larger cache-tier devices (e.g. 4-600GB) would be preferable but this will also increase the memory usage (800MB per 100GB on Hybrid and 1.4GB per 100GB on All-Flash).

Whether you do need a better ratio here really depends on how you are going to be using this (e.g. what % used of vsandatastore and which FTM) and what the workload profile is like - this might be completely fine for some workloads but not enough juice for others.

Some good articles on this subject:

https://blogs.vmware.com/virtualblocks/2017/01/18/designing-vsan-disk-groups-cache-ratio-revisited/

http://www.yellow-bricks.com/2016/02/16/10-rule-vsan-caching-calculate-vm-basis-not-disk-capacity/

https://storagehub.vmware.com/export.../vmware-r-vsan-tm-design-and-sizing-guide

Bob

[Knowledgable]

Thanks for the insight. Will keep the disk groups at two and increase the caching tier with some larger disks.

As a side note, what would be the recommened brand of reliable SSDs for cache. Made a lil worried when you said "you hope its not low end SSDs".

[TheBobkin]

Hello Knowledgable​,

Sorry there, on re-reading, I made an error in the above - base consumption is static, not per DG. Essentially this means each identical DG added adds the amount of memory used with one DG minus 5426MB (in the kb Hybrid example, 3 DGs = ~2x the consumption of 1 DG).

I do think that adding bigger cache-tiers would still be the better solution over spreading disks into smaller DGs but do of course weigh up the option and potentially find out if the smaller devices (with less capacity per DG) are enough for the workload (especially if they have already been purchased and wouldn't have a good re-purpose).

Most of the OEMS/Vendors make/sell drives in a full range of capabilities (and price-ranges) so saying 'Brand A make better drives than Brand B' wouldn't really suffice as an answer. The vSAN HCL references the durability and performance class of all certified SSD devices so using this as a starter point for comparison would be wise, referencing the vendors spec sheets for the devices and storage-forums/reviews of these would be the next logical step. Do also check whether the devices are going to be near EOL (for various reasons) and supported capabilities (e.g. All-Flash cache-tier aswell as Hybrid cache-tier) which can potentially be a further judge of quality and at the very least allow these to be re-used for the same purpose if upgrading to All-Flash (You didn't mention whether current cluster is HY or AF).

Bob

[Knowledgable]

Thanks for taking the time to review. I did forget to mention that it is a Hybrid cluster.

What i was considering was leaving the existing disk groups at 2, however reducing the amount of disks per group to be more in line with that of the SSD cache. I will look at the HCL as a starter as recommended.