vSAN transient storage usage

[tim-may]

Hi everyone,

I am currently using vSphere 6.7U3q with vSAN in my environment.

I was wondering which events in vSAN trigger to use transient storage as part of the 30% slack space.

I understand that changing a policy with Raid1 FTT1 to Raid5 FFT1 obviously requires vSAN to "recreate" the data and therefore create transient overhead until the operation is finished. But what about changing the policy from Raid1 FTT1 to Raid1 FTT2? I would consider vSAN to just create a new set of additional components rather recreating all the copies. And what about increasing disks?

In my concreate issue I see a hugh spike in transient storage usage in "Cluster-Monitoring-vSAN-Capacity-Usage breakdown before dedup and compression" after expanding a single disk which I would not expect because of thin provisioning and the fact that transient storage signals to be temporary for something like policy changes or rebalance.

Thanks in advance.

[TheBobkin · Accepted solution]

@tim-may , "But what about changing the policy from Raid1 FTT1 to Raid1 FTT2? I would consider vSAN to just create a new set of additional components rather recreating all the copies. And what about increasing disks?"

This depends on the starting layout and location of the data, the available space on capacity-tier disks, the size of these free spaces and where these free space reside in relation to the where the Object that is being changed data resides.

If the cluster already has a reasonable amount of data on it then larger FTT=1,FTM=RAID1 may have not been stored plainly as data1+data2+witness across nodes - these could have for instance been stored without witness components (e.g. using votes on data-components instead) and been stored as data1-1+data1-2+data2-1+data2=2 across 4 nodes, then when you try to change it to FTT=2,FTM=RAID1 it couldn't just simply place the new data-components and/or witness components due to the available Fault Domains (nodes here) already having data-components from this Object on them and thus it would violate affinity/anti-affinity of replica placement. Let's assume this is a 5-node cluster for convenience: 4 nodes already have data-components on them and maybe there is not sufficient space on the 5th node to place a full replica - vSAN may have to move data off that 5th node to place the whole new replica (and note this is still forced to use votes here, no witness components), this will incur resync (and thus transient usage) in addition to the space for the new data-replica placement.

This is just one example, there are really a lot of different possibilities here but the summary is that changing from FTT=1,FTM=RAID1 to FTT=2,FTM=RAID1 will only just need to resync the new data-replica + witness component if the layout of the Object and the available space in the cluster (and specifically where this free space is, in how big chunks per node and taking into account the current component placement for this Object). Additional things being needed to be moved around prior to new replica placement is more/less likely based on how large the Object is, how many nodes (or Fault Domains) there are in the cluster, how much free space there is on each node/FD and the size of these free spaces.