deduplication and Compression overview shows no savings at all

[FrankLuo]

We started to migrate VMs over the VSAN cluster which had deduplication and Compression turned on but interestingly, The savings is 0B and Ratio is 1x. I checked several days before, it was 1.16 or so. Anybody have the same experience

We are running 6.0U2

Thanks!

[virtualg_uk]

This might not be the case for you but deduplication on vSAN, as far as I am aware, is per disk group so there is no dedup between disk groups. Could this consideration be what's showing you such little deduplication?

[TheBobkin]

Hello Frank,

So to start off, the output in the Web Client looks more than a bit suspicious:

(the vCenter/vCSA and hosts are BOTH on 6.0 U2 and also all hosts on identical build?)

Used - Total 10.52TB

Dedup & compression overhead 9.24TB

Free 154.9TB

This is essentially a sub 1.00x ratio:

(10.52+9.24)/10.52  = 0.53x ratio

Essentially what this means is using dedupe/compression here is using more space than it is saving.

But there is far more to this than is obvious here, which I will attempt to address...

As per Graham's response, dedupe only works inter-Disk-group, not across the whole vsandatastore

"dedupe domains will be the same as a disk group; therefore, all redundant copies of data in the disk group will be reduced to a single copy, however redundant copies across disk groups will not be deduped."

https://blogs.vmware.com/virtualblocks/2016/03/11/virtual-san-6-2-cool-new-features/

What this means for the dedupe aspect is that only data blocks that are identical and ALSO residing on the same disk-group will be deduped (e.g. it will use one set of data for two or more blocks that have identical data).

A key factor in what is going to be identical is the applications and Operating Systems residing on the vsandatastore (e.g. if you have 100 different application VMs with completely different data structures and OSs vs 100 near-identical VMs, the identical ones will likely see huge dedupe benefits while the varying ones will see less dedupe benefits if any).

Now here is where it gets personal! (..to your current situation :smileygrin: 😞

You are currently only using a small fraction of the vsandatastore (let's disregard the dedupe overhead and say ~6.3% actual data usage).

If you have many hosts and disk-groups per host, the data that has been migrated to this cluster is going to be very sparsely distributed and as such it cannot really benefit from deduplication as any data that *might* be identical has a lower odd of also residing on the same disk-group.

Thus, as you increase the amount of utilised storage (assuming there is some data similarity between blocks) dedupe ratio and space saving will likely increase.

This will also take some time to be fully actualised as my understanding of this is that it only dedupes blocks once it reads/accesses blocks it has checked and knows there is a duplicate of.

Other possible factors are potentially at play here:

- Are any/many vmdks thick-provisioned? (Objects with a Storage Policy rule of 100% Object Space Reservation)

This can also occur due to being cloned/deployed from template that has thick-provisioned disks (though even default Storage Policy may show, this may not be applied in reality), reservation space per disk can be seen using this command via RVC at the cluster level: #vsan.disks_stats .

- Is this by any chance a VxRail implementation? (As I have seen some stuff thick-provisioned on older builds even when a thin-provisioned SP was applied)

Bob

[depping]

It seems to be a very large cluster? Could it be new VMs are placed on empty diskgroups and as such lowering the overal ratio? Usually when it fills up the ratio goes up.

[jbax]

I'm in a similar scenario of 0.00 saved bytes. My overhead is about 2.2TB on a 40TB cluster that's 65% full. I started with a new VxRail cluster (3.5 that's been upgraded to the latest version of 4.0) and have migrated my non-vsan cluster over to the VxRail cluster using cross-vcenter vmotions. I'm using the default vSAN policy but the vms were mostly thick provisioned on the old cluster.
(Based on your comment I have a bad feeling your going to say previously thick provisioned vms won't benefit from dedupe/compression.)

[TheBobkin]

Hello jbax

If they are still in reality thick-provisioned (proportionalCapacity = 100) they will not benefit.

I have seen this occur even when it says that the Default vSAN Storage Policy has been applied (and even more times when MARVIN policies are involved).

To start:

Identifying and fixing Thick Objects:

All commands run via RVC at the cluster level:

List disks (so that you can focus on disks with high % RESERVED space, this indicates Objects that are indeed Thick, copy+paste this info for later):

#vsan.disks_stats .

Check policy applied to all VMs:

#spbm.check_compliance ./resourcePool/vms/*

(any Objects that say 'Unknown' here I would be immediately be suspicious of, you can look at their disks via Edit Settings on the VM in the Web Client, click drop-down for the disk in question and it will show you what Storage Policy it is using, if it says 'Datastore Default' this is NOT the same as vSAN Default Storage Policy and I bet will be thick)

Display disk layout for all objects:

#vsan.vm_object_info ./resourcePool/vms/*

(copy-paste into notepad and look for any Objects with "proportionalCapacity = 100", or you can use grep against .txt on a host)

If you have a ton of Objects/disks that have "proportionalCapacity = 100" then create a new Storage Policy identical to the policy they are using (except with a new name), apply this policy to the objects and go back to step one to see if the 'Reserved' space on disks has decreased, if it has then we are on the right track.

Bob

-o- If you found this comment useful or answer please select as 'Answer' and/or click the 'Helpful' button, please ask follow-up questions if you have any -o-

[jbax]

DING DING! I found a lot of "proportionalCapacity = 100" and "proportionalCapacity =  [0, 100]". Not sure what the [0,100] means but I tested a new policy and it updated the vm to 0. Instead of moving all my machines to the new policy i just edited the current policy (changed iops limit from 0 to 0) and reapplied to all the vms (in stages). So far we're showing a ratio of 1.55x (almost 7TB saved!). Thanks!

[TheBobkin]

Hello jbax,

Glad to hear that worked, I advised creating a new identical policy (in all but name) as I have had mixed results with just re-applying same policy though it looks that adding a new rule-set is enough to do the trick, so good to know for future.

Did you also see a large corresponding decrease in the % 'Reserved' on disks as seen via RVC vsan.disks_stats?

Bob

-o- If you found this comment useful or answer please select as 'Answer' and/or click the 'Helpful' button, please ask follow-up questions if you have any -o-