Thin provisioned disks grow up to its limits

[bagara]

Hi everyone!

I am using Esxi 6.7 on one HP server with the image "HPE Customized Image ESXi 6.7.0 Update 1 version 670.U1.10.3.5 released on October 2018 and based on ESXi 6.7.0 Update 1 Vmkernel Release Build 10302608". I am runnig two VM with Windows 2016 Server:

VM 1) ControladorDominio: has two disks "thin provisioned". The first one is 300 GB for the SO and 1,2 TB for data.

VM 2) Contabilidad: only one disk "thin provisioned" with 300 GB.

I am using Iperius Backup to backup both VMs with CBT option and all was working perfectly. The backups only took the space used currently: for example, the VM's disk of 1,2 TB only took 100 GB, which is the real space used. The same for the 300 GB disk: only 40 GB used and so was the backup's size.

One day and suddenly, the backups for the VM ControladorDominio take the full size: 1,2 TB + 300 GB = 1.5 TB aprox. But the real size inside the VM has not changed and is 100 GB in one disk and 40 GB in the other. I thought it could be a problem of Iperius Backup. But when I go to the Esxi web client, I see that the datastore has increased from 180 GB to 1,58 TB.

In this capture you can see the free space of the C and E drives. The used space is 1.47 TB when it should be  122 GB aprox.

The VM Contabilidad still is ok (49,87 used space)

Here you can check that hard drives are thin provisioned:

This is a real problem, because the hard disk we use to backup is 1TB capacity and the datastore has only 233 GB free.

What is the problem? Why did this happen to ControladorDominio? And the most important, can I free the unused space as it was before?

Thank you!

[Ardaneh]

Hi

Did you check the snapshots?

Generally, when you use thin provisioning, the VM knows how much space it can use but when you take a snapshot and your snapshot remains on the VM, it can grow more and more and will fill your whole datastore. We had this problem once, so we decided to not use Thin provisioning and make our datastores according to best practices (20% of your storage must be unused when you create your LUNs or vVOLs)

So my recommendation is that check your VMs snapshot and be sure that your snapshots will be deleted after 72 hrs (according to best practices)

I hope it can help you

[bagara]

Hi!

Thank you for your attention!

Yeah, I checked snapshots. There is no one

So, is there a way to regain the uncommitted space?

[Ardaneh · Accepted solution]

Yes, there is.

If you have a vCenter, it can be easily done, but if you don't, you can use this method only for Thin Disks:

Make an SSH connection to your ESXi

Go to the path of your VM which you want to reclaim space from it (for example /vmfs/volumes/DS1/MyVM)

use vmkfstools command with -K (not -k) to reclaim the space: (your VM must be shut down)

vmkfstools -K "YOUR VM vmdk file"

Note:

If you have a Windows-based VM, you should use sdelete tools first and then run the vmkfstools command. you can use this link to download sdelete:

SDelete - Windows Sysinternals | Microsoft Docs

This a related video:

Free up ESXi local drive VMFS datastore space after deleting data in a thin provisioned Windows VM -...

-------------------------------------------------------------------------------------------

By vCenter, you can connect to your vSphere Client, go to datastores section and then choose "Edit Space Reclamation" from right-click menu

-------------------------------------------------------------------------------------------

If you don't have vCenter, You can use the esxcli command to make a configuration on your datastore:

https://anthonyspiteri.net/quick-look-vsphere-6-5-storage-space-reclamation/

[continuum]

FYI .... Snapshotmanager is NOT the place to look for when you need to know wether your VM has snapshots !!!
Look at the VMDK-settings instead.
If you see that there is a name-00000*.vmdk assigned you HAVE at least one snapshot.
Snapshotmanager only displays correctly in sunny weather conditions.

[bagara]

Thank you again for your answers!

I dig with SSH in the file system and found this:

As you can see, there are the files "WindowsServer2016STD_1.vmdk" and "WindowsServer2016STD_2.vmdk". The VM config shows:

So, does it mean that I have a snapshot? I already tried this procedure: VMware Knowledge Base

What should be my next step then?

Thank you for your help!

[Ardaneh]

So you don't have any snapshots there.

Did you attempt to reclaim the space as we talked about?

Please let me know if you could do that.

[bagara]

Hi!

I am executing sdelete in the VM, but it is taking a lot of time... When I have results, I will report them!

Thank you for your time!

[bagara]

I completed the vmkfstools proccess on one 300 Gb virtual disk, and it worked! I will repeat this with all the disks.

But we still do not know why this happened... Being the disks thin provisioned, not written the whole disk.... why did it grew to his full size?

Thank you all for your answers!

[malickware]

You have to check from the OS side as the vmkfstool is simply reclaiming the space which was marked as unused from the OS side.

[Ardaneh]

My only guess is "Snapshots", But as you mentioned there is no snapshot so I need to check more.

I'm glad that your problem was solved

[bagara]

Hi everyone!

This is an old post, but I have another question: Why is the auto space reclamation function not working in my system? As you can see in the screenshots, the datastore is in VMFS6 format....

Thank you in advance!

[continuum]

Are you running automatic defragmentation jobs inside the VM ?

[bagara]

Yeah! Both VMs are Windows 2016 Server and both are running automatic defragmentation.... disabling it!

When I exec this command:

[root@esxi:~] esxcli storage vmfs reclaim config get -u=5c5aac7c-37d86854-021c-b88303f1a8dc

I get this output:

   Reclaim Granularity: 1048576 Bytes

   Reclaim Priority: low

   Reclaim Method: priority

   Reclaim Bandwidth: 0 MB/s

Is everything correct?

Thank you so much!