vmkfstools --punchZero with delta disks

[nponeccop]

I have a Windows Server VM running in free ESXi 6 on a 450 GB drive.

The disk has one 9GB snapshot and 200 GB delta. But actual disk usage as seen by Windows is 75 GB.

So at least 100 GB is wasted. Can I reclaim the space?

I can try zerofilling free space with sdelete -z and run vmkfstools -K, but will it work on disks with deltas? I remember VMWare Workstation/Player were quite limited.

[a_p_]

The "--punchzero" option is only useful for thin provisioned virtual disks, because punchzero works on the file system level.

The disk has one 9GB snapshot and 200 GB delta.

Snapshots are stored in delta files, and I don't understand the the different sizes you mentioned.

It might help if you'd provide a list of the VM's files as well as the free disk space on the datastore. For the file list, please run ls -lisa in the VM's folder from the command (e.g. via putty), and paste the output to a reply post.

Andréline

[nponeccop]

Yes I have a thin-provisioned disk with snapshots.

Technically the first snapshot of a disk is stored in a file, and deltas to subsequent snapshots are stored in delta files.

624957572      0 -rw-r--r--    1 root     root            13 Dec 10 16:33 MyVM-aux.xml

46143620   1024 -rw-------    1 root     root          8684 Dec 10 17:10 MyVM.nvram

16783492      0 -rw-r--r--    1 root     root            77 Dec 10 16:25 MyVM.vmsd

4200580      8 -rwxr-xr-x    1 root     root          3464 Dec 10 17:10 MyVM.vmx

83892356      8 -rw-------    1 root     root          3797 Dec  6 19:16 MyVM.vmxf

54532228 2688000 -rw-------    1 root     root     2752286720 Aug 22 18:31 MyVM0-000001-delta.vmdk

58726532      0 -rw-------    1 root     root           347 Aug 22 18:29 MyVM0-000001.vmdk

92280964 254379008 -rw-------    1 root     root     260483878912 Dec 10 17:10 MyVM0-000002-delta.vmdk

96475268      0 -rw-------    1 root     root           327 Dec 10 16:57 MyVM0-000002.vmdk

8394884 9732096 -rw-------    1 root     root     429496729600 Aug 22 13:15 MyVM0-flat.vmdk

12589188      0 -rw-------    1 root     root           498 Dec 10 13:15 MyVM0.vmdk

Below is a relevant (IMO) extract from there:

2       2       MyVM0-000001-delta.vmdk

254     260     MyVM0-000002-delta.vmdk

9       429     MyVM0-flat.vmdk

I created a 429GB thin provisioned disk.

Then I installed Windows. 9 GB was used if seen in Windows. Created a snapshot.

Then I installed VMWare tools and whatnot. Created another snapshot.

Then I played alot which included copying a 40 GB file around many times.

So in the end I have:

9 gb "base" file.

2 gb 01-delta

250 gb 02-delta.

However in VM the NTFS filesystem only contains 70 GB worth of data. Is there a way to optimize 02-delta somehow so it takes less space on disk?

[a_p_ · Accepted solution]

I'm not aware of a tool which is able to cleanup delta files. The allocated blocks in the delta file are actually caused by the guest OS (Windows), which demands disk space in the guest's file system, and ESXi just allocated the space if the data blocks are not already in use.

With the file sizes you posted and the 450GB physical disk, I'm afraid that it is not possible to delete/consolidate the snapshots easily. Assuming you don't need the snapshots anymore, the easiest way to cleanup the disk space would be to - at least temporarily - add a second disk (datastore) to the ESXi host, and clone the virtual disk (with consolidating the snapshots) manually using the vmkfstools command line tool. If you have an option to present an NFS share to the ESXi host, this could also be used for the cloning.

André

[nponeccop]

I ended up powering down my VM for 2h and doing vmkfstool -i MyVM-00002.vmdk Smaller.vmdk -d thin

Apparently vmkfstools uses an inefficient buffering so it copied at some 20 MB/sec instead of theoretical 50 MB/sec, as seen in vSphere Client Disk Performance charts. So copying to another physical drive is strongly advised, as it will speed up things 4-fold instead of theoretical 2-fold. Well, at least in my case of old Gen6 HP server with mechanical SAS drives.

As for why blocks are not marked free automatically - I think the root reason is complexity of SCSI standards for thin provisioned storage. You can search vmkb for UNMAP and see all kinds of slow performance and misbehaviour. So we need to wait another 10 years before VMWare paravirtualized adapter properly exposes UNMAP and guest filesystem drivers utilize it without errors or slowdown. Or a genius could invent a conceptually simpler way of doing storage than SCSI standards 1000+ pages long .

[gvmwz]

I have the same problem and didn't find any tool to do this job, so I write my own GitHub - gvmwz/vmfsSparsePunchZero: vmkfstools -K for snapshots

Hope others may find it useful.

[continuum]

Hello ak ka
This seems to be a very useful tool.
Can you please explain how to build it ?
Thank you
Regards Ulli

[gvmvvz]

I have uploaded a prebuilt binary to Release Initial release · gvmwz/vmfsSparsePunchZero · GitHub , you can copy it to ESXi host and run (make backup before proceeding)

./vmfsSparsePunchZero "testvm-000001.vmdk"

mv "testvm-000001-delta.vmdk.new" "testvm-000001-delta.vmdk"

[continuum]

Thank you
Wil van Antwerpen found what was missing.
After adding
#include <string.h>
I was able to built it on a Ubuntu system.
Regards Ulli