Sdelete a thin provisioned Disk

[myesko]

Hello, im preparing to go through the process of reclaiming some space on thin provisioned vmdks. 

The process will be to sdelete -z the drive first, then set the datamover to the old legacy datamover and Storgage VMotion the disk to reclaim.

Its a Windows 2012 VM.

Sdelete will basically put zeros in the blocks and can expand your thin provisioned drive to the max amount.

But I have a large vmdk that is 16Tb, it was thin provisioned.  The space in Windows got used up a while back, so the vmdk is really close to 16TB.  The users cleaned up about 8 TB, so windows now has 8 TB free, but my vmdk is still around 16Tb hence the wanting to clean up.

My concern is, when I run sdelete -z, will the vmdk grow an addition 8Tb up to 24TB from zeroing out the free space?  I only have about 3 to 4 TB free on the datastore so if it grew an additional 8TB I would run out of space.

Thanks

Mike

[a_p_]

A thin provisioned virtual disk can only grow up to its provisioned size, i.e. its configured/provisioned size.

André

[compwizpro · Accepted solution]

What is the VM hardware version and ESXi version you are running?  Are you on Server 2012 or 2012 R2?

Starting with Windows Server 2012 (R2 I believe) and on VM hardware version 11 with a PVscsi controller, Windows should be able to pass TRIM commands to the ESX upon file deletions which should shrink the VMDK if thin provisioned.  In ESX 6.0, this will not work with CBT enabled by will work in 6.5.  There is a known limitation where depending on the NTFS allocation size, the UNMAP commands might be mis-aligned causing UNMAP commands to be discarded in ESX but that is also fixed in 6.5 and in your case when deleting 8TB of files, should still reclaim more than what you are getting now.

Since you are looking to re-claim at the VMDK level and not the array level, running sdelete might not benefit you here. 

Once you verified the prerequisites above, start by running the defrag / optimize task in Windows to make sure your virtual disk is being detected as Thin-Provisioned.

[myesko]

We are running Esxi 6.0 U3.  The VM hardware level is 11.  The Scsi Controller is a LSI Logic SAS.

So Windows 2012 server is definitely not passing any TRIM commands as the VMDK is close to 16Tb and windows is only using 8TB after deleting 8TB.

So right now my concern is getting that 8TB back on the VMDK level.  We also know we need to reclaim at the LUN level, we occasionally run the Scsi unmap commands to get that space back, looking forward to upgrading to 6.7 at some poing and upgrading the datastores to VMFS 6 so that will be automatic:)

But for now I need to get the 8TB back on the VMDK, so the plan is to run sdelete -z to zero things out in Windows, then change the datamover to the legacy datamover, this will allow Storage VMotion to reclaim the space without any downtime.

But my concern with sdelete will it make the vmdk grow larger then the 16TB?

Thanks

Mike

[myesko]

Hi Andre, so the vmdk is already close to the max of 16TB that we have it configured for.  If it can only grow to 16 TB, but windows has 8TB free and someone dumps 1TB of data in there, then im guessing bad things will start happening if the vmdk cant grow?

[Dave_the_Wave]

The sdelete alone from Sysinternals is not going to shrink thin disks.

You need to Putty into the host, and hole punch the vmdk.

I have explained this here:

Re: HD space for VM (in Web Client)

[compwizpro]

As Andre mentioned, your VMDK will not grow beyond the max size (16TB) even with the 8TB free space in Windows.  The file system behaves oddly where it will write data to logical free space on the drive even after files have been deleted causing the VMDK to bloat beyond the used space on the NTFS file system.  This is where TRIM helps.

If TRIM commands are not working, trying changing the SCSI controller to PVSCSI.  I think that is a requirement for TRIM to be passed through.

If you want to pass TRIM commands to the array from the OS, you will need to set an advanced setting on the ESX host called  EnableBlockDelete to 1.  This will allow you to reclaim deleted blocks on thin LUNs on your SAN when data gets deleted within the guest OS.

YMMV since there caveats with UNMAP in 6.0 in terms of block misalignment, etc. 

Did you verify in Windows Defrag utility the drives are displaying as Thin-Provisioned instead of Hard Disk?

[myesko]

Thanks for all the info, its been helpful, I was not aware of the ability to use Defrag to optimize and trim the vmdk down.

Ive enabled the EnableBlockDelete

Im playing with that now and experimenting.  One problem im having is in 2012 Server R2 the disks aren't showing up as Thin Provisioned.  This seems to be because I have CBT enabled.  So im going to try turning that off and see what happens on some smaller VMs.

Will let you know

Thanks Again

Mike

[Dave_the_Wave]

When I want to "shrink" my thin disked Windows VMs, I sdelete them and then vMotion them to another host (the hole punching is done if you want to keep the VM on the same host).

If you have a way for a guest 2012R2 to do it from within its VM and without it ever being migrated out, please do tell.

[myesko]

Hi Dave, so looks like you don't have to sdelete and SvMotion anymore.  That is what I use to do, but was having issues where the SvMotion wasn't giving back the space due to the VMFS Datastores block sizes being the same. 

But I did my first test and was successful with the suggestion from compwiz and it worked.

So what I needed to do.

First you have to enable the EnableBlockDelete on the hosts by setting them to 1

I had to turn off CBT on the VM in order for this to work.  This is because im on Esxi 6.0.  My understanding is if your on 6.5 or 6.7 you don't have to do this.  I cant keep that off as we use CBT for backups, but my plan is to just disable, get my space back, then turn it back on, most likely my back will need to run a Full again though.

Probably going to upgrade to 6.5 or 6.7 on there next update.  Heard there is still some buggy issues

After that I was able to go into 2012 server and open the Defrag optimizer, then click optimizer and it worked.  My vmdk shrunk down.

In my test I got back about 1 TB which is what I expected.  So it took a while, not sure if it really bogs down your storage system or not so you might want to plan for off peak hours.  It did take a while for mine,

FYI, I also had to update my VM hardware to version 11, the one I tested on was version 8.

When you open the defrag optimizer in Windows 2012, the disks should have a media type of Thin provisioned.

Mine were not until I updated the VM hardware and disabled cbt.

I did not have to change the Scsi card to PVScsci in my testing.

Thanks everyone for the help, Im going to do a little more testing, but don't forsee anymore issues

Mike

[Dave_the_Wave]

So my understanding is that with the EnableBlockDelete enabled, the host will accept commands from the Guest OS, in this case 2012's Defrag and Optimize Drives, and will perform the hole punching scheduled from 2012 ?

Likewise there would be no harm in having EnableBlockDelete enabled anyways, and the admin can still choose to either let the GOS do it all, or sdelete/vmotion manually. I suppose a reason why it isn't enabled by default is so no VM can hijack a host's storage, as you said.

I couldn't find any Changed Block Tracking params in my ESXi6.0 Windows VMs, and Veritas Backup Exec 16 never told me to add it, so I guess I am good there.

In summary:

1. The host must have EnableBlockDelete=1

2. If ESXi6.0, the VM's Changed Block Tracking must not be enabled (or not present)

3. Let 2012's Defrag and Optimize Drives run as it likes?

[compwizpro]

That's great that solution worked for you!

To be clear, the EnableBlockDelete is only needed if you want to pass UNMAP through to the storage backing your VMFS datastore (storage array).  This is helpful in re-claiming space on your array if you are using thin LUNs or if you have dedup and compression enabled.  It shouldn't affect reclaiming storage from your VMDK.

The process gets more efficient and compatible with other settings the with each newer version of ESXi.