ollie2001
Contributor
Contributor

Thin provisioning issue

Jump to solution

Evening all

We recently P2V'd 4 Dell servers which all went OK.

When it came to thin provisioning the storage, we've hit a slight snag. No matter what we do, we can't claw back any space through thin provisioning.

For example, we have one server, 62GB C drive. We've performed the following:

Defragged

Chkdsk'd

Sdelete -c (and even -z followed by another -c)

Then proceeded to migrate to another datastore as thin provisioned. We appear to get 0 space back even though Windows reports there being 31.6GB free disk space.

Don't get me wrong, in the past doing the 4 above steps has always clawed back the free space but just this time round on all 4 servers we can't reclaim anything.

I've even converted back to thick and then to thin in case something was being funny.

Using ESX 4 fully patched across all hosts.The only thing I can think is that when we've thin provisioned in the past, we were running ESX 4 with no patches and we've recently updated everything to the current release but I can't really believe a patch/update would have broken something so fundamental.

Any idea's?

0 Kudos
1 Solution

Accepted Solutions
jfield
Enthusiast
Enthusiast

There is a very simple workaround for this problem, that I found a week or two ago.

Do the usual "sdelete -c C:" stuff, then migrate it as Thin provisioned to a datastore with a **different blocksize**.

Then it all works just how it always did.

It's only when you migrate it to a datastore with the same block size that it is broken.

--

Jules

Jules@Jules.FM

www.Jules.FM

View solution in original post

0 Kudos
29 Replies
ollie2001
Contributor
Contributor

No idea's folks?

I've logged a VMWare support call and I think they're stumped to. Not a great start to our P2V's...

0 Kudos
ollie2001
Contributor
Contributor

VMWare have washed their hands of it as they don't support shrinking a thin provisioned disk, once it's gone it's gone in their eyes.

Surely someone must have a clue?

0 Kudos
amvmware
Expert
Expert

What OS's are you running on these servers?

What storage are you using - does the storage support zero page detection?

0 Kudos
golddiggie
Champion
Champion

Do the VM's have ANY snapshots?? Whenever you take a snap ESX/ESXi allocates/provisions up to twice the amount of storage/space used in the VM. So, if the VM is using 20GB of space, you'll see 40GB provisioned from the console. Try deleting the snapshots for the VM's and see what happens to the provisioned space...

VMware VCP4

Consider awarding points for "helpful" and/or "correct" answers.

Hosted Systems Engineer IV (VMware environment)
Brewing beer again!
0 Kudos
ollie2001
Contributor
Contributor

Windows Server 2003 R2 32bit SP2

We're using VMFS3 datastores with standard VDMK's (windows basic disks) for our VM's.

As I've previously said, this has worked fine in the past on ESXi 4 vanilla, but since updating to Update 2 and all other patches, it's stopped working.

We haven't updated our vCenter in line with the Update 2 release, would this make a difference?

0 Kudos
ollie2001
Contributor
Contributor

No, no snapshots on any of the VM's involved.

0 Kudos
amvmware
Expert
Expert

Yes it would - Vmware recommend you upgrade vCenter first and then the hosts - i Would upgrade vCenter and then see what happens.

0 Kudos
ollie2001
Contributor
Contributor

I've updated vCenter to the latest version for Update 2 (build 258672) and still no joy Smiley Sad

Next week I'm going to try and wipe one of our ESX hosts and take it back to v4.0 vanilla and try another thin provision on that host without any updates and see what happens.

Very frustrating!

Thanks for the comments so far!

0 Kudos
ollie2001
Contributor
Contributor

Things are going from bad to worse...

I tried migrating a thin provisioned VM (a completely different one) from one datastore to another.

It was thin provisioned before hand (in fact, it was clone from a thin provisioned template) with a single 50GB C drive, before the move it was using 24GB of that and afterwards the used storage has gone up to 52GB! (I'm guessing the extra 2GB is the swap file).

So it's basically turned it into a thick disk even though it's stating it's still thin.

And no, no data had changed on the VM to incur this drastic change is usage, the OS is still reporting over 30GB free, I even did a sdelete -c before the move just in case.

Heck, I've even tried creating a new LUN and formatting it separately and migrating over to it with the same results Smiley Sad something's really broke here!

Still really can't believe we're the only organisation in this boat, but I guess there's a first for everything.

Will post back my results next week from wiping a host and installing 4.0 bare on it.

Cheers

0 Kudos
ollie2001
Contributor
Contributor

Spoke to VM Support today.

It appears that the Sdelete tool is in fact the cause of the problem! It would appear a VMWare update (between 4.0 RTM and Update 2) now see's "0's" in a VDMK as actual data, so of course when you run Sdelete -c and it 0's out all the free space, ESX is now thinking all of that is data and thusly expanding the thin provisioned disk to it's maximum.

We're still working on the case but to prove it I'll be rebuilding an ESX host this week with 4.0 with no updates and see how we get on.

Thanks all

0 Kudos
ollie2001
Contributor
Contributor

Just to give everyone an update, VMWare have confirmed it's a bug and are working on it.

They've confirmed it as I rebuilt 2 x ESX servers as 4.0 with no patches and thin provisioned fine on those using the SDelete utility.

No ETA on a fix yet but if I hear anything I'll report back.

0 Kudos
BearHuntr
Contributor
Contributor

You are not the only one seeing this issue, I recently updated my VDI Infrastructure to ESXi 4.1 and I'm now seeing the same thing. Using the script we had always used successfully in the past to defrag and zerowipe yields almost no disk shrinking after an svmotion. For a 10 GB disk, it is showing it as 9.9 GB even though there is over 2 GB available in the OS (Win XP.) What's strange is that my server infrastructure was also updated to ESXi 4.1 using the same disk media a couple months ago and thin provisioning works just fine there. The only thing that I could think of being different between the 2 environments is that I had enabled Storage IO Control on my VDI datastores, while I only have Enterprise licensing on my server, so I didn't have that feature. I have since disable SIOC, but it doesn't seem to be making any difference in shrinking a thin disk. It's a little embarrassing to be instructing my desktop team how to shrink a disk and not have it work properly!

Please keep us informed on what support tells you, I may open a ticket myself since I'm running ESXi 4.1. Thanks!

Scott.

0 Kudos
ollie2001
Contributor
Contributor

At least I'm not alone!

Sorry, I've been on leave for a few weeks and in that time VMWare closed my support call without my permission. Even though I had an out of office on my email. I've rung them today to re-open it but haven't received any confirmation of that yet.

They asked me to generate the system diagnostic logs which I'm having trouble with at the moment as our hosts don't have any internal storage (just the USB module) and they're whinging they don't have enough space to generate the logs, so I've emailed them requesting a way forward from that as well.

If you have any luck with your support ticket, I'm all ears!

We're upgrading to 4.1 at the end of the week and was hoping that might resolve things but if you're already at 4.1 and having the issue I fear it might not fix it after all.

Thanks!

0 Kudos
ollie2001
Contributor
Contributor

OK, had the final word(s) from VM.

They won't support it, so game over. It will be supported in ESX v5 which should be released mid-2011.

It sucks...really sucks...

Thanks for everyone's comments!

0 Kudos
ollie2001
Contributor
Contributor

A huge update for this thread, "jfield" over at this thread http://communities.vmware.com/message/1696468#1696468

Has come up with the solution! Create another datestore with a different block size and re-attempt the above sdelete / svMotion steps and it works as before!!

Just tested from 4MB block size to 8MB and works like a charm now Smiley Happy

Thanks all!

0 Kudos
jfield
Enthusiast
Enthusiast

There is a very simple workaround for this problem, that I found a week or two ago.

Do the usual "sdelete -c C:" stuff, then migrate it as Thin provisioned to a datastore with a **different blocksize**.

Then it all works just how it always did.

It's only when you migrate it to a datastore with the same block size that it is broken.

--

Jules

Jules@Jules.FM

www.Jules.FM

0 Kudos
ollie2001
Contributor
Contributor

Good stuff buddy, cheers Smiley Happy Answered!

0 Kudos
depping
Leadership
Leadership

Working on providing you with an answer on WHY this is actually happening.

Duncan

0 Kudos
depping
Leadership
Leadership

I also mentioned this in the other thread on this topic, but for completeness sake:

It most definitely has to do with the type of  datamover used. When a different blocksize is used for the destination  the legacy datamover is used which is the FSDM. When the blocksize is  equal the new datamover is used which is FS3DM. FS3DM decides if is will  use VAAI or just the software component, in either case unfortunately  the zeroes will not be gobbled. I have validated it and reported it to  engineering that this is desireable. The team will look into it but  unfortunately I cannot make any promises if or when this feature would  be added.

Duncan (VCDX)

Available now on Amazon: vSphere 4.1 HA and DRS technical deepdive

0 Kudos