Unable to shrink or otherwise compact the virtual disk

[Haelix]

Hello.

I have the following situation. I wanted to backup my VM (vmdk + adjacent files) by zipping it in the host OS, but it was too big. So I wanted to exclude some huge data that I had downloaded (~8GB) by temporarily moving it somewhere outside the guest.

The problem is that the vmdk doesn't shrink. After removing 8GB of data the guest filesystem amounts to ~4GB of data however the size of the vmdk is still 16GB (the same as it was at its highest). I repeatedly tried the "compact" and "defragment" VM tools (in the host OS) but size didn't change. Then I found out about vmware-tools utility in the guest OS (ubuntu), but "shrink" function is not available since I use ext4. I don't understand the essences of "journaling file systems" but it seems that there exists this limitation.

Then I took a different approach. Since there was no way to reclaim the unused virtual disk space, I tried to wipe it using "sfill" utility from the secure-delete ubuntu package. I was expecting - since the actual filesystem data i have is 4GB - that wiping the free space would work towards a smaller resulting zip file - since there is a lot of redundancy with the free space zeroed. No luck: same ~10GB resulting zip file.

Yet another thing surprised me during the wiping process. Wiping works by writing a huge file with zeroes in it. Since my vmdk is not pre-allocated I was aware that I needed to stop the wiping at some point, in order not to expand the vmdk unnecessarily. I carefully monitored guest used filesystem space and vmdk size, but surprise: at some point the zero-file got to 16GB; add the 4GB of ubuntu files and the vmdk should have been over 20GB, but it was only ~15GB. It's as though the zeroes were nowhere written... judging also by the similarly-sized zipped vmdk.

Could someone shed light into this confusion, and maybe some advice?

a) is there a "natural" way to shrink vmdk's that host an ext4 filesystem? so that the vmdk file is smallest possible

b) does VMware Player use some kind of compression for the data in the vmdk? or otherwise how to explain vmdk file size sometimes being smaller than guest filesystem usage

Thank you in advance and sorry for the long post - you may just read the questions at the end

-- Haelix

[Haelix]

Hasn't anyone tried to shrink virtual disk with ext4 filesystem?

Like for example someone using Ubuntu 10.04 Lucid?

Anyone?

[Haelix]

Please, doesn't anyone know anything about this?

[wila]

Hi,

I repeatedly tried the "compact" and "defragment" VM tools (in the host OS) but size didn't change. Then I found out about vmware-tools utility in the guest OS (ubuntu), but "shrink" function is not available since I use ext4. I don't understand the essences of "journaling file systems" but it seems that there exists this limitation.

I have not looked at shrinking ext4 partitions myself yet, but is there are message somewhere that this is not supported for ext4?

Then I took a different approach. Since there was no way to reclaim the unused virtual disk space, I tried to wipe it using "sfill" utility from the secure-delete ubuntu package. I was expecting - since the actual filesystem data i have is 4GB - that wiping the free space would work towards a smaller resulting zip file - since there is a lot of redundancy with the free space zeroed. No luck: same ~10GB resulting zip file.

Wiping free space does not shrink the virtual disk in the host OS, the supported way to do this is to use the shrink option in vmware-tools as you found.

Could someone shed light into this confusion, and maybe some advice?

a) is there a "natural" way to shrink vmdk's that host an ext4 filesystem? so that the vmdk file is smallest possible

The most natural way might be to clone your disk to a new virtual disk. Then it will only take up the size of your data, not any zero-d out space that is now still using disk on your original ext4 partition.

b) does VMware Player use some kind of compression for the data in the vmdk? or otherwise how to explain vmdk file size sometimes being smaller than guest filesystem usage

No it doesn't really compress, but zero's do not have to take up space on the host OS. Overwriting existing data with zero's doesn't reclaim the space on the host OS. That's exactly what the shrink process is supposed to do for you, but as it doesn't appear to support ext4, you're out of luck with using that tool.

Hope this helps,

--
Wil
_____________________________________________________
VI-Toolkit & scripts wiki at http://www.vi-toolkit.com

Contributing author at blog www.planetvm.net

Twitter: @wilva

[Haelix]

Thanks, what you say makes sense, what do you mean by "cloning" the disk? It has to be something done from within the guest OS right?

I will probably end up switching to ext3... I wonder if I can do that with a system partition.

[wila]

Hi,

From what I understand it should be possible to change your ext4 partition back to ext2 and then to ext3 (maybe even directly to ext3).. but I'm not sure if I would do that in your case.

Ext4 shrinking support is most likely on its way.

As for how-to shrink now? Look at the link here

http://vsphere-land.com/tips-tricks/re-sizing-virtual-disks

I'm basically talking about method 5.

Hope this helps,

--
Wil
_____________________________________________________
VI-Toolkit & scripts wiki at http://www.vi-toolkit.com

Contributing author at blog www.planetvm.net

Twitter: @wilva

[admin]

This appears to be a known bug (no KB article yet). VMware Tools doesn't support ext4 partitions (the default for Ubuntu 9.10). The issue affects all VMware products e.g. Workstation, Fusion, ESX.

For ESX, there's a patch available:

• KB 1017740 - VMware ESXi 4.0, Patch ESXi400-201006202-UG: Updates Tools

No mention of available fixes for all other products.

[continuum]

Like with any other filesystem that can not be handled by the VMware-tools the procedure goes like this ...

1. inside the guest: create a wiperfile for every single partition - or in other words - do this for any entry in fstab separately

Standard Unix:

This works on any Unix variant - Linux, NetBSD, Solaris, etc.:

dd if=/dev/zero of=/0bits bs=20971520 # bs=20m

rm /0bits

see http://theatomicmoose.ca/g4u/#shrinkimg

2. power down the guest and use vmware-vdiskmanager -k disk-with-ext4.vmdk

This is how I shrink all my OpenBSD-VMs




_________________________

VMX-parameters- WS FAQ -[ MOAcd|http://sanbarrow.com/moa241.html] - VMDK-Handbook

You also find me in the support crew of PHD Virtual Backup

[Haelix]

Hello and thank you for you suggestion.

I am wondering, does the zeroing expand the vmdk to its full maximum size? I have configured a 250GB hard disk but currently only 15GB are allocated. It would be a dissapointment for this to happen.

[osde_info]

also see http://vizz.info/2010/09/vmware-tools-cant-shrink-edxubuntu-10-x-ext4-partitions/

regards

clive

http://vizz.info

[Haelix]

Hi,

I uninstalled the vmware-tools that came with my installation of vmware player and installed open-vm-tools from the Ubuntu repositories. Looks like this version (8.4.0) is able to shrink my ext4 partition.

This ext4 thing seems to make a lot of mess in conjunction with virtualization. Besides the enormous growth of the vmdk, I get very poor disk performance overall. It looks like data disk access patterns are very scattered. I'm trying to compact, defragment & everything else that i can in order to make things better.

[Haelix]

Oh and thanks again to everybody for advice.

[wila]

Good to hear you got the main issue resolved.

>It looks like data disk access patterns are very scattered. I'm trying to compact, defragment & everything else that i can in order to make things better.

You can actually find out if the files are fragmented. In the host OS run:

filefrag -v FILE

Where FILE is the name of your virtual disk (the vmdk file)

Hope this helps,

Forum Upgrade Notice - the VMware Communities forums will be upgraded the weekend of December 12th. The forum will be in read-only mode from Friday, December 10th 6 PM PST until Sunday, December 12th 2 AM PST.
--
Wil
_____________________________________________________
VI-Toolkit & scripts wiki at http://www.vi-toolkit.com

Contributing author at blog www.planetvm.net

Twitter: @wilva

[Anatol56]

Hi,
I Try the VMware Workstation 11 before i was using 10.04, so my host Systems are Ubuntu 14.04 64-bit and Windows 7 64-Bit so i install the new Version of VMware 11 on both systems and after i used view days i like to Compact an Guest OS witch was a Ubuntu 14.10 64-bit i user sudo dd if=/dev/zero of=/0bits bs=20971520 # bs=20m
in the Guest OS to free unused diskspace and then delete the 0bits file with sudo rm /0bits.
the powerd off the VM and go to Options / Harddisk and use the Compact funktion after 5 Seconds it says the Disk was successfully compacted but when i watch on the size still was and will be 65GB so it does nothing!

So i check a cuople of things to get sure its not the Filesystem itself (i mean bad sectors on the Real Harddrive) but this is all ok no errrors where found and anything other works fine.
So i Uninstall the VMware 11 and Reinstall the 10.04 Version and again Compacting this VMDK and after doing it with Vmware 10.04 the size is only 3.6GB so in my Point i think theres a Problem with VMware 11.

Can anyone out there reproduce or got the same Problems?

By the way, i also try to Compact the disk on The Windows 7 Host with VMware 11 and 10 and its the same problem vmware 11 dose not working but vmware 10 Compacts fine.

Here are some Data about the system im Working on:

AMD 4x 3,6 GHz

8GB Ram

2 Hard drives 1. 500Gb for the Systems

and 2. 1TB As Data/Work Drive the Working Drive is Formatet on NTFS.
(I also try to shrink the VMDK on a Ext4 Partition and it is not Working! Only with the VMware Version 10.04 it still Compact the VMDK)

So in that Case Can you help me or tell me what im Doing wrong?

Greetz Anatol

[DavidWoodward]

Anatol56, I found a work around for the WS11 and Player 7 bug.

The VDDK has an application that shrinks and defragments virtual disks.

Shrink/Compact command:  vmware-vdiskmanager.exe -k "<path to vmdk>"

Defrag command: vmware-vdiskmanager.exe -d "<path to vmdk>"

https://my.vmware.com/group/vmware/get-download?downloadGroup=VDDK55U2