Observation regarding thin provisioning

[mikelane]

I just created a VM with a thin provisioned 750Gb hard drive.

/dev/sda1 has used only 1.4Gb of space so far (after installation of OS, VMWare Tools and Applications) yet my vmdk is 25Gb.

Is there really 23.5Gb of overhead when creating a large thin provisioned disk of this size?

I guess I am just surprised that there is so much overhead ... it's not like losing 24Gb out of 750 is such a big deal though.

I do like the flexibility that thin provisioning gives me though - they take a long time to convert or download though ...

[RaySperle]

What is Thin Provisioning?

[AndreTheGiant]

Which kind of OS have you installed? Windows 2008 or Vista?

Thin disk are disk from the first block to the last used block, so if you have fragmentation your thin disk could be greater than the used space.

To resolve the problem you have to shrink the disk.

Andre

[mikelane]

Thin provisioning is where you specify the size of your virtual hard drive (for example 100Gb) but the actual size that the virtual disk uses on your storage is just the actual space that the OS and installed files actually occupy.

[mikelane]

I am using Ubuntu JeOS, which is roughly 300Mb installed.

After that I copied and extracted and installed VMWare Tools, installed a LAMP stack, and then uploaded a small application which was also extracted and installed.

I assumed that the size of the vmdk was the overhead of having such a large hard drive configured ... I am not yet familiar with shrinking linux disks (if that can be done).

[AndreTheGiant]

I am not yet familiar with shrinking linux disks (if that can be done).

I depends by your filesystem type.

Are you using ext3?

Andre

[mikelane]

Yes, I am using ext3

[AndreTheGiant]

Sorry, I do not know if there is a "shrink" utility for ext3.

I remember that only reiserfs can do (but probably there are some tools for the other linux filesystems).

Andre

[Peter_vm]

If I remember correctly, in thin provisioned disks, physical space used reflects number of virtual disk sectors that data was written to, not virtual filesystem used space.

So If you had written (for example) 1GB of data to a file, then deleted that file, physical space on host disk will still account for that 1GB.

[DSTAVERT]

I would be careful creating thin disks especially where there is so little actually used. There are cases where copying, or moving etc. where the disk will grow to the provisioned size. Don't get caught out. Especially with Linux where it is so easy to mount additional drives to gain space. IMHO create enough space for your OS and applications. Add data space as an additional drive or drives. Get the flexibility up front.

[mikelane]

The application in question is an apache webserver - so how would I add drives and link them together so to speak?

I'm in the process of putting as much of the setup of the webserver into a script so I can easily recreate the server at this point.

I do not know how to mount a secondary drive just for /var/www data or how to mount additional drives for /var/www should the first one fill up ... if I remember correctly you can add a large number of disks using scsi so there should not be any real limitation that I would run in to there ...

The point that you make about copying and moving files growing the disk is enough to concern me with this application as media is uploaded to the webserver and converted to .flv (a "youtube" kind of server - so the last thing I want is for the disks to grow on the basis of this kind of operation).

[DSTAVERT · Accepted solution]

It isn't so much about uploading files to the server it is about needing to move or copy your vmdks like when you are running out of space on the host.

To break up data from os add a second vmdk. Partition and format the disk from within the VM.

fdisk /dev/sdb

mkfs.ext3 /dev/sdb1

mkdir /var/www/html/videos

mount /dev/sdb1 /var/www/html/videos

It is easy to add additional data drives as needed. Just create additional directories /var/www/html/videos1 /var/www/html/videos2 etc.

You will need to use the man pages for fdisk directions but it is very simple.

[mikelane]

Just to make sure that I understand this process correctly ...

On my webserver the .flv files are written to /var/www/media/video on the "operating system" disk where apache2 is installed (and this folder will quickly fill up if I am sizing my OS vmdk as recommended).

When I add my "data" disk(s) I need a folder on this disk to become an extension of /var/www/media/video on the OS disk.

So in therory I could create /var/www/media/video (or similar) on the "data" disk and create a symbolic link between /dev/sda1/var/www/media/video and /dev/sdb1/var/www/media/video?

[DSTAVERT]

What I am suggesting is that your OS partition should be sized for the OS and the basic website. 10GB 20GB or similar. The data drive where the videos are stored can be any size but it could be expanded at will based on need at the time using additional VMDKs.

The /var/www/media/video is just a directory and you mount the data disk to that directory. The new data disk is attached and visible in the video directory. Looking at the file structure you would not be aware that the video directory was a separate drive. The VMDK that the videos were stored on could be on a totally separate storage device NFS or iSCSI.

[mikelane]

That process worked for mounting a single drive - so I assume that it will work for additional drives too - thanks very much