Virtual Appliances

I've been looking at the community virtual appliances and the uncompressed sizes range from 8 MB to over 700 MB.

What is everyone's opinions on a good size for one? Obviously, the smaller the better and the size will all depend on what is inside, but at what point does it become too big?

Hi Tyler,

I personally believe that you should use as metric the actual stardard size for USB keys.

I think that the most sold today is the 256MB. So, I woudn't create a Virtual Appliance occupying more that this size (including VMware Player and documentation).

HTH

Alessandro Perilli, CISSP, MVP

http://www.alessandroperilli.com

Blogging about IT Security on http://www.securityzero.com

Blogging about Virtualization on http://www.virtualization.info

Thanks. That brings up an interesting question of usable disk space for your appliance.

Assuming that you want your total image to be 256 MB, you allow the appliance to have 64 MB of memory and you are using a Linux/BSD OS. You still need to have around 64 MB of swap space, which leaves you with around 192 MB of usable disk space (actually less than that bc of file system internal stuff). You lose even more if you give it 128 MB of memory.

Granted, you could always go with less swap to increase the amount of usable disk space and I think if you build your appliance correctly it should never hit any swap, but I don't think you can build any modern OS without swap (I've never tried).

Trevlix - don't read Allesandros suggestion to strictly.

I doubt you will have good results when you start with:

"I got 25MB to fill - what can I do that fits ...?"

I'd suggest that you think about a task and a convincing solution first - if your VM can find someones "Love of your life" in 5 seconds and this function depends on a large image-collection of little green frogs - 300MB for a image-database is more than justified.

It doesn't make sense to pay the 256MB size with a 50% divorce-rate 5 years later :smileywink:

Ulli

Don't worry, I'm not reading the suggestion too strictly. I was looking for suggestions and he was actually thinking along the same lines as me.

My worry is that the size of an appliance may be an offset for the audiance my appliance is being built for (consumers). I want to have the right functionality but don't want it to be too big to deter downloads and installations. Right now, I'm shooting for 256-500 MB total size uncompressed which I think is a good trade-off and \_should_ give me all of the space for the functionality I require.

Anyone want to post what they are shooting for size-wise? Just curious here.

Assuming all said till now is a winning strategy I really believe the problem here is the starting focus.

I have the impressions many are trying to "simply" fit a whole Linux distro inside the virtual appliance with some nifty application on top.

I'm not sure this is enough to win $100,000 (and sincerely I hope not).

Coming from the IT security planet an appliance is an extreme tailored OS around a hardware set, doing nothing but the task advertised.

This approach leads to a very small, very fast, very simple-to-manage (well, apart Cisco stuffs) boxes companies not having money to invest in know-how are happy to buy.

I think this approach should be applied in a virtual appliance too, considering average RAM available in an average desktop computer and average media vectors (IMHO thinking the Virtual Appliance will move around by CDRom is not a positive starting point).

So, at the end of the day, I'm wondering how space could take a Linux kernel without anything but 1 or 2 packages (let's say even 10 of them) to provide the feature you're looking for. More than 192MB?

(just consider Damn Small Linux is desktop distribution working in 50mb space).

BTW: you may find interesting VMware Challenge suggestions:

http://www.virtualization.info/2006/03/vmware-challenge-suggestions.html

HTH

Alessandro Perilli, CISSP, MVP

http://www.alessandroperilli.com

Blogging about IT Security on http://www.securityzero.com

Blogging about Virtualization on http://www.virtualization.info

Allesandro - You make very good points and I agree with it all. I sincerely hope that the winning appliance is not a full Linux install. To me, that kinda defeats the purpose of the contest.

As I am designing mine, one of my tasks is to get my chosen distro down to as small as possible. I think there are a number of things you can do in order to do this, including recompiling the kernel, removing un-necessary packages and files, etc. I'm really very excited to see what comes out of the contest in this area especially since I've been spending alot of time on this.

I also want to say that I don't agree with your thumb drive idea. That never occured to me and really opens up some new possibilities for virtual appliances.

oops - double post

This goes against the norm, smaller the better. When you surf through the entries are you going to down load a 2 gig double zipped monster or a 20-30 meg VM. The smaller, cleaner and more dedicated to its task the VM is, will result in more down loads even if it is just to see what it does.

A VM based on the Linux Disk Router Project, could be very small maybe 3-5 megs.

>A VM based on the Linux Disk Router Project, could be very small maybe 3-5 megs.

Actually, there's one out there with FreeSco, configured, @ 1.41 MB. Check out vmguru.com :smileyhappy:

Ok. so its a 2GB limit. Does that include everything the appliance needs? What if the appliance needs a passthrough lun or a vmdk mounted for storing data?

is that to be included in the 2GB as well?

thanks

Ok. so its a 2GB limit. Does that include everything

the appliance needs? What if the appliance needs a

passthrough lun or a vmdk mounted for storing data?

is that to be included in the 2GB as well?

Bear in mind that the whole idea of the appliance is that it should be pre-configured; the idea is that you should be able to download the thing and it should do whatever it does without any additional configuration on the part of the end user. Consider in particular that the Appliance should be able to be run in VMware Player, which has no UI for making configuration changes. So your entry should probably not include any instructions like "Download this appliance, then add a new .vmdk file, then...".

Anyway I think the rule is that the file limit applies to the compressed size of your entry, period. I.e. whatever it is you upload to our server to be judged (and therefore, whatever a potential user would download to use) has to fit in the size limit. If your VM had a growable .vmdk that started out empty but grew once it was used, I think that would be ok as long as the VM started out under the limit.

I doubt that there can be many "really" usufull apps that come fully preconfigured. Even the most basic apps usually still have "some" configuration that has to be done to get it to work. The simpler the better. I hope that the VApp (VirtualApp:) that wins will be the most popular (=most used) one. It should be noted that one of the three entry categories in the competition is "Consumer". (Not just for Developers or Server admins.) I belive that is the most important category. If we can come up with an VApp that will have MILLIONS of users, that would truly be worth the effort (and prize.)

Sam

thanks for that clarification.

as long as it runs on Player, it qualifies right?

but what about in an enterprise level environment where more data could be stored or perhaps the retention period for data is longer and so it occupies more space?

in that case, 2GB won't be nearly enough i think.

OK... Not sure how much clearer this can be...

Your Zip file which you submit must be no larger than 2 GB. If it extracts out and is 100 Gb in size it is still legitimate! (not that I believe anything could be compressed quite that much, but I'm trying to make a point here).

Thought I would post that my VM w/tools is 202 megs, without tools installed 87 megs. I will submit both versions, maybe?

Including Tools is a good idea, but it's not mandatory.

That's an awfully big difference, much larger than the amount of useful stuff we install as part of the Tools. Are you sure you don't still have the Tools tar file and/or the vmware-tools-distrib directory wherever you unpacked the tar? The install file is a lot bigger than the stuff you need, because we include kernel modules for a whole bunch of supported guests. Only one set of modules is actually needed for a given guest OS though.

The binaries and modules that you actually need shouldn't be more than a handful of megabytes in size, so you can probably figure out what else is being added that you don't actually need and remove it.

I found that in my install the VMWare tools were the biggest thing because I had to install with the tar file. However, I trimmed it down to only a couple of MBs by getting rid of a bunch of modules that I would never use.

I like to use a seperate vmdk for all of my volatile data (/tmp, /var, etc.) This not only means that I can easily purge everything after tools installation and/or testing, it allows the possibility of the end-user replacing the drive with a larger version if they have more data (say, in a database) than I expected.

Sorry for the slow reply, I did not realise anyone had responded to me.

I have cleaned out all the tar files in the distro. Also all the temp files. The compression is winzip and these amounts are from the Windows Platform, on my Fedora 4, the tar file is a little smaller at 82 megs and then about the same at 202 with the tools.

Thanks

As noted by others in this thread, you should be able to trim that down a lot by removing extraneous modules that are not used by your guest distro.

OK - Here's something that no-one has mentioned.

While building my VM & updating the packages installed, the used space has increased very minimally, however the size of my expandable vmdk has increased significantly - as has the compressed (zipped) vmdk.

What I think is happening is that the non-zeroed free space has increased (due to temp files etc that have been deleted), thus increasing the size of the compressed image.

To test this I uncompressed an image and deleted ~250mb of cached rpms, confirmed the amount of free space had increased by the same amount with df, then recompressed the image - result: same sized image (within 5mb).

Does anyone know of a linux tool to zero out free space? I know that this will increase the size of the uncompressed image (It's not massive anyway), but my real aim is to reduce the size of the compressed image as much as possible.

-Thoughts???

-Andrew

OK - after much effort on google....

In each filesystem that you need to zero out, run the following command:

dd if=/dev/zero of=filler bs=1000

what this does is fills all free space in the filesystem with a single file called filler, which is filled with binary zeros. Once the command predictably dies and moans that the disk is full, you must delete the file:

rm filler

I have run this on a VM (without deleting anything else) and reduced the zipped size successfully from 786881 kb to 390247 kb, which is a hell of a lot easier to download....

-Andrew

In each filesystem that you need to zero out, run the

following command:

dd if=/dev/zero of=filler bs=1000

Once the command

predictably dies and moans that the disk is full, you

must delete the file:

rm filler

I have run this on a VM (without deleting anything

else) and reduced the zipped size successfully from

786881 kb to 390247 kb, which is a hell of a lot

easier to download....

Thanks for the posting. I'll be trying it out in the near future.

I'm going to presume that you're using non-preallocated disks. In VMware Server, at least, there's an option to defragment a virtual disk. What happens if you defragment after zapping the drive space?

Here are some other techniques that I'm using.

I've got two identical VMs, one is a dev system, the other is the "final". As I get things working on my dev box, I mount the other's disks and copy the files into it. This avoids a lot of disk writes to the pristine copy.

Some people use RAM-disks for /tmp, but I've created an independent non-presistent disk. Either has the feature that /tmp is cleared after each reboot; mine allows the possibility of pre-loading /tmp with caches and such.

I've also thought about making my boot drive non-persistent, and putting all of my data on a second drive that contains /home. That would make it easier to clean up \*everything*, not just /tmp.

I'm going to presume that you're using

non-preallocated disks.

That is correct, however this doesn't seem to make the vmdk any/much bigger...

In VMware Server, at least,

there's an option to defragment a virtual disk. What

happens if you defragment after zapping the drive

space?

Probably nothing. Haven't tried it. I thought that the defrag option just defags the vmdk within NTFS, although I could be wrong - it has happened in the past... (in fact I welcome confirmation or correction on this).

Some people use RAM-disks for /tmp, but I've created

an independent non-presistent disk. Either has the

feature that /tmp is cleared after each reboot; mine

allows the possibility of pre-loading /tmp with

caches and such.

I like your idea - I might have to use that in my next release - not just for tmp, but for /var/cache/yum as well. My entry has already been posted and I have no plans to re-submit...

I've also thought about making my boot drive

non-persistent, and putting all of my data on a

second drive that contains /home. That would make it

easier to clean up \*everything*, not just /tmp.

Maybe, but that could be a pain for configs & updates...

-Andrew

http://www.global-domination.org

I've also thought about making my boot drive

non-persistent, and putting all of my data on a

second drive that contains /home. That would make

it easier to clean up \*everything*, not just /tmp.

Maybe, but that could be a pain for configs &

updates...

Actually, I think it would make things easier. One disk would be a non-persistent root, containing your typical 3rd-party LAMP installation,

the other would be persistant and have your code, data, etc. Strategic parts of the first disk's filesystem would be sym-linked into the second drive; i.e. "/var/http" would be a sym-link to "/home/http", etc. You could update the two disks independently of each other, easing your migration headaches. There are versions of Knoppix that, IIRC, do something similar to this.