.vmss files on ramdisk?

[jonmccune]

Hello,

I have between 50 and 100 linux VMs on ESX Server 3.0.2 that boot from a customized CD image and run with only a ramdisk (i.e., no writable virtual block devices). Each VM is configured with 80MB RAM. Presently I am able to boot these VMs, take snapshots, and then revert-to-snapshot. In my experiments, reverting to snapshot takes between 10 and 20 seconds for a single VM. I would love to get this down to under 1 second. I have two questions:

1. Can I use one snapshot for all VMs? Each revert-to-snapshot entails an 80 MB read from disk (from what I can tell), and each VM has its own snapshot, meaning that any caching of VMSS contents (if it even does this?) by the VMM is unlikely

to help. The only necessary customization of a VM is for networking purposes.

2. Can I store my one snapshot on a ramdisk inside the service console? I noticed that the service console has 20 64 MB ramdisks available.

I realize that I could answer my own questions by experimenting with creating ramdisks and editing the VMX files of my VMs, but I'm hoping to save time by getting some advice from somebody experienced first.

Thanks!

-Jon

[wila]

1. Can I use one snapshot for all VMs? Each revert-to-snapshot entails an 80 MB read from disk (from what I can tell), and each VM has its own snapshot, meaning that any caching of VMSS contents (if it even does this?) by the VMM is unlikely

to help. The only necessary customization of a VM is for networking purposes.

If you only want to revert to the original disk after usage you could throw away the snapshot and re-use the original disk from before the snapshot. It sounds like it should be possible to do this quicker... if that's really what you are after.

2. Can I store my one snapshot on a ramdisk inside the service console? I noticed that the service console has 20 64 MB ramdisks available.

I would strongly advise against it, The console is used for managing all your VMs and is "just a VM" itself. It is supposed to be mean and lean. By doing this you would probably loose support on the host.

hth,

Wil

[jonmccune]

Hi Wil,

Thanks for the input. Are you suggesting that I perform a fresh boot of each VM? I have been using the snapshots because they capture the VM's state once it is already booted. Given the snapshot performance I observe, perhaps it would be faster to optimize the boot process for my VMs?

What about using one snapshot for all my VMs?

Thanks,

-Jon

[wila]

Well little differences in what you want can mean totally different solutions. First let's step back a bit.

The way it works with taking a snapshot of a VM is that ESX will use a Copy On Write layer for the snapshots.

Basically like this; quiesce your VMs, then close ALL files, create a new disk that will contain delta files (a sort of redo logs). When commiting/deleting the snapshot it goes the other way around and the delta files are either discarded completely OR merged with the original disk.

You also have the option to add the current memory contents of your VM to this (those are your .vmss files, the "vmware suspend state" files, basically your RAM contents)

That should answer your first question, you cannot just use the memory of VM 1 and apply it to VM 2.

The normal way to speed the snapshot process itself up is by using faster hardware, most likely by getting a faster storage array. It doesn't have to take 10 seconds, i'm sure, it doesn't take that long here.

The only necessary customization of a VM is for networking purposes.

Can't you use DHCP for that? Sounds like it would do the trick.

Another solution might be to approach this in a knoppix sort of way, so instead of using the COW filesystem from the snapshot use it in the VM and just reset the VM when you want to start fresh.

Luckily you are using linux so i'm sure there's ways to get around it. It doesn't have the additional "We want to have ALL the ram you assigned to me at boottime as in Windows" tax for example which does put some extra strain on the host during the boot process.

--

Wil

[jonmccune]

My understanding is that the CoW layer and REDO logs are for vmdk disk images. These VMs don't have any disk images, except for an ISO file backing their virtual CDROM drives. Thus, a snapshot without including the memory contents would essentially be an empty file.

What is the major obstacle to using the memory (VMSS contents) of VM 1 and applying it to VM 2? Two virtual machines can be identical.

Interesting that your snapshots are faster. The machine I'm using is fairly powerful (4x1.8 GHz Xeon, 8 GB RAM), and the snapshots are being stored on a locally attached SCSI disk.

I can use DHCP for the network, if I'm able to `revertsnapshot` my VMs to an identical state. If I can't use a single VMSS file to `revertsnapshot` more than one VM, though, then I can do DHCP before taking each VM's snapshot in the first place.

Thanks,

-Jon

[jonmccune]

I meant to say .vmsn files, not .vmss files. My understanding is that .vmsn files store snapshot information (e.g., RAM contents) and that .vmss files store state when a VM is "paused". I am interested in creating one .vmsn file and using it to rapidly spawn a lot of identical VMs.

Sorry for the confusion,

-Jon

[wila]

.vmsn files / vmss files, i knew what you meant and was too lame myself to verify although i wondered about it.

You are correct on the .vmdk snapshot files, they are for the disk only. If you boot from .iso, then that should not be needed.

As for memory contents on systems being the same for all of them. That would appear impossible to me, to have an exact copy of the memory on all machine being the same? No, statistically impossible and i don't like statistics.

Even if you assign all the machines the same IP address, same machine name, same everything, they will be different as they won't have the exact same state at the same time. One VM for example will get a few CPU cycles more as another one, the MAC address will be different and plenty more.

If i where you i would go down the DHCP path.

With using one base vmsn, that might work if all your VMs are based of the exact same master VM, but i would expect network problems and more fun like that. I also don't see it get you more speed in commiting the snapshot.

[wila]

Jon,

Maybe what you need is a "linked clone", apparently something that is a feature in Lab Manager and vmware workstation, but it might work on ESX. I have no experience with this myself yet.

Quote from: http://www.pcmag.com/article2/0,2704,1785619,00.asp

"The linked clone feature makes it possible to set up a shared store of predefined VM templates on the network; individual users then create linked clones of these using very little local storage."

and:

http://communities.vmware.com/thread/104583

http://communities.vmware.com/thread/43300;jsessionid=08B0C16E5814158679324C3DDDE5D111?tstart=0