Well, I wrote a while back about portability of PV virtual machines, but I've also just had some experience with portability of fully-virtualized VMs (HVM in Xen language). I have a mixture of ESX and Xen for my virtual machines here, and I like to be able to move VMs back and forth. I've recently moved a couple of Linux-based VMs from ESX to Xen, which turned out to be very, very easy. ESX, it seems, creates two files for each disk - the descriptor file, and a "flat" file. The descriptor file simply tells ESX about the disk and then points over to the flat file for the data. The flat file is nothing more than a raw file - it has a partition table and then the data in each of the partitions. So, to move a VM from ESX to Xen, all you have to do is copy out the Flat file to the Xen box and point Xen at the flat file. You can also tell Xen to use the previous MAC address from VMware.

Going the other way isn't all that hard, either. You can copy the raw file from a Xen VM over to the ESX box, then you just need to create a descriptor file for it. I haven't done this, yet, though I'm sure I will, soon. I doubt you can do it from the VI Client, which will mean a little command line work, but I always liked getting my hands dirty on the CLI. The only trick to this will be that VMware limits the MAC addresses you can set on VMs, which means you may end up with a new MAC address if the original VM has a Xen-ranged MAC address. This is slightly annoying, though understandable, and shouldn't cause too many problems.

Anyway, that's it for now - just nice to know that I can move these things around without too much trouble! Using an NFS datastore would help, too, because I could access VMs from both ESX and Xen. I'm just concerned about NFS performance (though VMFS performance isn't anything to brag about), and I'm concerned about getting an NFS server setup that's not a single point-of-failure.

I now have five of six of the SuperMicro machines that I own running ESXi. I'll be working on getting the other one running, too - I'm going to try to boot ESXi off PXE on that one and use it for fiddling with ESXi settings. Among the SuperMicro machines I have running are the SuperServer 5013C-i (P4-based processor), a SuperMicro X5DPA-TGM+ motherboard, and a SuperMicro X5DL8-GG motherboard. They all connect to an Openfiler 2.2 machine via the iSCSI software initiator and share that volume for VMs. Next year I'm going to try to replace these five machines (and a few more) with a couple of 8-core machines and decide what hypervisor to run. Anyway, kudos to VMware for releasing ESXi under a free license - working for me!

I'm still ticked off at them, though, for removing support for permissions at the Pool and VM levels!

I've spent the past week playing with ESXi, specifically figuring out which hardware that's not on the official VMware HCL I could get the stuff to run on. First, I found that Dell PCs run ESXi beautifully, straight off a USB stick. I followed the directions out there for transferring from the install image to USB stick and that worked very well. But, I doubt my users are going to understand when I tell them their PCs have been commandeered to run my virtual machines...so...back to the data center.

After clearing out that little bug VMware had in one of their builds, I found that I could successfully install ESXi on the following hardware, none of which is on the HCL:

• SuperMicro SuperServer 5013C-i - These are probably 4 or 5 year old P4-based machines. I have three of them with 2-4GB of RAM each and a P4 3.2GHz CPU. The on-board SATA controller is the Intel ICH5R, which happens to be supported by the ata_piix driver present in the ESX(i) kernel. The catch is that the bios must be set up with SATA in "Enhanced" mode and note "legacy" mode, otherwise the SATA connections are seen as IDE drives, not SCSI drives, which prevents VMware from loading them correctly.

• SuperMicro X5DPA-TGM+ - This is a motherboard in a SuperChassis that I have here that has 2 x 2.8GHz Xeon processors and 4GB of RAM. It has an onboard Intel Pro 100 and an on-board Intel Pro 1000 ethernet interface, and runs ESXi remarkably well. I don't think I'll be running more than a handful of VMs on it at a time, but I'll take all the ESX(i) servers I can get! The on-board SATA controller is the Adaptec ICH6 controller, which is also supported under the ata_piix chipset. This has to be set up in the BIOS in "combined" mode with the SATA controller in "Native" mode in order for ESXi to see it correctly.

• SuperMicro X5DL8-GG -This is a PCI-X motherboard with 2 x 2.8 GHz Xeon processors. This one is a bit more challenging to get running, but I was able to do it at intermittent intervals. First, the on-board Adaptec SCSI controller is the 7902 chipset, which you would think would be supported under the aic79xx or aic7xxx module with ESXi. Unfortunately, these modules crash when loading, so you can't use the on-board SCSI controller. There's no on-board SATA controller, and IDE is out of the question for ESXi, so it has to be an add-in one. I have a couple of LSI Logic 22030-R cards that seemed to work okay. I was also able to boot off USB at one point, however my USB-based KVM system interferes with this older motherboard's ability to see the USB storage device correctly, so it only worked when I wasn't using the USB KVM. Of course, SuperMicro has stopped updating the BIOS for this MB, so there's little hope that USB BIOS support will ever work correctly on it, but maybe I'll be able to kludge my way around this one with the right combo of add-in controllers, USB devices, etc.

• Also got ESXi running on a whitebox Intel D865GLC motherboard with 1GB of RAM. The machine has a SATA controller but I booted it off a USB stick as I don't have a SATA disk in the machine. I'm guessing the SATA controller is probably compatible with the ata_piix module in ESXi, but we'll see - I may try that later. This machine has an Intel Pro 100 network card that is recognized by the e100 driver in ESXi. 2.4GHz processor.

Well, I was pretty excited to find out that VMware is now giving away ESXi for free! I have several servers that could benefit from this. While I use Xen in some places, it's only real useful where I have VT-enabled servers so that I can PV & HVM domUs. ESXi gives me the ability to run VMs on some of these others servers where I can't justify purchasing ESX but want the ability to run non-PV VMs.

After spending the past day or so experimenting with ESXi, I've managed to get it to run on some older P4-based SuperMicro servers (SuperServer 5013C-i). ESXi seems to support the SATA ICH5 controllers on these motherboards, so installation is pretty seamless. I've also tried it out on a SuperMicro X5DL8-GG motherboard with less success. First, loading the 7xxx and/or 79xx drivers on this platform fails, even though the motherboard has the on-board Adaptec SCSI adapter (7902, I think). So, on-board SCSI is out. Next, I tried a USB flash drive installation, but this also didn't prove out - the latest BIOS version on this board (circa 2005) still has some issues with USB boot support. I already have a hard enough time with my USB-based KVM switch on this machine, and adding a thumb drive to the mix didn't make the situation any better. So, my only option now is an add-in card supported by ESXi. Problem is that I don't want to spend a lot of money, but most of the chipsets supported by ESXi are "expensive" chipsets, so I need to find a compromise. Also, the chassis has a hot-swap SATA backplane, so a SATA or SAS controller is my best bet. Anyway, I'm sure I'll get that figured out.

Other than that I have a few previous generation Dell servers that may end up running this - a couple of them run VMware Server right now, and a move to ESXi would be a good upgrade, assuming they're supported.