Finally...success for PXE-booting ESXi. Many of the sites I found with instructions on ESXi PXE booting require that you VC and insert a little program into ESXi that contacts the server to get configuration information. While this is very, very cool, for those of us who do not have VC, it isn't very useful.

I noticed early on, when booting ESXi from a USB device, that after you configure ESXi and save the configuration, there's a local.tgz file written out to the /bootbank directory, which is on the partition of the USB stick from which ESXi boots. I extracted this file and found all of the customizations that I had made to my ESXi machine. So, I followed some of the ESXi PXE boot instructions and got all of the correct tarballs on the PXE server, then threw in the local.tgz file that had all the customizations. This worked - mostly. The hostname and datastore configuration was all there, but there was one minor annoyance: the SSL keys and SSH keys for the host didn't "stick" - they changed every time the machine was booted.

Next, I stumbled upon the vim-cmd command, which seems to command just about every aspect of the ESXi host. The vim-cmd command has many, many "sublevels" of commands - the hostsvc/firmware/backup_config command was particularly interesting. This writes all of the necessary local configuration information to a file and then spits out an HTTP URL you can use to download the file. Very, very useful - you get a "configBundle-<hostname>.tgz" file, which contains a Manifest.xml and the local.tgz file. I then made the mistake of trying to extract the local.tgz file and throw that on the PXE server...which results in a PSOD.

Finally, I came across the following document: http://docs.google.com/View?docid=ddcwgcd6_4fs6s7jcf. In this document are PXE instructions, and among the ones at the end it says to put the entire configBundle-<hostname>.tgz file on the PXE server. Voila! Instead of extracting the local.tgz file, I just needed to take the entire configBundle file generated by the vim-cmd command and boot the PXE clients with that file. And everything works now!

On the PXE side I'm using pxelinux and am able to create individual configuration files per MAC address. I then create an individual config file for each ESXi host, and each config file specifies the correct configBundle for each host and now I can boot all of my ESXi servers from the networks. I also created an NFS datastore that allows me to share VMs between hosts (easily - and access the datastore without the need for a VMFS filesystem).

As far as the effort to run ESXi on Whitebox machines, this is fantastic - most of the problems getting ESXi up and running are due to ESXi's limited support for hardware - mainly for storage controllers. While I understand VMware's reasoning behind this, I'm not opposed to working around that particular issue with things like this, and I'm glad VMware has created a hypervisor that can be PXE booted. Instead of having to have a compatible storage controller, you just need a compatible network controller, which you can come by pretty easily and inexpensively.

Well, I have a vision for hypervisors and VMMs, but one that will likely not happen. See, I have a few legacy applications around the office here that must run on the Sparc architecture, and usually must run on Solaris on Sparc. Now, recently I've started using binary translation applications provided by a certain vendor to run these applications on 64-bit Linux. While this works very well, it occured to me that it would be very, very cool to combine a Sparc emulator into one of the hypervisor sets.

There's a project out there called Qemu that provides Linux binary emulation/translation as well as full system emulation. The project will emulate Sparc, x86(64), PPC, mips, sh6, etc., CPUs for either running Linux binaries on these architectures or for running full system emulators. The degree to which each architecture is supported, especially on the system emulation side, is pretty limited, but it would be really, really cool if you could pull up your favorite virtual machine manager and not only have the choice of which O/S you want to install, but what CPU architecture you'd like to run it on.

I realize there would be several objections to this. First, it kind of violates the line between "virtualization" and "emulation." Virtualization is simply splitting the available architecture between multiple O/Ss and controlling access and isolating the O/Ss from one another. Emulation, on the other hand, requires that CPU instructions be translated from one architecture to the physical architecture running underneath it all. Another issue that would come up is performance, especially given that the emulation must be done.

Still, it doesn't seem all that different to me. I'm just an IT guy who wants to be able to run many virtual machines or guests on a single piece of hardware. You can call it what you like underneath the hood, but it would be really nice to be able to choose the architecture of the guest machine.

Like I said, probably a dream that may never come true, but now it's out there...