Blog Posts

Finally...success for PXE-booting ESXi. Many of the sites I found with instructions on ESXi PXE booting require that you VC and insert a little program into ESXi that contacts the server to get configuration information. While this is very, very cool, for those of us who do not have VC, it isn't very useful.

I noticed early on, when booting ESXi from a USB device, that after you configure ESXi and save the configuration, there's a local.tgz file written out to the /bootbank directory, which is on the partition of the USB stick from which ESXi boots. I extracted this file and found all of the customizations that I had made to my ESXi machine. So, I followed some of the ESXi PXE boot instructions and got all of the correct tarballs on the PXE server, then threw in the local.tgz file that had all the customizations. This worked - mostly. The hostname and datastore configuration was all there, but there was one minor annoyance: the SSL keys and SSH keys for the host didn't "stick" - they changed every time the machine was booted.

Next, I stumbled upon the vim-cmd command, which seems to command just about every aspect of the ESXi host. The vim-cmd command has many, many "sublevels" of commands - the hostsvc/firmware/backup_config command was particularly interesting. This writes all of the necessary local configuration information to a file and then spits out an HTTP URL you can use to download the file. Very, very useful - you get a "configBundle-<hostname>.tgz" file, which contains a Manifest.xml and the local.tgz file. I then made the mistake of trying to extract the local.tgz file and throw that on the PXE server...which results in a PSOD.

Finally, I came across the following document: http://docs.google.com/View?docid=ddcwgcd6_4fs6s7jcf. In this document are PXE instructions, and among the ones at the end it says to put the entire configBundle-<hostname>.tgz file on the PXE server. Voila! Instead of extracting the local.tgz file, I just needed to take the entire configBundle file generated by the vim-cmd command and boot the PXE clients with that file. And everything works now!

On the PXE side I'm using pxelinux and am able to create individual configuration files per MAC address. I then create an individual config file for each ESXi host, and each config file specifies the correct configBundle for each host and now I can boot all of my ESXi servers from the networks. I also created an NFS datastore that allows me to share VMs between hosts (easily - and access the datastore without the need for a VMFS filesystem).

As far as the effort to run ESXi on Whitebox machines, this is fantastic - most of the problems getting ESXi up and running are due to ESXi's limited support for hardware - mainly for storage controllers. While I understand VMware's reasoning behind this, I'm not opposed to working around that particular issue with things like this, and I'm glad VMware has created a hypervisor that can be PXE booted. Instead of having to have a compatible storage controller, you just need a compatible network controller, which you can come by pretty easily and inexpensively.

I now have five of six of the SuperMicro machines that I own running ESXi. I'll be working on getting the other one running, too - I'm going to try to boot ESXi off PXE on that one and use it for fiddling with ESXi settings. Among the SuperMicro machines I have running are the SuperServer 5013C-i (P4-based processor), a SuperMicro X5DPA-TGM+ motherboard, and a SuperMicro X5DL8-GG motherboard. They all connect to an Openfiler 2.2 machine via the iSCSI software initiator and share that volume for VMs. Next year I'm going to try to replace these five machines (and a few more) with a couple of 8-core machines and decide what hypervisor to run. Anyway, kudos to VMware for releasing ESXi under a free license - working for me!

I'm still ticked off at them, though, for removing support for permissions at the Pool and VM levels!

I've spent the past week playing with ESXi, specifically figuring out which hardware that's not on the official VMware HCL I could get the stuff to run on. First, I found that Dell PCs run ESXi beautifully, straight off a USB stick. I followed the directions out there for transferring from the install image to USB stick and that worked very well. But, I doubt my users are going to understand when I tell them their PCs have been commandeered to run my virtual machines...so...back to the data center.

After clearing out that little bug VMware had in one of their builds, I found that I could successfully install ESXi on the following hardware, none of which is on the HCL:

• SuperMicro SuperServer 5013C-i - These are probably 4 or 5 year old P4-based machines. I have three of them with 2-4GB of RAM each and a P4 3.2GHz CPU. The on-board SATA controller is the Intel ICH5R, which happens to be supported by the ata_piix driver present in the ESX(i) kernel. The catch is that the bios must be set up with SATA in "Enhanced" mode and note "legacy" mode, otherwise the SATA connections are seen as IDE drives, not SCSI drives, which prevents VMware from loading them correctly.

• SuperMicro X5DPA-TGM+ - This is a motherboard in a SuperChassis that I have here that has 2 x 2.8GHz Xeon processors and 4GB of RAM. It has an onboard Intel Pro 100 and an on-board Intel Pro 1000 ethernet interface, and runs ESXi remarkably well. I don't think I'll be running more than a handful of VMs on it at a time, but I'll take all the ESX(i) servers I can get! The on-board SATA controller is the Adaptec ICH6 controller, which is also supported under the ata_piix chipset. This has to be set up in the BIOS in "combined" mode with the SATA controller in "Native" mode in order for ESXi to see it correctly.

• SuperMicro X5DL8-GG -This is a PCI-X motherboard with 2 x 2.8 GHz Xeon processors. This one is a bit more challenging to get running, but I was able to do it at intermittent intervals. First, the on-board Adaptec SCSI controller is the 7902 chipset, which you would think would be supported under the aic79xx or aic7xxx module with ESXi. Unfortunately, these modules crash when loading, so you can't use the on-board SCSI controller. There's no on-board SATA controller, and IDE is out of the question for ESXi, so it has to be an add-in one. I have a couple of LSI Logic 22030-R cards that seemed to work okay. I was also able to boot off USB at one point, however my USB-based KVM system interferes with this older motherboard's ability to see the USB storage device correctly, so it only worked when I wasn't using the USB KVM. Of course, SuperMicro has stopped updating the BIOS for this MB, so there's little hope that USB BIOS support will ever work correctly on it, but maybe I'll be able to kludge my way around this one with the right combo of add-in controllers, USB devices, etc.

• Also got ESXi running on a whitebox Intel D865GLC motherboard with 1GB of RAM. The machine has a SATA controller but I booted it off a USB stick as I don't have a SATA disk in the machine. I'm guessing the SATA controller is probably compatible with the ata_piix module in ESXi, but we'll see - I may try that later. This machine has an Intel Pro 100 network card that is recognized by the e100 driver in ESXi. 2.4GHz processor.

That's it, so far. Less successful tests have been done on the Dell D600 laptops, which brings the Purple Screen of Death on trying to boot. My quest to build a massive ESXi server base with all the spare hardware I have lying around will go on!