Finally...success for PXE-booting ESXi. Many of the sites I found with instructions on ESXi PXE booting require that you VC and insert a little program into ESXi that contacts the server to get configuration information. While this is very, very cool, for those of us who do not have VC, it isn't very useful.
I noticed early on, when booting ESXi from a USB device, that after you configure ESXi and save the configuration, there's a local.tgz file written out to the /bootbank directory, which is on the partition of the USB stick from which ESXi boots. I extracted this file and found all of the customizations that I had made to my ESXi machine. So, I followed some of the ESXi PXE boot instructions and got all of the correct tarballs on the PXE server, then threw in the local.tgz file that had all the customizations. This worked - mostly. The hostname and datastore configuration was all there, but there was one minor annoyance: the SSL keys and SSH keys for the host didn't "stick" - they changed every time the machine was booted.
Next, I stumbled upon the vim-cmd command, which seems to command just about every aspect of the ESXi host. The vim-cmd command has many, many "sublevels" of commands - the hostsvc/firmware/backup_config command was particularly interesting. This writes all of the necessary local configuration information to a file and then spits out an HTTP URL you can use to download the file. Very, very useful - you get a "configBundle-<hostname>.tgz" file, which contains a Manifest.xml and the local.tgz file. I then made the mistake of trying to extract the local.tgz file and throw that on the PXE server...which results in a PSOD.
Finally, I came across the following document: http://docs.google.com/View?docid=ddcwgcd6_4fs6s7jcf. In this document are PXE instructions, and among the ones at the end it says to put the entire configBundle-<hostname>.tgz file on the PXE server. Voila! Instead of extracting the local.tgz file, I just needed to take the entire configBundle file generated by the vim-cmd command and boot the PXE clients with that file. And everything works now!
On the PXE side I'm using pxelinux and am able to create individual configuration files per MAC address. I then create an individual config file for each ESXi host, and each config file specifies the correct configBundle for each host and now I can boot all of my ESXi servers from the networks. I also created an NFS datastore that allows me to share VMs between hosts (easily - and access the datastore without the need for a VMFS filesystem).
As far as the effort to run ESXi on Whitebox machines, this is fantastic - most of the problems getting ESXi up and running are due to ESXi's limited support for hardware - mainly for storage controllers. While I understand VMware's reasoning behind this, I'm not opposed to working around that particular issue with things like this, and I'm glad VMware has created a hypervisor that can be PXE booted. Instead of having to have a compatible storage controller, you just need a compatible network controller, which you can come by pretty easily and inexpensively.
