Virtually Nick

Community ESX(i) Driver Project

nick.couchman — Wed, 25 Feb 2009 17:39:20 GMT

Well, we've started up a community project on SourceForge.net for adding drivers to ESX/ESXi (see link below). I've got a couple of Wiki pages out there describing how to get things going, and I've checked the code into SVN for the first driver that I ported - the Realtek 8169 Gigabit Ethernet driver. I also have an initial binary download for this driver for anyone who wants to try it out. The download is in the form of a .tgz file that you can add to the boot configuration of your existing ESXi system (see the Wiki for detailed instructions). It has the r8169.o module and an updated simple.map file.

We're looking for people to help in developing and testing these drivers. Check out the project page:

http://www.sourceforge.net/projects/open-vdrivers

http://open-vdrivers.sourceforge.net

More ESX Drivers!

nick.couchman — Wed, 04 Feb 2009 04:52:49 GMT

Well, I think I've finally got it to work - I've modified the RTL8169 driver to be supported under VMware ESX and ESXi! I'll post the patch here before too long, and I hope this will be the first of many additional ESX drivers to come!

ESXi + PXE

nick.couchman — Sat, 20 Dec 2008 22:57:32 GMT

Finally...success for PXE-booting ESXi. Many of the sites I found with instructions on ESXi PXE booting require that you VC and insert a little program into ESXi that contacts the server to get configuration information. While this is very, very cool, for those of us who do not have VC, it isn't very useful.

I noticed early on, when booting ESXi from a USB device, that after you configure ESXi and save the configuration, there's a local.tgz file written out to the /bootbank directory, which is on the partition of the USB stick from which ESXi boots. I extracted this file and found all of the customizations that I had made to my ESXi machine. So, I followed some of the ESXi PXE boot instructions and got all of the correct tarballs on the PXE server, then threw in the local.tgz file that had all the customizations. This worked - mostly. The hostname and datastore configuration was all there, but there was one minor annoyance: the SSL keys and SSH keys for the host didn't "stick" - they changed every time the machine was booted.

Next, I stumbled upon the vim-cmd command, which seems to command just about every aspect of the ESXi host. The vim-cmd command has many, many "sublevels" of commands - the hostsvc/firmware/backup_config command was particularly interesting. This writes all of the necessary local configuration information to a file and then spits out an HTTP URL you can use to download the file. Very, very useful - you get a "configBundle-<hostname>.tgz" file, which contains a Manifest.xml and the local.tgz file. I then made the mistake of trying to extract the local.tgz file and throw that on the PXE server...which results in a PSOD.

Finally, I came across the following document: http://docs.google.com/View?docid=ddcwgcd6_4fs6s7jcf. In this document are PXE instructions, and among the ones at the end it says to put the entire configBundle-<hostname>.tgz file on the PXE server. Voila! Instead of extracting the local.tgz file, I just needed to take the entire configBundle file generated by the vim-cmd command and boot the PXE clients with that file. And everything works now!

On the PXE side I'm using pxelinux and am able to create individual configuration files per MAC address. I then create an individual config file for each ESXi host, and each config file specifies the correct configBundle for each host and now I can boot all of my ESXi servers from the networks. I also created an NFS datastore that allows me to share VMs between hosts (easily - and access the datastore without the need for a VMFS filesystem).

As far as the effort to run ESXi on Whitebox machines, this is fantastic - most of the problems getting ESXi up and running are due to ESXi's limited support for hardware - mainly for storage controllers. While I understand VMware's reasoning behind this, I'm not opposed to working around that particular issue with things like this, and I'm glad VMware has created a hypervisor that can be PXE booted. Instead of having to have a compatible storage controller, you just need a compatible network controller, which you can come by pretty easily and inexpensively.

Virtual Machine Portability

nick.couchman — Fri, 19 Dec 2008 14:48:45 GMT

Well, I wrote a while back about portability of PV virtual machines, but I've also just had some experience with portability of fully-virtualized VMs (HVM in Xen language). I have a mixture of ESX and Xen for my virtual machines here, and I like to be able to move VMs back and forth. I've recently moved a couple of Linux-based VMs from ESX to Xen, which turned out to be very, very easy. ESX, it seems, creates two files for each disk - the descriptor file, and a "flat" file. The descriptor file simply tells ESX about the disk and then points over to the flat file for the data. The flat file is nothing more than a raw file - it has a partition table and then the data in each of the partitions. So, to move a VM from ESX to Xen, all you have to do is copy out the Flat file to the Xen box and point Xen at the flat file. You can also tell Xen to use the previous MAC address from VMware.

Going the other way isn't all that hard, either. You can copy the raw file from a Xen VM over to the ESX box, then you just need to create a descriptor file for it. I haven't done this, yet, though I'm sure I will, soon. I doubt you can do it from the VI Client, which will mean a little command line work, but I always liked getting my hands dirty on the CLI. The only trick to this will be that VMware limits the MAC addresses you can set on VMs, which means you may end up with a new MAC address if the original VM has a Xen-ranged MAC address. This is slightly annoying, though understandable, and shouldn't cause too many problems.

Anyway, that's it for now - just nice to know that I can move these things around without too much trouble! Using an NFS datastore would help, too, because I could access VMs from both ESX and Xen. I'm just concerned about NFS performance (though VMFS performance isn't anything to brag about), and I'm concerned about getting an NFS server setup that's not a single point-of-failure.

PV Portability

nick.couchman — Wed, 27 Aug 2008 22:39:00 GMT

It's been a while since I compiled a Linux kernel from scratch. I was upgrading a Gentoo VM today and was reconfiguring the new kernel and saw that, not only is VMI PV support built in to the standard kernel, but so is Xen PV support. And, you can compile both in at the same time. Combine that with Xen's support for VMDK files, and it looks like I now have the possibility of creating PV virtual machines and moving them back and forth between my Xen hosts and my ESX/ESXi hosts. If I use an NFS share, I may be able to have the same set of VMs accessible in both places at the same time.

More WhiteBox Success

nick.couchman — Wed, 27 Aug 2008 01:05:00 GMT

I now have five of six of the SuperMicro machines that I own running ESXi. I'll be working on getting the other one running, too - I'm going to try to boot ESXi off PXE on that one and use it for fiddling with ESXi settings. Among the SuperMicro machines I have running are the SuperServer 5013C-i (P4-based processor), a SuperMicro X5DPA-TGM+ motherboard, and a SuperMicro X5DL8-GG motherboard. They all connect to an Openfiler 2.2 machine via the iSCSI software initiator and share that volume for VMs. Next year I'm going to try to replace these five machines (and a few more) with a couple of 8-core machines and decide what hypervisor to run. Anyway, kudos to VMware for releasing ESXi under a free license - working for me!

I'm still ticked off at them, though, for removing support for permissions at the Pool and VM levels!

Virtual Appliances: Increasing Open Source Popularity

nick.couchman — Wed, 20 Aug 2008 18:43:00 GMT

For those of you who aren't real familiar with me, I'm an avid open source fan - I believe that open source software is a fantastic thing and has proved itself to have many, many benefits. Microsoft actively combats the popularity of open source software by trying to throw things like TCO and ROI at us and then see what sticks. Most of it doesn't stick.

One of the things that I think has created a new burst in the popularity of open source software is the concept of virtual appliances, especially VMware Virtual Appliance Marketplace. If you look at the appliances available in the marketplace, they are predominately Linux appliances, and most of the ones that aren't Linux appliances are other open source operating systems, like OpenSolaris, FreeBSD, NetBSD, OpenBSD, etc.

Why? Licensing. While there are a couple of evaluation copies of Windows-based appliances floating around out there, people don't like the idea of downloading something only to have it expire in 60, 90, 180 days, etc. They want something that they can download and try, then use in production if they decide they like it. Microsoft licensing doesn't lend itself well (at all??) to this type of usage. Open Source licensing does. And the virtual appliance concept gives those who may shy away from the "trouble" of installing Linux, Solaris, *BSD, etc., the opportunity to download and run a machine and see what it's like.

Score another victory for the open source community, and keep posting those open virtual appliances out there!

Is there an echo in here??

nick.couchman — Tue, 19 Aug 2008 13:21:00 GMT

One thing that I've noticed in the forums recently here at VMware is that there seem to be a lot of people who like to respond to a discussion and post exactly the same thing that the people before them posted on a thread. It's like a bad echo in the forums - seriously, if you can't post something original on a thread, wait until you have something original to add to an issue. I'm not claiming to be guilt-free in this respect - I'm sure I've done something similar, but it's something I'll be a lot more careful about in the future. I know everyone wants the points, but get them posting something new and not just repeating what everyone else said.

Can I be a consultant, too?!

nick.couchman — Mon, 18 Aug 2008 20:26:00 GMT

In my recent roamings around the VMware Communities, I notice that there are a lot more people with the little VMware logo by their name. In digging a little deeper it appears that VMware has added a bunch of consultants to their payroll. Maybe I can find a way to get in on the action, too!

Really Annoyed

nick.couchman — Sat, 16 Aug 2008 19:08:00 GMT

Well, I was very impressed that VMware had released ESXi for free, and I have just as suddenly become very unimpressed with them. In ESX 3.0 with VI Client 2.0, you could edit permissions for the host, a resource pool, and/or a VM. I went to adjust permissions on my ESX 3.5 server yesterday and noticed that the tab is now suddenly gone. I went into the ESXi 3.5 client (VI Client 2.5) and found the same. So, I posted a thread and was told that this feature is no longer available in 3.5. I opened an SR, then went over to the VMware KB and found the following arctile, 1004552, which states that this was removed by design! ARE YOU KIDDING ME?! I upgraded to 3.5 and had a feature taken away? I'm not paying any less for 3.5! Also, the price actually went UP recently! And now VMware is removing features?? This makes me really, really mad! The article stated that some similar functionality "may" be restored in the future, but that doesn't help me now. I don't have budget for purchasing Virtual Center, and, reading over the release notes that I read when I installed 3.5, I don't see any mention of this feature being removed.

VMware: please, please, please add it back - removing it is so silly I cannot believe you would do that!

ESXi on Whitebox Hardware

nick.couchman — Fri, 15 Aug 2008 18:58:00 GMT

I've spent the past week playing with ESXi, specifically figuring out which hardware that's not on the official VMware HCL I could get the stuff to run on. First, I found that Dell PCs run ESXi beautifully, straight off a USB stick. I followed the directions out there for transferring from the install image to USB stick and that worked very well. But, I doubt my users are going to understand when I tell them their PCs have been commandeered to run my virtual machines...so...back to the data center.

After clearing out that little bug VMware had in one of their builds, I found that I could successfully install ESXi on the following hardware, none of which is on the HCL:

SuperMicro SuperServer 5013C-i - These are probably 4 or 5 year old P4-based machines. I have three of them with 2-4GB of RAM each and a P4 3.2GHz CPU. The on-board SATA controller is the Intel ICH5R, which happens to be supported by the ata_piix driver present in the ESX(i) kernel. The catch is that the bios must be set up with SATA in "Enhanced" mode and note "legacy" mode, otherwise the SATA connections are seen as IDE drives, not SCSI drives, which prevents VMware from loading them correctly.

SuperMicro X5DPA-TGM+ - This is a motherboard in a SuperChassis that I have here that has 2 x 2.8GHz Xeon processors and 4GB of RAM. It has an onboard Intel Pro 100 and an on-board Intel Pro 1000 ethernet interface, and runs ESXi remarkably well. I don't think I'll be running more than a handful of VMs on it at a time, but I'll take all the ESX(i) servers I can get! The on-board SATA controller is the Adaptec ICH6 controller, which is also supported under the ata_piix chipset. This has to be set up in the BIOS in "combined" mode with the SATA controller in "Native" mode in order for ESXi to see it correctly.

SuperMicro X5DL8-GG -This is a PCI-X motherboard with 2 x 2.8 GHz Xeon processors. This one is a bit more challenging to get running, but I was able to do it at intermittent intervals. First, the on-board Adaptec SCSI controller is the 7902 chipset, which you would think would be supported under the aic79xx or aic7xxx module with ESXi. Unfortunately, these modules crash when loading, so you can't use the on-board SCSI controller. There's no on-board SATA controller, and IDE is out of the question for ESXi, so it has to be an add-in one. I have a couple of LSI Logic 22030-R cards that seemed to work okay. I was also able to boot off USB at one point, however my USB-based KVM system interferes with this older motherboard's ability to see the USB storage device correctly, so it only worked when I wasn't using the USB KVM. Of course, SuperMicro has stopped updating the BIOS for this MB, so there's little hope that USB BIOS support will ever work correctly on it, but maybe I'll be able to kludge my way around this one with the right combo of add-in controllers, USB devices, etc.

Also got ESXi running on a whitebox Intel D865GLC motherboard with 1GB of RAM. The machine has a SATA controller but I booted it off a USB stick as I don't have a SATA disk in the machine. I'm guessing the SATA controller is probably compatible with the ata_piix module in ESXi, but we'll see - I may try that later. This machine has an Intel Pro 100 network card that is recognized by the e100 driver in ESXi. 2.4GHz processor.

That's it, so far. Less successful tests have been done on the Dell D600 laptops, which brings the Purple Screen of Death on trying to boot. My quest to build a massive ESXi server base with all the spare hardware I have lying around will go on!

A Dream, Perhaps...

nick.couchman — Fri, 15 Aug 2008 18:48:00 GMT

Well, I have a vision for hypervisors and VMMs, but one that will likely not happen. See, I have a few legacy applications around the office here that must run on the Sparc architecture, and usually must run on Solaris on Sparc. Now, recently I've started using binary translation applications provided by a certain vendor to run these applications on 64-bit Linux. While this works very well, it occured to me that it would be very, very cool to combine a Sparc emulator into one of the hypervisor sets.

There's a project out there called Qemu that provides Linux binary emulation/translation as well as full system emulation. The project will emulate Sparc, x86(64), PPC, mips, sh6, etc., CPUs for either running Linux binaries on these architectures or for running full system emulators. The degree to which each architecture is supported, especially on the system emulation side, is pretty limited, but it would be really, really cool if you could pull up your favorite virtual machine manager and not only have the choice of which O/S you want to install, but what CPU architecture you'd like to run it on.

I realize there would be several objections to this. First, it kind of violates the line between "virtualization" and "emulation." Virtualization is simply splitting the available architecture between multiple O/Ss and controlling access and isolating the O/Ss from one another. Emulation, on the other hand, requires that CPU instructions be translated from one architecture to the physical architecture running underneath it all. Another issue that would come up is performance, especially given that the emulation must be done.

Still, it doesn't seem all that different to me. I'm just an IT guy who wants to be able to run many virtual machines or guests on a single piece of hardware. You can call it what you like underneath the hood, but it would be really nice to be able to choose the architecture of the guest machine.

Like I said, probably a dream that may never come true, but now it's out there...

ESXi

nick.couchman — Sat, 09 Aug 2008 13:36:00 GMT

Well, I was pretty excited to find out that VMware is now giving away ESXi for free! I have several servers that could benefit from this. While I use Xen in some places, it's only real useful where I have VT-enabled servers so that I can PV & HVM domUs. ESXi gives me the ability to run VMs on some of these others servers where I can't justify purchasing ESX but want the ability to run non-PV VMs.

After spending the past day or so experimenting with ESXi, I've managed to get it to run on some older P4-based SuperMicro servers (SuperServer 5013C-i). ESXi seems to support the SATA ICH5 controllers on these motherboards, so installation is pretty seamless. I've also tried it out on a SuperMicro X5DL8-GG motherboard with less success. First, loading the 7xxx and/or 79xx drivers on this platform fails, even though the motherboard has the on-board Adaptec SCSI adapter (7902, I think). So, on-board SCSI is out. Next, I tried a USB flash drive installation, but this also didn't prove out - the latest BIOS version on this board (circa 2005) still has some issues with USB boot support. I already have a hard enough time with my USB-based KVM switch on this machine, and adding a thumb drive to the mix didn't make the situation any better. So, my only option now is an add-in card supported by ESXi. Problem is that I don't want to spend a lot of money, but most of the chipsets supported by ESXi are "expensive" chipsets, so I need to find a compromise. Also, the chassis has a hot-swap SATA backplane, so a SATA or SAS controller is my best bet. Anyway, I'm sure I'll get that figured out.

Other than that I have a few previous generation Dell servers that may end up running this - a couple of them run VMware Server right now, and a move to ESXi would be a good upgrade, assuming they're supported.