Manual Automation

Bye, Bye ESXi

Virtual_JTW — Wed, 20 May 2009 19:19:07 GMT

What a long, frustrating trip it's been! Don't get me wrong, I really like the idea of ESXi: thin, fast install, small foot-print, BIOS-like host configuration, no Console OS (COS) to patch or support, can run from embedded USB key, etc, etc. But, my experience in supporting and managing an ESXi-based VI production environment tells a different story.

I've decided to convert all of my hosts from ESXi to ESX "Classic". There are three primary reasons:

Support
Reliability
Compatibility

Support

Without the COS it's difficult to execute commands and view logs files "real-time". I've had more than one VMware support engineer complain about this during a trouble-shooting session (so it must be true!). There are alternatives: using the unsupported trick to get to the command line from the host's console, hacking SSH to open it up (which is also unsupported), capturing logs/diagnostic bundles via vCenter Server, RCLI, VIMA, etc. But none of the alternatives are as fast/clean/easy as SSH'ing right into the COS and working from there.

Reliability

I purchased eleven Hewlett-Packard USB keys w/unlicensed ESXi to embed in my ProLiant DL380 G5s. When I upgraded them via VMware Update Manger (VUM), the entire installation on the key became corrupted. I was not even able to revert back to the previous ESXi image on some keys. HP has since issued a customer advisory and I have replaced all of the keys: http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c01605187

Unfortunately, this experience still leaves me with a less-than-fuzzy feeling for running ESXi on said embedded USB keys in critical production environments.

Compatibility

ESXi seems to lag ESX Classic in updates - specifically when it comes to compatibility. This implies that ESX Classic is developed, tested and certified against first. I manage two VI environments located in different datacenters. I use SRM at the primary for DR/fail-over to the secondary (see previous articles). It took around a month for VMware to release a patch for ESXi Update 3 that made it compatible with SRM 1.0 Update 1. Read your compatibility guides! More on this in a future article.

Many third-party tools and scripts require the COS. There are many examples of this: Snap Hunter, Vizioncore vOptimizer Pro, etc.

Unlike ESX Classic, HA in ESXi requires a ScratchConfig folder created on separate VMFS datastores for each host. This may not be a big deal for smaller clusters, but for a cluster with many servers, many datastores will be required.

Finally, for those of us that run HP servers to host ESXi, we have a specific firmware/ISO that contains the HP management providers. Unfortunately, even with the built-in providers you still can't monitor disk status - which is, of course, the one hardware component that fails the most often(!). As of this writing, here's what I've been ale to determine:

Licensed ESXi

HP only supports ESXi with the proper "management providers" on Update 2 and Update 3 (as evidenced by VMware's downloads section of their web site). ESXi 3.5 Update 4 is not yet supported.
Using VMware Update Manger to upgrade ESXi instances to Update 4 effectively breaks SIM manageability.

Free ESXi

A new installable image is available for Update 4 with the management providers.
Upgrading exiting hosts via the VMware Infrastructure Update tool effectively breaks SIM manageability. These hosts will have to be reinstalled from the ISO.

Once you're sure you have the right ESXi firmware image installed, it's time to add the host to HP Systems Insight Manager (SIM) for hardware monitoring. I was able to add only 3 out of about 20 of my ESXi hosts successfully. HP support wasn't able to help me out. The main suggestions I got were to reinstall(?) and to call VMware. With ESX Classic you install the Insight agent in the COS, add the host to SIM, and you're done. It just works.

Conclusion

Like I mentioned previously, I still really like the idea of ESXi. Once I saw that Hitachi was embedding a virtualization solution in their servers I knew it was only a matter of time before VMware came out with something similar.

I have many free ESXi installable instances. This is a great solution in cases where the budget it tight or non-existent. Utilizing the free ESXi still gives you many of the benefits of virtualization making it a better way to go than bare metal OS installation in most cases.

I think embedded/thin is the future. I hope vSphere 4 embedded improves on the issues described above.

5-20-2009 UPDATE:
Not being one to spread any FUD, I would to add to my comment "Using VMware Update Manger to upgrade ESXi instances to Update 4 effectively breaks SIM manageability". According to this article on the Yellow Bricks blog (http://www.yellow-bricks.com/2009/05/14/updating-an-esxi-server-with-vendor-agents/), VUM has the intelligence to download and apply the correct ESXi firmware image (i.e. the one with whatever OEMs management providers are pre-installed). And I believe it because Ducan is "the man". Note that I used the Virtual Infrastructure Update Tool, not VUM, so that may have made a difference. Regardless, I do know that after updating the image HP Insight Manager was no longer seeing all of the hardware components and in some cases failed to communicate to the server completely. This hasn't to me happened on my ESX "classic" hosts.

Home Lab Build – Part 1

Virtual_JTW — Mon, 08 Sep 2008 14:38:39 GMT

My home lab has changed dramatically over the years – driven mostly by what I was working on at the time and the availability of hardware. I hadn’t updated my lab in quite a while so I decided it was time. I was also inspired by Chad’s post as to how cheaply I could build a server or two: http://virtualgeek.typepad.com/virtual_geek/2008/06/building-a-home.html

The Hardware

Motherboard: $30; ECS NFORCE6M-A rev 3.0 (http://www.newegg.com/Product/Product.aspx?Item=N82E16813135083). This thing is sweet – capable of 32GB! Not sure I’ll ever need that much but, wait… what am I saying; of course I’ll need that extra memory some day.
CPU: $64; CPU AMD Athlon 64 X2 4800+ Dual-Core 2.5GHz AM2 purchased from local computer shop.
RAM: $94.50; 4GB = 2x2GB DDR2-800 (PC2-6400) 2GB Supertalent also purchased from local computer shop. I prefer to do this when the price is the same or close to NewEgg within a few bucks.
Case w/PS: $39.50; ATX RAIDMAX Elite Black ATX/Micro ATX Case 380 watt power supply also purchased from local shop. Cheap case, thin metal – you get what you pay for especially when it comes to computer cases.
Video: $0; I had 2 cheapo SiS PCI cards lying around.
HDD: $0; I’m using ESXi so local storage is not necessary.
NIC: $0; I had 4 Intel 1000MT Server NICs I repurposed from other systems I’m not using. I put 2 in each server.
USB Key: $0; I had 2x2GBers I wasn’t using.
Total Spent = $228 per server. Not bad!

I decided to go with AMD to keep the costs down. Note that the motherboard has since been delisted at NewEgg. ECS has a similar model but it’s more expensive. Now to be fair, this means that all of your storage is going to have to be on a third server. I already had a storage server in my home lab but it needed some updating:

Motherboard: $0; P4 2.6GHz – repurposed from an older PC I wasn’t using.
CPU Heatsink: $14; purchased from local computer shop. Need to replace original since the fan exhaust was directed in the wrong direction per the design of the original case.
Power Supply: $67; 580 watt from local shop.
HDD: $64; purchased another 250GB SATA3 drive to fill out my SATA RAID 4 port PCI adapter with 3 other drives.
Total Spent = $145

Hey, this is getting expensive! I sold some older systems and parts I wasn’t using on EBay to help cover some of the costs. The dominos finally stopped falling.

The Install

I installed Windows Server 2008 as the storage server OS on 2 RAID1/mirrored 160GB IDE drives. The 4 250GB HDDs are setup in a RAID5 logical drive.

For ESX, I used ESXi installed on a USB key per these instructions: http://communities.vmware.com/blogs/Knorrhane/2008/01/21/installing-esx-3i-on-usb-stick

Okay great, so now I have two ESX servers up and running and the backend storage running. Now I just need to create a datastore on the first ESX server. ESX supports Fibre, iSCSI and NFS storage types. Microsoft provides good NFS support in Windows so that seems like the easiest way to go. I installed the File Storage role in Windows Server 2008 and included the Server for NFS feature.

Snag! Microsoft no longer provides User Name Mapping for NFS – you basically need to install the Unix integration component for Active Directory. Well, my domain controller was going to be installed in a VM. I can’t create a VM w/o storage, so now what?

Stay tuned and for the answer revealed in Part 2!

Environment Stabilization: Hardware

Virtual_JTW — Mon, 09 Jun 2008 03:41:52 GMT

There's nothing like going in to work Monday morning only to find that one of your ESX hosts is listed as "not responding" in VirtualCenter. Using HP's iLO, I tried restarting the management network. No change. The VMs were still running and functioning normally. The host was still running - there just seemed to be a communications problem between the host and VirtualCenter. After a quick call to VMware technical support, they had me restart the VirtualCenter server service and voila, communications were restored and the host's status in VirtualCenter returned to normal.

I didn't spend a lot of time doing a root-cause analysis as this environment was not in production yet. But I suspected there was a network interruption from which host-VC communications never recovered.

Now let me just say something about VMware technical support. I've worked support incidents with many hardware and software vendors over the years and have to say that VMware has their act together when it comes to product support. I'm not saying they're perfect, but I've received consistent quality support from these guys going back to my ESX Server 1.5 days. They're worth the money and I wouldn't run a Virtual Infrastructure environment without them.

So a week later, it happens again. I open another case with VMware referencing the previous. This time, restarting the management network or the VirtualCenter server service doesn't work. The support tech reviews some additional logs and is basically stumped. The only thing he had left for me to do was to manually shut down the VMs running on this host and reboot the host. This fixes the problem, but doesn't really explain why it happened in the first place. The tech is going to review a new set of logs I just uploaded and let me know if he finds anything. While there wasn't much more that could be done at this point, this always seems to me like a "don't call me, I'll call you" kind of resolution.

Before he has a chance to call me back, it happens a third time on the same host! Same symptoms, same results. The same tech doesn't find anything in the logs from the previous incident so he escalates to senior-level VirtualCenter support. We discovered a new symptom - the host seems to have lost connectivity with the storage, even though the VMs are still running fine (strange but true).

The senior tech said something that jogged my memory and I remembered that while this server survived our 4-day hardware burn-in test, we had problems connecting to the management console very early on to the point where we had to pull the USB key fob and reinstall it. (Keep in mind we're running ESXi.)

To be safe, I installed a new USB key fob and the problem has not occurred again. It's been about three weeks since writing this entry. Moral of the story: don't automatically rule-out the hardware even when the problem appears to be with the software.