VMware
5 Replies Last post: Feb 16, 2009 9:57 PM by Dave.Mishchen…  

Warning -- faulty HP ESXi USB Keys! posted: Nov 26, 2008 3:12 PM

Click to view MitC's profile Novice
MitC
21 posts since
Jun 8, 2006

I just came across this HP advisory, and I didn't see any postings about it in the forums. If I missed one, I apologize. I have experienced this problem myself.

There is a known problem with some USB flash drives that HP shipped. Please see the advisory below. Note that the color of the USB key seems to be the determining factor.

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c01605187

In case the link goes dead, here's the text of the advisory:

Release Date: 2008-11-13

Last Updated: 2008-11-13

DESCRIPTION

HP has identified an issue that may occur when updating the configuration of a ProLiant server running HP Integrated VMware ESX Server 3i on certain USB Keys. Two types of VMware ESX 3i Server USB Keys have been shipped. However, this issue ONLY occurs when using the USB Key with the green aluminum casing. The USB Key with the black plastic casing functions properly.

The following two issues were detected during testing:

Issue 1: Configuration settings are lost upon rebooting supported ProLiant servers running VMware ESX 3i Server because there is not sufficient time during shutdown to write all configuration data to the affected USB Key.

A second issue may occur that is unrelated to Issue 1, but is related to the USB Key:

Issue 2: On rare occasions, the server may not boot properly due to a hardware issue related to the USB key.

The USB Keys that are affected were shipped between March 31, 2008 and September 15, 2008. All USB Keys (with green aluminum casing) have been purged from inventory. Effective September 16, 2008, the VMware ESX Server 3i USB Keys with green aluminum casing were replaced with a VMware ESX Server 3i USB Key encased in black plastic. The spare kit part number remains the same (490749-001).

SCOPE

Any of the following ProLiant servers running HP Integrated VMware ESX Server 3i Version 3.5 Update 1 or Update 2 on certain USB Keys. Affected USB Keys can be identified by the green aluminum casing.

Affected servers:

  • ProLiant ML350 G5 Server
  • ProLiant ML370 G5 Server
  • ProLiant DL360 G5 Server
  • ProLiant DL380 G5 Server
  • ProLiant DL385 G2 Server
  • ProLiant DL580 G5 Server
  • ProLiant DL585 G2 Server
  • ProLiant BL460c G5 Server Blade
  • ProLiant BL465c Server Blade
  • ProLiant BL465c G5 Server Blade
  • ProLiant BL480c G5 Server Blade
  • ProLiant BL680c G5 Server Blade
  • ProLiant BL685c Server Blade

The complete list of supported HP ProLiant servers with VMware ESX Server 3i is available at the following URL:

http://h71028.www7.hp.com/enterprise/cache/505363-0-0-0-121.html

RESOLUTION

Customers can obtain a replacement USB Key under warranty by contacting HP Customer Support and requesting Spares Kit 490749-001. HP will also provide the instructions on how to obtain the Recovery CD Image for the replacement VMware ESX Server 3i USB Key.

Note: The only VMware ESX Server 3i USB Keys that are affected are the USB Keys with the green aluminum casing.

How to Obtain a Replacement USB Key

In North America, contact Customer Support at 1-800-HP-INVENT.

OR

For Customer Support outside of North America, click on the following URL for a list of Customer Support Centers and appropriate phone numbers:

http://welcome.hp.com/country/us/en/wwcontact.html

When Calling HP Customer Support

Refer to Customer Advisory c01605187 VMware ESX Server 3i - CUSTOMER ACTION REQUIRED: Configuration Information May Not Be Retained After a Reboot When Running VMware ESX Server 3i USB Key Version 3.5 Update 1 or Update 2 from Certain USB Keys on ProLiant Servers.

The Customer Support Agent will assist customers with the process of obtaining a Recovery ISO Image to create a Recovery CD. With the Recovery CD, customers can create new VMware ESX Server 3i USB Key(s). Before calling HP Customer Service, the customer should have the following required information available:

  1. VMware ESX Server 3i USB Key(s) Spares Kit 490749-001 (Only USB Keys with green aluminum casing are affected).
  2. Serial Number for each ProLiant BL/DL/ML server that is using an affected VMware ESX Server 3i USB Key.
  3. Customer Advisory c01605187.

HP Integrated VMware ESXi Server Product and Licensing:

VMware licensing and product serial number information is required to reestablish the VMware ESXi Server Licensing. The information provided here is taken from the HP integrated VMware ESX Server 3i Getting Started Guide. To reference the Getting Start Guide in full refer to the following URL:

http://h20000.www2.hp.com/bc/docs/support/SupportManual/c01399744/c01399744.pdf?jumpid=reg_R1002_USEN

The HP integrated VMware ESXi products use a serial number based license technology. The serial number that enables the software and provides additional license entitlements is delivered electronically on the USB key. View the Support Information section of the Direct Console User Interface or the VMware Infrastructure Client to obtain the USB Key for the serial number information.

HP strongly recommends recording a copy of the product serial number after the initial system boot. Your serial number contains important license information and should be recorded and saved for later reference.

Backing Up the VMware ESX Server 3i Configuration Files

The instructions listed below are provided to customers that may want to save the current VMware ESX Server 3i configuration files before applying the new image to the USB Key. However, this is a manual procedure as there is no mechanism to backup VMware ESX Server 3i configuration files that reside on a USB Key.

IMPORTANT: The following instructions can be used to back up the VMware configuration files.

Download and install the VMware Remote CLI software located at the following URL:

http://www.vmware.com/download/download.do?downloadGroup=VI-RCLI-U2

Refer to the VMware ESX Server 3i Embedded Setup Guide, Chapter 3 - Backing and Restoring ESX Server 3i located at the following URL:

http://www.vmware.com/pdf/vi3_35/esx_3i_e/r35/vi3_35_25_3i_setup.pdf

To backup the VMware ESX Server 3i configuration files there is a Remote Command Line (RCLI) command that allows backup, copy, and restore of configuration information, specifically vicfg-cfgbackup.

This command has the following main options:

    • save : save all state information to a local file (local to the RCLI system).

    • load : restore all state information from a local file.
Tags: esxi, usb, faulty

Re: Warning -- faulty HP ESXi USB Keys!

1. Jan 9, 2009 4:19 AM in response to: MitC
Click to view omahaomaha's profile Lurker
omahaomaha
1 posts since
Jan 9, 2009
Thanks, this is very helpful. I experienced the same issue with the USB key for VMware ESXi and HP Xenserver select edition. Now I know it's a true hardware issue.

Re: Warning -- faulty HP ESXi USB Keys!

2. Feb 13, 2009 4:48 AM in response to: MitC
Click to view Simon.H's profile Enthusiast
Simon.H
48 posts since
Oct 1, 2007
Thanks for posting this MitC! :D

I stumbled across this discussion after an HP ESXi server (with a green aluminium internal USB memory stick) had some trouble the other day.

I had been rearranging the virtual networks (to split some test networks onto a new vSwitch) and basically it said one of the networks was still in use, even though it had no VMs attached. I had also regenerated the SSL self-signed certfificate which I think also needs a reboot to be take effect. However after the reboot ESXi came up but the networking didn't (e.g. I couldnt' ping the host) and I was gettting nasty messages like "VmHostRaidProviders Can't found any controller...", "Peer table full for sfcbd" and "init fn user failed with: Out of resources!".

The upshot was that I got it working again after using the console to change the IP address on the management network to a new value and restarting the network (and then changing it back afterwards. Since then everything else seems OK - it retained the new SSL, passwords etc, although it did lose the VM inventory.

I have spoken to HP who are sending a replacement USB key (part 490749-001) though I think this is a blank key and I expect getting the image sorted could be "challenging".


One thought though - the HP advisory says:

"Configuration settings are lost upon rebooting supported ProLiant servers running VMware ESX 3i Server because there is not sufficient time during shutdown to write all configuration data to the affected USB Key"

So this would suggest the memory is not actually faulty, just not fast enough. Therefore my question is, can we tell ESXi to give more time to write the configuration data to the USB key before rebooting?

I don't really understand how the /bootbank files get updated. All the files in the /bootbank directory have the same date (last year) except for local.tgz which is dated yesterday (more recently than the above fix, perhaps when something else was changed?). Does local.tgz get created when you do a reboot; if so, by what?

Re: Warning -- faulty HP ESXi USB Keys!

3. Feb 14, 2009 12:41 AM in response to: Simon.H
Click to view Dave.Mishchenko's profile Guru User Moderators vExpert
Dave.Mishchen…
9,151 posts since
Nov 15, 2005
I don't really understand how the /bootbank files get updated. All the files in the /bootbank directory have the same date (last year) except for local.tgz which is dated yesterday (more recently than the above fix, perhaps when something else was changed?). Does local.tgz get created when you do a reboot; if so, by what?

The files you see in /bootbank are the firmware files that ESXi extracts each time it boots up. It's the 32 MB of compressed files that make up the code for ESXi. Those files will be static until you upgrade your version. When you do that, the new version of ESXi is written to /altbootbank/ and when ESXi reboots, altbootbank become the new bootbank, and the old bootbank becomes altbootbank.

Local.tgz is a backup of the configuration of ESXi. At a minute past the hour, ESXi will go through /etc looking for all config files which have a corresponding .#<filename>. For example, on the FS there is both a file called /etc/vmware/config and /etc/vmware/.#config. Because .#config exists, the backup script for ESXi will put that into local.tgz (or state.tgz for ESXi installable).

So this would suggest the memory is not actually faulty, just not fast enough.
I run ESXi on a $10 memory stick and it's plenty fast enough for the backup to happen. There's just something inherently wrong with the stick HP handed out.

Re: Warning -- faulty HP ESXi USB Keys!

4. Feb 14, 2009 2:12 AM in response to: Dave.Mishchen…
Click to view Simon.H's profile Enthusiast
Simon.H
48 posts since
Oct 1, 2007
Dave

Thanks for your comment - it looks like backup.sh is called by reboot (without params) too. There is a line in there that copies local.tgz from /tmp (RAM) to the USB stick and then does a sync- but it only does a copy if the files are different (to save wear on the USB flash memory). So perhaps if the hourly autobackup had already run it wouldn't be writing just before shutdown.

As you imply HP's speed comment may just be a red herring for a more general fault, though I have been using it since October (rebooted several times) without any problem. It is ironic though - you pay for a premium USB device (with fancy aluminium casing no less) exactly so that you don't have this kind of failure! :0

A related problem (without wanting to divert this thread too much) I recently changed my host IP address through the console and one thing I did notice was that /etc/hosts still has the old address - this may be a result of the reboot error too - presumably that's wrong, but how should I fix it? Perhaps I should reset everything when I get my replacement USB key as a precaution.

PS. Dave thanks for your website too B-)

Re: Warning -- faulty HP ESXi USB Keys!

5. Feb 16, 2009 9:57 PM in response to: Simon.H
Click to view Dave.Mishchenko's profile Guru User Moderators vExpert
Dave.Mishchen…
9,151 posts since
Nov 15, 2005
You can access the hosts file with vifs.pl - http://www.vm-help.com/esx/esx3i/esx_3i_rcli/vifs.php.

Actions

VMware Beta Programs

Want to be Considered for Future Beta Programs?

Learn More

VMware Developer

Download SDKs, APIs, videos,
training, and more in the Developer community.

Learn More

Developer
Sample Code

Increase your developer productivity with VMware API sample code.

Learn More

VMworld
Sessions & Labs

Online access to the latest VMworld Sessions & Labs and online services.

Learn more

Purchase PSO Credits Online

Purchase credits to redeem training and consulting services online.

Buy Now

Community Hardware Software

View reported configurations or report your own.

Learn More

Only VMware ... Delivers Nexus 1000V

Ensure consistent, policy-based network capabilities to virtual machines across your data center.

Learn More

Communities