All VMs down due to firmware problem - HBA firmware upgrade

[hernandez80]

Hello,

Before explainations, here is the context :

At least 4 ESX servers, version 3.5 u4. All HP Blade 680c G5 with Emulex LPe 1105 HBAs (Firmware 2.72a)

On 19th of August, we experienced a san disconnection on 4 blade servers in the same enclosure. (Our fault, this was an isolated cluster and all VMs went down, bad luck )

I've opened a SR at vmware and they statued that it was a hardware fault.

here are the logs, that appeared at the exact same time on all four ESX servers :

Aug 19 19:30:39 esx3 vmkernel: 29:08:52:41.233 cpu6:1235)StorageMonitor: 196: vmhba3:2:9:0 status = 2/0 0x5 0x21 0x0

Aug 19 19:30:39 esx3 vmkernel: 29:08:52:41.233 cpu6:1235)<4>lpfc0:0457:INe:Adapter Hardware Error Data: x40000000 x7ff88 x6000000f

Aug 19 19:30:39 esx3 vmkernel: 29:08:52:41.335 cpu6:1235)StorageMonitor: 196: vmhba3:2:22:0 status = 0/1 0x0 0x0 0x0

Error code lpfc:0:0457 seems to point to :

"0457 INe: Adapter Hardware Error

DESCRIPTION: The driver received an interrupt indicting a possible hardware problem.

Data: (1) status (2) status1 (3) status2

SEVERITY: Error

LOG: Always

ACTION: This error could indicate a hardware or firmware problem. If the problem persists, report the error to Technical Support."

cf

After opening a call at HP, they told me that we have to upgrade to latest firmware version, 2.80a4, available since 14 aug 09 (!!!) - and no similar issue reported ..

Now, i'm in the process of upgrading my firmwares, on all ESX servers in all enclosures ...

I'm just searching for the best way to do it, as it appear that it is not anymore possible to do it with lputil.

HBanyware seems the best way, i'm actually working on it with our SAN team..

I would be interrested in some good links, and of course if anybody had same kind of problems ?

best regards

Patrick

[thecakeisalie]

Emulex download page for VMware has the apps that you need to get the firmware upgrade on the hosts completed.

Note that for the HP rebranded Emulex HBA's you will need to download the firmware from the HP website for your mezzanine cards, not from the Emulex site. Also note that because you have a c-class blade chassis that you will require the chassis firmware upgrade, onboard administrator upgrade, and virtual interconnect/switch upgrades as well as the HBA firmware ugprades.

Most times, using the HP Firmware Maintenance CD will upgrade all of the components within the blade, iirc this includes the HBAs. However, according to the on-CD compat list, these are outdated compared to the version they're asking you to upgrade to, and you will still require the .scexe offline upgrade in conjunction with the Firmware Maintenance CD to upgrade to the August release.

From firmware-8.60-0/iso/_autorun/HTMLFILES/en/cdcontent-fw.htm

<span style="font-weight: bold">Category: Firmware - Storage Fibre Channel</span>
Offline	HP Firmware Flash for QLogic Fibre Channel Host Bus Adapters	2009.04.15	cp010721.scexe
Offline	HP Firmware Flash for Emulex Fibre Channel Host Bus Adapters	2009.06.24	CP010725.scexe
Offline	HP Firmware Flash for Brocade Fibre Channel Host Bus Adapters	2009.05.18	CP010983.scexe

[hernandez80]

Hello,

Thanks for the reply.

Emulex download page gives tools that should help, but not here.

I tried with hbanyware and lputil locally on my hosts, but since U4, it seems that the drivers make these tools unable to see the HBAs.

• Driver lpfcdfc is incompatable with libdfc. The driver must be at DFC Rev 1.19

• Emulex Fibre Channel Host Adapters Detected: 0

We are also in the process of upgrading the entire enclosure (mezz boards, interconnect bays,OAs, ILOs, servers...), but that takes some time. Only problem is that Emulex Firmware

Actually, for servers, the 8.60 firmware CD does not include the 2.80a4 emulex firmware (just some days difference between firmware CD and Emulex firmware..).

I hope this will soon be updated.

[thecakeisalie]

You can actually download the offline update to use with the firmware CD for upgrading the Emulex HBA's - just grab the 2.80xx download, put it on a 'floppy' (image), then boot to the firmware CD and load it from the floppy (aka virtual media).

[dinny · Accepted solution]

Hiya,

There is a new version of the hbanyware software downloadable from the emulex site that does work specifically with esx3.5 u4.

(lputil is no longer supported with esx 3.5 u4)

It is a bit of a pain to use compared to lputil, as you have to specify the WWN in most of the hbanyware commands though.

I found it easier to allow hbanyware remote through the ESX firewall (see readme with hbanyware for the syntax) then administer them all remotely running hbanyware from my Windows VCB server.

Dinny

[hernandez80]

I've built a dedicated server in order to have the HBanyware managing all hosts.

It seems to work perfectly, we now have to set correctly our firewalls ..

Thanks for information.

[gr8fnma]

Just to let you know that you can still use lputil on ESX3.5 Update4 to update your firmware by uninstalling the Emulex driver

rpm -qa | grep elx

rpm -e elxvmwarecorekit-<kit version> to uninstall

rpm -i elxvmwarecorekit-<kit version>.rpm to re-install

You can query the rpm in between to make sure the rpm status

execute lputil and you should see your HBAs

Hope this helps