iandekit
Contributor
Contributor

ESXi 7.0.3 u3f doesn't work with Emulex LPe12000-S 8Gb Fibre Channel Adapter

Just upgraded from ESXi u3d 19482537 to u3f 20036589 but now it can't
see my fibre channel storage anymore.

This is on a Dell R640 with a Emulex LPe12000-S 8Gb Fibre Channel Adapter lpfc
card installed. Was a Sun card originally SG-XPICE2FC-EM8-Z and has been working fine until the u3f upgrade.

One system I still have running the u3d and it see this:

[root@bondi:~] esxcfg-scsidevs -a
vmhba0 lsi_mr3 link-n/a sas.5d0946600d28f000 (0000:19:00.0) Broadcom PERC H740P Mini
vmhba1 vmw_ahci link-n/a sata.vmhba1 (0000:00:11.5) Intel Corporation Lewisburg SATA AHCI Controller
vmhba2 vmw_ahci link-n/a sata.vmhba2 (0000:00:17.0) Intel Corporation Lewisburg SATA AHCI Controller
vmhba3 lpfc link-up fc.20000090fa0cfd92:10000090fa0cfd92 (0000:65:00.0) Emulex Corporation Emulex LPe12000-S 8Gb Fibre Channel Adapter
vmhba4 lpfc link-up fc.20000090fa0cfd93:10000090fa0cfd93 (0000:65:00.1) Emulex Corporation Emulex LPe12000-S 8Gb Fibre Channel Adapter

But the u3f system doesn't see that:

[root@indigo:~] esxcfg-scsidevs -a
vmhba0 lsi_mr3 link-n/a sas.5d09466003def900 (0000:19:00.0) Broadcom PERC H740P Mini
vmhba1 vmw_ahci link-n/a sata.vmhba1 (0000:00:11.5) Intel Corporation Lewisburg SATA AHCI Controller
vmhba2 vmw_ahci link-n/a sata.vmhba2 (0000:00:17.0) Intel Corporation Lewisburg SATA AHCI Controller

[root@bondi:~] vmkchdev -l |grep 10df
0000:65:00.0 10df:fc40 10df:fc42 vmkernel vmhba3
0000:65:00.1 10df:fc40 10df:fc42 vmkernel vmhba4

[root@indigo:~] vmkchdev -l |grep 10df
0000:65:00.0 10df:fc40 10df:fc42 vmkernel
0000:65:00.1 10df:fc40 10df:fc42 vmkernel

I don't see this card explicitly mentioned in the HCL though it worked fine until today.

The lpfc driver was updated in u3f...

old 14.0.169.25-5vmw.703.0.35.19482537
new 14.0.543.0-1OEM.700.1.0.15843807

Is it possible to reinstall the old driver from u3d or is there something I can do to make

it work under u3f?

If not, may have to rollback to u3d... which would be a shame as I use this Essentials licence platform as a pre-production testing system for our Standard licence setup.  Don't really want to have to buy replacement hardware...

Ian D

 

 

8 Replies
pirx2020
Contributor
Contributor

That looks like a problem we had with 6.7 and HPE ISO. But this was with a Emulex 12.x lpfc version, this version was broken and we did not see any LUNs of our storage. It was working with older and newer versions, but not with certain versions. VMware's inbox driver was always working (but old). But you are on version 14.x. Maybe you have to update the FC adapter firmware for the most recent driver?

 

PowerMax, VMAX AFA, VMAX3: Emulex Driver version 12.8.317.0 has issues connecting to arrays | Dell D...

Emulex Drivers for VMware ESXi Release Notes (broadcom.com)

 

From the 2 versions you posted, it seems the first is also native VMware inbox driver and the new one is the latest one from Emulex. Emulex driver just suck and they are broken more or less constantly.

 

There are other Emulex lpfc driver versions available at Herunterladen VMware vSphere - VMware Customer Connect

You can try one of them or download the one you had before from here: https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/lpfc/VMW_bootbank_lpfc_14.0...

 

 

 

 

 

 

 

0 Kudos
iandekit
Contributor
Contributor

Looking at the Drivers list Release_Notes_lpfc-14.0.326.12 for a slightly older lpfc driver than u3f provides:

https://customerconnect.vmware.com/en/downloads/details?downloadGroup=DT-ESXI70-BROADCOM-LPFC-140326...

it says that that support for the LPe12000 series HBA's was discontinued in this driver.

The u3f lpfc 14.0.543 driver release notes don't mention this at all.

The  release notes for u3f as a whole doesn't say anything about this at all!

Anyway following your advice (many thanks) I reinstalled the u3d driver for lpfc, and it seems to work again,

though I'm not sure if there will be issues ongoing and whether that driver will work in future ESXi releases.

[root@indigo:~] esxcli software vib list |grep lpfc
lpfc 14.0.543.0-1OEM.700.1.0.15843807 EMU VMwareCertified 2022-07-20


[root@indigo:~] esxcli software vib remove -n lpfc
Removal Result
Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
Reboot Required: true
VIBs Installed:
VIBs Removed: VMW_bootbank_lpfc_14.0.169.25-5vmw.703.0.35.19482537
VIBs Skipped:

[root@indigo:/tmp] esxcli software vib install -v /tmp/VMW_bootbank_lpfc_14.0.169.25-5vmw.703.0.35.19482537.vib
Installation Result
Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
Reboot Required: true
VIBs Installed: VMW_bootbank_lpfc_14.0.169.25-5vmw.703.0.35.19482537
VIBs Removed:
VIBs Skipped:

 

Then rebooted.   Lets see how that goes...

 

pirx2020
Contributor
Contributor

According to VMware Compatibility Guide - I/O Device Search the supported versions for SG-XPCIE1FC-EM8-Z are either native driver 14.0.169.25-5vmw or Emulex 12.8.614.16.

But this is always a mess, usually the storage vendor tells what driver/fw on server side are supported - at least that's what HPE is telling us over an over. So in the end it's trial and error. Using native VMware drivers is probably a good idea as VMware then also takes care of issues. Which is a problem if you use the async vendor drivers

 

 

0 Kudos
iandekit
Contributor
Contributor

FYI, I updated another u3d ESXi server to u3f and found the lpfc driver wasn't updated until I applied the "Non-Critical Host Patches",

in which case I _then_ lost access to the SAN until I manually rolled the lpfc driver back.

0 Kudos
Jabadakkas
Contributor
Contributor

Thanks for sharing this issue. I can confirm the same issue applies to LPe16000 HBAs as well. I excluded all OEM variants of the lpfc driver in my vLCM baseline as a result. The downside of this approach is that you can easily create a patch baseline with dynamic criteria, but exclusions are static. So this means we need to keep updating the baseline with new exclusions. How did you work around this problem?

0 Kudos
pirx2020
Contributor
Contributor

That's still my problem too. I created a custom extension baseline with the drive version, but then I also need to create a custom non-critical baseline without the newer driver. 

And for vSAN I've a similar problem, the async driver for HPE E208i-a ctrl in latest non-critical baseline is not on the HCI for vSAN, so I get a warning. I've I install the old one and create a custom extension baseline I again have to clone the predefined non-critical and blacklist the current driver there.

And each cluster has it's onw issues and custom baselines. What a mess.

0 Kudos
Jabadakkas
Contributor
Contributor

Also keep in mind that sometimes the vendor updates the drivers in their custom ESXi images. For example we noticed that when we want to upgrade from 6.7 to 7 we shouldn't use the latest version of their ESXi image A06 because it contains a version of the lpfc driver that doesn't work. Using the A04 version works just fine.

0 Kudos
fehret
Enthusiast
Enthusiast

Hi all,

Had the exact same problem after updated to U3g this morning, such a mess!

Luckily my cluster was strong enough to hold the load.

Thanks for all the commands, this helped too. 😉

0 Kudos