ESXi 6.5 U1 on HP ProLiant BL460c Gen9 fails: There was a problem with the Network Device specified on the command line. Error: No NIC found with MAC address:

[UCL]

Hello.

I'm using VMware-ESXi-6.5.0-Update1-7388607-HPE-650.U1.10.2.0.23-Feb2018.iso but end up at a black screen with white text.

exception.HandledError: Error (see log for more info):

There was a problem with the Network Device specified on the command line. Error: No NIC found with MAC address "0a:1a:4c:a0:00:54"..

I have read this one ESXi 6.5 upgrade There was a problem with the Network Device specified on the command line. Error: N...

but how do I proceed from here?

Best Regards

Peter

[a_p_]

What's the exact network adapter type/model in the blade?

André

[UCL]

Broadcom Corporation QLogic 57840 10 Gigabit Ethernet Adapter

(as per Web Client)

   Driver Info:

         Bus Info: 0000:06:00.0

         Driver: bnx2x

         Firmware Version: bc 7.15.24

         Version: 2.713.30.v60.9

(as per esxcli network nic get -n vmnic0)

net-bnx2x                      2.713.30.v60.9-1OEM.600.0.0.2494585  QLogic     VMwareCertified   2018-05-29

(as per  esxcli software vib list)

Peter

[a_p_]

From what I saw by comparing the HPE custom images, HPE replaced the QLogic drivers with the Feb-2018 release.

However, I'm not really sure whether this is the root cause in this case. What confuses me is the MAC address you've posted. It's looks like a locally administered address rather than a vendor address. Do the other blade servers show similar MAC addresses, where bit 2 in the first octet is "1".?

André

[UCL]

You can ignore the mac address. It is assigned virtually from a range of MACs through HP OneView

/Peter

[UCL]

I suspecting firmware and currently upgrading the C7000 Interconnects and the firmware on the OAs. I'll try to ESXi upgrade again and return when done.

HPE Support document - HPE Support Center

"

When using a Virtual Connect (VC) domain running Virtual Connect Firmware Version 4.60 or 4.61, following a reset, reboot or power-cycle of a server blade or VC Module, the Stacking Link(s) between VC Modules either within a single enclosure or across multiple enclosures may be lost and/or Uplink(s) may not form a LACP Group, resulting in a network outage. Additionally, latency may be seen in FCoE Traffic.

This occurs due to an issue found in the control traffic handler within Virtual Connect Modules.

"

/Peter

[UCL]

No. No change. Same message.

What log am I supposed to look into anyway? And how do I solve this problem?

Best Regards

Peter

[UCL]

When I upgraded from 6.0 to 6.5 these Qlogic VIB was in HP bulletins xml file:

QLogic_bootbank_misc-cnic-register_1.713.30.v60.1-1OEM.600.0.0.2494585

QLogic_bootbank_net-bnx2x_2.713.30.v60.9-1OEM.600.0.0.2494585

QLogic_bootbank_net-bnx2_2.2.6b.v60.2-1OEM.600.0.0.249458

QLogic_bootbank_qlnativefc_2.1.57.1-1OEM.600.0.0.2768847

QLogic_bootbank_scsi-bnx2i_2.713.30.v60.5-1OEM.600.0.0.2494585

QLogic_bootbank_net-cnic_2.713.30.v60.6-1OEM.600.0.0.2494585

QLogic_bootbank_scsi-bnx2fc_1.713.30.v60.6-1OEM.600.0.0.2494585

Now, in 6.5 to 6.5 Update 1 only this one is present:

QLogic_bootbank_qlnativefc_2.1.57.1-1OEM.600.0.0.2768847

and no bnx2 nor any bnx2x

Ideas are welcome.

/Peter

[UCL]

Very odd..........

This leads me HPE Support document - HPE Support Center

to a HPE Custom Image for VMware ESXi 6.5 U1 Install CD from 2018-02-09 which does NOT contain the drivers.

Further below I can see a Pre-Gen9 Custom Image which does contain those drivers but the Host is a HP ProLiant BL460c Gen9 not a pre-Gen9

It looks like I should have been lead to a HPE Customized ESXi Image vSphere 6.5 U1 OCt 2017 but I get to one from 2018.

Ehh???

/Peter

[a_p_]

It's currently not easy to follow HPE with all of their different images (PereGen9, Gen9+), and driver versions within these images.

The October 2017 image still came with the BNX2X drivers, which have been replaced with the QLFE3 drivers in the current images.

The VMware driver page as well as HPE's vibsdepot already contain updated QLEF3 drivers (QLG-qcnic-ESXi6.5-8484591.zip). Although there's nothing specific regarding your issue in the release notes, it may be worth a try to create a custom image using the new drivers, because all newer images come with the QLEF3 driver package.

André

[UCL]

Ok. How would you suggest I create a custom image? Somehow register which VIB are installed today and compare with whats on the new one, manually adding those that are not present, keeping those that are replacing other or ?

I haven't done this before.

/Peter

[a_p_]

There are several ways to create a custom image. With vCenter Server in place it you can use the integrated Image Builder to do this. An example can be found at https://docs.vmware.com/en/VMware-Validated-Design/4.0/com.vmware.vvd.sddc-upgrade.doc/GUID-9248EB76...​.

Simply replace the bundles mentioned in the example with the HPE Offline Bundle (VMware-ESXi-6.5.0-Update1-7388607-HPE-650.U1.10.2.0.23-Feb2018-depot.zip), and the driver package that I've mentioned before (please note that the driver download comes as a .zip archive, which contains several files, including the driver package itself).

Alternatively you can use Image Builder from the command line (requires VMware.PowerCLI to be installed).

André

[UCL]

Should I be worried VMware Knowledge Base since it states that: "qfle3 is a native driver that replaces the vmklinux net-bnx2x driver, but does not support HW iSCSI and SW FcOE." ?

/Peter

[UCL]

Oh golly.

Still on 6.5 and I maintenance mode:

Just lost contact to all Datastores on SAN after

esxcli system module set --enabled=true --module=qflge

esxcli system module set --enabled=true --module=qfle3

Rebooting after

esxcli system module set --enabled=false --module=qflge

esxcli system module set --enabled=false --module=qfle3

Fingers crossed.

[UCL]

Ok, I got access to the datastores back.

Enabling the native drivers is not the way forward and I suppose replacing the BNX2X drivers with QLEF3 drivers from the zip is also no-no or what do you think André?

/Peter

[UCL]

As I see it, comparing the Offline Bundles from HPE HPE Custom Image for VMware ESXi 6.5 U1 Offline Bundle  and HPE Custom Image for VMware ESXi 6.5 Offline Bundle

That

bnxtroce--20.8.3.0-1OEM.650.0.0.4598673--BCM--VMwareCertified--2017-05-10

i40en--1.5.6-1OEM.650.0.0.4598673--INT--VMwareCertified--2017-05-10

igbn--1.4.1-1OEM.600.0.0.2768847--INT--VMwareCertified--2017-05-10

ixgben--1.6.5-1OEM.600.0.0.2768847--INT--VMwareCertified--2017-05-10

lsi-mr3--7.702.17.00-1OEM.650.0.0.4598673--Avago--VMwareCertified--2017-05-10

qcnic--1.0.10.0-1OEM.650.0.0.4598673--QLC--VMwareCertified--2017-05-10

qedrntv--3.0.7.5.1-1OEM.650.0.0.4598673--QLC--VMwareCertified--2017-05-10

qfle3--1.0.60.0-1OEM.650.0.0.4240417--QLC--VMwareCertified--2017-05-10

qfle3f--1.0.45.0-1OEM.650.0.0.4598673--QLC--VMwareCertified--2017-05-10

qfle3i--1.0.13.0-1OEM.650.0.0.4598673--QLC--VMwareCertified--2017-05-10

are new in U1

and

misc-cnic-register--1.713.30.v60.1-1OEM.600.0.0.2494585--QLogic--VMwareCertified--2017-02-02

net-bnx2--2.2.6b.v60.2-1OEM.600.0.0.2494585--QLogic--VMwareCertified--2017-02-02

net-bnx2x--2.713.30.v60.9-1OEM.600.0.0.2494585--QLogic--VMwareCertified--2017-02-02

net-cnic--2.713.30.v60.6-1OEM.600.0.0.2494585--QLogic--VMwareCertified--2017-02-02

net-ixgbe--4.5.2-1OEM.600.0.0.2494585--INT--VMwareCertified--2017-02-02

scsi-bnx2fc--1.713.30.v60.6-1OEM.600.0.0.2494585--QLogic--VMwareCertified--2017-02-02

scsi-bnx2i--2.713.30.v60.5-1OEM.600.0.0.2494585--QLogic--VMwareCertified--2017-02-02

are missing in U1 but present in "U0"

So is the road forward to take the U1 Offline Bundle and add these 7 drives from "U0" to a new U1 Offline bundle or ?

/Peter

[a_p_]

In case of a test environment, I'd say go for it, and add the BNX2X driver package. However, I'm not sure whether this will work, nor whether it supported.

The only thing I can tell is that the October 6.5U1 Gen9+ custom image (VMware-ESXi-6.5.0-Update1-6765664-HPE-650.U1.10.1.5.26-Oct2017) came with these drivers.

Maybe it's worth opening a support call with HPE to find out whether they are aware of the issue with the new driver package, and if there's a fix/workaround for it.

André

[UCL]

Hello André.

I'll go ahead and open a support case with HP.

I'll report back here with whatever findings HP may provide me with.

Best Regards

Peter

[UCL]

Hello André.

As promised the answer from HP:

"

Looking at the detailed description you provided I do see one thing to note. As mentioned, you're using the 536FLB adapter (QLogic 57840 10 Gigabit Ethernet Adapter).
This adapter used to use the bnx2x driver but since February this year it's no longer supported when using ESXi 6.5.
Instead the native driver qfle3 should be used. You can find information regarding this change and corrective action in the following article on how to replace the driver https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-a00044039en_us&docLocale=en_US. This is the reason why it's not included in the later releases of the HPE custom image.

I would recommend that you proceed with replacing the bnx2x driver with the native driver and test if you're able to proceed with the update to 6.5 U1 or if it still fails.

"

I try going down that path tomorrow. The current version of qfle3 is 1.0.2.7, so I'd better upgrade that one first.

/Peter

[UCL]

Not the proper way to go.

Activating qfle3 makes the Host lose contact to all the datastores.

So HPE writes:

“

This adapter used to use the bnx2x driver but since February this year it's no longer supported when using ESXi 6.5.
Instead the native driver qfle3 should be used. You can find information regarding this change and corrective action in the following article on how to replace the driverhttps://support.hpe.com/hpsc/doc/public/display?docId=emr_na-a00044039en_us&docLocale=en_US. This is the reason why it's not included in the later releases of the HPE custom image.

I would recommend that you proceed with replacing the bnx2x driver with the native driver and test if you're able to proceed with the update to 6.5 U1 or if it still fails.

“

But a newer bnx2x driver is available in VMwares own Offline Bundle for ESXi 6.5 Update 1.

Feels like a Catch-22. Why does HPE not support bnx2x but VMware does and https://kb.vmware.com/s/article/2147565 describes that not all functions of bnx2x are replaced in qfle3?

If I use qfle3, I can’t see the datastores. If I use bnx2x I’m on an unsupported setup from HPEs point-of-view.

Any ideas on how to proceed?

/Peter