ESX6.0upd1 and Brocade BR-1020 : extreme issues, crashing, not possible to install (Proliant DL380 Gen9)

[LubomirZvolens1]

Guys, I'm struggling with ESX6 installation on HP DL380Gen9 (brand new servers). I have several of those, they all behave the same.

Upgraded with 2015.06 Proliant Support Pack (BIOS 1.40, ILO 2.20 - later upgraded to v2.30 Oct2015 without change, SmartArray B140i don't remember which version but latest possible, there was no change from June till today). CPU 8core E5-2630 v3,  quad-port integrated Gigabit ethernet 331i, 128GB RAM (8x 16GB modules, they are ok), boot from 16GB SD card. Today I uploaded onboard NICs to latest firmware dated 01/Oct/2015, no change.

I installed two dual-port 10Gbit adapters in each server. Brocade BR-1020, all four ports are connected with twinax DAC to Brocade MP-8000e switch in plain simple Ethernet mode, nothing fiber-channel, FCoe, DCB here. When system boots up, Brocades show Adapter Firmware Version (Option ROM Version) 2.2.0.1 : I guess this might be problem, I'm trying to investigate how to update it but so far I found info "driver uploads newest firmware".

No matter what I do, I can't succeed ESX6.0b,  6.0 Upd1 or HP Custom ESX6.0 images to start installation. They all crash second later after displaying "Initializing Storage Stack", see screenshots please. No dump, no PSOD, nothing, just bang reboot. If I disable BOTH Brocades in BIOS, I can install and run v6.0 without any issues from the same installation images, all three. I can manually install BR-1020 driver in running ESX (VMware Compatibility Guide: I/O Device Search), but after enabling adapters in BIOS, it's all the same crash and ESX6 can NOT start and reboots exactly in the same place, the same way. I tried to delete all Brocade drivers from running system so that only net-bna would remain, no change.

Everytime it crashes, there is new "Uncorrectable Machine Check Exception" recorded in Integrated Management Log. Tried to find something reasonable on internet, related to Brocades, but failed (lots of generic stuff).

I probably tried all settings in BIOS. I intentionally flipped every Performance Option there is, set servers to High Performance Mode without any C-States as I found it in some HP discussions, disabled onboard SmartArray B140i, disabled PXE boot from network cards, there really is very little - if any - I haven't touched in BIOS. Of course loaded system default and manufacturer default settings, nothing helps me at all.

Now the news : ESX5.1 and ESX5.5 both install and run flawlessly on the same servers, adapters, firmware and BIOS settings. Without single glitch. It's ONLY v6.0 that makes problems. ESX5.1 was running there for about 2 weeks, so there is nothing wrong with the hardware. I can easily push 2GB/s from iscsi storage through two 10Gbit ports without missing a beat all day long.

Any ideas what to disable / change so I could use ESX v6.0 ? I'm overlooking something very easy and obvious.

images : VMware-ESXi-6.0.0-2494585-HP-600.9.3.30.2-Jun2015.iso

VMware-VMvisor-Installer-6.0.0.update01-3073146.x86_64.iso

and I already deleted the original VMware 6.0.0b which I don't remember what it was

[LubomirZvolens1]

haven't found way how to update Brocade 1020 BIOS from ESX  (firmware, they call it boot code - for google search) and of course I fail to boot Brocade/Qlogic LiveCD v3.2.6.0. It requires legacy boot mode and does not work in UEFI, in my case this is the result when attempting to boot in legacy :  [attachment]

There is something Qlogic calls "VMware ESXi 5.5/6.0 BCU - ESX CLI plug-in". BCU is linux-based tool that is used to update boot code plus other functionality, according to documentation. My problem is I can't get this plug-in work. Forget  VMware KB: ESXi 5.0 ESXCLI plug-in (BCU) support for Brocade     forget all the "usual"  ESXCLI BROCADE BCU or ESXCLI BCU...  this is not v5.0...

so I pulled out two adapters from server, brought it home and updated both in long-forgotten-behind-the-table Z79 based desktop with Windows2012. Boot code changed from 2.2.0.1 as seen on screenshots above to 3.2.6.0 which is the latest available as of this writing. Will report if something changed in several hours when they are back in servers.

Also, what is supposed to be UEFI-based "Multi-Boot Firmware Image" ain't startable too. After unpacking it, I see BCU.EFI and brocade_adapter_boot_fw_v3-2-6-0  files BIOS detects BFU.EFI when I map it over ILO but it refuses to perform any action with it.

The same adapters, the same boot code (firmware 2.2.0.1) work flawlessly on my white-box with ESX v6.0. It's Asus X99-W motherboard, i7-5820k, 64GB RAM, 2x Brocade BR1020, absolutely without any problems for a month.

The funny thing doesn't end here !! VMware packaged v3.2.4 drivers in v3.2.6 package. You got to be kidding me !

http://www.vmware.com/resources/compatibility/detail.php?deviceCategory=io&productid=38221

https://my.vmware.com/group/vmware/details?downloadGroup=DT-ESXI55-QLOGIC-BNA-3260&productId=353

when you download package "VMware ESXi 5.5 bna 3.2.6.0 NIC Driver for QLogic BR-Series HBA and CNA" and unpack it, there actually is file "net-bna-3.2.4.0-1OEM.550.0.0.1331820.x86_64.vib"  and not only the file version is old, but also inside the file it says RELEASE DATE 2014-02-14.

HOW HARD CAN IT BE ???

[LubomirZvolens1]

talking to myself   seems like the drama is over.

Brocade adapters with updated boot code v3.2.6.0 are getting through installation phase of ESX v6 and after manual driver installation (esxcli software vib install -v) they show up as networking adapters. While I didn't perform extensive testing, at first glance and after some gigabytes transferred left and right it seems to be OK. Because they are on VMware HCL, hopefully there will be no more headaches - I've had enough.

I'm using driver v3.2.5 because REAL v3.2.6 is nowhere to be found as described above (qlogic webpage takes you to vmware.com download and there is v3.2.4 only, despite name saying v3.2.6). I'm pretty sure this will be corrected shortly.

Nobody can explain me why no-name whitebox works OK with these adapters (UEFI BIOS of course) ;  and why the same DL380 Gen9 servers are OK with ESX5.1 and 5.5.

To be honest, I didn't suspect Brocade 1020 firmware / boot code to be the culprit.

Took me three days. And I still have to pull out 14 adapters from servers and update them in external enclosure with Windows 2012. Duh.

[JMIT28]

Hi LubomirZvolens1,

I was wonder if you still have the Firmware files to update Brocade 1020 firmware. I have been looking all over the internet but without luck.

Multi-Boot Code for BR- Series Adapters LiveCD

brocade_adapter_boot_fw_v3-2-7-0.zip

Im trying to follow this article: https://nbctcp.wordpress.com/2018/05/17/upgrade-brocade-or-qlogic-br-1020-in-esxi/