VMware Cloud Community
Kliwer16
Contributor
Contributor
Jump to solution

ESXi 6.7 and BCM57810

Hi. I have an ESXi 6.7 host and I need to add BCM57810 card to it, but with the card installed I get pink screen with

MEM_ALLOC bora/vmkernel/main/memmap.c:3950

cr0=0x8001003f cr2=0x0 cr3=0x100168000 cr4=0x38

I updated firmware of the card to the latest I could find: 14.07.06

I installed 1.0.69.1 version of qfle3 driver.

Every time I try to boot with the card inserted it fails. Without the card the host is working without problems. What can I do?

1 Solution

Accepted Solutions
SupreetK
Commander
Commander
Jump to solution

This PSOD looks like an issue with the qfle3 driver. Can you run the below commands and share the output?

esxcfg-nics -l

esxcli software vib list | egrep -i 'qfle|bnx2'

Cheers,

Supreet

View solution in original post

Reply
0 Kudos
18 Replies
SupreetK
Commander
Commander
Jump to solution

Can you share the screenshot of the complete PSOD screen?

Cheers,

Supreet

Reply
0 Kudos
Kliwer16
Contributor
Contributor
Jump to solution

Sure.

20181007_171906.jpg

Reply
0 Kudos
SupreetK
Commander
Commander
Jump to solution

What is the server make and model? Are you using a VMware standard image or a hardware vendor custom image for ESXi?

Cheers,

Supreet

Reply
0 Kudos
Kliwer16
Contributor
Contributor
Jump to solution

It's not a server. I am using GA-H81M-S1, Intel Pentium G3450, 4GB RAM. Without BCM57810 it was working fine for months now. BCM57810 is working fine in Windows or Linux on the same PC. VMware standard image.

Reply
0 Kudos
SupreetK
Commander
Commander
Jump to solution

With untested, unsupported and non-certified hardware (or a assembly of hardware), we are bound to have such issues. That's the reason, there is a hardware compatibility list in place.

Cheers,

Supreet

Kliwer16
Contributor
Contributor
Jump to solution

Yeah, and I do not have issues with this board itself. I't a home lab that have been running fine. It's ESXi that won't start with BCM57810 installed. So, I would appreciate if someone who was experiencing similar issues with that card could help. Or if somebody knows where to look in logs to get more info about what makes it crash.

Reply
0 Kudos
SupreetK
Commander
Commander
Jump to solution

It is failing during memory allocation and since it is failing even before the dump system gets initialized, I don't think any of the logs would be of any help in this regard.

Cheers,

Supreet

Reply
0 Kudos
Kliwer16
Contributor
Contributor
Jump to solution

Thanks for that info. It seems I'll have to switch to something else to do my test. It's a shame as ESXi in my opinion is most user friendly. Well, not so much since vSphere Client is no longer available for current versions but still. Thanks again for your time.

Reply
0 Kudos
bluefirestorm
Champion
Champion
Jump to solution

What is the boot mode set in the BIOS? Is it UEFI or BIOS? Is CSM turned on or off? It might make a difference since this is a problem occurring at boot up time.

The CR4 register value indicates PAE is expected (address size wider than 32-bits, i.e. access memory location beyond 4GB boundary), so it might work with UEFI mode.

Reply
0 Kudos
Kliwer16
Contributor
Contributor
Jump to solution

Thanks, that got me into playing with RAMs. The thing is, when BCM57810 is connected, the MB does not see one slot of RAM making it only 2GB for ESXi. I tried using 2x 4GB (still with the card on only one is recognized) and ESXi loaded a little further but got another error. Anyway, thanks for your help.

20181008_184606.jpg

Reply
0 Kudos
SupreetK
Commander
Commander
Jump to solution

This PSOD looks like an issue with the qfle3 driver. Can you run the below commands and share the output?

esxcfg-nics -l

esxcli software vib list | egrep -i 'qfle|bnx2'

Cheers,

Supreet

Reply
0 Kudos
bluefirestorm
Champion
Champion
Jump to solution

You have gotten further but not by much :smileylaugh: ; less than 16 seconds based on the VM kernel uptime of the PSOD screenshots.

The thing is, when BCM57810 is connected, the MB does not see one slot of RAM making it only 2GB for ESXi.

Is this evident already at the UEFI/BIOS level (i.e. going into UEFI/BIOS setup, it already only sees one stick of RAM? Was this problem also present with the Windows and Linux setups?

I am assuming that you inserted the card on the PCIe x16 slot. Try inserting the card on the PCIe x1 slots assuming that there are no physical impediments such as capacitors and other motherboard components that the "teeth" of the PCIe card connectors will hit. Also try reducing the amount of video memory setup in the UEFI for the integrated graphics of the CPU. These steps are to try to remedy the situation of seeing only one slot of RAM when the card is inserted.

Reply
0 Kudos
Kliwer16
Contributor
Contributor
Jump to solution

Well, those 16 seconds allowed me to notice that ESXi saw 4GB RAM when 8GB was inserted. Yes, it is the same at UEFI BIOS level. I didn't check in Linux or Windows but as it is like this in BIOS, I don't think it was different. Yes, the card is inserted in 16x slot. The card is 8x so I can't try it in 1x slot. I did reduce video memory to minimum and there was no difference.

Reply
0 Kudos
Kliwer16
Contributor
Contributor
Jump to solution

esxcfg-nics -l (obviously without the card connected):

[root@localhost:~] esxcfg-nics -l

Name    PCI          Driver      Link Speed      Duplex MAC Address       MTU    Description

vmnic0  0000:03:00.0 ne1000      Down 0Mbps      Half   68:05:ca:17:b4:6d 6000   Intel Corporation Gigabit CT Desktop Adapter

vmnic1  0000:04:00.0 r8168       Up   1000Mbps   Full   50:3e:aa:02:3b:ec 6000   Realtek Semiconductor Co., Ltd. TG-3468 Gigabit PCI Express Network Adapter

vmnic2  0000:02:00.0 r8168       Down 0Mbps      Half   40:8d:5c:3a:9f:b7 6000   Realtek Semiconductor Co., Ltd. Onboard Ethernet

esxcli software vib list

[root@localhost:~] esxcli software vib list | egrep -i 'qfle|bnx2'

qfle3                          1.0.69.1-1OEM.670.0.0.8169922         QLC      VMwareCertified     2018-10-07

net-bnx2                       2.2.4f.v60.10-2vmw.670.0.0.8169922    VMW      VMwareCertified     2018-06-27

net-bnx2x                      1.78.80.v60.12-2vmw.670.0.0.8169922   VMW      VMwareCertified     2018-06-27

qfle3f                         1.0.25.0.2-14vmw.670.0.0.8169922      VMW      VMwareCertified     2018-06-27

qfle3i                         1.0.2.3.9-3vmw.670.0.0.8169922        VMW      VMwareCertified     2018-06-27

scsi-bnx2fc                    1.78.78.v60.8-1vmw.670.0.0.8169922    VMW      VMwareCertified     2018-06-27

scsi-bnx2i                     2.78.76.v60.8-1vmw.670.0.0.8169922    VMW      VMwareCertified     2018-06-27

Reply
0 Kudos
bluefirestorm
Champion
Champion
Jump to solution

Looking at this Dell webpage

https://www.dell.com/en-us/shop/dell-qlogic-57810-dual-port-10gb-direct-attach-sfp-network-adapter-f...

From the picture and the "Overview" mentioning PCIe x1 support, you should be able to try the PCIe x1, unless the different OEM versions have different specs. The only issue would be if there are motherboard components such as capacitors that get in the way of card connector teeth.

The other alternative is to force a passthru by manually editing the esx.conf (assuming it is still the same with ESXi 6.7) by specifying "passthru" for the PCIe slot where the card is to be inserted.

https://kb.vmware.com/s/article/1010789

If it is in passthru mode, the ESXi driver should not load and thus avoid the PSOD once the card is inserted. May not be exactly what you have originally in mind, but if this successfully boots the ESXi with the 57810 card inserted, at the very least you could try the passthrough through a VM.

Reply
0 Kudos
SupreetK
Commander
Commander
Jump to solution

For the above PSOD, let us uninstall the qfle3 related drivers so that the card is forced to use the bnx2x driver. Run the below commands and reboot the host. Once host is rebooted, check if the qfle3 drivers are uninstalled. Post validation, insert the card and see if you still hit the same PSOD.

esxcli software vib remove --vibname=qfle3

esxcli software vib remove --vibname=qfle3f

esxcli software vib remove --vibname=qfle3i

Cheers,

Supreet

Reply
0 Kudos
Kliwer16
Contributor
Contributor
Jump to solution

I tried upgrading qfle3f and qfle3i drivers as I upgraded only qfle3 and now it booted fine, no PSOD. It's still a mystery why only half of RAM is recognized on this system, but doesn't matter much as this card is only needed for some tests. Thank you for your help and pointing me in the right direction.

Alfoele
Contributor
Contributor
Jump to solution

Hi, I have the same problem, with card inserted the systems sees only 1 ram slot. Did you understand how solve the problem using the nic card with full ram? Thank you

Reply
0 Kudos