Broadcom NIC causing RAM loss in BIOS

[miallen]

Not sure where to post this because it's really a hardware issue but ...

I bought a used “Dell 0FCGN Broadcom 5720 2Port 1Gbps PCIe Ethernet Adapter Card 48-5” for a newish Dell Inspiron 3671 to run VMware ESXi because ESXi doesn’t support the on-board Realtek NIC.

When I boot with the Broadcom card in slot 0, I get “The amount of system memory has changed” and the BIOS sees only 8GB and not the full 16GB or RAM.

If I pull the Broadcom card out, the BIOS sees the full 16GB.

Note that I have successfully installed ESXi and installed a VM which runs great on this machine. The only problem is that I can only access one stick of RAM. The other is knocked out by the Broadcom card.

I've tried disabling stuff in the BIOS like the realtek card, wireless, bluetooth and so on without any effect.

Would it help (or hurt) if I installed the VIB described on this page for the BCM5720 adapters:

  https://www.broadcom.com/products/ethernet-connectivity/network-adapters/1gb-nic-ocp/bcm5720-2p

Presumably after I install this I have to log in with ssh and run some command to update the firmware on the card?

Any ideas?

If this NIC is not right for the system, can anyone recommend a good card for this low-budget SOHO rig?

Thanks,

Mike

[scott28tt]

Not sure what this has to do with ESXi given that the BIOS changes when you install the card…

[e_espinel]

Hello.
The first thing would be to validate if the Broadcom 5720 2Port 1Gbps PCIe card is supported in the Dell Inspiron 3671 Desktop.
Try to update the firmware of the Dell and also the firmware of the card, the firmware can be found on the manufacturer's website.

VMware supports this card in version 7.0 and 6.7

https://www.vmware.com/resources/compatibility/search.php?deviceCategory=io&details=1&partner=7_bpar...

[miallen]

Unclear how I would validate that some old card works with a new desktop. I would say nobody is going to say yes to that. I think I'm on my own. That's ok. If I can't get it to work, I'll just buy another NIC from the compatibility list and keep trying ...

Note that it's a Vostro 3671 (I mixed up a digit in the service tag).

I updated the firmware on the Dell before trying anything.

I was able to install the VIB from the aforementioned broadcom page:

] vim-cmd hostsvc/maintenance_mode_enter
] unzip BCM-ESX-bnxtrocecli-219.0.1.0.zip 
Archive: BCM-ESX-bnxtrocecli-219.0.1.0.zip
inflating: BCM-ESX-bnxtrocecli-219.0.1.0.vib
] esxcli software vib install -v /vmfs/volumes/<guid>/BCM-ESX-bnxtrocecli-219.0.1.0.vib -f
Installation Result
Message: Operation finished successfully.
Reboot Required: false
VIBs Installed: BCM_bootbank_bcm-esx-bnxtrocecli_219.0.1.0-00
VIBs Removed: 
VIBs Skipped: 
] esxcli software vib list | grep bnxt
bcm-esx-bnxtrocecli 219.0.1.0-00 BCM PartnerSupported 2022-04-09
bnxtnet 216.0.50.0-41vmw.703.0.20.19193900 VMW VMwareCertified 2022-04-08
bnxtroce 216.0.58.0-23vmw.703.0.20.19193900 VMW VMwareCertified 2022-04-08

But it's not clear what this is doing. What is a "bootbank"?

Maybe when I reboot something will happen?

[bluefirestorm]

My guess is that the introducing the NIC, the BIOS and NIC (or any PCIe device for that matter) has to negotiate which address range(s) to use for MMIO and probably ended up somewhere above 8GB but below 16GB; and the BIOS ended up saying that above 8GB address range should not be used by any OS, be it ESXi or Windows. (You can see why this isn't an issue because of ESXi).

So even buying different ESXi certified NICs, it will be just another roll of the dice. You could try to find NICs that are qualified by Dell for this Vostro model (even for Windows as it should not cut off the addressable space to 8GB or any number below the maximum RAM it can take) and hope that it is also qualified for ESXi.

There is not much in way of configurations options.

You could try disabling the legacy boot ROM options.
https://dl.dell.com/topicspdf/vostro-3671-desktop_owners-manual2_en-us.pdf#page=20&zoom=auto,-204,78...

The other one is to disable VT-d (OTOH, this might be what you want to passthrough some PCIe device such as a GPU).

You could also try removing the GPU and other PCIe devices so that there is more wiggle room for MMIO address negotiation during POST.

On the other extreme, switching to BIOS mode instead of UEFI mode (so that all MMIO address space would be below 4GB, but you will also lose UEFI stuff like secure boot) but there does not appear to be an option for that in the manual. Note that switching to BIOS mode that means boot disk would have to be in MBR (not GPT) and I don't know if ESXi still supports MBR for boot device.

[miallen]

Legacy boot roms was already disabled.

Disabling VT-d had no effect.

There are no other off-board PCI devices installed (I also tried disconnecting the WLAN M.2 card).

Does anyone think an Intel I350 card might behave differently or is this more a problem with the host machine and not the NICs?

After much searching for a firmware update, it looks like the BCM5720 does not have an update near as I can tell. And it's not clear that would have helped anyway.

I guess I'll just have to limp along with 8GB RAM for now until I can figure out a good combination of host / NIC that is good for an ESXi home lab.

[bluefirestorm]

Have you tried moving the card on to different slots? Typically, the x16 slot would be directly connected to the CPU PCIe lanes, while the other slots will be routed through the chipset.

I think this is the first time I have come across adding a PCIe device cuts the addressable RAM space.

There are two parts of the equation, the NIC firmware and the motherboard firmware.

If you think about it, the Broadcom 5720 NIC boots up fine for many other servers in the HCL and likely other servers/desktops that are not in the VMware HCL without cutting off addressable RAM space. The probability is also high that any other NIC would also cut off the addressable RAM space. So it is either a bug of the Vostro BIOS or an intended feature to discourage Vostro users to add in cards that Dell won't support (even if it is Dell branded). Keep in mind Vostro target market is really small/medium sized businesses and they would not want to be swamped with support calls; or they can just answer back "it is not supported" if someone does try to get support for situations like this.

[miallen]

It's a relief to meet someone as cynical as I am 😆

I did try the other slots. The x16 slot was a little different in that I also lost keyboard and mouse.

I guess I won't be buying another NIC then. I'll have to buy a new machine. Every two years or so I buy a new machine, rotate everything and retire one. I'll sort it out then.

But I really like the smallish tower, energy efficient, quiet machines for my SOHO lab. That's ultimately the problem. Apparently those machines are not good hosts for ESXi compatible hardware.

[bluefirestorm]

I was not being cynical but trying to be realistic. I think I went a bit too far with the "intended feature". But I have read posts in other forums wherein a large system vendor laptop (I think it was Lenovo) would not work with WiFi modules not sourced from them; so any attempt to upgrade the WiFi is futile unless you go through their channel.

Just my two cents, with the way things are with ESXi 7.0 (i.e. no more vmklinux drivers), it will only become more difficult to run ESXi on bare metal using non-HCL hardware especially using consumer-grade desktops. Even some of the Intel NICs in Dell Optiplex machines are not getting detected by ESXi although the NIC model is in the HCL; so there are sub-models even within the Intel NIC model. A closer bet would be the workstations such as Dell Precision/HP Z as some of these chipsets/motherboards would be same/similar to an entry level PowerEdge/Proliant server tower; but Dell Precision machines are not quiet and tend to have non-standard components (e.g. power supply, connectors). I think some still have success with Intel NUCs and that would fit the small form factor and quiet criteria.

[miallen]

Just to follow-up, I was never able to get the broadcom card to work. Reasoning it was just a problem with the machine, I decided to buy a Dell Precision 3420 tower for $163.21 USD shipped from a refurb company on Ebay. It has an integrated Intel I219-LM (on the I/O compatibility list).

Surprisingly, the machine actually works great. It's near mint and they gave me a power cable among other things.

The only hitch was that the 3420 would not boot the EFI ISO of ESXi 7u3. I had to burn a DVD and boot from the optical drive.

Note: This particular machine does not have a VGA port by default. There's a special VGA cable assembly specifically for this machine that connects the mobo to a VGA port that goes into a punch-out. I ordered one for $10 but I haven't tried it yet.

Thanks to bluefirestorm for suggesting the Dell Precision.

[bluefirestorm]

I assume you were trying to boot from the ESXi 7u3 EFI ISO from a USB thumb drive.

With most (maybe all) Dell machines, you can press F12 (instead of F2 that goes to UEFI/BIOS setup) upon powering on and it creates a boot menu. If it detects a bootable USB thumb drive, it will be listed in this boot menu that is created on the fly. I tend to just use dd on Linux to create bootable USB thumb drive from bootable ISO files.

Anyway, looks like you got your ESXi installation sorted out.

[miallen]

EFI ISOs would not show up under F12 on the Dell T3420. I use Fedora Image Writer to create live USB sticks which has always worked fine (except for Windows images I use WoeUSB). It's just something about that 3420 BIOS (which was updated incidentally). An EFI ISO would not show up. Fedora live was not listed either. CentOS 7 was because it's MBR. But the optical drive was listed so the DVD worked. Strange. But yes it's all sorted out now. Thanks.

[ASKINT]

I've got exactly the same issue with LENOVO M720s. ONe out of 4 DIMM slots doersn't work with BCM5720. Tried all described with no luck

[bluefirestorm]

ASKINT,

Does the M720s have an on-board Intel I219-V not recognised by ESXi 7.x?
https://psref.lenovo.com/syspool/Sys/PDF/ThinkCentre/ThinkCentre_M720_SFF/ThinkCentre_M720_SFF_Spec....

If yes, you could try one of the community drivers for ESXi 7.0 instead of using the BCM5720. There is a list of specific submodels of I219-V.
https://flings.vmware.com/community-networking-driver-for-esxi#requirements

[ASKINT]

Yes I've just realised I can use the one that is built-in LAN (Intel). It works perfectly with https://vmware.lenovo.com/ ESXi 7.0.3.

Thank you! Now I have a Home Lab for £200 with 64gb RAM 🙂