4.1i Hypervisor and Broadcom NICs

[AndersN]

I have a server board with 2 x 1GB NICs based On Broadcom 5780 (HT2000 chipset) that I would like to run ESXi Hypervisor, but so far no luck.

The NICs reports PCI ID 14e4:166a, the tg3 driver is loaded, mapped to vmnic0 and 1, but not associated with the driver.

So, has anyone had any luck with this?

[AndersN]

Am I the only one using these NICs?

[DSTAVERT]

Did you get ESXi installed or are you stuck at installing? If you are installed ESXi won't install without a supported NIC so that would indicate that it recognized the NIC. Let us know where you are stuck?????

[AndersN]

I'm not really "stuck", but the NICs aren't associated with the driver in the install stage.

I've tried ESXi 3.5, 4.0, 4.1 and 4.1u1. Both free versions and 60 days trial versions. No joy.

I  also downloaded the latest driver from VMware and applied to the 4.x  versions. Nope, nothing. I defined it in 2 ways, first via pci.ids and  then via the xml way. Same result.

On my server it loads tg3.o  automatically, nicely and cleanly, lspci -p from the install console shows the pci ids mapped to  vmnic0 and vmnic1. But neither is associated with the driver.

If I boot up from the same USB stick on a computer with only nVidia NICs and no Broadcom devices, it never loads tg3.o.

So the system must correctly identify the NICs, but for some reason not making use of them correctly in installation mode.

I'll do a test install to USB with a e1000/pro card and then see if the Broadcom NICs are useable in the installed system.

lspci -p from install console:
000:003:05.0 9005:0286 9005:029b 10/ 10/0xa0 A V aacraid vmhba1
000:004:04.0 14e4:166a 14e4:166a 11/ 11/0xa8 A V         vmnic0
000:004:04.1 14e4:166a 14e4:166a 11/ 11/0xb8 A V         vmnic1

lspci from openSUSE Linux, same machine:

0000:04:04.0 Ethernet controller [0200]: Broadcom Corporation NetXtreme BCM5780 Gigabit Ethernet [14e4:166a] (rev 03)
0000:04:04.1 Ethernet controller [0200]: Broadcom Corporation NetXtreme BCM5780 Gigabit Ethernet [14e4:166a] (rev 03)

[golddiggie]

Once you have it installed, if you're able to get an eval for vCenter too, use VUM to update the host to the very latest level. There are patches post update 1 that might resolve your NIC issue...

OR, you could just get a NIC (dual or quad port) that IS on the HCL and supported right out of the gate... I prefer the second option personally.

[DSTAVERT]

Did ESXi install? Can you assign the IP address in the DCUI console?

[peetz]

This is very odd behaviour...

This device is in /etc/simple.map on a ESXi 4.1U1 system and is mapped to the tg3-driver there. This triggers the loading of this driver, but maybe it does not really support this device?!

Have you checked vmkernel.log for the tg3 initilization messages?

Andreas

- VMware Front Experience Blog

[peetz]

Instead oy buying new hardware you could also just wait a few days until ESXi 5.0 is released.

The ESXi 5.0 RC includes a much newer tg3-driver (version 3.110g) that will probably support the 5780 better than the driver that is in 4.1U1.

Andreas

- VMware Front Experience Blog

[AndersN]

The NIC is in the HCL, see this page.

[AndersN]

DSTAVERT: I haven't installed it yet, but I expect it to install since it seems like I have to put in a Intel e1000 NIC for the installation.

peetz: I have tried the 3.110 driver that I found in the driver download section here. No luck. Will 5.0 be released as the free Hypervisor as well as the full licenced version?

The strange thing is that this very motherboard installs the BCM5780 NICs flawlessly in both older and newer versions of Linux, i586 and x86_64 as well as Windows XP, Windows Server 2003 & 2008 and Windows 7. In short, I've only seen the problem with ESXi Hypervisor.

[DSTAVERT]

I would check to see if there is updated firmware for the motherboard. It is certainly possible that there are minor differences in the chipset or how it is implemented by the MB manufacturer that causes an issue. I would see whether you can change the interrupts used by the embedded NICs. Most server manufacturers have some facility in the bios to allow this.

ESXi 5 will have a free version. It should be available within the next few weeks.

[AndersN]

Now then...

ESXi Hypervisor installed just fine on to a USB stick, however, the Broadcom NICs doesn't show up anywhere else than already mentioned..

The only available NICs are the Intel PRO/1000 and a Realtek 8169.

I have the latest available firmware for the m/b, and unfortunately there are no options for interrupts. But I can't see that that should be an issue since both onboard NICs works flawlessly in all o/s but ESXi..

[peetz]

Yes, ESXi 5.0 will also be available in a free edition (like ESXi 4.1).

Have you looked at the vmkernel-messages?

To do this, press ALT-F12 on the console after VMkernel has been fully initialized. Scroll back on this screen with Pg-Up until you see the tg3 initialization messages. Any clue there?

Andreas

- VMware Front Experience Blog

[AndersN]

Everything containing '04:04' in /var/log/messages (I've omitted the timestamps and 'vmkernel:' to save some typing..)

Code:

cpu2:4659)PCI: 2538 000:004:04.0 named 'vmnic0' (was '')
cpu2:4659)PCI: 2538 000:004:04.1 named 'vmnic1' (was '')
cpu2:4659)PCI: Trying 0000:04:04.0
cpu2:4659)PCI: Announcing 0000:04:04.0
cpu2:4659)VMK_PCI: 746: device 000:004:04.0 capType 1 capIndex 72
cpu2:4659)VMK_PCI: 746: device 000:004:04.0 capType 5 capIndex 88
cpu2:4659)VMK_PCI: 746: device 000:004:04.0 capType 7 capIndex 64
cpu2:4659))<3>tg3: (0000:04:04.0) phy probe failed, err -19
cpu2:4659)VMK_PCI: 746: device 000:004:04.0 capType 3 capIndex 80
cpu0:4659)PCI: Trying 0000:04:04.1
cpu0:4659)PCI: Announcing 0000:04:04.1
cpu0:4659)VMK_PCI: 746: device 000:004:04.1 capType 1 capIndex 72
cpu0:4659)VMK_PCI: 746: device 000:004:04.1 capType 5 capIndex 88
cpu0:4659)VMK_PCI: 746: device 000:004:04.1 capType 7 capIndex 64
cpu0:4659))<3>tg3: (0000:04:04.1) phy probe failed, err -19
cpu0:4659)VMK_PCI: 746: device 000:004:04.1 capType 3 capIndex 80
cpu3:4679)PCI: Trying 0000:04:04.0
cpu3:4679)PCI: Announcing 0000:04.04.0
cpu3:4679)PCI: Trying 0000:04:04.1
cpu3:4679)PCI: Announcing 0000:04.04.1
cpu1:4679)PCI: Trying 0000:04:04.0
cpu1:4679)PCI: Announcing 0000:04.04.0
cpu1:4679)PCI: Trying 0000:04:04.1
cpu1:4679)PCI: Announcing 0000:04.04.1

Everything containing 'tg3' in /var/log/messages (I've omitted the timestamps and 'vmkernel:' to save some typing..)

Code:

cpu2:4659)Loading module tg3 ...
cpu2:4659)Elf: 2187: <tg3> provides name-space <esx:nover>
cpu2:4659)Elf: 1612: <tg3> : R_X86_64_32 relocation(s) ignored
cpu2:4659)module heap vmklnx_tg3: creation succeded. id = 0x4100b8c00000
cpu2:4659)PCI: driver tg3 is looking for devices
<6>tg3.o:v3.86.0.1-vmw (November 9, 2007)
cpu0:4659)<3>: Problem fetching invariants of chip, aborting.
cpu0:4659)PCI: driver tg3 claimed 0 device
cpu0:4659)Mod: 4163 Initialization of tg3 succeded with module ID 30.
cpu0:4659)tg3 loaded successfully.

The "<3>: Problem fetching invariants of chip, aborting." tells me that there's something missing in the source for the driver, I've tried browsing the header and c-code from the oss source package for the tg3 driver, but since I'm not a c programmer, I have some difficulties to actually follow the code.. If it was only written in php..:-)

[AndersN]

I just noticed something I overlooked earlier.

If I just use 'lspci' without the '-p', the console shows the Broadcom NICs with bus id, pci id AND the full name of the NICs. But with the '-p' switch the driver is not bound to the NICs.

I also tested with v5.0 RC, it gives the exact same result as with v4.1u1.

[AndersN]

Is there anybody out there that can make the necesarry source code changes to enable the driver to correctly use the BCM5780 NICs?

I have looked in the code, but I'm just not knowledgeable enough in programming to locate the problem.

[tonybourke]

I haven't been able to get ESXi 4.x to run with the Broadcoms, however ESXi 5.0 installed with the broadcom NIC (it used the tg3 driver, and it was already in the default install disk).

[AndersN]

Will try that Tony, thank you!

[AndersN]

Tony Bourke wrote:

I haven't been able to get ESXi 4.x to run with the Broadcoms, however ESXi 5.0 installed with the broadcom NIC (it used the tg3 driver, and it was already in the default install disk).

Tried that, still the exact same problem as with v4.1. Highly annoying..

I guess I have to give up and put in some Intel PRO/1000 cards in the box.