HP NC380T Recognized by hardware but not by ESXi

[seanacais]

I'm running ESXi 4.1 Update 3 on a lab machine.

I recently purchased some HP NC380T Dual Gigabit network cards and installed them.

I have no issues seeing the cards in BIOS, and just about everywhere I look in the logs the system is aware the card is there except that they are not assigned a vmnic number.  I expect these ports to be assigned vmnic2 and vmnic3 since the motherboard has two gigabit ethernet ports on it (and they are working fine)

So I'm really at a loss as to why these ports are not being recognized so I can use them in ESXi.  Any insight is appreciated!!!

Please find below additional information which should be helpful in understanding the problem.

Thank you,

Kevin

Only the motherboard ports are shown as NICs.

~ # esxcfg-nics -l

Name    PCI           Driver      Link Speed     Duplex MAC Address       MTU    Description

vmnic0  0000:02:00.00 tg3         Up   1000Mbps  Full   00:1e:8c:51:62:26 1500   Broadcom Corporation NetXtreme BCM5721 Gigabit Ethernet

vmnic1  0000:03:00.00 tg3         Up   1000Mbps  Full   00:1e:8c:51:63:10 1500   Broadcom Corporation NetXtreme BCM5721 Gigabit Ethernet

But lspci is showing that the NIC itself is seen at the PCI level:

~ # lspci | grep vmnic

000:002:00.0 Network controller: Broadcom Corporation NetXtreme BCM5721 Gigabit Ethernet [vmnic0]

000:003:00.0 Network controller: Broadcom Corporation NetXtreme BCM5721 Gigabit Ethernet [vmnic1]

000:006:04.0 Network controller: Broadcom Corporation NC380T PCI Express Dual Port Multifunction Gigabit Server Adapt [vmnic2]

000:007:05.0 Network controller: Broadcom Corporation NC380T PCI Express Dual Port Multifunction Gigabit Server Adapt [vmnic3]

And ethtool is showing the correct PCI addresses and drivers being assigned:

~ # ethtool -i vmnic0

driver: tg3

version: 3.110h.v41.1

firmware-version: 5721-v3.55a

bus-info: 0000:02:00.0

~ # ethtool -i vmnic1

driver: tg3

version: 3.110h.v41.1

firmware-version: 5721-v3.55a

bus-info: 0000:03:00.0

~ # ethtool -i vmnic2

driver: bnx2

version: 2.0.22f.v41.2

firmware-version: bc 1.9.6

bus-info: 0000:06:04.0

~ # ethtool -i vmnic3

driver: bnx2

version: 2.0.22f.v41.2

firmware-version: bc 1.9.6

bus-info: 0000:07:05.0

The following is from /var/log/messages.  I see the error message, but I'm not sure exactly what it means and google didn't help.  What is really odd is that the bnx2-cnic driver appears to be picking these ports up, but I do not see the ports listed in the storage adapters either.

Jun 12 14:56:47 vmkernel: 0:00:00:16.962 cpu7:4618)PCI: Announcing 0000:06:04.0

Jun 12 14:56:47 vmkernel: <6>Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v2.0.22f.v41.2 (Feb 11, 2011)

Jun 12 14:56:47 vmkernel: 0:00:00:16.962 cpu7:4618)socket buffer cache creation succeeded for vmnic%d

Jun 12 14:56:47 vmkernel: 0:00:00:16.962 cpu7:4618)VMK_PCI: 746: device 000:006:04.0 capType 1 capIndex 72

Jun 12 14:56:47 vmkernel: 0:00:00:16.962 cpu7:4618)VMK_PCI: 746: device 000:006:04.0 capType 7 capIndex 64

Jun 12 14:56:47 vmkernel: 0:00:00:16.962 cpu7:4618)VMK_PCI: 746: device 000:006:04.0 capType 5 capIndex 88

Jun 12 14:56:47 vmkernel: 0:00:00:16.963 cpu7:4618)<6>vmnic2: Broadcom NetXtreme II BCM5706 1000Base-T (A2) PCI-X 64-bit 100MHz found at mem d0

Jun 12 14:56:47 vmkernel: 0:00:00:16.963 cpu7:4618)PCI: driver bnx2 claimed device 0000:06:04.0

Jun 12 14:56:47 vmkernel: 0:00:00:16.963 cpu7:4618)PCI: Registering network device 0000:06:04.0

Jun 12 14:56:47 vmkernel: 0:00:00:16.963 cpu7:4618)VMK_PCI: 646: Device 000:006:04.0 name: vmnic2

Jun 12 14:56:47 vmkernel: 0:00:00:16.963 cpu7:4618)Uplink: 11078: Device vmnic2 not yet opened

Jun 12 14:56:47 vmkernel: 0:00:00:16.963 cpu7:4618)LinPCI: LinuxPCI_DeviceClaimed: Device 6:20 claimed.

Jun 12 14:56:47 vmkernel:

Jun 12 14:56:47 vmkernel: 0:00:00:16.963 cpu7:4618)PCI: Trying 0000:07:05.0

Jun 12 14:56:47 vmkernel: 0:00:00:16.963 cpu7:4618)PCI: Announcing 0000:07:05.0

Jun 12 14:56:47 vmkernel: 0:00:00:16.963 cpu7:4618)VMK_PCI: 746: device 000:007:05.0 capType 1 capIndex 72

Jun 12 14:56:47 vmkernel: 0:00:00:16.963 cpu7:4618)VMK_PCI: 746: device 000:007:05.0 capType 7 capIndex 64

Jun 12 14:56:47 vmkernel: 0:00:00:16.963 cpu7:4618)VMK_PCI: 746: device 000:007:05.0 capType 5 capIndex 88

Jun 12 14:56:47 vmkernel: 0:00:00:16.963 cpu7:4618)<6>vmnic3: Broadcom NetXtreme II BCM5706 1000Base-T (A2) PCI-X 64-bit 100MHz found at mem d4

Jun 12 14:56:47 vmkernel: 0:00:00:16.963 cpu7:4618)PCI: driver bnx2 claimed device 0000:07:05.0

Jun 12 14:56:47 vmkernel: 0:00:00:16.963 cpu7:4618)PCI: Registering network device 0000:07:05.0

Jun 12 14:56:47 vmkernel: 0:00:00:16.963 cpu7:4618)VMK_PCI: 646: Device 000:007:05.0 name: vmnic3

Jun 12 14:56:47 vmkernel: 0:00:00:16.963 cpu7:4618)Uplink: 11078: Device vmnic3 not yet opened

Jun 12 14:56:47 vmkernel: 0:00:00:16.963 cpu7:4618)LinPCI: LinuxPCI_DeviceClaimed: Device 7:28 claimed.

Jun 12 14:56:47 vmkernel:

Jun 12 14:56:47 vmkernel: 0:00:00:16.963 cpu7:4618)PCI: driver bnx2 claimed 2 devices

Jun 12 14:56:47 vmkernel: 0:00:00:16.963 cpu7:4618)Uplink: 13003: Opening device vmnic2

Jun 12 14:56:47 vmkernel: 0:00:00:16.963 cpu0:4129)WARNING: VMK_VECTOR: 104: Bad vector 0 specified

Jun 12 14:56:47 vmkernel: 0:00:00:16.963 cpu0:4129)WARNING: vmklinux26: vmklnx_request_irq: Couldn't register vector 0x0

Jun 12 14:56:47 vmkernel: 0:00:00:16.963 cpu0:4129)Uplink: 12466: Device open handler failed for : vmnic2

Jun 12 14:56:47 vmkernel:

Jun 12 14:56:47 vmkernel: 0:00:00:16.963 cpu5:4618)Uplink: 13026: Device vmnic2 failed to open

Jun 12 14:56:47 vmkernel: 0:00:00:16.963 cpu5:4618)Uplink: 12209: Failed to close vmnic2: not found

Jun 12 14:56:47 vmkernel: 0:00:00:16.963 cpu5:4618)Uplink: 13003: Opening device vmnic3

Jun 12 14:56:47 vmkernel: 0:00:00:16.963 cpu0:4129)WARNING: VMK_VECTOR: 104: Bad vector 0 specified

Jun 12 14:56:47 vmkernel: 0:00:00:16.963 cpu0:4129)WARNING: vmklinux26: vmklnx_request_irq: Couldn't register vector 0x0

Jun 12 14:56:47 vmkernel: 0:00:00:16.963 cpu0:4129)Uplink: 12466: Device open handler failed for : vmnic3

Jun 12 14:56:47 vmkernel:

Jun 12 14:56:47 vmkernel: 0:00:00:16.963 cpu5:4618)Uplink: 13026: Device vmnic3 failed to open

Jun 12 14:56:47 vmkernel: 0:00:00:16.963 cpu5:4618)Uplink: 12209: Failed to close vmnic3: not found

Jun 12 14:56:47 vmkernel: 0:00:00:16.963 cpu5:4618)Mod: 4163: Initialization of bnx2 succeeded with module ID 32.

Jun 12 14:56:47 vmkernel: 0:00:00:16.963 cpu5:4618)bnx2 loaded successfully.

Jun 12 14:56:47 vmkernel:

Jun 12 14:56:48 vmkernel: sysboot: Loading VMkernel Module 'cnic'

Jun 12 14:56:48 vmkernel: 0:00:00:17.406 cpu6:4626)Loading module cnic ...

Jun 12 14:56:48 vmkernel: 0:00:00:17.406 cpu6:4626)Elf: 2187: <cnic> provides name-space <esx:nover>

Jun 12 14:56:48 vmkernel: 0:00:00:17.407 cpu6:4626)module heap : Initial heap size : 1048576, max heap size: 68665344

Jun 12 14:56:48 vmkernel: 0:00:00:17.407 cpu6:4626)module heap vmklnx_cnic: creation succeeded. id = 0x4100bc000000

Jun 12 14:56:48 vmkernel: 0:00:00:17.407 cpu6:4626)module skb heap : Initial heap size : 524288, max heap size: 23068672

Jun 12 14:56:48 vmkernel: 0:00:00:17.408 cpu6:4626)module skb heap : creation succeeded

Jun 12 14:56:48 vmkernel: <6>Broadcom NetXtreme II CNIC Driver cnic v1.10.2q.v41.9 (Feb. 11, 2011)

Jun 12 14:56:48 vmkernel: 0:00:00:17.408 cpu6:4626)<6>cnic: Added CNIC device: vmnic3

Jun 12 14:56:48 vmkernel: 0:00:00:17.408 cpu6:4626)<6>cnic: Added CNIC device: vmnic2

Jun 12 14:56:48 vmkernel: 0:00:00:17.408 cpu6:4626)Mod: 4163: Initialization of cnic succeeded with module ID 33.

Jun 12 14:56:48 vmkernel: 0:00:00:17.408 cpu6:4626)cnic loaded successfully.

I've also installed the recommended vmware drivers

~ # esxupdate query --vib-view | grep bnx2
deb_vmware-esx-drivers-net-bnx2_400.2.0.7d-6vmw.3.26.800380                 installed     2012-08-02T20:55:21+00:00
deb_vmware-esx-drivers-scsi-bnx2i_400.1.8.11t5.rc2.8.1-4vmw.3.26.800380     installed     2012-08-02T20:55:21+00:00
deb_vmware-esx-drivers-net-bnx2x_400.1.54.1.v41.1-3vmw.3.26.800380          installed     2012-08-02T20:55:21+00:00
cross_vmware-esx-drivers-net-bnx2_400.2.2.3m.v41.2-1vmw.0.0.00000           uninstalled   2013-06-11T21:29:41.637931+00:00
cross_vmware-esx-drivers-scsi-bnx2i_400.2.76.56.v41.2-1vmw.0.0.00000        uninstalled   2013-06-11T21:30:39.798582+00:00
cross_vmware-esx-drivers-net-bnx2x_400.1.76.54.v41.1-1vmw.0.0.00000         uninstalled   2013-06-11T21:31:37.731451+00:00
cross_vmware-esx-drivers-net-bnx2_400.2.0.22f.v41.2-1vmw.0.0.00000          installed     2013-06-11T21:46:02.744078+00:00
cross_vmware-esx-drivers-net-bnx2x_400.1.62.15.v41.2-1vmw.0.0.00000         installed     2013-06-11T21:46:41.777374+00:00
cross_vmware-esx-drivers-scsi-bnx2i_400.1.9.1t.v41.2-1vmw.0.0.00000         installed     2013-06-11T21:47:14.932194+00:00
~ #

Finally, I haven't done any specific configuration with the bnx2 module

~ # esxcfg-module -i bnx2
esxcfg-module module information
input file: /usr/lib/vmware/vmkmod/bnx2.o
License: GPL
Version:
Name-space:
Required name-spaces:
  com.vmware.vmkapi@v1_1_0_0
Parameters:
  disable_msi: int
   Disable Message Signaled Interrupt (MSI)
  force_netq: array of int
   Option used for 5709/5716 only: Enforce the number of NetQueues per port (allowed values: -1 to 7 queues: 1-7 will force the number of NetQueues for the  given device, 0 to disable NetQueue, -1 to use the default driver NetQueues value) [Maximum supported NIC's = 32] [example usage: force_net=-1,0,1,2: This corresponds to the first 5709/5716 to use the default number of NetQueues, disable NetQueue on the second 5709/5716, use 1 NetQueue on the third 5709/5716use 2 NetQueues on the fourth 5709/5716]
  heap_initial: int
   Initial heap size allocated for the driver.
  heap_max: int
   Maximum attainable heap size for the driver.
  skb_heap_initial: int
   Initial private socket buffer heap size allocated for the driver.
  skb_heap_max: int
   Maximum attainable private socket buffer heap size for the driver.
  stop_on_tx_timeout: int
   For debugging purposes, prevent a chip  reset when a tx timeout occurs

Again, any insight or suggestions are welcome!

Kevin

[a_nut_in]

Jun 12 14:56:47 vmkernel: 0:00:00:16.963 cpu0:4129)WARNING: VMK_VECTOR: 104: Bad vector 0 specified

Jun 12 14:56:47 vmkernel: 0:00:00:16.963 cpu0:4129)WARNING: vmklinux26: vmklnx_request_irq: Couldn't register vector 0x0

Looks like IRQ assignment is failing.

Try this and see if this helps:

1. Run the following command "esxcfg-module -s 'disable_msi=1' bnx2" without the quotes and reboot and check

2. Try updating the drivers to this and check

3. Check and change the IRQ assignments in BIOS and see if that helps

[seanacais]

I forgot to mention that I had tried the disable_msi=1 prior to my post with no luck.

The different driver did not make a difference either.

~ # ethtool -i vmnic2

driver: bnx2

version: 2.1.5d.v40.1

firmware-version: bc 1.9.6

bus-info: 0000:06:04.0

There is not a lot of choices for setting the interrupts on the motherboard BIOS.  The only choices I had for setting the interrupt where 18h, 19h, BBS, and auto and that  was in the NIC setup, not the motherboard BIOS.  For the record, the MB is an ASUS KFSN4-DRE/SAS with Dual Quad Core Opteron 8360.  I tried all three with the new driver and alternating disable_msi between 1 and 0.  I saw no difference in any configuration in the messages file.

Kevin

[a_nut_in]

Try

esxcfg-module -s force_netq=0,0,0,0 bnx2

Then reboot and check

If that does not work, try

esxcfg-module -s force_netq=1,1,1,1 bnx2

Regards

a

[seanacais]

There is no joy in that solution 😞

Again I see no difference in the messages file.

I did note that the command line you gave removed the disable_msi option from the module options.

Using the command line

esxcfg-module -s 'force_netq=0,0,0,0 disable_msi=1' bnx2

I set both options and tried that combination as well.

Unfortunately there was no change in the behavior.

I swapped out the NC380T with a different card and ran through all the scenarios that have been discussed.  Same behavior in all cases.

Kevin

[a_nut_in]

Not sure this is going to help but worth a try

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=32...

<snip>

Customers using this driver CD should update all four drivers included

</snip>

Could you try uninstalling the earlier drivers, reboot the system and then try installing all four drivers in order and check?

[seanacais]

These are the drivers that I had started with.  All four drivers had been updated as specified in the instructions.

I did reinstall this set of drivers and attempted the force_netq option on the driver but haven't seen any change in behavior.

Kevin

[a_nut_in]

Well nothing else I can think of, to be honest here's where a ticket with VMware should be the next steps. IRQ seems to be the culprit, but not sure if that's a driver or firmware or ESXi causing the device not to load.

You could try one other thing though:

1. Power off all VM's on the host
2. Power down the host
3. Remove the power cable
4. Press and hold the power button for about 10 secs
5. Connect the power cable back on
6. Power on the system and check

Long shot and honestly, I don't expect it to work, but worth a try!

[seanacais]

a_nut_in:

there are no VM's on the host.  The power cable has been removed multiple times when I swapped out the cards.  I think I just have to accept that this motherboard is not going to work with these NIC's.  Too bad.  I got them for a good price on e-bay!

I really appreciate the help!!

Thanks,

Kevin