Can't add 10GBe physical nic

[ElliottN]

I've been having a dog of a time trying to add a dual 10GBe nic to ESXi.  I've tried installing the drivers and everything appears that it is installing but ESXi doesn't seem to be loading the drivers for some reason.  This is an Intel NUC NUC7i5BNK host with a OWC Mercury Helios 3  Thunderbolt 3 PCIe Expansion Chassis housing the NIC.  The NIC and the expansion chassis is known good on other OS's so everything points to a configuration issue on ESXi or in the bios of the NUC.  I've enabled thunderbolt at boot so I think that rules out the BIOS.

[root@esxi1:~] lspci -v | grep -A1 -i ethernet

0000:00:1f.6 Ethernet controller Network controller: Intel Corporation Ethernet Connection (4) I219-V [vmnic0]

         Class 0200: 8086:15d8

--

0000:06:00.0 Ethernet controller Network controller: Intel(R) 82599 10 Gigabit Dual Port Network Connection

         Class 0200: 8086:10fb

--

0000:06:00.1 Ethernet controller Network controller: Intel(R) 82599 10 Gigabit Dual Port Network Connection

         Class 0200: 8086:10fb

[root@esxi1:~] vmkload_mod -l |grep drivername

[root@esxi1:~] vmkload_mod -l |grep ixgben

[root@esxi1:~] esxcfg-nics -l

Name    PCI          Driver      Link Speed      Duplex MAC Address       MTU    Description

vmnic0  0000:00:1f.6 ne1000      Up   1000Mbps   Full   94:c6:91:15:0b:e4 9000   Intel Corporation Ethernet Connection (4) I219-V

[root@esxi1:~] esxcli software vib install -d "/vmfs/volumes/datastore1/VMW-ESX-

6.7.0-ixgben-1.7.10-offline_bundle-10105563.zip"

Installation Result

   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.

   Reboot Required: true

   VIBs Installed: INT_bootbank_ixgben_1.7.10-1OEM.670.0.0.8169922

   VIBs Removed: INT_bootbank_ixgben_1.7.1-1OEM.670.0.0.7535516

   VIBs Skipped:

Still no joy after reboot

[root@esxi1:~] esxcli hardware pci list

0000:06:00.0

   Address: 0000:06:00.0

   Segment: 0x0000

   Bus: 0x06

   Slot: 0x00

   Function: 0x0

   VMkernel Name:

   Vendor Name: Intel Corporation

   Device Name: 82599EB 10-Gigabit SFI/SFP+ Network Connection

   Configured Owner: VMkernel

   Current Owner: VMkernel

   Vendor ID: 0x8086

   Device ID: 0x10fb

   SubVendor ID: 0x8086

   SubDevice ID: 0x7a11

   Device Class: 0x0200

   Device Class Name: Ethernet controller

   Programming Interface: 0x00

   Revision ID: 0x01

   Interrupt Line: 0xff

   IRQ: 255

   Interrupt Vector: 0x00

   PCI Pin: 0x00

   Spawned Bus: 0x00

   Flags: 0x3219

   Module ID: -1

   Module Name: None

   Chassis: 0

   Physical Slot: 4294967295

   Slot Description:

   Passthru Capable: true

   Parent Device: PCI 0:5:1:0

   Dependent Device: PCI 0:6:0:0

   Reset Method: Function reset

   FPT Sharable: true

0000:06:00.1

   Address: 0000:06:00.1

   Segment: 0x0000

   Bus: 0x06

   Slot: 0x00

   Function: 0x1

   VMkernel Name:

   Vendor Name: Intel Corporation

   Device Name: 82599EB 10-Gigabit SFI/SFP+ Network Connection

   Configured Owner: VMkernel

   Current Owner: VMkernel

   Vendor ID: 0x8086

   Device ID: 0x10fb

   SubVendor ID: 0x8086

   SubDevice ID: 0x7a11

   Device Class: 0x0200

   Device Class Name: Ethernet controller

   Programming Interface: 0x00

   Revision ID: 0x01

   Interrupt Line: 0xff

   IRQ: 255

   Interrupt Vector: 0x00

   PCI Pin: 0x01

   Spawned Bus: 0x00

   Flags: 0x3219

   Module ID: -1

   Module Name: None

   Chassis: 0

   Physical Slot: 4294967295

   Slot Description:

   Passthru Capable: true

   Parent Device: PCI 0:5:1:0

   Dependent Device: PCI 0:6:0:1

   Reset Method: Function reset

   FPT Sharable: true

[root@esxi1:~] esxcfg-nics -l

Name    PCI          Driver      Link Speed      Duplex MAC Address       MTU    Description

vmnic0  0000:00:1f.6 ne1000      Up   1000Mbps   Full   94:c6:91:15:0b:e4 9000   Intel Corporation Ethernet Connection (4) I219-V

[TomAtTritium]

I am using Dell / Intel X520-DA2 Dual Port 10Gbe SFP Network Adapter NIC 942V6 Low Profile together with ESXi 6.7 U3.

NIC shows as "82599EB 10-Gigabit SFP+ Network Connection" in hardware but are not available as a NIC.

I checked the log..

cat /var/log/vmkernel.log  | grep ixgben

and found that the SFP+ that i am trying to use is unsupported and that has stopped the 10GB from initializing properly:

2019-09-10T20:49:43.630Z cpu51:2098034)ixgben: ixgben_InitSharedCode:185: Unsupported SFP+ or QSFP detected. Try to replace a supported module then reload the driver.

I will try another SFP+ or DAC, and update this post with result.

[4n3w]

Hey there,

I have what looks like the same issue as ElliottN with some X520-DA2 Dual Port 10GE NIC (FWIW with correct and genuine Intel SFP+ modules) in an Intel NUC8i5BEH (with latest BIOS firmware) machine connected via a Sonnet Echo Express SEL Thunderbolt 3 PCIe expansion chassis. I know this card works just fine because when I boot into ubuntu I'm able to use the card without error.

I've tried many versions of the ixgben driver, all have yielded the same results (PCI device is detected but esxcfg-nics -l only yields the built in ethernet adapter -- here's a list of the drivers I've tried so far:

1.7.1-1OEM.670.0.0.7535516

1.7.1.16-1vmw.670.3.73.14320388

1.7.10-1OEM.670.0.0.8169922

1.7.15-1OEM.670.0.0.8169922

1.7.17-1OEM.670.0.0.8169922

1.7.20-1OEM.670.0.0.8169922

Every time I reboot after esxcli software vib install I get only the built-in ethernet adapter, and not the ixgben driven adapter.

Also, when I run vmkload_mod -l | grep ixgben I've never seen the ixgben driver get loaded, although I do see the ixgben driver eg:

esxcli software vib list | grep ixgben

ixgben                         1.7.10-1OEM.670.0.0.8169922           INT     VMwareCertified   2019-09-14

Also, here's the evidence the PCI devices are detected.

lspci -v | grep -A1 -i ethernet

0000:00:1f.6 Network controller Ethernet controller: Intel Corporation Ethernet Connection (6) I219-V [vmnic0]

  Class 0200: 8086:15be

--

0000:07:00.0 Network controller Ethernet controller: Intel(R) 82599 10 Gigabit Dual Port Network Connection

  Class 0200: 8086:10fb

--

0000:07:00.1 Network controller Ethernet controller: Intel(R) 82599 10 Gigabit Dual Port Network Connection

  Class 0200: 8086:10fb

I'm relatively new to ESXi/vsphere but I've learned a lot troubleshooting this. Curious if there's something I'm missing to add the card? I don't see any errors like TomAtTritium does regarding the SFP+ modules. Any help would be appreciated.

Failing that, maybe a recommendation for a card that will work without this much hassle.

Thanks,

Andrew

[ashishsingh1508]

Most probably an unsupported SFP

OR

FW is not compatible

[4n3w]

Thanks for the reply. Al least for me the SFP+ modules are Intel FTLX8571D3BCVIT1 which are explicitly mentioned as compatible by Intel for this card (SFP+ Modules, SFP Modules, and Cables Compatible with Intel® Ethernet... ).

So, ashishsingh1508 can you kindly explain what firmware you're referring to? On the nic itself? I believe the firmware is current for the card.

[fizzyade]

I suspect this is because it's an old version of the card, I'm currently having the same problem.  I've ordered a newer version which will allow me to update the firmware in it.

You can tell it's the old version by the sub device id of 7a11, I believe this has a smaller flash on it and therefore cannot be updated past 16.5.20, that's the last update package I was able to find for my card that has the 7a11 sub id in the flash configuration file.

[TomHowarth]

not wanting to teach anybody to suck eggs here, but have you actually verified that this card and sfp are supported by VMware for the version if ESXi that you are attempting to install. the single source of truth is the VMware Hardware Compatibility list that is found here: VMware Compatibility Guide - System Search 

[4n3w]

Ugh, I had checked before but I just looked up the VID/DID; when I entered the subvendor ID I see that nothing comes up for that card. Well, thanks TomHowarth​ I'm just happy to understand why the card just wasn't working.

I had updated the firmware on the card earlier tonight but to no avail; so moral of the story: check the darn SVID.

[fizzyade]

Been trying various things, I have another X520 which is newer and has later firmware on, but it still doesn't work (and yes it's in the HCL for 7.2u2 which I'm running, VID/DID/SVID/SDID all match).

In vmkwarning.log I see the following messages relating to that PCI device:

0:00:00:05.352 cpu0:2097152)WARNING: PCI: 453: 0000:07:00.0: Failed to add BAR[2] (IO f=0x1 0x4020-0x4040) - out of resources on parent: 0000:06:01.0

0:00:00:05.352 cpu0:2097152)WARNING: PCI: 476: 0000:07:00.0: Failed to add BAR[2] (IO f=0x1 0x4020-0x4040) status: Limit exceeded

0:00:00:05.352 cpu0:2097152)WARNING: PCI: 453: 0000:07:00.1: Failed to add BAR[2] (IO f=0x1 0x4000-0x4020) - out of resources on parent: 0000:06:01.0

0:00:00:05.352 cpu0:2097152)WARNING: PCI: 476: 0000:07:00.1: Failed to add BAR[2] (IO f=0x1 0x4000-0x4020) status: Limit exceeded

0:00:00:05.360 cpu0:2097152)WARNING: PCI: 679: 0000:07:00.0: Unable to free BAR[2] (IO f=0x1 0x4020-0x4040): Limit exceeded

0:00:00:05.360 cpu0:2097152)WARNING: PCI: 679: 0000:07:00.0: Unable to free BAR[4] (MEM64 f=0x4 0x90100000-0x90104000): Not found

0:00:00:05.360 cpu0:2097152)WARNING: PCI: 679: 0000:07:00.1: Unable to free BAR[2] (IO f=0x1 0x4000-0x4020): Limit exceeded

0:00:00:05.360 cpu0:2097152)WARNING: PCI: 679: 0000:07:00.1: Unable to free BAR[4] (MEM64 f=0x4 0x90300000-0x90304000): Not found

Any other ideas?

[fizzyade]

This knowledge base entry contains the answer to my error I believe. VMware Knowledge Base

[scott28tt]

Moderator note: Moved to ESXi

[4n3w]

I'm stubborn and did the same as you fizzyade ​I bought a newer x520 as well and I get the same errors as you.

vmkernel.log:0:00:00:05.391 cpu0:2097152)WARNING: PCI: 452: 0000:07:00.0: Failed to add BAR[2] (IO f=0x1 0x4020-0x4040) - out of resources on parent: 0000:06:01.0

vmkernel.log:0:00:00:05.391 cpu0:2097152)WARNING: PCI: 475: 0000:07:00.0: Failed to add BAR[2] (IO f=0x1 0x4020-0x4040) status: Limit exceeded

vmkernel.log:0:00:00:05.391 cpu0:2097152)WARNING: PCI: 452: 0000:07:00.1: Failed to add BAR[2] (IO f=0x1 0x4000-0x4020) - out of resources on parent: 0000:06:01.0

vmkernel.log:0:00:00:05.391 cpu0:2097152)WARNING: PCI: 475: 0000:07:00.1: Failed to add BAR[2] (IO f=0x1 0x4000-0x4020) status: Limit exceeded

vmkernel.log:0:00:00:05.398 cpu0:2097152)WARNING: PCI: 678: 0000:07:00.0: Unable to free BAR[2] (IO f=0x1 0x4020-0x4040): Limit exceeded

vmkernel.log:0:00:00:05.398 cpu0:2097152)WARNING: PCI: 678: 0000:07:00.0: Unable to free BAR[4] (MEM64 f=0x4 0x90080000-0x90084000): Not found

vmkernel.log:0:00:00:05.398 cpu0:2097152)WARNING: PCI: 678: 0000:07:00.1: Unable to free BAR[2] (IO f=0x1 0x4000-0x4020): Limit exceeded

vmkernel.log:0:00:00:05.398 cpu0:2097152)WARNING: PCI: 678: 0000:07:00.1: Unable to free BAR[4] (MEM64 f=0x4 0x90200000-0x90204000): Not found

vmkernel.log:2019-09-27T22:14:35.641Z cpu6:2097563)DEBUG (ne1000): mapped BAR[0]: start = c0b00000, size = 20000, mapped to 450185f40000

vmkwarning.log:0:00:00:05.391 cpu0:2097152)WARNING: PCI: 452: 0000:07:00.0: Failed to add BAR[2] (IO f=0x1 0x4020-0x4040) - out of resources on parent: 0000:06:01.0

vmkwarning.log:0:00:00:05.391 cpu0:2097152)WARNING: PCI: 475: 0000:07:00.0: Failed to add BAR[2] (IO f=0x1 0x4020-0x4040) status: Limit exceeded

vmkwarning.log:0:00:00:05.391 cpu0:2097152)WARNING: PCI: 452: 0000:07:00.1: Failed to add BAR[2] (IO f=0x1 0x4000-0x4020) - out of resources on parent: 0000:06:01.0

vmkwarning.log:0:00:00:05.391 cpu0:2097152)WARNING: PCI: 475: 0000:07:00.1: Failed to add BAR[2] (IO f=0x1 0x4000-0x4020) status: Limit exceeded

vmkwarning.log:0:00:00:05.398 cpu0:2097152)WARNING: PCI: 678: 0000:07:00.0: Unable to free BAR[2] (IO f=0x1 0x4020-0x4040): Limit exceeded

vmkwarning.log:0:00:00:05.398 cpu0:2097152)WARNING: PCI: 678: 0000:07:00.0: Unable to free BAR[4] (MEM64 f=0x4 0x90080000-0x90084000): Not found

vmkwarning.log:0:00:00:05.398 cpu0:2097152)WARNING: PCI: 678: 0000:07:00.1: Unable to free BAR[2] (IO f=0x1 0x4000-0x4020): Limit exceeded

vmkwarning.log:0:00:00:05.398 cpu0:2097152)WARNING: PCI: 678: 0000:07:00.1: Unable to free BAR[4] (MEM64 f=0x4 0x90200000-0x90204000): Not found

We're not the only ones having this issue: https://medium.com/@adrian_65383/ok-so-i-got-that-10gtek-and-plugged-it-in-but-having-the-same-issue...

Any chance this can be addressed in an ESXi release, or would this have to be solved by a BIOS update by Intel?

[fizzyade]

That’s actually my post on medium!

I did solve this.....I backed up the hard disks of my virtual machines, wiped the NUC and installed Proxmox. Booted up straight away recognising the x520 in the Node Lite enclosure.

works like a dream.  Probably not a useful “solution“ for a lot of people, but i haven’t looked back, loving lxc containers as well.

i Bought a 4 port Intel  Ethernet card (1G) off eBay cheap to check in ESXi that the node lite was actually functional with a card in and that it wasn’t part of the problem:  ESXi booted up and all 4 ports appeared in

[ez12a323]

Ressurrected, but I'm experiencing this issue trying to use Auto Deploy to upgrade some Mac Pro hosts to 6.7 U3. The Mac Pro 6,1 host, the Sonnet Twin 10G Thunderbolt Adapter (at least the VID:DID), appear on VMware's compatibility list. The other mac hosts are running fine on 6.5 Update 2. This was running a local ESXi 6.5 U2 installation with no problems.

I've tried the 1.7.10 (working on 6.5 U2), 1.7.15 (shown on for ESXi U3), and 1.7.20 (latest available on VMWare.com)

vmkwarning.log:

"

0:00:00:06.830 cpu0:2097152)WARNING: PCI: 452: 0000:60:00.0: Failed to add BAR[0] (MEM64 f=0xc 0x0-0x200000) - out of resources on parent: 0000:5f:03.0

0:00:00:06.830 cpu0:2097152)WARNING: PCI: 475: 0000:60:00.0: Failed to add BAR[0] (MEM64 f=0xc 0x0-0x200000) status: Limit exceeded

0:00:00:06.830 cpu0:2097152)WARNING: PCI: 225: PF mem (0x0 - 0x1fffff) out of range

0:00:00:06.830 cpu0:2097152)WARNING: PCI: 452: 0000:60:00.1: Failed to add BAR[0] (MEM64 f=0xc 0x0-0x200000) - out of resources on parent: 0000:5f:03.0

0:00:00:06.830 cpu0:2097152)WARNING: PCI: 475: 0000:60:00.1: Failed to add BAR[0] (MEM64 f=0xc 0x0-0x200000) status: Limit exceeded

0:00:00:07.846 cpu0:2097152)WARNING: PCI: 678: 0000:60:00.0: Unable to free BAR[0] (MEM64 f=0xc 0x0-0x200000): Limit exceeded

0:00:00:07.846 cpu0:2097152)WARNING: PCI: 678: 0000:60:00.0: Unable to free BAR[4] (MEM64 f=0xc 0x0-0x4000): Limit exceeded

0:00:00:07.847 cpu0:2097152)WARNING: PCI: 678: 0000:60:00.1: Unable to free BAR[0] (MEM64 f=0xc 0x0-0x200000): Limit exceeded

0:00:00:07.847 cpu0:2097152)WARNING: PCI: 678: 0000:60:00.1: Unable to free BAR[4] (MEM64 f=0xc 0x0-0x4000): Limit exceeded

"

lspci -vvv output:

"

0000:60:00.0 Network controller Ethernet controller: Intel(R) Ethernet Controller 10G X550

         Class 0200: 8086:1563

0000:60:00.1 Network controller Ethernet controller: Intel(R) Ethernet Controller 10G X550

         Class 0200: 8086:1563

"

I've tried doing a PRAM reset as well if that would resolve.

I cant seem to get the NICs to load anymore..even if I try to boot using the same image profile as the running/working nodes.