I've been having a dog of a time trying to add a dual 10GBe nic to ESXi. I've tried installing the drivers and everything appears that it is installing but ESXi doesn't seem to be loading the drivers for some reason. This is an Intel NUC NUC7i5BNK host with a OWC Mercury Helios 3 Thunderbolt 3 PCIe Expansion Chassis housing the NIC. The NIC and the expansion chassis is known good on other OS's so everything points to a configuration issue on ESXi or in the bios of the NUC. I've enabled thunderbolt at boot so I think that rules out the BIOS.
[root@esxi1:~] lspci -v | grep -A1 -i ethernet
0000:00:1f.6 Ethernet controller Network controller: Intel Corporation Ethernet Connection (4) I219-V [vmnic0]
Class 0200: 8086:15d8
--
0000:06:00.0 Ethernet controller Network controller: Intel(R) 82599 10 Gigabit Dual Port Network Connection
Class 0200: 8086:10fb
--
0000:06:00.1 Ethernet controller Network controller: Intel(R) 82599 10 Gigabit Dual Port Network Connection
Class 0200: 8086:10fb
[root@esxi1:~] vmkload_mod -l |grep drivername
[root@esxi1:~] vmkload_mod -l |grep ixgben
[root@esxi1:~] esxcfg-nics -l
Name PCI Driver Link Speed Duplex MAC Address MTU Description
vmnic0 0000:00:1f.6 ne1000 Up 1000Mbps Full 94:c6:91:15:0b:e4 9000 Intel Corporation Ethernet Connection (4) I219-V
[root@esxi1:~] esxcli software vib install -d "/vmfs/volumes/datastore1/VMW-ESX-
6.7.0-ixgben-1.7.10-offline_bundle-10105563.zip"
Installation Result
Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
Reboot Required: true
VIBs Installed: INT_bootbank_ixgben_1.7.10-1OEM.670.0.0.8169922
VIBs Removed: INT_bootbank_ixgben_1.7.1-1OEM.670.0.0.7535516
VIBs Skipped:
Still no joy after reboot
[root@esxi1:~] esxcli hardware pci list
0000:06:00.0
Address: 0000:06:00.0
Segment: 0x0000
Bus: 0x06
Slot: 0x00
Function: 0x0
VMkernel Name:
Vendor Name: Intel Corporation
Device Name: 82599EB 10-Gigabit SFI/SFP+ Network Connection
Configured Owner: VMkernel
Current Owner: VMkernel
Vendor ID: 0x8086
Device ID: 0x10fb
SubVendor ID: 0x8086
SubDevice ID: 0x7a11
Device Class: 0x0200
Device Class Name: Ethernet controller
Programming Interface: 0x00
Revision ID: 0x01
Interrupt Line: 0xff
IRQ: 255
Interrupt Vector: 0x00
PCI Pin: 0x00
Spawned Bus: 0x00
Flags: 0x3219
Module ID: -1
Module Name: None
Chassis: 0
Physical Slot: 4294967295
Slot Description:
Passthru Capable: true
Parent Device: PCI 0:5:1:0
Dependent Device: PCI 0:6:0:0
Reset Method: Function reset
FPT Sharable: true
0000:06:00.1
Address: 0000:06:00.1
Segment: 0x0000
Bus: 0x06
Slot: 0x00
Function: 0x1
VMkernel Name:
Vendor Name: Intel Corporation
Device Name: 82599EB 10-Gigabit SFI/SFP+ Network Connection
Configured Owner: VMkernel
Current Owner: VMkernel
Vendor ID: 0x8086
Device ID: 0x10fb
SubVendor ID: 0x8086
SubDevice ID: 0x7a11
Device Class: 0x0200
Device Class Name: Ethernet controller
Programming Interface: 0x00
Revision ID: 0x01
Interrupt Line: 0xff
IRQ: 255
Interrupt Vector: 0x00
PCI Pin: 0x01
Spawned Bus: 0x00
Flags: 0x3219
Module ID: -1
Module Name: None
Chassis: 0
Physical Slot: 4294967295
Slot Description:
Passthru Capable: true
Parent Device: PCI 0:5:1:0
Dependent Device: PCI 0:6:0:1
Reset Method: Function reset
FPT Sharable: true
[root@esxi1:~] esxcfg-nics -l
Name PCI Driver Link Speed Duplex MAC Address MTU Description
vmnic0 0000:00:1f.6 ne1000 Up 1000Mbps Full 94:c6:91:15:0b:e4 9000 Intel Corporation Ethernet Connection (4) I219-V
I am using Dell / Intel X520-DA2 Dual Port 10Gbe SFP Network Adapter NIC 942V6 Low Profile together with ESXi 6.7 U3.
NIC shows as "82599EB 10-Gigabit SFP+ Network Connection" in hardware but are not available as a NIC.
I checked the log..
cat /var/log/vmkernel.log | grep ixgben
and found that the SFP+ that i am trying to use is unsupported and that has stopped the 10GB from initializing properly:
2019-09-10T20:49:43.630Z cpu51:2098034)ixgben: ixgben_InitSharedCode:185: Unsupported SFP+ or QSFP detected. Try to replace a supported module then reload the driver.
I will try another SFP+ or DAC, and update this post with result.
Hey there,
I have what looks like the same issue as ElliottN with some X520-DA2 Dual Port 10GE NIC (FWIW with correct and genuine Intel SFP+ modules) in an Intel NUC8i5BEH (with latest BIOS firmware) machine connected via a Sonnet Echo Express SEL Thunderbolt 3 PCIe expansion chassis. I know this card works just fine because when I boot into ubuntu I'm able to use the card without error.
I've tried many versions of the ixgben driver, all have yielded the same results (PCI device is detected but esxcfg-nics -l only yields the built in ethernet adapter -- here's a list of the drivers I've tried so far:
1.7.1-1OEM.670.0.0.7535516 |
1.7.1.16-1vmw.670.3.73.14320388 |
1.7.10-1OEM.670.0.0.8169922 |
1.7.15-1OEM.670.0.0.8169922 |
1.7.17-1OEM.670.0.0.8169922 |
1.7.20-1OEM.670.0.0.8169922 |
Every time I reboot after esxcli software vib install I get only the built-in ethernet adapter, and not the ixgben driven adapter.
Also, when I run vmkload_mod -l | grep ixgben I've never seen the ixgben driver get loaded, although I do see the ixgben driver eg:
esxcli software vib list | grep ixgben
ixgben 1.7.10-1OEM.670.0.0.8169922 INT VMwareCertified 2019-09-14
Also, here's the evidence the PCI devices are detected.
lspci -v | grep -A1 -i ethernet
0000:00:1f.6 Network controller Ethernet controller: Intel Corporation Ethernet Connection (6) I219-V [vmnic0]
Class 0200: 8086:15be
--
0000:07:00.0 Network controller Ethernet controller: Intel(R) 82599 10 Gigabit Dual Port Network Connection
Class 0200: 8086:10fb
--
0000:07:00.1 Network controller Ethernet controller: Intel(R) 82599 10 Gigabit Dual Port Network Connection
Class 0200: 8086:10fb
I'm relatively new to ESXi/vsphere but I've learned a lot troubleshooting this. Curious if there's something I'm missing to add the card? I don't see any errors like TomAtTritium does regarding the SFP+ modules. Any help would be appreciated.
Failing that, maybe a recommendation for a card that will work without this much hassle.
Thanks,
Andrew
Most probably an unsupported SFP
OR
FW is not compatible
Thanks for the reply. Al least for me the SFP+ modules are Intel FTLX8571D3BCVIT1 which are explicitly mentioned as compatible by Intel for this card (SFP+ Modules, SFP Modules, and Cables Compatible with Intel® Ethernet... ).
So, ashishsingh1508 can you kindly explain what firmware you're referring to? On the nic itself? I believe the firmware is current for the card.
I suspect this is because it's an old version of the card, I'm currently having the same problem. I've ordered a newer version which will allow me to update the firmware in it.
You can tell it's the old version by the sub device id of 7a11, I believe this has a smaller flash on it and therefore cannot be updated past 16.5.20, that's the last update package I was able to find for my card that has the 7a11 sub id in the flash configuration file.
not wanting to teach anybody to suck eggs here, but have you actually verified that this card and sfp are supported by VMware for the version if ESXi that you are attempting to install. the single source of truth is the VMware Hardware Compatibility list that is found here: VMware Compatibility Guide - System Search
Ugh, I had checked before but I just looked up the VID/DID; when I entered the subvendor ID I see that nothing comes up for that card. Well, thanks TomHowarth I'm just happy to understand why the card just wasn't working.
I had updated the firmware on the card earlier tonight but to no avail; so moral of the story: check the darn SVID.
Been trying various things, I have another X520 which is newer and has later firmware on, but it still doesn't work (and yes it's in the HCL for 7.2u2 which I'm running, VID/DID/SVID/SDID all match).
In vmkwarning.log I see the following messages relating to that PCI device:
0:00:00:05.352 cpu0:2097152)WARNING: PCI: 453: 0000:07:00.0: Failed to add BAR[2] (IO f=0x1 0x4020-0x4040) - out of resources on parent: 0000:06:01.0
0:00:00:05.352 cpu0:2097152)WARNING: PCI: 476: 0000:07:00.0: Failed to add BAR[2] (IO f=0x1 0x4020-0x4040) status: Limit exceeded
0:00:00:05.352 cpu0:2097152)WARNING: PCI: 453: 0000:07:00.1: Failed to add BAR[2] (IO f=0x1 0x4000-0x4020) - out of resources on parent: 0000:06:01.0
0:00:00:05.352 cpu0:2097152)WARNING: PCI: 476: 0000:07:00.1: Failed to add BAR[2] (IO f=0x1 0x4000-0x4020) status: Limit exceeded
0:00:00:05.360 cpu0:2097152)WARNING: PCI: 679: 0000:07:00.0: Unable to free BAR[2] (IO f=0x1 0x4020-0x4040): Limit exceeded
0:00:00:05.360 cpu0:2097152)WARNING: PCI: 679: 0000:07:00.0: Unable to free BAR[4] (MEM64 f=0x4 0x90100000-0x90104000): Not found
0:00:00:05.360 cpu0:2097152)WARNING: PCI: 679: 0000:07:00.1: Unable to free BAR[2] (IO f=0x1 0x4000-0x4020): Limit exceeded
0:00:00:05.360 cpu0:2097152)WARNING: PCI: 679: 0000:07:00.1: Unable to free BAR[4] (MEM64 f=0x4 0x90300000-0x90304000): Not found
Any other ideas?
This knowledge base entry contains the answer to my error I believe. VMware Knowledge Base
Moderator note: Moved to ESXi
I'm stubborn and did the same as you fizzyade I bought a newer x520 as well and I get the same errors as you.
vmkernel.log:0:00:00:05.391 cpu0:2097152)WARNING: PCI: 452: 0000:07:00.0: Failed to add BAR[2] (IO f=0x1 0x4020-0x4040) - out of resources on parent: 0000:06:01.0
vmkernel.log:0:00:00:05.391 cpu0:2097152)WARNING: PCI: 475: 0000:07:00.0: Failed to add BAR[2] (IO f=0x1 0x4020-0x4040) status: Limit exceeded
vmkernel.log:0:00:00:05.391 cpu0:2097152)WARNING: PCI: 452: 0000:07:00.1: Failed to add BAR[2] (IO f=0x1 0x4000-0x4020) - out of resources on parent: 0000:06:01.0
vmkernel.log:0:00:00:05.391 cpu0:2097152)WARNING: PCI: 475: 0000:07:00.1: Failed to add BAR[2] (IO f=0x1 0x4000-0x4020) status: Limit exceeded
vmkernel.log:0:00:00:05.398 cpu0:2097152)WARNING: PCI: 678: 0000:07:00.0: Unable to free BAR[2] (IO f=0x1 0x4020-0x4040): Limit exceeded
vmkernel.log:0:00:00:05.398 cpu0:2097152)WARNING: PCI: 678: 0000:07:00.0: Unable to free BAR[4] (MEM64 f=0x4 0x90080000-0x90084000): Not found
vmkernel.log:0:00:00:05.398 cpu0:2097152)WARNING: PCI: 678: 0000:07:00.1: Unable to free BAR[2] (IO f=0x1 0x4000-0x4020): Limit exceeded
vmkernel.log:0:00:00:05.398 cpu0:2097152)WARNING: PCI: 678: 0000:07:00.1: Unable to free BAR[4] (MEM64 f=0x4 0x90200000-0x90204000): Not found
vmkernel.log:2019-09-27T22:14:35.641Z cpu6:2097563)DEBUG (ne1000): mapped BAR[0]: start = c0b00000, size = 20000, mapped to 450185f40000
vmkwarning.log:0:00:00:05.391 cpu0:2097152)WARNING: PCI: 452: 0000:07:00.0: Failed to add BAR[2] (IO f=0x1 0x4020-0x4040) - out of resources on parent: 0000:06:01.0
vmkwarning.log:0:00:00:05.391 cpu0:2097152)WARNING: PCI: 475: 0000:07:00.0: Failed to add BAR[2] (IO f=0x1 0x4020-0x4040) status: Limit exceeded
vmkwarning.log:0:00:00:05.391 cpu0:2097152)WARNING: PCI: 452: 0000:07:00.1: Failed to add BAR[2] (IO f=0x1 0x4000-0x4020) - out of resources on parent: 0000:06:01.0
vmkwarning.log:0:00:00:05.391 cpu0:2097152)WARNING: PCI: 475: 0000:07:00.1: Failed to add BAR[2] (IO f=0x1 0x4000-0x4020) status: Limit exceeded
vmkwarning.log:0:00:00:05.398 cpu0:2097152)WARNING: PCI: 678: 0000:07:00.0: Unable to free BAR[2] (IO f=0x1 0x4020-0x4040): Limit exceeded
vmkwarning.log:0:00:00:05.398 cpu0:2097152)WARNING: PCI: 678: 0000:07:00.0: Unable to free BAR[4] (MEM64 f=0x4 0x90080000-0x90084000): Not found
vmkwarning.log:0:00:00:05.398 cpu0:2097152)WARNING: PCI: 678: 0000:07:00.1: Unable to free BAR[2] (IO f=0x1 0x4000-0x4020): Limit exceeded
vmkwarning.log:0:00:00:05.398 cpu0:2097152)WARNING: PCI: 678: 0000:07:00.1: Unable to free BAR[4] (MEM64 f=0x4 0x90200000-0x90204000): Not found
We're not the only ones having this issue: https://medium.com/@adrian_65383/ok-so-i-got-that-10gtek-and-plugged-it-in-but-having-the-same-issue...
Any chance this can be addressed in an ESXi release, or would this have to be solved by a BIOS update by Intel?
That’s actually my post on medium!
I did solve this.....I backed up the hard disks of my virtual machines, wiped the NUC and installed Proxmox. Booted up straight away recognising the x520 in the Node Lite enclosure.
works like a dream. Probably not a useful “solution“ for a lot of people, but i haven’t looked back, loving lxc containers as well.
i Bought a 4 port Intel Ethernet card (1G) off eBay cheap to check in ESXi that the node lite was actually functional with a card in and that it wasn’t part of the problem: ESXi booted up and all 4 ports appeared in
Ressurrected, but I'm experiencing this issue trying to use Auto Deploy to upgrade some Mac Pro hosts to 6.7 U3. The Mac Pro 6,1 host, the Sonnet Twin 10G Thunderbolt Adapter (at least the VID:DID), appear on VMware's compatibility list. The other mac hosts are running fine on 6.5 Update 2. This was running a local ESXi 6.5 U2 installation with no problems.
I've tried the 1.7.10 (working on 6.5 U2), 1.7.15 (shown on for ESXi U3), and 1.7.20 (latest available on VMWare.com)
vmkwarning.log:
"
0:00:00:06.830 cpu0:2097152)WARNING: PCI: 452: 0000:60:00.0: Failed to add BAR[0] (MEM64 f=0xc 0x0-0x200000) - out of resources on parent: 0000:5f:03.0
0:00:00:06.830 cpu0:2097152)WARNING: PCI: 475: 0000:60:00.0: Failed to add BAR[0] (MEM64 f=0xc 0x0-0x200000) status: Limit exceeded
0:00:00:06.830 cpu0:2097152)WARNING: PCI: 225: PF mem (0x0 - 0x1fffff) out of range
0:00:00:06.830 cpu0:2097152)WARNING: PCI: 452: 0000:60:00.1: Failed to add BAR[0] (MEM64 f=0xc 0x0-0x200000) - out of resources on parent: 0000:5f:03.0
0:00:00:06.830 cpu0:2097152)WARNING: PCI: 475: 0000:60:00.1: Failed to add BAR[0] (MEM64 f=0xc 0x0-0x200000) status: Limit exceeded
0:00:00:07.846 cpu0:2097152)WARNING: PCI: 678: 0000:60:00.0: Unable to free BAR[0] (MEM64 f=0xc 0x0-0x200000): Limit exceeded
0:00:00:07.846 cpu0:2097152)WARNING: PCI: 678: 0000:60:00.0: Unable to free BAR[4] (MEM64 f=0xc 0x0-0x4000): Limit exceeded
0:00:00:07.847 cpu0:2097152)WARNING: PCI: 678: 0000:60:00.1: Unable to free BAR[0] (MEM64 f=0xc 0x0-0x200000): Limit exceeded
0:00:00:07.847 cpu0:2097152)WARNING: PCI: 678: 0000:60:00.1: Unable to free BAR[4] (MEM64 f=0xc 0x0-0x4000): Limit exceeded
"
lspci -vvv output:
"
0000:60:00.0 Network controller Ethernet controller: Intel(R) Ethernet Controller 10G X550
Class 0200: 8086:1563
0000:60:00.1 Network controller Ethernet controller: Intel(R) Ethernet Controller 10G X550
Class 0200: 8086:1563
"
I've tried doing a PRAM reset as well if that would resolve.
I cant seem to get the NICs to load anymore..even if I try to boot using the same image profile as the running/working nodes.