Environment
Problem Statement
ESXi fails to list the Intel X540 (ixgben) as configurable network ports via a Thunderbolt3 to Thunderbolt2 Apple Adapter.
[root@esxinuc:~] esxcli network nic list
Name PCI Device Driver Admin Status Link Status Speed Duplex MAC Address MTU Description
------ ------------ ------ ------------ ----------- ----- ------ ----------------- ---- --------------------------------------------------------
vmnic0 0000:00:1f.6 ne1000 Up Down 0 Half d4:5d:df:07:4f:59 1500 Intel Corporation Ethernet Connection (2) I219-V
vmnic1 0000:05:00.0 igbn Up Up 1000 Full d4:5d:df:07:4f:5a 1500 Intel Corporation I210 Gigabit Network Connection
vmnic2 0000:44:00.0 tg3 Up Down 0 Half 38:c9:86:22:e4:b0 1500 Broadcom Corporation NetXtreme BCM57762 Gigabit Ethernet
The vmkwarning.log outputs PCI MMIO limit exceeded warnings:
0:00:00:05.485 cpu0:2097152)WARNING: PCI: 452: 0000:40:00.0: Failed to add BAR[2] (IO f=0x1 0xa020-0xa040) - out of resources on parent: 0000:3f:03.0
0:00:00:05.485 cpu0:2097152)WARNING: PCI: 475: 0000:40:00.0: Failed to add BAR[2] (IO f=0x1 0xa020-0xa040) status: Limit exceeded
0:00:00:05.485 cpu0:2097152)WARNING: PCI: 221: IO (0xa000 - 0xa01f) out of range
0:00:00:05.485 cpu0:2097152)WARNING: PCI: 452: 0000:40:00.1: Failed to add BAR[2] (IO f=0x1 0xa000-0xa020) - out of resources on parent: 0000:3f:03.0
0:00:00:05.485 cpu0:2097152)WARNING: PCI: 475: 0000:40:00.1: Failed to add BAR[2] (IO f=0x1 0xa000-0xa020) status: Limit exceeded
0:00:00:05.496 cpu0:2097152)WARNING: PCI: 678: 0000:40:00.0: Unable to free BAR[2] (IO f=0x1 0xa020-0xa040): Limit exceeded
0:00:00:05.496 cpu0:2097152)WARNING: PCI: 678: 0000:40:00.0: Unable to free BAR[4] (MEM64 f=0xc 0x2fd0200000-0x2fd0204000): Limit exceeded
0:00:00:05.496 cpu0:2097152)WARNING: PCI: 678: 0000:40:00.1: Unable to free BAR[2] (IO f=0x1 0xa000-0xa020): Limit exceeded
0:00:00:05.496 cpu0:2097152)WARNING: PCI: 678: 0000:40:00.1: Unable to free BAR[4] (MEM64 f=0xc 0x2fd0600000-0x2fd0604000): Limit exceeded
[root@esxinuc:~] lspci | grep 0000:3f:03.0
0000:3f:03.0 PCI bridge: Intel Corporation DSL5520 Thunderbolt 2 Bridge [Falcon Ridge 4C 2013]
- TB2 Bridge via Apple TB3-to-TB2 Adapter (Parent)
0:00:00:05.496 cpu0:2097152)WARNING: PCI: 678: 0000:40:00.0: Unable to free BAR[4] (MEM64 f=0xc 0x2fd0200000-0x2fd0204000): Limit exceeded
0:00:00:05.496 cpu0:2097152)WARNING: PCI: 678: 0000:40:00.1: Unable to free BAR[2] (IO f=0x1 0xa000-0xa020): Limit exceeded
- Promise SANLink2 10GbE (Intel Ethernet Controller 10 Gigabit X540-AT2)
It appears that I'm experiencing a PCI MMIO issue with the adapter and verified this by reviewing Memory Addressing Limits for ESXi 5.1.x, 5.5.x 6.x and 7.0 (2087943)
I couldn't figure out a way to change this on the Intel NUC with the latest BIOS.
I never posted about my testing with newer TB3 based Mac machines while using the TB3-to-TB2 adapter, since previously, nothing would be detected via the Thunderbolt3 bus.
Below is an example of a TB3 MacBook Pro 15,1 using the TB3-to-TB2 adapter with the Promise SANLink2 10GbE working fine after the ESXi670-202004002 patch:
(Same HW configuration that fails on the NUC):
[root@esxinuc:/var/log] esxcli network nic list
Name PCI Device Driver Admin Status Link Status Speed Duplex MAC Address MTU Description
------ ------------ ------ ------------ ----------- ----- ------ ----------------- ---- ---------------------------------------------------------
vmnic0 0000:0a:00.0 ixgben Up Up 10000 Full 00:01:55:14:6e:8f 1500 Intel Corporation Ethernet Controller 10 Gigabit X540-AT2
vmnic1 0000:0a:00.1 ixgben Up Up 10000 Full 00:01:55:14:6e:90 1500 Intel Corporation Ethernet Controller 10 Gigabit X540-AT2
vmnic2 0000:0e:00.0 tg3 Up Up 1000 Full 38:c9:86:22:e4:b0 1500 Broadcom Corporation NetXtreme BCM57762 Gigabit Ethernet
[root@esxinuc:/var/log] lspci | grep X540
0000:0a:00.0 Ethernet controller: Intel(R) Ethernet Controller 10 Gigabit X540-AT2 [vmnic0]
0000:0a:00.1 Ethernet controller: Intel(R) Ethernet Controller 10 Gigabit X540-AT2 [vmnic1]
[root@esxinuc:/var/log] esxcli hardware platform get
Platform Information
UUID: 0x28 0x64 0xf3 0xd1 0x83 0xfe 0x57 0x7c 0x93 0x36 0x13 0x66 0xfc 0x9e 0xb6 0x42
Product Name: MacBookPro15,1
Vendor Name: Apple Inc.
Serial Number: <omitted>
Enclosure Serial Number: <omitted>
BIOS Asset Tag:
IPMI Supported: false
If my MacBook Pro 15,1 can do exactly what I'm trying to accomplish via ESXi; this should be achievable on the NUC as well. Off to a new adventure.
Wow, a blast from the past! I'm fizzyade.
I did indeed give up, nothing I did or tried could get the thing to work in ESXi, I spent quite a bit of money on various different cards with all the same results before I realised the issue with the MMIO mapping.
I installed Proxmox and it all worked straight away, off the bat, without any tinkering. Been running it since that post and haven't had any issues.
I have the same issue than you, my setup is NUC7i7BNH with a SONNET Echo Express SEL I and a SFC5122 10gb card. I get the same MMIO / PCI BAR @limit error, even if the card is listed with lspci. I tested the same setup but booting with Ubuntu 20 and it works perfectly, the two 10gb ports show up and work.
So the issue is with ESX. It doesnt handle well the PCI cards installed in TB2/3 enclosures.
In ESXi 6.7 the Sonnet TB3 10G Solo Adapter works flawless with the 1.0.2.0 Driver Bundle. However, the same Setup won't work anymore with ESXi 8.0.
I tried different driver versions (even downgrading from 1.0.3.0 to the before-working 10.2.0).
The adapter is listed by lspci. But the firmware can't be initialized (FW kickoff failed).