13 Replies Latest reply on Dec 3, 2019 5:28 PM by ez12a323

    Can't add 10GBe physical nic

    ElliottN Lurker

      I've been having a dog of a time trying to add a dual 10GBe nic to ESXi.  I've tried installing the drivers and everything appears that it is installing but ESXi doesn't seem to be loading the drivers for some reason.  This is an Intel NUC NUC7i5BNK host with a OWC Mercury Helios 3  Thunderbolt 3 PCIe Expansion Chassis housing the NIC.  The NIC and the expansion chassis is known good on other OS's so everything points to a configuration issue on ESXi or in the bios of the NUC.  I've enabled thunderbolt at boot so I think that rules out the BIOS.

       

      [root@esxi1:~] lspci -v | grep -A1 -i ethernet

      0000:00:1f.6 Ethernet controller Network controller: Intel Corporation Ethernet Connection (4) I219-V [vmnic0]

               Class 0200: 8086:15d8

      --

      0000:06:00.0 Ethernet controller Network controller: Intel(R) 82599 10 Gigabit Dual Port Network Connection

               Class 0200: 8086:10fb

      --

      0000:06:00.1 Ethernet controller Network controller: Intel(R) 82599 10 Gigabit Dual Port Network Connection

               Class 0200: 8086:10fb

      [root@esxi1:~] vmkload_mod -l |grep drivername

      [root@esxi1:~] vmkload_mod -l |grep ixgben

      [root@esxi1:~] esxcfg-nics -l

      Name    PCI          Driver      Link Speed      Duplex MAC Address       MTU    Description

      vmnic0  0000:00:1f.6 ne1000      Up   1000Mbps   Full   94:c6:91:15:0b:e4 9000   Intel Corporation Ethernet Connection (4) I219-V

      [root@esxi1:~] esxcli software vib install -d "/vmfs/volumes/datastore1/VMW-ESX-

      6.7.0-ixgben-1.7.10-offline_bundle-10105563.zip"

      Installation Result

         Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.

         Reboot Required: true

         VIBs Installed: INT_bootbank_ixgben_1.7.10-1OEM.670.0.0.8169922

         VIBs Removed: INT_bootbank_ixgben_1.7.1-1OEM.670.0.0.7535516

         VIBs Skipped:

       

      Still no joy after reboot

       

      [root@esxi1:~] esxcli hardware pci list

      0000:06:00.0

         Address: 0000:06:00.0

         Segment: 0x0000

         Bus: 0x06

         Slot: 0x00

         Function: 0x0

         VMkernel Name:

         Vendor Name: Intel Corporation

         Device Name: 82599EB 10-Gigabit SFI/SFP+ Network Connection

         Configured Owner: VMkernel

         Current Owner: VMkernel

         Vendor ID: 0x8086

         Device ID: 0x10fb

         SubVendor ID: 0x8086

         SubDevice ID: 0x7a11

         Device Class: 0x0200

         Device Class Name: Ethernet controller

         Programming Interface: 0x00

         Revision ID: 0x01

         Interrupt Line: 0xff

         IRQ: 255

         Interrupt Vector: 0x00

         PCI Pin: 0x00

         Spawned Bus: 0x00

         Flags: 0x3219

         Module ID: -1

         Module Name: None

         Chassis: 0

         Physical Slot: 4294967295

         Slot Description:

         Passthru Capable: true

         Parent Device: PCI 0:5:1:0

         Dependent Device: PCI 0:6:0:0

         Reset Method: Function reset

         FPT Sharable: true

       

       

      0000:06:00.1

         Address: 0000:06:00.1

         Segment: 0x0000

         Bus: 0x06

         Slot: 0x00

         Function: 0x1

         VMkernel Name:

         Vendor Name: Intel Corporation

         Device Name: 82599EB 10-Gigabit SFI/SFP+ Network Connection

         Configured Owner: VMkernel

         Current Owner: VMkernel

         Vendor ID: 0x8086

         Device ID: 0x10fb

         SubVendor ID: 0x8086

         SubDevice ID: 0x7a11

         Device Class: 0x0200

         Device Class Name: Ethernet controller

         Programming Interface: 0x00

         Revision ID: 0x01

         Interrupt Line: 0xff

         IRQ: 255

         Interrupt Vector: 0x00

         PCI Pin: 0x01

         Spawned Bus: 0x00

         Flags: 0x3219

         Module ID: -1

         Module Name: None

         Chassis: 0

         Physical Slot: 4294967295

         Slot Description:

         Passthru Capable: true

         Parent Device: PCI 0:5:1:0

         Dependent Device: PCI 0:6:0:1

         Reset Method: Function reset

         FPT Sharable: true

       

      [root@esxi1:~] esxcfg-nics -l

      Name    PCI          Driver      Link Speed      Duplex MAC Address       MTU    Description

      vmnic0  0000:00:1f.6 ne1000      Up   1000Mbps   Full   94:c6:91:15:0b:e4 9000   Intel Corporation Ethernet Connection (4) I219-V

        • 1. Re: Can't add 10GBe physical nic
          TomAtTritium Lurker

          I am using Dell / Intel X520-DA2 Dual Port 10Gbe SFP Network Adapter NIC 942V6 Low Profile together with ESXi 6.7 U3.

          NIC shows as "82599EB 10-Gigabit SFP+ Network Connection" in hardware but are not available as a NIC.

           

          I checked the log..

          cat /var/log/vmkernel.log  | grep ixgben

          and found that the SFP+ that i am trying to use is unsupported and that has stopped the 10GB from initializing properly:

          2019-09-10T20:49:43.630Z cpu51:2098034)ixgben: ixgben_InitSharedCode:185: Unsupported SFP+ or QSFP detected. Try to replace a supported module then reload the driver.

           

          I will try another SFP+ or DAC, and update this post with result.

          • 2. Re: Can't add 10GBe physical nic
            4n3w Lurker

            Hey there,

             

            I have what looks like the same issue as ElliottN with some X520-DA2 Dual Port 10GE NIC (FWIW with correct and genuine Intel SFP+ modules) in an Intel NUC8i5BEH (with latest BIOS firmware) machine connected via a Sonnet Echo Express SEL Thunderbolt 3 PCIe expansion chassis. I know this card works just fine because when I boot into ubuntu I'm able to use the card without error.

             

            I've tried many versions of the ixgben driver, all have yielded the same results (PCI device is detected but esxcfg-nics -l only yields the built in ethernet adapter -- here's a list of the drivers I've tried so far:

             

             

            1.7.1-1OEM.670.0.0.7535516
            1.7.1.16-1vmw.670.3.73.14320388
            1.7.10-1OEM.670.0.0.8169922
            1.7.15-1OEM.670.0.0.8169922
            1.7.17-1OEM.670.0.0.8169922
            1.7.20-1OEM.670.0.0.8169922

             

            Every time I reboot after esxcli software vib install I get only the built-in ethernet adapter, and not the ixgben driven adapter.

             

            Also, when I run vmkload_mod -l | grep ixgben I've never seen the ixgben driver get loaded, although I do see the ixgben driver eg:

             

            esxcli software vib list | grep ixgben

            ixgben                         1.7.10-1OEM.670.0.0.8169922           INT     VMwareCertified   2019-09-14

             

             

            Also, here's the evidence the PCI devices are detected.

             

            lspci -v | grep -A1 -i ethernet

            0000:00:1f.6 Network controller Ethernet controller: Intel Corporation Ethernet Connection (6) I219-V [vmnic0]

              Class 0200: 8086:15be

            --

            0000:07:00.0 Network controller Ethernet controller: Intel(R) 82599 10 Gigabit Dual Port Network Connection

              Class 0200: 8086:10fb

            --

            0000:07:00.1 Network controller Ethernet controller: Intel(R) 82599 10 Gigabit Dual Port Network Connection

              Class 0200: 8086:10fb

             

            I'm relatively new to ESXi/vsphere but I've learned a lot troubleshooting this. Curious if there's something I'm missing to add the card? I don't see any errors like TomAtTritium does regarding the SFP+ modules. Any help would be appreciated.

             

            Failing that, maybe a recommendation for a card that will work without this much hassle.

             

            Thanks,

            Andrew

            • 3. Re: Can't add 10GBe physical nic
              ashishsingh1508 Enthusiast
              vExpert

              Most probably an unsupported SFP

               

              OR

               

              FW is not compatible

              • 4. Re: Can't add 10GBe physical nic
                4n3w Lurker

                Thanks for the reply. Al least for me the SFP+ modules are Intel FTLX8571D3BCVIT1 which are explicitly mentioned as compatible by Intel for this card (SFP+ Modules, SFP Modules, and Cables Compatible with Intel® Ethernet... ).

                 

                So, ashishsingh1508 can you kindly explain what firmware you're referring to? On the nic itself? I believe the firmware is current for the card.

                • 5. Re: Can't add 10GBe physical nic
                  fizzyade Novice

                  I suspect this is because it's an old version of the card, I'm currently having the same problem.  I've ordered a newer version which will allow me to update the firmware in it.

                   

                  You can tell it's the old version by the sub device id of 7a11, I believe this has a smaller flash on it and therefore cannot be updated past 16.5.20, that's the last update package I was able to find for my card that has the 7a11 sub id in the flash configuration file.

                  1 person found this helpful
                  • 6. Re: Can't add 10GBe physical nic
                    TomHowarth Guru
                    vExpertUser Moderators

                    not wanting to teach anybody to suck eggs here, but have you actually verified that this card and sfp are supported by VMware for the version if ESXi that you are attempting to install. the single source of truth is the VMware Hardware Compatibility list that is found here: VMware Compatibility Guide - System Search 

                    1 person found this helpful
                    • 7. Re: Can't add 10GBe physical nic
                      4n3w Lurker

                      Ugh, I had checked before but I just looked up the VID/DID; when I entered the subvendor ID I see that nothing comes up for that card. Well, thanks TomHowarth I'm just happy to understand why the card just wasn't working.

                       

                      I had updated the firmware on the card earlier tonight but to no avail; so moral of the story: check the darn SVID.

                      • 8. Re: Can't add 10GBe physical nic
                        fizzyade Novice

                        Been trying various things, I have another X520 which is newer and has later firmware on, but it still doesn't work (and yes it's in the HCL for 7.2u2 which I'm running, VID/DID/SVID/SDID all match).

                         

                        In vmkwarning.log I see the following messages relating to that PCI device:

                         

                        0:00:00:05.352 cpu0:2097152)WARNING: PCI: 453: 0000:07:00.0: Failed to add BAR[2] (IO f=0x1 0x4020-0x4040) - out of resources on parent: 0000:06:01.0

                        0:00:00:05.352 cpu0:2097152)WARNING: PCI: 476: 0000:07:00.0: Failed to add BAR[2] (IO f=0x1 0x4020-0x4040) status: Limit exceeded

                        0:00:00:05.352 cpu0:2097152)WARNING: PCI: 453: 0000:07:00.1: Failed to add BAR[2] (IO f=0x1 0x4000-0x4020) - out of resources on parent: 0000:06:01.0

                        0:00:00:05.352 cpu0:2097152)WARNING: PCI: 476: 0000:07:00.1: Failed to add BAR[2] (IO f=0x1 0x4000-0x4020) status: Limit exceeded

                        0:00:00:05.360 cpu0:2097152)WARNING: PCI: 679: 0000:07:00.0: Unable to free BAR[2] (IO f=0x1 0x4020-0x4040): Limit exceeded

                        0:00:00:05.360 cpu0:2097152)WARNING: PCI: 679: 0000:07:00.0: Unable to free BAR[4] (MEM64 f=0x4 0x90100000-0x90104000): Not found

                        0:00:00:05.360 cpu0:2097152)WARNING: PCI: 679: 0000:07:00.1: Unable to free BAR[2] (IO f=0x1 0x4000-0x4020): Limit exceeded

                        0:00:00:05.360 cpu0:2097152)WARNING: PCI: 679: 0000:07:00.1: Unable to free BAR[4] (MEM64 f=0x4 0x90300000-0x90304000): Not found

                         

                        Any other ideas?

                        • 9. Re: Can't add 10GBe physical nic
                          fizzyade Novice

                          This knowledge base entry contains the answer to my error I believe. VMware Knowledge Base

                          • 10. Re: Can't add 10GBe physical nic
                            scott28tt Champion
                            Community WarriorsUser ModeratorsVMware Employees

                            Moderator note: Moved to ESXi

                            • 11. Re: Can't add 10GBe physical nic
                              4n3w Lurker

                              I'm stubborn and did the same as you fizzyade I bought a newer x520 as well and I get the same errors as you.

                               

                              vmkernel.log:0:00:00:05.391 cpu0:2097152)WARNING: PCI: 452: 0000:07:00.0: Failed to add BAR[2] (IO f=0x1 0x4020-0x4040) - out of resources on parent: 0000:06:01.0

                              vmkernel.log:0:00:00:05.391 cpu0:2097152)WARNING: PCI: 475: 0000:07:00.0: Failed to add BAR[2] (IO f=0x1 0x4020-0x4040) status: Limit exceeded

                              vmkernel.log:0:00:00:05.391 cpu0:2097152)WARNING: PCI: 452: 0000:07:00.1: Failed to add BAR[2] (IO f=0x1 0x4000-0x4020) - out of resources on parent: 0000:06:01.0

                              vmkernel.log:0:00:00:05.391 cpu0:2097152)WARNING: PCI: 475: 0000:07:00.1: Failed to add BAR[2] (IO f=0x1 0x4000-0x4020) status: Limit exceeded

                              vmkernel.log:0:00:00:05.398 cpu0:2097152)WARNING: PCI: 678: 0000:07:00.0: Unable to free BAR[2] (IO f=0x1 0x4020-0x4040): Limit exceeded

                              vmkernel.log:0:00:00:05.398 cpu0:2097152)WARNING: PCI: 678: 0000:07:00.0: Unable to free BAR[4] (MEM64 f=0x4 0x90080000-0x90084000): Not found

                              vmkernel.log:0:00:00:05.398 cpu0:2097152)WARNING: PCI: 678: 0000:07:00.1: Unable to free BAR[2] (IO f=0x1 0x4000-0x4020): Limit exceeded

                              vmkernel.log:0:00:00:05.398 cpu0:2097152)WARNING: PCI: 678: 0000:07:00.1: Unable to free BAR[4] (MEM64 f=0x4 0x90200000-0x90204000): Not found

                              vmkernel.log:2019-09-27T22:14:35.641Z cpu6:2097563)DEBUG (ne1000): mapped BAR[0]: start = c0b00000, size = 20000, mapped to 450185f40000

                              vmkwarning.log:0:00:00:05.391 cpu0:2097152)WARNING: PCI: 452: 0000:07:00.0: Failed to add BAR[2] (IO f=0x1 0x4020-0x4040) - out of resources on parent: 0000:06:01.0

                              vmkwarning.log:0:00:00:05.391 cpu0:2097152)WARNING: PCI: 475: 0000:07:00.0: Failed to add BAR[2] (IO f=0x1 0x4020-0x4040) status: Limit exceeded

                              vmkwarning.log:0:00:00:05.391 cpu0:2097152)WARNING: PCI: 452: 0000:07:00.1: Failed to add BAR[2] (IO f=0x1 0x4000-0x4020) - out of resources on parent: 0000:06:01.0

                              vmkwarning.log:0:00:00:05.391 cpu0:2097152)WARNING: PCI: 475: 0000:07:00.1: Failed to add BAR[2] (IO f=0x1 0x4000-0x4020) status: Limit exceeded

                              vmkwarning.log:0:00:00:05.398 cpu0:2097152)WARNING: PCI: 678: 0000:07:00.0: Unable to free BAR[2] (IO f=0x1 0x4020-0x4040): Limit exceeded

                              vmkwarning.log:0:00:00:05.398 cpu0:2097152)WARNING: PCI: 678: 0000:07:00.0: Unable to free BAR[4] (MEM64 f=0x4 0x90080000-0x90084000): Not found

                              vmkwarning.log:0:00:00:05.398 cpu0:2097152)WARNING: PCI: 678: 0000:07:00.1: Unable to free BAR[2] (IO f=0x1 0x4000-0x4020): Limit exceeded

                              vmkwarning.log:0:00:00:05.398 cpu0:2097152)WARNING: PCI: 678: 0000:07:00.1: Unable to free BAR[4] (MEM64 f=0x4 0x90200000-0x90204000): Not found

                               

                              We're not the only ones having this issue: https://medium.com/@adrian_65383/ok-so-i-got-that-10gtek-and-plugged-it-in-but-having-the-same-issue-its-seen-by-esxi-bu…

                               

                              Any chance this can be addressed in an ESXi release, or would this have to be solved by a BIOS update by Intel?

                              • 12. Re: Can't add 10GBe physical nic
                                fizzyade Novice

                                That’s actually my post on medium!

                                 

                                I did solve this.....I backed up the hard disks of my virtual machines, wiped the NUC and installed Proxmox. Booted up straight away recognising the x520 in the Node Lite enclosure.

                                 

                                works like a dream.  Probably not a useful “solution“ for a lot of people, but i haven’t looked back, loving lxc containers as well.

                                 

                                i Bought a 4 port Intel  Ethernet card (1G) off eBay cheap to check in ESXi that the node lite was actually functional with a card in and that it wasn’t part of the problem:  ESXi booted up and all 4 ports appeared in

                                • 13. Re: Can't add 10GBe physical nic
                                  ez12a323 Novice

                                  Ressurrected, but I'm experiencing this issue trying to use Auto Deploy to upgrade some Mac Pro hosts to 6.7 U3. The Mac Pro 6,1 host, the Sonnet Twin 10G Thunderbolt Adapter (at least the VID:DID), appear on VMware's compatibility list. The other mac hosts are running fine on 6.5 Update 2. This was running a local ESXi 6.5 U2 installation with no problems.

                                   

                                  I've tried the 1.7.10 (working on 6.5 U2), 1.7.15 (shown on for ESXi U3), and 1.7.20 (latest available on VMWare.com)

                                   

                                  vmkwarning.log:

                                  "

                                  0:00:00:06.830 cpu0:2097152)WARNING: PCI: 452: 0000:60:00.0: Failed to add BAR[0] (MEM64 f=0xc 0x0-0x200000) - out of resources on parent: 0000:5f:03.0

                                  0:00:00:06.830 cpu0:2097152)WARNING: PCI: 475: 0000:60:00.0: Failed to add BAR[0] (MEM64 f=0xc 0x0-0x200000) status: Limit exceeded

                                  0:00:00:06.830 cpu0:2097152)WARNING: PCI: 225: PF mem (0x0 - 0x1fffff) out of range

                                  0:00:00:06.830 cpu0:2097152)WARNING: PCI: 452: 0000:60:00.1: Failed to add BAR[0] (MEM64 f=0xc 0x0-0x200000) - out of resources on parent: 0000:5f:03.0

                                  0:00:00:06.830 cpu0:2097152)WARNING: PCI: 475: 0000:60:00.1: Failed to add BAR[0] (MEM64 f=0xc 0x0-0x200000) status: Limit exceeded

                                  0:00:00:07.846 cpu0:2097152)WARNING: PCI: 678: 0000:60:00.0: Unable to free BAR[0] (MEM64 f=0xc 0x0-0x200000): Limit exceeded

                                  0:00:00:07.846 cpu0:2097152)WARNING: PCI: 678: 0000:60:00.0: Unable to free BAR[4] (MEM64 f=0xc 0x0-0x4000): Limit exceeded

                                  0:00:00:07.847 cpu0:2097152)WARNING: PCI: 678: 0000:60:00.1: Unable to free BAR[0] (MEM64 f=0xc 0x0-0x200000): Limit exceeded

                                  0:00:00:07.847 cpu0:2097152)WARNING: PCI: 678: 0000:60:00.1: Unable to free BAR[4] (MEM64 f=0xc 0x0-0x4000): Limit exceeded

                                  "

                                   

                                  lspci -vvv output:

                                  "

                                  0000:60:00.0 Network controller Ethernet controller: Intel(R) Ethernet Controller 10G X550

                                           Class 0200: 8086:1563

                                   

                                   

                                  0000:60:00.1 Network controller Ethernet controller: Intel(R) Ethernet Controller 10G X550

                                           Class 0200: 8086:1563

                                  "

                                   

                                   

                                  I've tried doing a PRAM reset as well if that would resolve.


                                  I cant seem to get the NICs to load anymore..even if I try to boot using the same image profile as the running/working nodes.