Install ESXi 6.5 on R730 with Intel X710

[CKF1028]

Hello All,

After install ESXi 6.0 via "VMware-VMvisor-Installer-6.0.0.update02-4192238.x86_64-Dell_Customized-A04" on Dell PowerEdge R730, everything is okay !!!

But, if I try to install ESXi 6.5 via "VMware-VMvisor-Installer-6.5.0-4564106.x86_64-Dell_Customized-A00" on Dell PowerEdge R730, my vmnic0 and vmnic1 (Intel X710 DP 10Gb DA/SFP+) are not working !!!

Does anybody know the solution ?

Thanks for your reply.

[root@R730:~] esxcli network nic list

Name    PCI Device    Driver  Admin Status  Link Status  Speed  Duplex  MAC Address        MTU  Description                                          

------  ------------  ------  ------------  -----------  -----  ------  -----------------  ----  ---------------------------------------------------------

vmnic0  0000:01:00.0  i40en  Up            Up          10000  Full    f8:bc:12:05:85:d0  1500  Intel Corporation Ethernet Controller X710 for 10GbE SFP+

vmnic1  0000:01:00.1  i40en  Up            Up          10000  Full    f8:bc:12:05:85:d2  1500  Intel Corporation Ethernet Controller X710 for 10GbE SFP+

vmnic2  0000:0c:00.0  igbn    Up            Down            0  Half    f8:bc:12:05:85:f0  1500  Intel Corporation Gigabit 4P X710/I350 rNDC          

vmnic3  0000:0c:00.1  igbn    Up            Up            1000  Full    f8:bc:12:05:85:f1  1500  Intel Corporation Gigabit 4P X710/I350 rNDC          

vusb0  Pseudo        cdce    Up            Up            100  Full    18:fb:7b:5d:d5:ee  1500  DellTM iDRAC Virtual NIC USB Device              

[root@R730:~] esxcli network nic get -n vmnic0

  Advertised Auto Negotiation: false

  Advertised Link Modes: 1000BaseT/Full, 10000BaseT/Full, 10000BaseT/Full, 40000BaseCR4/Full, 40000BaseSR4/Full

  Auto Negotiation: false

  Cable Type:

  Current Message Level: -1

  Driver Info:

        Bus Info: 0000:01:00:0

        Driver: i40en

        Firmware Version: 5.04 0x800024bc 17.5.11

        Version: 1.1.0

  Link Detected: true

  Link Status: Up

  Name: vmnic0

  PHYAddress: 0

  Pause Autonegotiate: false

  Pause RX: false

  Pause TX: false

  Supported Ports:

  Supports Auto Negotiation: false

  Supports Pause: false

  Supports Wakeon: true

  Transceiver:

  Virtual Address: 00:50:56:52:57:61

  Wakeon: MagicPacket(tm)

[cjckalb]

Just a quick heads up:

• A new Intel X(L)710 firmware 6.01 has been released on Aug 22 - with no release notes to be found: Download Non-Volatile Memory (NVM) Update Utility for Intel® Ethernet Converged Network Adapter XL71...
• There's also a new Linux based driver 2.1.26 that corresponds to said firmware: https://downloadmirror.intel.com/24411/eng/releasenotes.txt​ - some of the issues fixed sound interesting, but as I do not know the root cause of our problems yet, I cannot really tell whether they would be fixed by that driver.
• No mention of the firmware/driver on the VMware HCL or driver downloads so far (which isn't really surprising, given how recent they are).
• My VMware SR regarding the X(L)710/i40e(n) problems has now supposedly resulted in an internal ticket to VMware dev/engineering, unfortunately I don't know any details as of yet.

[TCG2]

I just finished updating my 6 host vSAN cluster to the i40en 1.4.3 drivers. This has cut latency by over 50% and seems to have helped the general behavior of the cluster. I was also able to enable flow control although it is unclear as to if the NIC is actually obeying this configuration.

I was previously on 2.0.6 i40e drivers and this driver still seemed to have some issues.

I am running a Dell branded x710 so I don't have access to 6.x firmware at the moment.

[AdamSw]

Hi cjckalb,

Did you manage to resolve your issue?

I have experienced something similar on i40e 2.0.6 however this happens rather rarely and therefore I have not managed to see whether disabling LRO/TSO helps.

Do you by any chance have managed to simplify the scenario that gets the PSOD to reproduce constantly? Having that would make it easier to get VMware/Intel to fix it.

==

Thanks,

Adam

cjckalb

[cjckalb]

Hi Adam,

we did not use i40e for too long, so we did not have the chance to isolate what exactly caused the PSODs. I would suggest trying the latest i40en 1.4.3 + the latest firmware 6.01 instead, as that combination would not cause PSODs for us. Even with the latest driver + firmware, it would however reset the interface intermittently (like once or twice a month). Unfortunately, we could not isolate the exact traffic pattern and/or other conditions causing this aswell. The last thing VMware support suggested was turning off TSO. We have since completely switched to QLogic (Broadcom) NICs a couple of days ago, since neither Intel nor VMware seemed too interested in determining and fixing the root cause, and we could not waste any more time on this issue.

[TheHevy]

I am sorry to hear that you have had this issue with the Intel Ethernet CNA X710.

This issue has been addressed with the latest i40en driver and 6.01 firmware.

Here are the links to the download pages.

https://my.vmware.com/group/vmware/details?downloadGroup=DT-ESXI65-INTEL-I40EN-156&productId=615#pro...

https://downloadcenter.intel.com/download/24769/Ethernet-Non-Volatile-Memory-NVM-Update-Utility-for-...

Brian Johnson

Solutions Architect -- Intel Corp

[iway_cr]

Just tried the new i40en drivers (1.5.6) but had to switch back to the i40e_2.0.7 immediatly again, as VLAN tagging was no longer working properly with the i40en drivers.

Our X710-DA4 cards are of course already on the 6.01 NVM firmware and TSO/TSO6/LRO is disabled on our hosts. (due to problems and PDODs with both the i40e and i40en driver back in the days...and we never dared to enabled it again)

[iway_cr]

Just got this PSOD today with 6.01 NVM firmware and i40e_2.0.7 driver. (UseHwTSO, UseHwTSO6 and TcpipDefLROEnabled disabled)

Did now also disable Vmxnet3HwLRO and Vmxnet3SwLRO, but don't think it's worth much. Really seems like today is the time to dump these NICs....(like everything that comes from Intel lately)

[MEGEDV]

Did anyone ever get his Intel X710 NICs to work without issues using esxi 6.0 or 6.5 ?

So far we have tried multiple driver / firmware combinations, multiple vmware versions etc... for nearly two years now... nothing ever was fully stable...
During times with high traffic load (usually during the nightly backups) the NIC sometimes just stops passing traffic to the vm and never recovers until the VM is reset.

[aritri]

I'm currently experiencing similiar problems with HPE branded cards (562SFP+).

Log entries show the following:

2018-03-02T11:15:56.116Z cpu12:66212)i40en: i40en_HandleMddEvent:6969: Malicious Driver Detection event 0x02 on TX queue 0 PF number 0x00 VF number 0x00

2018-03-02T11:15:56.116Z cpu12:66212)i40en: i40en_HandleMddEvent:6995: TX driver issue detected, PF reset issued

Driver/firmware versions tested so far:

driver 1.4.3 and firmware 5.60

driver 1.5.6 and firmware 5.60

driver 1.5.6 and firmware 6.01

Driver 1.5.6 is HPE version 2018.02.12

Firmware 6.01 is HPE version 10.2.5

There is a new driver version 1.5.8 listed on the HCL but the recommended version from HPE is 1.5.6.

Support request with VMware is open but no solution so far...

HPE firmware: https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX_87c83853cb5a4bc5949e9b0dd5#tab5

HPE driver : Drivers & Software - HPE Support Center.

[hostasaurus]

Seeing the same exact thing on a brand new install of Dell R740 servers with the dual port x710.  This card is a piece of sh*t and should just be thrown in the trash.  Even with the 1.5.8 driver + 6.01 firmware combo, and disabling the optimizations, still dies under load if vlan tagging is being used.  That of course required deviating from Dell's supported firmware updates as their latest only puts 6.00 on the card.  Intel has known about these issues for years and hasn't solved it.

[Vibs1983]

Hi,

We have the same issue and it's really frustrating as this has caused some of our productions hosts to go offline.. I hope it gets fixed soon.

Intel X710 / i40en driver on vmware - any fix? |Intel Communities

Regards

[HendersonD]

We recently received 8 brand new HPE DL380 Gen10 servers with the HPE branded cards (562SFP+)

We are experiencing the same issues as everyone else. In the ESXi console we associated two of these nics for the management network. After assigning an IP address, mask, gateway, and DNS servers we restart a host. During the restart we have a continuous ping going to the management network IP address. Part way through booting up the ping returns. After about 30 seconds the ping drops and never comes back. If we remove one of the nics, the ping returns. In other words, with two of these nics as part of the management network, they will not pass traffic. We are on driver version 1.5.8 and firmware version 10.2.5

Any solutions yet? I have an open ticket with VMWare and HPE for nearly two weeks

[skaWll]

Hello,

So far, we don't have such problems on Dell r740xd running ESXi 6.5 U1 with Intel x710 network adapters :

• firmware version : 18.3.6
• driver type : i40en
• driver version : 1.5.6
• no TSO desactived or whatever
• connected to Cisco Nexus 5000 network switchs

On those x710 adapters, we're using lacp lags, vlan tagging and jumbo frames : no problems yet (ESXi are up and running since ~ 20 days without network outages or psod).

We're not using SR-IOV, maybe VXLAN in a near future.

I'll keep you posted if we face any problems !

[FreddyFredFred]

Maybe the suggestions here will help?

http://thenicholson.com/where-did-my-host-go/

[HendersonD]

Problem solved, I figured out the issue this morning. The two Juniper switch ports used for the management network have jumbo frames enabled. Once ESXi was installed on the new HPE server, the first step is to go into the ESXi console and setup the management network. Nics are assigned, IP addresses added etc. When this is done ESXi in the background is creating a vSwitch and Kernel port. The vSwitch was made just fine and it had its MTU set for 9000 which is correct for Jumbo Frames. The Kernel port was also made but its MTU was set to 1500, not 9000. This cannot be seen from the ESXi console, you have to go into the web interface to see this. Once I changed the kernel port to 9000 for MTU I could add the second nic and it worked perfect. When ESXi made the vSwitch and Kernel port it should have set both to an MTU of 9000 and it did not.

[HendersonD]

My issue was actually not solved and it was not a Jumbo Frames mismatch. It turns out the Intel X710 based nics do not handle LLDP properly, this post gives the details

Where did my host go.... - Virtual Ramblings

A few more posts that do not give this nic glowing reviews

https://lonesysadmin.net/2018/02/28/intel-x710-nics-are-crap/

vSphere 6.5 with Intel X710 network adapter

We just put in one Broadcom nic for testing and it seems to be working fine. May end up swapping all of our Intel nics for Broadcoms

[rafficvmware]

Does this issue resolved?. I saw new driver version i40en version 1.5.8 released on Feb 2018. Does this resolves the issue?

[Ljugtomten2]

I'd say this is resolved with the following driver update: Download VMware vSAN (

VMware ESXi 6.7 i40en 1.7.1 NIC Driver for Intel Ethernet Controllers X710,XL710,XXV710 and X722 family)

I built a server using a SuperMicro X11SPi-TF which have two 10GBE Intel X722 + X557 on board.

Before updating the driver I had two occurances where the network would not respond (IPMI using a dedicated NIC would respond), and using esxcli to disable/enable the NICs would resume normal operation leading to a driver error so my HW was OK.

After updating the driver with the one in the link above, everying is as it should with no network interruptions for several weeks.

[aritri]

This is for ESXi 6.7 only. There is still no solution for ESXi 6.5.

Our last support case with vmware was solved by replacing the cards with different ones.

[skaWll]

Fresh news for the i40en driver for ESXi 6.5 (version 1.7.5) :

https://my.vmware.com/group/vmware/details?downloadGroup=DT-ESXI65-INTEL-I40EN-175&productId=614

Haven't be able to test it already.

We're currently using the 1.5.8 driver, and we're suffering some network outage from time to time (the Malicious Driver Detection bug).