Configure Intel x520 10GbE flow control

[RobVM]

I'm testing a Dell R710 host running ESX 5.1 that has an Intel X520 10GbE adapter for iSCSI traffic. The model # is 82599EB. I am connecting to Force10 S4810 switches. The switches have jumbo frames and flow control enabled. For some reason I cannot enable flow control on the NIC. If I run ethtool I see that autonegotiate is on, rx off tx off. If I try to enable rx on tx on, the command completes but they both stay off.

The nic ports are configured with the EqualLogic MEM and a standard vswitch.

Anyone familiar with these NICs and/or switches?

[MKguy · Accepted solution]

If the switches support 802.3x flow control autonegotiation and you enabled that, you actually shouldn't have to configure anything on the hosts.

Try to disable autonegotiation prior to forcibly enabling flow control on the ESXi-side; it might just switch back automatically as long as autoneg is enabled:

# ethtool --pause vmnic0 autoneg off

It may also not hurt to reset the link after the changes by re-plugging the cables or via re-initiating link autonegotiation from the host like:

# esxcfg-nics --auto vmnic0

Also check what NIC driver version you're running/have installed:

# ethtool -i vmnic0
[...]
# esxcli software vib list | grep ixgbe
net-ixgbe             2.0.84.8.2-10vmw.500.0.0.469512     VMware           VMwareCertified   2011-12-29

The latest ixgbe driver for your NIC available seems to be 3.11.32 released last month, update it if necessary:

https://my.vmware.com/group/vmware/details?downloadGroup=DT-ESXI50-INTEL-IXGBE-31132&productId=285

[RobVM]

Hey MKguy,

If you were here and I ever gave anyone a high-give or fist bump, I would totally give you one or both. The new driver and a reboot fixed the problem! Yesterday I flashed the firmware on the card to 13.5.7 which did not help (but I'm glad I did it anyway as the card was several revs behind). I got the firmware from Dell and used the Dell Repository Manager to create a boot disc with the update. Anyway Iinstalled the new driver today and after reboot FlowControl (Pause) is enabled. So you're right, after the new driver was installed, flowcontrol came on automatically on the NIC.

Now I can take a look at my iSCSI SAN traffic and see if this fixes my intermittent latency and re-transmit issues.

Thanks!

TL;DR, everyone running ESXi 5.1 with an Intel X520 NIC for iSCSI should update their firmware/drivers

[STK2]

We are about to deploy a new environment with Dell R720s and Intel X520 NICs. We will definitely check out the firmware and update.

As a matter of interest, were you running the Dell customised ESXi 5.1 image?

Thanks

Sean

[RobVM]

No, I didn't use the customized Dell ISO to install ESXi 5.1. The Dell ISO seems to be for recovery, not fresh installs, haven't seen the fresh installs version yet. Also read this KB if upgrading your hosts from version 5: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=203616...

If you are ordering new servers, you can get them pre-installed with ESXi from Dell onto flash media. Even so you would need to update the ixgbe driver as it is newer than the ESXi 5.1 RTM driver. You can build an update disc for the x520 firmware using Dell Repository Manager.

[smr222]

Had the very same issue with an Intel X520-T2 in Dell PowerEdge R710 running Dell Customized Image of ESXi 6.0 (Dell Version: A00, Build# 2494585) connected to a Dell Networking N4032.

Flow control started working after upgrading the boxed net-ixgbe driver (3.7.13.7.14iov-20vmw.600.0.0.2494585) to version 3.21.6-1OEM.600.0.0.2159203 (Intel, VMwareCertified).

VMware I/O Compatibility Guide for the Intel X520-T2 is your friend

http://www.vmware.com/resources/compatibility/detail.php?deviceCategory=io&productid=18611&VID=8086&...