Intel X550T can not link at 10Gb

[ciscomast]

Hello,

Please, can someone tell me if there are patch or configuration that can force  an Intel X550T Dual 10Gb nic controller to connect to 10Gb when the ESXi OS it is loaded?

We have two servers with identical hardware and OS configuration and in one of them the X550T ports connect at 10Gb without problems and the other one only connects at 1Gb.

Both servers has the same driver configuration:

server1:

esxcli network nic list

Name    PCI Device    Driver  Admin Status  Link Status  Speed  Duplex  MAC Address         MTU  Description

------  ------------  ------  ------------  -----------  -----  ------  -----------------  ----  -------------------------------------------------

vmnic0  0000:04:00.0  igbn    Up            Up            1000  Full    a4:bf:01:19:af:87  1500  Intel Corporation I350 Gigabit Network Connection

vmnic1  0000:04:00.3  igbn    Up            Down             0  Half    a4:bf:01:19:af:88  1500  Intel Corporation I350 Gigabit Network Connection

vmnic2  0000:02:00.0  igbn    Up            Up            1000  Full    a4:bf:01:0e:8e:c6  1500  Intel Corporation I350 Gigabit Network Connection

vmnic3  0000:02:00.1  igbn    Up            Down             0  Half    a4:bf:01:0e:8e:c7  1500  Intel Corporation I350 Gigabit Network Connection

vmnic4  0000:02:00.2  igbn    Up            Down             0  Half    a4:bf:01:0e:8e:c8  1500  Intel Corporation I350 Gigabit Network Connection

vmnic5  0000:02:00.3  igbn    Up            Down             0  Half    a4:bf:01:0e:8e:c9  1500  Intel Corporation I350 Gigabit Network Connection

vmnic6  0000:81:00.0  ixgbe   Up            Up            1000  Full    a0:36:9f:db:b5:20  9000  Intel(R) Ethernet Controller 10G X550T

vmnic7  0000:81:00.1  ixgbe   Up            Up            1000  Full    a0:36:9f:db:b5:21  9000  Intel(R) Ethernet Controller 10G X550T

ethtool -i vmnic6

driver: ixgbe

version: 4.4.1-iov

firmware-version: 0x80000492

bus-info: 0000:81:00.0

ethtool -i vmnic7

driver: ixgbe

version: 4.4.1-iov

firmware-version: 0x80000492

bus-info: 0000:81:00.1

server2:

esxcli network nic list

Name    PCI Device    Driver  Admin Status  Link Status  Speed  Duplex  MAC Address         MTU  Description

------  ------------  ------  ------------  -----------  -----  ------  -----------------  ----  -------------------------------------------------

vmnic0  0000:04:00.0  igbn    Up            Up             100  Half    a4:bf:01:19:58:07  1500  Intel Corporation I350 Gigabit Network Connection

vmnic1  0000:04:00.3  igbn    Up            Down             0  Half    a4:bf:01:19:58:08  1500  Intel Corporation I350 Gigabit Network Connection

vmnic2  0000:02:00.0  igbn    Up            Up            1000  Full    a4:bf:01:0e:8e:ca  1500  Intel Corporation I350 Gigabit Network Connection

vmnic3  0000:02:00.1  igbn    Up            Down             0  Half    a4:bf:01:0e:8e:cb  1500  Intel Corporation I350 Gigabit Network Connection

vmnic4  0000:02:00.2  igbn    Up            Down             0  Half    a4:bf:01:0e:8e:cc  1500  Intel Corporation I350 Gigabit Network Connection

vmnic5  0000:02:00.3  igbn    Up            Down             0  Half    a4:bf:01:0e:8e:cd  1500  Intel Corporation I350 Gigabit Network Connection

vmnic6  0000:81:00.0  ixgbe   Up            Up           10000  Full    a0:36:9f:b9:de:b4  9000  Intel(R) Ethernet Controller 10G X550T

vmnic7  0000:81:00.1  ixgbe   Up            Up           10000  Full    a0:36:9f:b9:de:b5  9000  Intel(R) Ethernet Controller 10G X550T

ethtool -i vmnic6

driver: ixgbe

version: 4.4.1-iov

firmware-version: 0x80000492

bus-info: 0000:81:00.0

ethtool -i vmnic7

driver: ixgbe

version: 4.4.1-iov

firmware-version: 0x80000492

bus-info: 0000:81:00.1

We swap the X550T controller by a new one and the problem remains the same.

When the server it is booting, during the BIOS load and until the ESXi loads the ixgbe driver, the X550T port are connected at 10Gb at the switch. Just after ESXi loads the ixgbe driver, the link connection drops to 1Gb.

It seems not a physical problem, we swap cables, controller and switch ports and the problem remains.

Any ideas or help would be appreciated.

[crieder]

Hi,

same fault here on 2 fujitsu rx200s6 with 2port X550 installed

Only difference: on 1 machine there is 1 port up with 10Gb, the other one links with 1GB. On other machine both ports link with 1GB

Also tried already all possible combinations, but did not change anything

Server1:

esxcli network nic list

Name    PCI Device     Driver  Link  Speed  Duplex  MAC Address         MTU  Description

------  -------------  ------  ----  -----  ------  -----------------  ----  --------------------------------------------------------------

vmnic0  0000:001:00.0  igb     Up     1000  Full    00:26:2d:04:d7:be  1500  Intel Corporation 82575EB Gigabit Network Connection

vmnic1  0000:001:00.1  igb     Up     1000  Full    00:26:2d:04:d7:bf  1500  Intel Corporation 82575EB Gigabit Network Connection

vmnic2  0000:005:00.0  e1000e  Up     1000  Full    00:1b:21:71:0f:c1  1500  Intel Corporation 82571EB Gigabit Ethernet Controller (Copper)

vmnic3  0000:005:00.1  e1000e  Up     1000  Full    00:1b:21:71:0f:c0  1500  Intel Corporation 82571EB Gigabit Ethernet Controller (Copper)

vmnic4  0000:006:00.0  e1000e  Up      100  Full    00:1b:21:71:0f:c3  1500  Intel Corporation 82571EB Gigabit Ethernet Controller (Copper)

vmnic5  0000:006:00.1  e1000e  Up      100  Full    00:1b:21:71:0f:c2  1500  Intel Corporation 82571EB Gigabit Ethernet Controller (Copper)

vmnic6  0000:007:00.0  ixgbe   Up     1000  Full    a0:36:9f:d4:bf:a8  9000  Intel(R) Ethernet Controller 10G X550T

vmnic7  0000:007:00.1  ixgbe   Up     1000  Full    a0:36:9f:d4:bf:a9  9000  Intel(R) Ethernet Controller 10G X550T

ethtool -i vmnic6

driver: ixgbe

version: 4.4.1-iov

firmware-version: 0x80000492

bus-info: 0000:07:00.0

ethtool -i vmnic7

driver: ixgbe

version: 4.4.1-iov

firmware-version: 0x80000492

bus-info: 0000:07:00.1

Server2:

esxcli network nic list

Name    PCI Device     Driver  Link  Speed  Duplex  MAC Address         MTU  Description

------  -------------  ------  ----  -----  ------  -----------------  ----  --------------------------------------------------------------

vmnic0  0000:001:00.0  igb     Up     1000  Full    00:26:2d:04:ab:42  1500  Intel Corporation 82575EB Gigabit Network Connection

vmnic1  0000:001:00.1  igb     Up     1000  Full    00:26:2d:04:ab:43  1500  Intel Corporation 82575EB Gigabit Network Connection

vmnic2  0000:005:00.0  e1000e  Up     1000  Full    00:1b:21:71:10:85  1500  Intel Corporation 82571EB Gigabit Ethernet Controller (Copper)

vmnic3  0000:005:00.1  e1000e  Up     1000  Full    00:1b:21:71:10:84  1500  Intel Corporation 82571EB Gigabit Ethernet Controller (Copper)

vmnic4  0000:006:00.0  e1000e  Up      100  Full    00:1b:21:71:10:87  1500  Intel Corporation 82571EB Gigabit Ethernet Controller (Copper)

vmnic5  0000:006:00.1  e1000e  Up      100  Full    00:1b:21:71:10:86  1500  Intel Corporation 82571EB Gigabit Ethernet Controller (Copper)

vmnic6  0000:007:00.0  ixgbe   Up     1000  Full    a0:36:9f:d4:bf:44  9000  Intel(R) Ethernet Controller 10G X550T

vmnic7  0000:007:00.1  ixgbe   Up    10000  Full    a0:36:9f:d4:bf:45  9000  Intel(R) Ethernet Controller 10G X550T

ethtool -i vmnic6

driver: ixgbe

version: 4.4.1-iov

firmware-version: 0x80000492

bus-info: 0000:07:00.0

ethtool -i vmnic7

driver: ixgbe

version: 4.4.1-iov

firmware-version: 0x80000492

bus-info: 0000:07:00.1

Any ideas to solve welcome

[Sakkanan]

Problem may be at

1.Specific Switch Port

2. Patch panel port

3.patch cable

try

1. connecting to another port in the switch

2. connect to another patch panel

3. Replace the cable

[crieder]

Did add some more information above.

But it is NO Switch (Port) problem because at system startup NIC comes up with 10GB Link. It drops to 1GB after vmWare takes control over NIC
No Patch Panel between
Patch cables are new and also exchanged already

Also there is a third server (a new one) connected with onboard x540-at2 which connects flawless with 10GB

ethtool - i

driver: ixgbe

version: 3.21.4iov

firmware-version: 0x80000369

bus-info: 0000:02:00.1

[crieder]

Solution could be that simple sometimes:

Just limit port at switch side to 10GB Full Duplex, after this, Link in vmWare also keeps at 10GB