Highlighted
Contributor
Contributor

Chelsio t520-cr 10G nic not functional in ESXi 6.7

Hello!

I recently purchased two Chelsio t520-cr 10G nics to connect my ESXi box to my FreeNAS box.  I installed both cards and connected them, but the card was not found in ESXi.  I downloaded the driver from Chelsio and installed the cxl driver per the instructions.

The card is listed by lspci:

[root@orion:~] lspci | grep Chelsio

0000:01:00.0 Network controller: Chelsio Communications Inc. T520-CR Unified Wire Ethernet Controller

0000:01:00.1 Network controller: Chelsio Communications Inc. T520-CR Unified Wire Ethernet Controller

0000:01:00.2 Network controller: Chelsio Communications Inc. T520-CR Unified Wire Ethernet Controller

0000:01:00.3 Network controller: Chelsio Communications Inc. T520-CR Unified Wire Ethernet Controller

0000:01:00.4 Network controller: Chelsio Communications Inc. T520-CR Unified Wire Bus Enumerator

0000:01:00.5 Mass storage controller: Chelsio Communications Inc T520-CR Unified Wire Storage Controller

0000:01:00.6 Serial bus controller: Chelsio Communications Inc T520-CR Unified Wire Storage Controller

[root@orion:~]

and the driver appears to be loaded:

[root@orion:~] esxcli software vib list | grep cxl

cxl                            2.0.0.21-1OEM.650.0.0.4598673         Chelsio  VMwareCertified   2018-06-17

[root@orion:~]

and the kernel logs says it loaded the driver:

[root@orion:~] cat /var/log/vmkernel.log

...

2018-06-17T19:30:16.259Z cpu2:2099757)Loading module cxl ...

2018-06-17T19:30:16.265Z cpu2:2099757)Elf: 2101: module cxl has license ThirdParty

2018-06-17T19:30:16.337Z cpu2:2099757)Device: 192: Registered driver 'cxl' from 96

2018-06-17T19:30:16.337Z cpu2:2099757)Chelsio T5/T6 Native Network Driver - version 2.0.0.21

2018-06-17T19:30:16.338Z cpu2:2099757)Mod: 4962: Initialization of cxl succeeded with module ID 96.

2018-06-17T19:30:16.338Z cpu2:2099757)cxl loaded successfully.

[root@orion:~]

However, when I list the nics, the chelsio card is not listed.  Only the onboard gigabit ethernet(s):

[root@orion:~] esxcfg-nics -l

Name    PCI          Driver      Link Speed      Duplex MAC Address       MTU    Description

vmnic0  0000:03:00.0 igbn        Up   1000Mbps   Full   ac:1f:6b:46:89:d0 1500   Intel Corporation I350 Gigabit Network Connection

vmnic1  0000:03:00.1 igbn        Up   1000Mbps   Full   ac:1f:6b:46:89:d1 1500   Intel Corporation I350 Gigabit Network Connection

[root@orion:~]

I've rebooted several times with no luck.  The card itself has a steady red light which according to chelsio's documentation, just means that the card has power and it should go out when the driver is loaded.  The card that I installed in the FreeNAS box was detected and the light is out as they suggested it would be.

I'm really stuck. Any ideas are welcome!  Thanks!

0 Kudos
9 Replies
Highlighted
Contributor
Contributor

Does anyone have any ideas on this?  I'm kinda stuck here and I'm out of ideas for what to try next.

0 Kudos
Highlighted
VMware Employee
VMware Employee

Can you check if the driver is loaded.

vmkload_mod -l | grep cxl

If not try loading it manually. That may help you solve this issue

0 Kudos
Highlighted
Contributor
Contributor

Hi jterli - I ran the command you gave and cxl is indeed loaded.

I have been trying to get the driver from Chelsio to work.  There is a driver available from the VMware site that is for ESXi 6.5.  Do you think I should remove the Chelsio driver and install the driver for ESXi 6.5?

0 Kudos
Highlighted
VMware Employee
VMware Employee

In the driver downloaded from Chelsio, there is a bundle "VMW-ESX-6.5.0-cxl-2.0.0.21-offline_bundle-7758199".

Can you try using the vib from there instead of "cxl-2.0.0.21-1OEM.650.0.0.4598673.x86_64.vib"

This is same as the one you would download from VMware site.

Highlighted
Contributor
Contributor

Hi jterli,

I went ahead and uninstalled the Chelsio driver and installed the VIB from VMware and when I rebooted, it worked.  I didn't see your message, so I installed the following vib:

cxl-2.0.0.21-1OEM.650.0.0.4598673.x86_64.vib

I did see the offline bundle you mentioned, but the vib above worked when I rebooted.  Thank you so much for your help though!  Glad we got this resolved.

Thanks!

0 Kudos
Highlighted
Contributor
Contributor

Hello again!

As described in the thread, I was having difficulty getting ESXi to recognize my Chelsio t520-cr 10g NIC.  I was able to resolve this problem and the interfaces are now recognized by ESXi.  However, when I try to connect to an iSCSI block share from my FreeNAS box, ESXi cannot connect to the iSCSI share.  I am getting a lot of the following error on my FreeNAS box:

pastedImage_0.png

on my ESXi box, dmesg is showing the following:

pastedImage_1.png

I can ping both boxes from each other (but the ping times are variable - .2 ms - 5000 ms).  I cannot ssh between the boxes using the Chelsio interfaces.  The two interfaces are setup as follows:

FreeNAS  <--> ESXi 6.7

10.0.0.1    <--> 10.0.0.2

10.0.1.1    <--> 10.0.1.2

These two ports are on each server are connected to each other using Chelsio Twinax cables.

The onboard 1g nic's are setup on DHCP using 192.168.1.0/24 addresses.  If I try to ssh root@10.0.0.2 from my FreeNAS box, it will not connect.  Same from the ESXi box.  I'm thinking that there are some ethernet configuration settings that are incorrect that are causing a problem.  I have kept the MTU for all interfaces at the standard 1500 to try and eliminate that as an issue.

I'm pretty much out of ideas on this one.  Any help would be greatly appreciated!

0 Kudos
Highlighted
Contributor
Contributor

Hi Folks - any thoughts on next steps to debug?  I downloaded iperf and ran some tests.  Definitely highlights there's a problem, but I'm at a loss as to what else I should try to get these cards communicating at full speed.  Thanks for your help!

[root@orion:/vmfs/volumes/5aff1136-dfc79715-eb9e-ac1f6b4689d0/tools] ./iperf -c 10.0.0.1

------------------------------------------------------------

Client connecting to 10.0.0.1, TCP port 5001

TCP window size: 32.5 KByte (default)

------------------------------------------------------------

[  3] local 10.0.0.2 port 10337 connected with 10.0.0.1 port 5001

[ ID] Interval       Transfer     Bandwidth

[  3]  0.0-10.0 sec   334 MBytes   280 Mbits/sec

[root@orion:/vmfs/volumes/5aff1136-dfc79715-eb9e-ac1f6b4689d0/tools] ./iperf -c 10.0.1.1

------------------------------------------------------------

Client connecting to 10.0.1.1, TCP port 5001

TCP window size: 32.5 KByte (default)

------------------------------------------------------------

[  3] local 10.0.1.2 port 58809 connected with 10.0.1.1 port 5001

[ ID] Interval       Transfer     Bandwidth

[  3]  0.0-10.0 sec  2.98 GBytes  2.56 Gbits/sec

[root@orion:/vmfs/volumes/5aff1136-dfc79715-eb9e-ac1f6b4689d0/tools] ./iperf -c 10.0.0.1

------------------------------------------------------------

Client connecting to 10.0.0.1, TCP port 5001

TCP window size: 32.5 KByte (default)

------------------------------------------------------------

[  3] local 10.0.0.2 port 46568 connected with 10.0.0.1 port 5001

[ ID] Interval       Transfer     Bandwidth

[  3]  0.0-10.0 sec   540 MBytes   453 Mbits/sec

[root@orion:/vmfs/volumes/5aff1136-dfc79715-eb9e-ac1f6b4689d0/tools] ./iperf -c 10.0.1.1

------------------------------------------------------------

Client connecting to 10.0.1.1, TCP port 5001

TCP window size: 32.5 KByte (default)

------------------------------------------------------------

[  3] local 10.0.1.2 port 13216 connected with 10.0.1.1 port 5001

[ ID] Interval       Transfer     Bandwidth

[  3]  0.0-10.0 sec   183 MBytes   154 Mbits/sec

[root@orion:/vmfs/volumes/5aff1136-dfc79715-eb9e-ac1f6b4689d0/tools] ./iperf -c 10.0.0.1

------------------------------------------------------------

Client connecting to 10.0.0.1, TCP port 5001

TCP window size: 32.5 KByte (default)

------------------------------------------------------------

[  3] local 10.0.0.2 port 62274 connected with 10.0.0.1 port 5001

[ ID] Interval       Transfer     Bandwidth

[  3]  0.0-10.0 sec   231 MBytes   193 Mbits/sec

[root@orion:/vmfs/volumes/5aff1136-dfc79715-eb9e-ac1f6b4689d0/tools] ./iperf -c 10.0.1.1

------------------------------------------------------------

Client connecting to 10.0.1.1, TCP port 5001

TCP window size: 32.5 KByte (default)

------------------------------------------------------------

[  3] local 10.0.1.2 port 16665 connected with 10.0.1.1 port 5001

[ ID] Interval       Transfer     Bandwidth

[  3]  0.0-10.0 sec  3.25 GBytes  2.79 Gbits/sec

[root@orion:/vmfs/volumes/5aff1136-dfc79715-eb9e-ac1f6b4689d0/tools] ./iperf -c 10.0.1.1

------------------------------------------------------------

Client connecting to 10.0.1.1, TCP port 5001

TCP window size: 32.5 KByte (default)

------------------------------------------------------------

[  3] local 10.0.1.2 port 59133 connected with 10.0.1.1 port 5001

[ ID] Interval       Transfer     Bandwidth

[  3]  0.0-10.0 sec   162 MBytes   136 Mbits/sec

[root@orion:/vmfs/volumes/5aff1136-dfc79715-eb9e-ac1f6b4689d0/tools] ssh 10.0.0.1

FIPS mode initialized

[root@orion:/vmfs/volumes/5aff1136-dfc79715-eb9e-ac1f6b4689d0/tools]

0 Kudos
Highlighted
Contributor
Contributor

Hi folks - I'm still working on debugging this.  I've reached out to Chelsio support.  I'm hoping someone sees something that can help. 

0 Kudos
Highlighted
Contributor
Contributor

Did you ever figure this out?

0 Kudos