Hello!
I recently purchased two Chelsio t520-cr 10G nics to connect my ESXi box to my FreeNAS box. I installed both cards and connected them, but the card was not found in ESXi. I downloaded the driver from Chelsio and installed the cxl driver per the instructions.
The card is listed by lspci:
[root@orion:~] lspci | grep Chelsio
0000:01:00.0 Network controller: Chelsio Communications Inc. T520-CR Unified Wire Ethernet Controller
0000:01:00.1 Network controller: Chelsio Communications Inc. T520-CR Unified Wire Ethernet Controller
0000:01:00.2 Network controller: Chelsio Communications Inc. T520-CR Unified Wire Ethernet Controller
0000:01:00.3 Network controller: Chelsio Communications Inc. T520-CR Unified Wire Ethernet Controller
0000:01:00.4 Network controller: Chelsio Communications Inc. T520-CR Unified Wire Bus Enumerator
0000:01:00.5 Mass storage controller: Chelsio Communications Inc T520-CR Unified Wire Storage Controller
0000:01:00.6 Serial bus controller: Chelsio Communications Inc T520-CR Unified Wire Storage Controller
[root@orion:~]
and the driver appears to be loaded:
[root@orion:~] esxcli software vib list | grep cxl
cxl 2.0.0.21-1OEM.650.0.0.4598673 Chelsio VMwareCertified 2018-06-17
[root@orion:~]
and the kernel logs says it loaded the driver:
[root@orion:~] cat /var/log/vmkernel.log
...
2018-06-17T19:30:16.259Z cpu2:2099757)Loading module cxl ...
2018-06-17T19:30:16.265Z cpu2:2099757)Elf: 2101: module cxl has license ThirdParty
2018-06-17T19:30:16.337Z cpu2:2099757)Device: 192: Registered driver 'cxl' from 96
2018-06-17T19:30:16.337Z cpu2:2099757)Chelsio T5/T6 Native Network Driver - version 2.0.0.21
2018-06-17T19:30:16.338Z cpu2:2099757)Mod: 4962: Initialization of cxl succeeded with module ID 96.
2018-06-17T19:30:16.338Z cpu2:2099757)cxl loaded successfully.
[root@orion:~]
However, when I list the nics, the chelsio card is not listed. Only the onboard gigabit ethernet(s):
[root@orion:~] esxcfg-nics -l
Name PCI Driver Link Speed Duplex MAC Address MTU Description
vmnic0 0000:03:00.0 igbn Up 1000Mbps Full ac:1f:6b:46:89:d0 1500 Intel Corporation I350 Gigabit Network Connection
vmnic1 0000:03:00.1 igbn Up 1000Mbps Full ac:1f:6b:46:89:d1 1500 Intel Corporation I350 Gigabit Network Connection
[root@orion:~]
I've rebooted several times with no luck. The card itself has a steady red light which according to chelsio's documentation, just means that the card has power and it should go out when the driver is loaded. The card that I installed in the FreeNAS box was detected and the light is out as they suggested it would be.
I'm really stuck. Any ideas are welcome! Thanks!
Does anyone have any ideas on this? I'm kinda stuck here and I'm out of ideas for what to try next.
Can you check if the driver is loaded.
vmkload_mod -l | grep cxl
If not try loading it manually. That may help you solve this issue
Hi jterli - I ran the command you gave and cxl is indeed loaded.
I have been trying to get the driver from Chelsio to work. There is a driver available from the VMware site that is for ESXi 6.5. Do you think I should remove the Chelsio driver and install the driver for ESXi 6.5?
In the driver downloaded from Chelsio, there is a bundle "VMW-ESX-6.5.0-cxl-2.0.0.21-offline_bundle-7758199".
Can you try using the vib from there instead of "cxl-2.0.0.21-1OEM.650.0.0.4598673.x86_64.vib"
This is same as the one you would download from VMware site.
Hi jterli,
I went ahead and uninstalled the Chelsio driver and installed the VIB from VMware and when I rebooted, it worked. I didn't see your message, so I installed the following vib:
cxl-2.0.0.21-1OEM.650.0.0.4598673.x86_64.vib
I did see the offline bundle you mentioned, but the vib above worked when I rebooted. Thank you so much for your help though! Glad we got this resolved.
Thanks!
Hello again!
As described in the thread, I was having difficulty getting ESXi to recognize my Chelsio t520-cr 10g NIC. I was able to resolve this problem and the interfaces are now recognized by ESXi. However, when I try to connect to an iSCSI block share from my FreeNAS box, ESXi cannot connect to the iSCSI share. I am getting a lot of the following error on my FreeNAS box:
on my ESXi box, dmesg is showing the following:
I can ping both boxes from each other (but the ping times are variable - .2 ms - 5000 ms). I cannot ssh between the boxes using the Chelsio interfaces. The two interfaces are setup as follows:
FreeNAS <--> ESXi 6.7
10.0.0.1 <--> 10.0.0.2
10.0.1.1 <--> 10.0.1.2
These two ports are on each server are connected to each other using Chelsio Twinax cables.
The onboard 1g nic's are setup on DHCP using 192.168.1.0/24 addresses. If I try to ssh root@10.0.0.2 from my FreeNAS box, it will not connect. Same from the ESXi box. I'm thinking that there are some ethernet configuration settings that are incorrect that are causing a problem. I have kept the MTU for all interfaces at the standard 1500 to try and eliminate that as an issue.
I'm pretty much out of ideas on this one. Any help would be greatly appreciated!
Hi Folks - any thoughts on next steps to debug? I downloaded iperf and ran some tests. Definitely highlights there's a problem, but I'm at a loss as to what else I should try to get these cards communicating at full speed. Thanks for your help!
[root@orion:/vmfs/volumes/5aff1136-dfc79715-eb9e-ac1f6b4689d0/tools] ./iperf -c 10.0.0.1
------------------------------------------------------------
Client connecting to 10.0.0.1, TCP port 5001
TCP window size: 32.5 KByte (default)
------------------------------------------------------------
[ 3] local 10.0.0.2 port 10337 connected with 10.0.0.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 334 MBytes 280 Mbits/sec
[root@orion:/vmfs/volumes/5aff1136-dfc79715-eb9e-ac1f6b4689d0/tools] ./iperf -c 10.0.1.1
------------------------------------------------------------
Client connecting to 10.0.1.1, TCP port 5001
TCP window size: 32.5 KByte (default)
------------------------------------------------------------
[ 3] local 10.0.1.2 port 58809 connected with 10.0.1.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 2.98 GBytes 2.56 Gbits/sec
[root@orion:/vmfs/volumes/5aff1136-dfc79715-eb9e-ac1f6b4689d0/tools] ./iperf -c 10.0.0.1
------------------------------------------------------------
Client connecting to 10.0.0.1, TCP port 5001
TCP window size: 32.5 KByte (default)
------------------------------------------------------------
[ 3] local 10.0.0.2 port 46568 connected with 10.0.0.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 540 MBytes 453 Mbits/sec
[root@orion:/vmfs/volumes/5aff1136-dfc79715-eb9e-ac1f6b4689d0/tools] ./iperf -c 10.0.1.1
------------------------------------------------------------
Client connecting to 10.0.1.1, TCP port 5001
TCP window size: 32.5 KByte (default)
------------------------------------------------------------
[ 3] local 10.0.1.2 port 13216 connected with 10.0.1.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 183 MBytes 154 Mbits/sec
[root@orion:/vmfs/volumes/5aff1136-dfc79715-eb9e-ac1f6b4689d0/tools] ./iperf -c 10.0.0.1
------------------------------------------------------------
Client connecting to 10.0.0.1, TCP port 5001
TCP window size: 32.5 KByte (default)
------------------------------------------------------------
[ 3] local 10.0.0.2 port 62274 connected with 10.0.0.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 231 MBytes 193 Mbits/sec
[root@orion:/vmfs/volumes/5aff1136-dfc79715-eb9e-ac1f6b4689d0/tools] ./iperf -c 10.0.1.1
------------------------------------------------------------
Client connecting to 10.0.1.1, TCP port 5001
TCP window size: 32.5 KByte (default)
------------------------------------------------------------
[ 3] local 10.0.1.2 port 16665 connected with 10.0.1.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 3.25 GBytes 2.79 Gbits/sec
[root@orion:/vmfs/volumes/5aff1136-dfc79715-eb9e-ac1f6b4689d0/tools] ./iperf -c 10.0.1.1
------------------------------------------------------------
Client connecting to 10.0.1.1, TCP port 5001
TCP window size: 32.5 KByte (default)
------------------------------------------------------------
[ 3] local 10.0.1.2 port 59133 connected with 10.0.1.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 162 MBytes 136 Mbits/sec
[root@orion:/vmfs/volumes/5aff1136-dfc79715-eb9e-ac1f6b4689d0/tools] ssh 10.0.0.1
FIPS mode initialized
[root@orion:/vmfs/volumes/5aff1136-dfc79715-eb9e-ac1f6b4689d0/tools]
Hi folks - I'm still working on debugging this. I've reached out to Chelsio support. I'm hoping someone sees something that can help.
Did you ever figure this out?