Network problem connecting 10gbps back-to-back (direct connect without switch)

[simonloiselle]

Hi,

I've got a network problem plugins my ESXi to a storage using 10gpbs card to card (direct connect no-switch)

Here's my config

ESXi 5u1 running on Dell R720 using X520-T2 (ethernet)

2 x Nexenta running on Dell R710 using X520-T2 sas cable to Dell MD1200

Dell R720 vmkernel port 1 -> R710 #1

               vmkernel port 2 -> R710 #2

R710 #1 -> R710 #2

Both my R710 can ping each other using direct connect. Sadly my R720 can't ping either of my R710 or be pinged.

No firewall on Nexenta.

Anybody got some idea ?

Thanks.

So far I've tried :

Replace the network cable.

Try forcing  10000mbps Full on both R720 and R710 (in auto VMware detect 10000mbps Full)

Updating drivers in ESXi to 3.7.13-NAPI

Different MTU on both end.

Using a simple Cisco 1000mpbs swith work very well.

Some output :

#vmware -v
VMware ESXi 5.0.0 build-623860

#esxcfg-nics -l
vmnic1  0000:42:00.01 ixgbe       Up   10000Mbps Full   00:1b:21:d8:6a:d3 1500   Intel Corporation 82599EB 10 Gigabit TN Network Connection

#ethtool -i vmnic1
driver: ixgbe
version: 3.7.13-NAPI
firmware-version: 0x80000135
bus-info: 0000:42:00.1

#vmkping -D
PING 192.168.42.20 (192.168.42.20): 56 data bytes
64 bytes from 192.168.42.20: icmp_seq=0 ttl=64 time=0.036 ms
64 bytes from 192.168.42.20: icmp_seq=1 ttl=64 time=0.046 ms
64 bytes from 192.168.42.20: icmp_seq=2 ttl=64 time=0.038 ms

--- 192.168.42.20 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.036/0.040/0.046 ms

[marcelo_soares]

Send me a but more information.. interesting setup...

Complete outputs:

esxcfg-vmknic -l

esxcfg-nics -l

esxcfg-vswitch -l

IP of both Nexenta's. vmkPing test to both Nexenta's from ESX.

[simonloiselle]

Nexenta #1 192.168.42.10 to ESXi

Nexenta #2 192.168.42.12 to ESXi

# esxcfg-vmknic -l
Interface  Port Group/DVPort   IP Family IP Address                              Netmask         Broadcast       MAC Address       MTU     TSO MSS   Enabled Type
vmk0       Management Network  IPv4      10.10.42.100                            255.255.0.0     10.10.255.255   d4:ae:52:8e:65:3f 1500    65535     true    STATIC
vmk1       VMkernelNFS         IPv4      192.168.42.20                           255.255.255.0   192.168.42.255  00:50:56:75:a4:cc 1500    65535     true    STATIC
vmk2       VMkernelNFS2        IPv4      192.168.42.21                           255.255.255.0   192.168.42.255  00:50:56:75:cc:15 1500    65535     true    STATIC

# esxcfg-nics -l
Name    PCI           Driver      Link Speed     Duplex MAC Address       MTU    Description
vmnic0  0000:42:00.00 ixgbe       Up   10000Mbps Full   00:1b:21:d8:6a:d2 1500   Intel Corporation 82599EB 10 Gigabit TN Network Connection
vmnic1  0000:42:00.01 ixgbe       Up   10000Mbps Full   00:1b:21:d8:6a:d3 1500   Intel Corporation 82599EB 10 Gigabit TN Network Connection
vmnic2  0000:01:00.00 tg3         Up   1000Mbps  Full   d4:ae:52:8e:65:3f 1500   Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet
vmnic3  0000:01:00.01 tg3         Down 0Mbps     Half   d4:ae:52:8e:65:40 1500   Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet
vmnic4  0000:02:00.00 tg3         Down 0Mbps     Half   d4:ae:52:8e:65:41 1500   Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet
vmnic5  0000:02:00.01 tg3         Down 0Mbps     Half   d4:ae:52:8e:65:42 1500   Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet

# esxcfg-vswitch -l
Switch Name      Num Ports   Used Ports  Configured Ports  MTU     Uplinks
vSwitch0         128         4           128               1500    vmnic2,vmnic3

  PortGroup Name        VLAN ID  Used Ports  Uplinks
  VM Network            0        0           vmnic2,vmnic3
  Management Network    0        1           vmnic2,vmnic3

Switch Name      Num Ports   Used Ports  Configured Ports  MTU     Uplinks
vSwitch1         128         1           128               1500

  PortGroup Name        VLAN ID  Used Ports  Uplinks
  VM MDT                0        0

Switch Name      Num Ports   Used Ports  Configured Ports  MTU     Uplinks
vSwitch2         128         3           128               1500    vmnic1

  PortGroup Name        VLAN ID  Used Ports  Uplinks
  VM Private Storage    0        0           vmnic1
  VMkernelNFS           0        1           vmnic1

Switch Name      Num Ports   Used Ports  Configured Ports  MTU     Uplinks
vSwitch3         128         3           128               1500    vmnic0

  PortGroup Name        VLAN ID  Used Ports  Uplinks
  VMkernelNFS2          0        1           vmnic0

 
~ # vmkping 192.168.42.10
PING 192.168.42.10 (192.168.42.10): 56 data bytes

--- 192.168.42.10 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss
~ # vmkping 192.168.42.12
PING 192.168.42.12 (192.168.42.12): 56 data bytes

--- 192.168.42.12 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss

[simonloiselle]

From both of my Nexenta

nmc@nxtd01:/$ ping 192.168.42.20
no answer from 192.168.42.20

nmc@nxtd01:/$ ping 192.168.42.21
no answer from 192.168.42.21

nmc@nxtd02:/$ ping 192.168.42.20
no answer from 192.168.42.20

nmc@nxtd02:/$ ping 192.168.42.21
no answer from 192.168.42.21

(All update have been applied to nexenta)

[marcelo_soares]

Try this: place each nexenta on different subnets and configure the vmkernel interfaces accordingly. Having 2 different vmknics with the same subnet on different uplinks can generate this kind of problem, although is odd that none of the servers ping.

[simonloiselle]

Changed my second vmkernel to 172.16.42.20 / 255.255.0.0 then tried 255.255.255.0

Same result (failed)

Removed completly one of vmkernel (tried both of them) to connect only one nexenta at a time.

Same result (still failed)

[marcelo_soares]

Well... when trying ping from nexenta, check if the ICMP requests get to the vmkX:

tcpdump-uw -i vmkX

replace X with wathever vmk you are testing. If ICMP is not arriving, the problem is the physical layer.

[marcelo_soares]

You need to allow promiscuous mode on the vSwitch for this to work, forgot to mention.

[simonloiselle]

Promiscuous mode was already enabled.

Tried ping from both my nexenta during the capture

~ # tcpdump-uw -i vmk1
tcpdump-uw: WARNING: SIOCGIFINDEX: Invalid argument
tcpdump-uw: verbose output suppressed, use -v or -vv for full protocol decode
listening on vmk1, link-type EN10MB (Ethernet), capture size 96 bytes
tcpdump-uw: pcap_loop: recvfrom: Interrupted system call
0 packets captured
0 packets received by filter
0 packets dropped by kernel

Tried to change cable just in case (using CAT 7).

Does that mean faulty NIC card ?

Do you think I will have the same problem if I found 10gbps swith ?

Thanks for your time.

[marcelo_soares]

I think that maybe is not that simple connect directly the 10gbit cards without a switch. Maybe you can start looking for the problem on this approach.

If with 1Gbit switch it works, it should work with a 10GBit one...