NSX-T 3.0 - extremely low transfer rate between vms on dvs and vms in overlay on n-vds

[Czernobog]

- have to repost this, since yesterday the new great forum software did not seem to save the thread... -

I am managing a small NSX-T test environment, consisting of 3 ESXi hosts as transport nodes and an overlay transport zone, where a few overlay networks are configured.
The hosts have 4 x 1Gbps physical uplinks (no switches with 10gb available atm...), two connected to the vds, the rest to the n-vds.
The Uplink profile applied to the ESXi Hosts uses both n-vds uplinks in a LB configuration. It uses the global MTU configuration setting of MTU = 1600.
2 Edge Transport nodes are grouped in an Edge Cluster. On the edges, uplink-1 is connected to the transport vlan for the overlay networks, uplinks 2 and 3 are used for communication with the physical switches.

There is almost no load on the environment currently, be it compute, storage or network. The edge nodes are deployed in the Medium sizing configuration.

The problem I am facing is, that transfer rates between vms placed on the vds in a physical vlan and VMs placed in the overlay network are extremely bad. I can only get something needing low thorughput, like SSH, to work, but during file transfers using SSH or HTTP I get as low as 30kB/s before loosing the connection. Using performance views from the vcenter I did not notice any packet drops neither on the hosts' vmnics nor the VMs' nic. Ping between components returns roundtrip times of <1 ms.

Transfer rates between VMs inside the overlay network are fine, even when they are placed on different ESXi hosts in the cluster. I've also tried different source systems from the physical VLAN to do the transfer tests.

All VMs placed in the overlay, regardles of the ESXi host, seem to be affected. Transfers between systems placed in VLANs on the vds are not negatively affected either.

All switchports connecting the transport nodes are configured consistenlty in regards of VLAN trunks, MTU size being 9216 bytes.

I've used https://spillthensxt.com/how-to-validate-mtu-in-an-nsx-t-environment/ as a guideline to check for MTU consistency across all components and could not find an issue. I use 1600 on the vds and as a default value on the nsx-t uplink profiles used on the transport nodes as well as the edge nodes.

I'm kind of at a loss here what to troubleshoot next, and any tips are most welcome:)

[p0wertje]

Could you tell a bit more about the env ?

Maybe a small drawing ? 
Are you using virtual edges ?
How is routing done between vds vlan and overlay ?

[Czernobog]

The Edges are deployed as VMs. They are grouped in a cluster as a pair, with active-passive failover mode.

Here's a scetch of a single node, I hope this helps:

edit: added some info.

[p0wertje]

Thx.

Are the edge-node TEP and the esx teps in different subnets ?

NSX Edge VM can be deployed using VLAN-backed logical switches on the N-VDS of the host transport node. Host TEP and NSX Edge TEP must be in different subnets.

[Czernobog]

Here are the 3 profiles applied on the transport nodes (ignore _VDS, it is not applied anywhere).

The first one is applied on the transport node n-vds uplinks (vmnic2 and vmnic3).

The second one on the TEP interface of the edge node (fp-eth0). The third on the uplink ports fp-eth1 and fp-eth2.

Edit:

I've also just checked, and the transfer rate between overlay networks attached to different edge node clusters is also affected. The other ledge cluster is identically configured as the one described above, just that the logical switch uses another subnet for adressing.

Also - AFAIK the transport node TEPs as well as the edge TEPs can use IPs from the same subnet. I could not find contradictory information. I know that when you use NSX-T lower than 3.1 in regards of VLANs in the transport network, but this occurs only when you use the VDS 7 exclusively to channel all traffic. This is not the case here.

[shank89]

Spoiler

What sort of hardware are you running it on?

You are correct about the TEPs in 3.1, details on how that works can be found here https://www.lab2prod.com.au/2020/11/nsx-t-inter-TEP.html#more

How are you testing, have you run any iPerf tests or just copying and pasting files? 

Have you done any packet captures on the edge appliances and hosts whilst doing the transfers? Have you checked esxtop to see if anything looks off?  What switches,  is your routing mtu set correctly?

[Czernobog]

I used copying files to identify the problem. What would be the benefit of using iperf in this case?

The MTU is set correctly end to end, 1600 as per documentation. Wouldn't I get dropped packets, when the MTU is not set correctly somewhere along the way, sicne Geneve frames cannot be fragmented?

Also it does not seem to be an issue with the networking hardware, since the problem occurs even the source and target system are placed on the same ESXi host. Packets would not leave the host in this case.

[shank89]

Just trying to cover all bases, it can sometimes be difficult to pin point Without logs and eyes on the environment.  There is obviously something wrong somewhere.

[Czernobog · Accepted solution]

Just wanted to close this thread, it only started to work again after the existing NSX installation has been deleted and a fresh NSX-T 3.1 installation was deployed. This was done with PSO assistance, besides reviewing the NSX and vSphere configuration, we have also checked the configuration of the network hardware but sadly we could not find the origin of the issue.

Since this was not production, redeploying NSX was a viable solution.

[JamesCurtis]

I have the same problem. However, problems still exist after reinstalling NSX-T. And the problem is random. Some virtual machines will exist and some will not. These virtual machines are cloned from an Ubuntu jammy template. And this problem only exists when external traffic is sent to the virtual machine. If the - R parameter is added to iperf3, the virtual machine can send traffic from the internal to the external.
I have no way to solve it. Is there any progress?

[JamesCurtis]

In addition, even if three virtual machines are in a segment and conduct iperf3 tests on each other, such results will also occur. I'm sure the firewall of the virtual machine has been turned off.

[JamesCurtis]

If the vm initiatively initiates the iperf, the connection will be directly disconnected. Otherwise, it will not, but the speed is very slow 0k/s.

My laptop:

D:\Users\th>iperf3 -s -p 666
-----------------------------------------------------------
Server listening on 666
-----------------------------------------------------------
iperf3: error - unable to receive parameters from client: Connection reset by peer

The vm:

root@ks01:~# iperf3 -c 10.10.1.211 -p 666
iperf3: error - unable to send control message: Broken pipe

[JamesCurtis]

Latest test results. I found that if only one network card is left for each virtual machine, it seems that forward transmission and reverse transmission are normal.

[JamesCurtis]

Guys, I probably found the reason. The root of the problem is that all my virtual machines have two network cards. When they get the dhcp information of nsx-t, they specify two default routes.

root@ks01:~# ip route
default via 172.16.10.1 dev ens192 proto dhcp src 172.16.10.25 metric 100 
default via 172.16.10.1 dev ens160 proto dhcp src 172.16.10.101 metric 100 
10.10.1.1 via 172.16.10.1 dev ens192 proto dhcp src 172.16.10.25 metric 100 
10.10.1.1 via 172.16.10.1 dev ens160 proto dhcp src 172.16.10.101 metric 100 
172.16.10.0/24 dev ens192 proto kernel scope link src 172.16.10.25 metric 100 
172.16.10.0/24 dev ens160 proto kernel scope link src 172.16.10.101 metric 100 
172.16.10.1 dev ens192 proto dhcp scope link src 172.16.10.25 metric 100 
172.16.10.1 dev ens160 proto dhcp scope link src 172.16.10.101 metric 100 

Now let's test the flow transmission test on the 172.16.10.101 network card of the ens160. The results displayed in the virtual machine are as follows

root@ks01:~# iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 10.10.1.2, port 62401
[  5] local 172.16.10.101 port 5201 connected to 10.10.1.2 port 62402
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  64.3 KBytes   526 Kbits/sec                  
[  5]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.06  sec  64.3 KBytes  52.3 Kbits/sec                  receiver

Now, I turn off the ens192 network card, and the test results immediately return to normal.

Then turn on the ens192 network card. After testing the ens160, it was found that the traffic bandwidth was still normal. (I don't understand the reason here)
However, if the ens192 is tested at this time, the situation of 0bits/s will occur again.

Next, close the ens160 network card and test the ens192. The results show that it miraculously restored the normal bandwidth.

Next, close the ens160 network card and test the ens192. The result shows that the ens192 magically restores the normal bandwidth. If you open the ens160 network card again, the ens192 is still normal. At this time, the problem shifts to ens160.

On the surface, it seems that this fault will always remain on one of the network cards. I mark it as A. The fault will not disappear until other network cards (marked with X) are turned off. However, if the closed network card (X) is opened again, the fault will appear on the network card (X) opened again, and the network card A will not be affected.
If you repair network card X according to the idea in the previous paragraph, network card A will fail again after one operation.