TCP session fail in a Nested environment

[aigiorgi]

Hi all:

I'm setting up a NSX nested lab, very similar to the NSX HOL with a plane (not segmented) level 1 network.

After to deploy four Logical Distributed Switches (transit, web, app and db tear), one Logical Distributed Router and one Edge Router, I reach the point where I can ping from the controller center (outside) to an internal VM connected to a LDS VNI 5001 and ping and traceroute work perfectly, but TCP sessions not.

If I capture packet at edge router I can see the returning packets from the VM reaching the inside interface (transit), but the packets does not reach the outside interface.

The firewall permit all the traffic (both, Distributed Logical Firewall and Edge Firewall).

I tested with NSX 6.0 and 6.1 with same behavior.

dvFilter is installed in nested VMs.

ESXi have e1000e vnics.

vmware tools installed in nested esxi

Is very strange that the icmp traffic work end to end but tcp not.

Something have seen the same behavior before?

Any comment will be appreciated.

Thanks

Al

[vBenja]

HI AI,

check MTU settings between VTEP:

ping ++netstack=vxlan -d -s 1572 -I vmkX 1.2.3.4

and change vmxnet3 instead of e1000 for nested ESXi.

Benja

[SRoland]

I have seen this happening when the path is asymmetric. ICMP does not care, but TCP does. Do you have a topology drawing or something about this?

[brandonpremo]

I was troubleshooting this issue for over a week.

Nested ESXi NSX lab.

UDP / ICMP traffic worked fine, but I had issues with TCP traffic.

I had originally used Intel 1000 adapters for the Virtual ESXi hosts.

This gave me an issue with jumbo frames.

I moved to Intel 1000e and it fixed the jumbo issue.

I just moved to the vxmnet3 adapters and my TCP issues are gone.

Jumbo frames are also still working.

I don't have a good answer for what the issue was. My NSX lab connects to a larger lab.

There is a router that does NAT for the entire lab to reach the internet.

TCP connections from VMs running on the virtual ESXi hosts that were attached to regular DVS portgroups had no issues.

TCP connections from VMs connected to logical switches would work to any device inside the lab, but everything through the NAT router would not work.

I did packet traces at every point and eventually found that the servers were not sending SYN ACK to reply to the VMs SYN for the TCP session...

I'm not going to take the time to go deep into detail, all I will say is use vmxnet3 for your virtual ESXi hosts.