NSX DLR issue

[iforbes]

Hi. I'm running ESXi 6.5/NSX 6.3. I've got my distributed switching working fine. I can:

1) Ping vm's that are on the same logical network, but on different ESXi hosts

2) Ping vm's that are on the same logical network, and on same ESXi host

I deployed a DLR to test out L3 routing between vm's on different logical networks. My DLR has LIFs for :

-web-tier - lif = 192.169.100.1

-app-tier - lif = 192.168.200.1

The vm's have been configured with the applicable default gateways (as above).

From a routing perspective, I can:

1) Ping a vm from web-tier network to a vm on app-tier network when both vm's exist on the same ESXi host (and vice-verca for app-tier vm to web-tier vm)

2) Cannot ping a vm from web-tier network to a vm on app-tier network when both vm's exist on different ESXi host (and vice-verca)

This seems to be an issue passing L3 VTEP traffic from one ESXi host to another. As mentioned above, when the vm's that are on different ESXi hosts are participating in L2 switching, it's successful (i.e. VXLAN IO successfully traverses between ESXi VTEPs).

So, something is wrong with L3 routing, only between ESXi hosts.

[Sreec]

Can the Vm's ping the gateway IP when they are on different host ? Also do check the routing table on both the host .

[iforbes]

Yes. The vm's can ping the default gateway of the other vm (i.e. app-tier vm can ping the default gateway of web-tier vm and vice-verca) when they are on different ESXi hosts.The vm's can also ping their own default gateway. So, it looks like L3 traffic can make it to the DLR fine when vm's are located on different ESXi hosts. Something is messed once the routed IO is on the DLR AND the vm's live on different ESXi hosts.

[iforbes]

VDR default+edge-1 Route Table

Legend: [U: Up], [G: Gateway], [C: Connected], [I: Interface]

Legend: [H: Host], [F: Soft Flush] [!: Reject] [E: ECMP]

Destination      GenMask          Gateway          Flags    Ref Origin   UpTime     Interface

-----------      -------          -------          -----    --- ------   ------     ---------

0.0.0.0          0.0.0.0          192.168.10.1     UG       1   AUTO     13159      138800000002

192.168.10.0     255.255.255.248  0.0.0.0          UCI      1   MANUAL   13180      138800000002

192.168.100.0    255.255.255.0    0.0.0.0          UCI      1   MANUAL   13180      13880000000b

192.168.200.0    255.255.255.0    0.0.0.0          UCI      1   MANUAL   13180      13880000000a

[root@localhost:~]

[root@localhost:~] net-vdr -l --route default+edge-1

VDR default+edge-1 Route Table

Legend: [U: Up], [G: Gateway], [C: Connected], [I: Interface]

Legend: [H: Host], [F: Soft Flush] [!: Reject] [E: ECMP]

Destination      GenMask          Gateway          Flags    Ref Origin   UpTime     Interface

-----------      -------          -------          -----    --- ------   ------     ---------

0.0.0.0          0.0.0.0          192.168.10.1     UG       1   AUTO     3602       138800000002

192.168.10.0     255.255.255.248  0.0.0.0          UCI      1   MANUAL   3623       138800000002

192.168.100.0    255.255.255.0    0.0.0.0          UCI      1   MANUAL   3623       13880000000b

192.168.200.0    255.255.255.0    0.0.0.0          UCI      1   MANUAL   3623       13880000000a

The routing tables for each of the ESXi hosts. I'm trying to route a vm between 192.168.100.0/24 and 192.168.200.0/24 located on different ESXi hosts. It works fine when they are both on same ESXi host.

Thanks

[iforbes]

Anybody? I thought it was a VTEP issue but as mentioned when I ping between vm's on the same logical network (L2) but different ESXi servers it's successful. It's only unsuccessful when pinging between vm's on different networks (L3) and on different ESXi hosts. Routing between vm's on different networks but on the same ESXi host is successful. A vm can ping it's own default gateway AND the target vm's default gateway whether the vm's are both on the same ESXi server or different ESXi servers.

Thanks

[Sreec]

Routing table looks fine . Do you have any firewall rules configured in DLR control VM ? How is the connectivity from DLR to Uplink ?  Can you also perform a traceroute from one of the VM when they are on different host ?

[bayupw]

Could you try to use the traceflow and screenshot the result?

Here's an example

[iforbes]

No firewall rules created. When you say connectivity between DLR and uplink are you referring to the LIFs (i.e. default gateways for the vm's)? I can log into the DLR console and successfully ping those LIFs. I haven't configured the ESG yet (no real reason at this point), so although I have a LIF called transit-tier defined on the DLR with an IP (192.168.10.2) which is the uplink LIF to the ESG, I haven't yet configured the ESG. I gave the DLR a default gateway IP 192.168.10.1.

I've attached pics of the DLR LIF interfaces, and a tracert from one of the vm's (192.168.100.100) to the other vm (192.168.200.100). The odd part is I can successfully ping the gateway for the the other vm (192.168.200.1) but cannot ping the vm. Once again, this routing failure is only when the vm's are on different ESXi hosts. Works fine when they are on same ESXi host.

[iforbes]

Interesting tool! Looks like the traffic makes it to the DLR and just stops there. No delivered or dropped observations.

[bayupw]

Yeah it's wired that it just stops there.

I just did a test using similar scenario with DLR but in my case no uplink only 2 LIFs it works fine.

Not sure if this related but could you try to double check all time & NTP settings across these components: NSX Manager, vCenter, ESXi. Make sure their time are in sync.

Check DNS forward and reverse lookup those components too make sure everything is resolvable via DNS.

I had funny issues when time/NTP are not sync or DNS forwards/reverse is not configured properly.

If this is a new setup, try to re-deploy DLR

[iforbes]

NTP setting are consistent. Question. I noticed that the logical networks which create network port groups on the vDS, exist on the same physical uplinks and VLAN as my non-VXLAN vm's (i.e. vCenter, DNS, etc). Should that be the case? When an NSX distributed logical switch is created, how does vCenter/NSX Manager know what physical uplink(s) to place it (and thus the corresponding VXLAN vm's)?

I'm curious because the vdr-vdrPort (seen via esxtop) points to a physical uplink that is part of what's been defined for the VTEP vmk. I suppose my question is why is that vdr-vdrPort on a VTEP backed physical uplink and the VXLAN vm's on VLAN backed uplinks (i.e. vcenter, dns, etc)?

In the attached pic you can see the Web-Tier logical network is on the physical uplink that contains VLAN 60 (which is the Servers port group containing other vm's like vCenter, DNS, etc)

[bayupw]

VXLAN will use all uplink on the selected VDS.

How many vmnics do you have?
For example, if you have 4 vmnics: vmnic0, vmnic1, vmnic2, vmnic3

vmnic0 & vmnic2 for non-VXLAN

vmnic1 & vmnic3 for VXLAN

I would suggest you to create separate two VDS, one for non-VXLAN and and one for VXLAN

If you want to use all vmnics, you will need to present the VLAN for VXLAN on all vmnics as we don't override VXLAN dvPortGroup uplink configurations

Please note if you are planning to use NSX L2 Bridging, the VLAN that will be bridged need to be on the VDS for VXLAN (vmnic1 & vmnic3 in this example)

[iforbes]

Hi Bayu,

So, when you say VXLAN network I'm assuming you mean the network I've used for configuring the VTEP vmk? Is that where VXLAN networks should ultimately exist? Are there any rules with respect to multiple vDS switches and NSX, or should this be as simple as me creating a second vDS for non VXLAN and just moving my non-VXLAN vm's over there?

Could all of this be why I'm getting the routing issues?

Thanks!

[bayupw · Accepted solution]

Hi, yes VXLAN network I mean VTEP vmk.

So taking the previous example, let say you have a VDS using all 4 vmnics 0-3 but VXLAN/VTEP VLAN is only on vmnic1 & vmnic3

When configuring VXLAN, you select VDS, for single VTEP you will have one vmkernel using all vmnics, for multi-VTEP, you will have four vmkernels.

Let say you override the config and change the vmkernel dvPortGroup to only use vmnic1 & vmnic3 and set vmnic0 & vmnic2 to unused.

Now when you create a new VXLAN logical switch, a dvPortGroup for that will be created in the VDS and again will use all vmnics on that VDS for the uplinks.

To overcome this issue, migrate the non-VXLAN to new VDS, and edit and reduce the VDS uplinks in the VXLAN/VTEP VDS to 2 uplinks.

Yes this could be why you are getting the routing issues, the VXLAN for that particular traffic could be using the non-VXLAN uplinks which don't have VXLAN/VTEP VLAN

[iforbes]

Hi Bayu,

You solved it! I have 2 clusters (mgmt, compute). I had created a single vDS that included all hosts from both clusters. I was running into issues with distributed logical routing between hosts because vxlan networks were being placed on physical uplinks that had non-vxlan vlans.

So, I created 2 vDS as you suggested. The second vDS just has 2 uplinks and strictly dedicated to vxlan (i.e. backed by physical vlan for vxlan). When I prepared each cluster for vxlan, i pointed to this second vDS to use. So, now all vxlan vm's connect to the correct vDS and uplinks, and the vdr-vdrPort also connects to the same vDS and uplinks.

I tested and now I can successfully ping from one vxlan subnet to a different vxlan subnet across esxi hosts.

Thanks for your help. Much appreciated!