TEP interfaces on Nested Virtual ESXi to physical network

[ngkin2010]

Hi all,

I am starting to learn NSX-T and currently building a hand-on lab with a limited hardware resource.

Everything work fine, except when I was trying to route traffic between T1-gateway and T0-gateway (Physical Network).

I have deployed NSX-Edge Transport Node, assigned both vlan transport zone & overlay transport zone.

I have configured NSX-Host Transport Node, assigned the overlay transport zone.

Both nodes are assign TEP IP using the IP pool 192.168.30.210-192.168.30.211.

However, the TEP on NSX-Edge Transport Node cannot form GENEVE tunnel with NSX-Host Transport Node's TEP.

I am able to ping 192.168.30.211 from physical network. But I am not able to ping 192.168.30.210.

Look like the VMK10 (TEP) interface on NSX-Host Transport Node (Nested ESXi VM) failed to reach physical network.

Anyone have idea about how to troubleshoot?

===

My Setup is simple, everything are assigned on the same flat subnet (VM Network 0).

Thanks,

Ngkin

[shank89]

Hi Ngkin,

The Host TEP network and Edge TEP network must be on a different VLAN / subnet and route to each other or else the traffic will just drop and the tunnel will never form, if the edges are sitting on a esxi host prepared for nsx-t (nsx-t vibs installed on it).

From what I can see your host TEPs and Edge TEPs are on the same network, although cannot exactly see if the edge vm's are sitting on a host prepped for NSX-T.

If this is the case then this should resolve your issue.

[ngkin2010]

Dear Shank,

Thanks for your kindly information!

if the edges are sitting on a esxi host prepared for nsx-t (nsx-t vibs installed on it).

Yes, the edges are sitting on a esxi host prepared for nsx-t. But I didn't notice from documents about the restriction you mentioned.

I was referring to the "VMware NSX-T - Getting Started Hands-on Lab", and I see that VMware assign the same IP pool for Host's TEP and Node's TEP.

Host TEP network and Edge TEP network must be on a different VLAN / subnet and route to each other

I would try to assign different IP pools for Host TEP and Edge TEP respectively. Do I necessary to route the traffic through physical network? Or can I assign the TEPs to the segments created under T1 Router, and routed within NSX?

Thanks again.

Ngkin

[shank89 · Accepted solution]

So this will come back down to your configuration, a few questions;

• How many nics on the nested hosts?
• Will you be running a single vDS (NSXT3) or a vDS and n-VDS combo (NSXT2.5)?
• Will you be running a single N-VDS (2.5 and below)?

Basically if you have 4 pnics, you can have the edge virtual appliance attached to the vDS on trunked portgroups there, and then you may use the same VLAN and TEP range as the host transport nodes.   This will be due to the fact that the packets still route through the vDS uplinks and back into the host transport nodes on the n-vds/vds in NSX-T).

If you are running nested hosts with 2 pnics, in which case would mean you are ingesting both of them into either the N-VDS or VDS, in this case you would need a different TEP VLAN for the Edges and then another for the Host Transport Nodes.  If a packet is destined for one of or the other and sees multiple TEPs the packet is dropped.

Hopefully this clears things up for you, you may find some use with this link; NSX-T Edge: Flexible deployment options for NSX-T Data Center Edge VM.

Pay close attention to the images and how the VLAN assignments change when the scenario's change.

PS.

Overlay segments will only be able to route traffic to NSX-T endpoints when the geneve tunnels are up and running, so if you mean can you use a segment in NSX-T to route TEP traffic between all endpoints, the answer is no as they will not all be able to communicate.

[ngkin2010]

Hi Shank,

Thanks for your detailed explanation!! Now I realize that I can't simply put Host's TEP and Edge's TEP into same subnet if I am deploying collapsed computer / edge cluster esxi host.

I'll take some time to reconfigure my lab based on your information.

I will mark your reply as correct solution. Thanks again.

Thanks,

Ngkin

[shank89]

Glad to help