Nested NSX-T v3.1.2 (Tunnel Down)

[MrJazze]

For the past month, I have attempted multiple configurations of a Nested NSX-T environment where all have failed at same point; once I boot a VM connected to a overlay segment. The "Edge Node" tunnel goes down and then the "ESXi Host" also shows down. I've gotten this same result if I manually install ESXi hosts, using William Lam's Nested ESXi images, using single/multiple vmnics on edge/transport nodes, enable MAC Learning on overlay segment/nested vmnics, enabled promiscuous mode and forged transmits, tested tunnel passing traffic using vmkping both before and after VM boot; each were successful...and more.

In every case, as I've stated once the VM boots and attempts to communicate to vNic the tunnel drops. I've watched Youtube videos of others that were successful in nested deployments, but older versions of ESXi where used.

If you were successful in deploying Nested NSX-T with these versions (possible bug), I'd like to hear from you.
NSX Appliance 3.1.2
VCenter 7.0.2
ESXi 7.0.2

[Sreec]

Looks like you have MTU, TEP or VLAN reachability issues. Are we using the same TEP subnet for Host and Edges? 

1. vmkping -I  vmkernel -S vxlan destination IP -d -s 1572 

2. vmkping -I  vmkernel -S vxlan destination IP -d -s 1400 

Can you test the above commands from ESXI ( destination must be Edge TEP IP) 

[MrJazze]

@Sreec

Thanks for replying. Yes, I was able to ping edge TEP from transport nodes using mtu 8000 (everything from physical to virtual set at mtu 9000). Edge and hosts are on same VLAN, edge profile mtu set 9000.

I'm really at a total loss why it's not working. It was easier building my nested Tanzu lab. I've already decided to tear it all down and try again (which will be like the 7th time). I believe there is something different v3.1.2 specifically hindering nested deployments. I could be wrong, but I have tried several different setting. The few youtube videos I've found deployed NSX-T nested all used previous versions. I've even went through a video series in another language.

Any and all knowledge you can share is very much appreciated.

[CyberNils]

Try to change your edge profile mtu to 8940. That is what I use in my nested labs. 9000 MTU on physical network.

[MrJazze]

@Sreec / @CyberNils 

I again blew-away my lab and redeploy. Except this time I decided to take another look a Jeffrey Kusters's videos. His approach to deploying NSX-T in a nested environment is almost opposite of Mike's (NRDY Tech) approach; and yet both worked for them. Admittedly, Jeffrey's audio difficulties and handwritting was a huge distraction to my grasping the concepts. Were as Mike used Powerpoints and provided more background reasoning as to why a choice was made. Likewise, Mike deployed NSX 3.0, whereas Jeffrey used 3.1.

As for my lab, I now have a semi-working nested NSX-T 3.1.2 lab. I can attach a VMs (Linux/Windows) to a segment, boot it and not have the TEP interfaces go down (MTU 9000 everywhere). VMs can ping east/west and north/south, can resolve internal/external DNS. I can ping VMs from LAN, but VMs cannot browse the internet. I've checked firewall (pfsense) logs which confirmed VM IP address is reaching internal/external addresses and protocols. I also notice if I vmkping a Edge node from parent ESXi host it gets duplicate responses, but not from remote hosts. Yes, this is likely the reason we set the port group security settings to accept, but no one bothers to explain this fact.

I truly appreciate you guys assistance and feedback as I try to wrangle this doggy (lil Texas humor).

[Sreec]

Most likely you have a routing issue.

1. Can the pfsense router reach the internet?  

2. Do you have any default route pointing to the internet? 

3. Are we doing NAT anywhere?  

[MrJazze]

@Sreec 

Your questions 1 & 2 are a given as pfsense is my physical network router/firewall, but the 3rd question got me thinking. So I logged into pfsense and navigated to Firewall > NAT > Outbound. There it was set to "Automatic outbound NAT rule generation" where it was automatically NATing all my network VLANs to the WAN interface. PFsense is aware of the segments due to BGP, but not for NATing. 

You pegged it! Turns out I needed to explicitly create mappings to NAT all my NSX-T segment traffic when it is destined to the internet. I changed mode to "Hybrid Outbound NAT rule generation", then added entries for each segment network. (see attached screenshot)

I don't want to mark this post as solved until I figure out a way to write-up something about how I configured the network nesting for NSX-T. (coming soon)

[MrJazze]

Side Note:

Are there plans to correct how the Tier0 Uplink Segment is displayed in the Network Topology diagram? (see attachment)

[Sreec]

I'm unsure about that. However, you can select T0 edge in the same UI,  which will pop up the edge configurations, select Interfaces, and it will show all the interfaces.  If you have VRNI, you will certainly get full path visibility. 

[Blaise2]

Nested NSX-T v3.1.2 (Tunnel Down) For the past month, I have attempted multiple configurations of a Nested NSX-T environment where all have failed at same point; once I boot a VM connected to a overlay segment. The "Edge Node" tunnel goes down and then the "ESXi Host" also shows down. I've gotten this same result if I manually install ESXi hosts, using William Lam's Nested ESXi images, using single/multiple vmnics on edge/transport nodes, enable MAC Learning on overlay segment/nested vmnics, enabled promiscuous mode and forged transmits, tested tunnel passing traffic using vmkping both before and after VM boot; each were successful...and more.

My Balance Now

[stillmags]

I have this exact issue!! I was even on the phone with a vmware rep who had no idea! I was thinking to just install 3.0 and upgrade it to 3.1.2 But did your fix work for this issue about the tunnels being down and host? If so, can you tell me what exactly the fix was you performed? Would be a huge help to my brain right now haha

[MrJazze]

@stillmags / @Blaise2 ,

Apologies' for delayed response.

Yes, I was able to resolve tunnel down issue in nested environment. I've since experienced data corruption in cluster and had to rebuild. But chose to deploy NSX-T to physical environment instead of nested. But, what I can tell you from what I've learned with both deployments and Jeffery Kusters's videos is to use two VDS switches.

Put your edge nodes on VDS switch along with host vmotion, storage, vSAN; create two portgroups, one for TEP VLAN (assign same vlan as host tep) and another for VLAN Uplink (vlan trunk). Then create a second VDS switch for transport nodes (hosts) TEP traffic (vlan trunk). This does mean you need dedicated interfaces (physical or virtual) for each VDS. When creating the host profile designate the same TEP VLAN as edge nodes. (FYI: the host TEPs are not associated with portgroup, but rather the actual network interfaces (aka Uplink1, Uplink2, etc). For the edge node interfaces I created two N-VDS switches, one for overlay transport zone and second for vlan transport zone.

I think that's about it. If you need more info I'll try to respond sooner.

[stillmags]

Thanks for the reply. What I don't understand is vmware docs mention that the vlan actually does NOT matter in version 3.0 and higher. But it seems it still does matter. we were able to get it working when we put the host teps and edge teps in different vlans AND the IPs had to be routable with a gateway on the physical devices which makes NO sense to me because I thought the teps did not car about IPs. Thats what the 16 billion VNI's were for. This is definitely making my head hurt 🙂

[MrJazze]

@stillmags 

LOL, believe me I have felt the pain. I have my issues with most technology companies' product documentation. To put it simple, they're now written to cause a dependency to contact/pay for support. Marketing and sales has too much control in businesses today. I've been working with tech for 20+ years thus don't say this without background.

Mike @ youtube channel "NRDY Tech" finally explains the significance of host vs edge TEP VLANs. I decided not to use different VLANs because I only have layer 2 switches and didn't want to traffic routing through router and have to implement jumbo frames. If you do, then different VLANs is the way to go.

On a positive note, I've found NSX-T a very cool and powerful addon to vSphere clusters. I hope to achieve some level of automation in the near future. First, I have to get NSX-T load balancing and firewall figured out.

[JamesCurtis]

I also got the problem of tunnel closure. If I delete the switch of esxi and replace it with N-VDS switch, it can work normally. Switching back to the VDS switch again will fail. 

NSX Appliance 3.2.2.0.0.20737185
VCenter 7.0.3.01100
VMware ESXi, 7.0.3, 20328353

[JaSo2]

In the nested environment starting from this week, I'll be always using the logical segment for Edge TEP. First of all, there is a great KB, which should be mentioned:

https://kb.vmware.com/s/article/83743

From my customer-deployments experience , if I wanted to be 100% that TEPs will be connected I would use 2 VLANs (separate subnets routed over the underlay) - I had 0 problems with that. In nested environment this configuration did not work  and I faced very strange problems on L7 - some HTTP worked perfectly, some did not. Packet tracer showed that for the HTTP that did not work, during the handshake ACK got lost (but just for SOME websites...).

I had Edge connected to PG on VDS that was not controlled by NSX, hosts (TEPs) were on VDS controlled by NSX. After I put Edges + Hosts TEPs to one VLAN and used logical segment (supported since 3.1) for Edge TEP and not the portgroup everything started working... I guess there is some problem with GENEVE encapsulation / de-encapsulation in the nested environment (since I have vSwitch of a physical host under it and not a physical NIC).