NSX-T 3.1 North-South routing

[emistery]

Hi all,

I'm pretty new to NSX-T, and trying to set up a lab to gain some knowledge about it. I already set up some segments with VMs connected to it, a T1 router and a T0 router. The VMs can reach the uplink interface of the T0 router, but no further (the physical router). What are some troubleshooting steps I could try to do? All suggestions or tips are welcome.

[CyberNils]

Have you added a static route on your T0 to reach the physical router?

[p0wertje]

And also routes from you physical router to the t0 ?

[shank89]

Best bet is, ssh onto the edge, change to to t0 sr vrf.

To find which one it is, type get logical-routers.

Find the t0 sr, type in vrf x (x is the number of the t0sr).

Type in get route and see if you have a route pointing out, either default or specific prefixes.

On the ToR or whatever it is, as was mentioned, ensure you have a route to the prefixes in NSX-T.

After you confirm this, then we can assist further. 

Are you Intending to use a routing protocol or static routes going forward?

[emistery]

Yes. From T0 to the physical router and vice versa.

[emistery]

Yes I have static routes on both routers. Also I can't ping from both routers to each other.

[shank89]

From the T0 SR can you do a source ping to the next hop?

Ping x.x.x.x source uplinkip

[emistery]

Just did this, and I got a Destination Host Unreachable

[p0wertje]

I assume you use virtual edge nodes ?
Is the network where the physical router is connected available on the physical switch where the esxi host is connected ?
Did you check the config of the edge profiles to use the correct vlan id for that ?

[shank89 · Accepted solution]

It sounds like the edges are wired incorrectly.

Edge vms vnic in vcenter should be trunking portgroups. The first vnic is your management portgroup, this doesn't have to be a trunk.

Within nsx create some vlan backed segments, tag them with your uplink vlans, then on the t0 uplink interfaces use these segments as the connected to segments when assigning the uplink interface ip  addresses to the edges.

Retest the ping after this.

[emistery]

Yes, the ESXi hosts can connect to the physical router. I'm running a nested esxi cluster with a virtual pfsense appliance as the "physical" router

[shank89]

If its nested,  make sure the security settings on the trunking pg are all set to accept.

Maybe also provide a diagram on how you have wired it all up?

[emistery]

I could try to make a diagram, but it's all gotten a bit complicated. I'm pretty new to ESXi and just started with this project. The image I added is the wiring of the vswitch on the physical esxi host.

[shank89]

Ok, have you configured VLANs on the pfsense downlink interface?

Is routing to the pfsense working correctly otherwise?

Have you confirmed connectivity in the nested environment?

[emistery]

Yes. The other vlans on the pfsense work perfect. The for example, from my machine I can ping all the interfaces in the vlans on the pfsense router.

I'm pretty clueless on what I could do now.

Also, before I had the uplink interface of the T0 router on the same subnet as the TEP interfaces of the hosts and edge nodes, but I just changed that to seperate VLANs on the trunking interface.

[shank89]

If you spin up a VM within the nested host and attach it to a portgroup with a VLAN tag, can you hit the gateway on the pfsense from it? 

You can use the same subnet as the uplink pg if you wish.

[emistery]

Yes, it can reach the gateway from that portgroup. But I think I found the problem. The edge node is connected to a different portgroup on a different dswitch in order to make the TEP tunnels work. The pfsense router is not connected to that dswitch. What is the best way to make that work?

[shank89]

You need to define all the subinterfaces on the PFsense on that single trunked pg.

- 2 uplinks, edge tep, host tep .. these should be the networks you define / have sub interfaces for.

Then in NSX-T, when you prepare the hosts, in the uplink profiles for the host, define the VLAN and the same for the Edges uplink profiles.  This will set their TEP VLAN ID.

Then go to networking > segments > create VLAN backed segments, tagged as required for your uplink interfaces.

To > Interfaces > create your interfaces, assigned appropriate IP addresses, ensure the IP / mask you specify is being conected to the correct segment for that subnet.

As long as your Edge VM is wired correctly ( VM, 3 vnics, trunked PGs), and NSX-T is tagging the VLANs correctly, then the packets will egress the nested host and hit the PFsense if everything is wired correctly there aswell.

When you get the basics working, you can have a look at this https://www.lab2prod.com.au/2020/11/nsx-t-inter-tep.html for details on inter tep traffic (which is what it sounds like you were trying to achieve / single VLAN for edge and host TEPs using the same switch)

[emistery]

I can't choose my custom uplink profiles for the hosts and edge nodes.

[shank89]

You'll need to create new profiles