emistery
Enthusiast
Enthusiast

NSX-T 3.1 North-South routing

Jump to solution

Hi all,

I'm pretty new to NSX-T, and trying to set up a lab to gain some knowledge about it. I already set up some segments with VMs connected to it, a T1 router and a T0 router. The VMs can reach the uplink interface of the T0 router, but no further (the physical router). What are some troubleshooting steps I could try to do? All suggestions or tips are welcome.

 

0 Kudos
1 Solution

Accepted Solutions
shank89
Expert
Expert

It sounds like the edges are wired incorrectly.

Edge vms vnic in vcenter should be trunking portgroups. The first vnic is your management portgroup, this doesn't have to be a trunk.

Within nsx create some vlan backed segments, tag them with your uplink vlans, then on the t0 uplink interfaces use these segments as the connected to segments when assigning the uplink interface ip  addresses to the edges.

 

Retest the ping after this.

Shashank Mohan

VCAP-NV 2020 | VCP-DCV2019 | CCNP Specialist

https://lab2prod.com.au

View solution in original post

0 Kudos
67 Replies
CyberNils
Enthusiast
Enthusiast

Have you added a static route on your T0 to reach the physical router?



Nils Kristiansen
https://cybernils.net/
0 Kudos
p0wertje
Hot Shot
Hot Shot

And also routes from you physical router to the t0 ?

Cheers,
p0wertje | VCIX6-NV | JNCIS-ENT
Please kudo helpful posts and mark the thread as solved if solved
0 Kudos
shank89
Expert
Expert

Best bet is, ssh onto the edge, change to to t0 sr vrf.

To find which one it is, type get logical-routers.

Find the t0 sr, type in vrf x (x is the number of the t0sr).

Type in get route and see if you have a route pointing out, either default or specific prefixes.

On the ToR or whatever it is, as was mentioned, ensure you have a route to the prefixes in NSX-T.

After you confirm this, then we can assist further. 

Are you Intending to use a routing protocol or static routes going forward?

Shashank Mohan

VCAP-NV 2020 | VCP-DCV2019 | CCNP Specialist

https://lab2prod.com.au
0 Kudos
emistery
Enthusiast
Enthusiast

Yes. From T0 to the physical router and vice versa.

0 Kudos
emistery
Enthusiast
Enthusiast

Yes I have static routes on both routers. Also I can't ping from both routers to each other.

0 Kudos
shank89
Expert
Expert

From the T0 SR can you do a source ping to the next hop?

Ping x.x.x.x source uplinkip

Shashank Mohan

VCAP-NV 2020 | VCP-DCV2019 | CCNP Specialist

https://lab2prod.com.au
0 Kudos
emistery
Enthusiast
Enthusiast

Just did this, and I got a Destination Host Unreachable

0 Kudos
p0wertje
Hot Shot
Hot Shot

I assume you use virtual edge nodes ?
Is the network where the physical router is connected available on the physical switch where the esxi host is connected ?
Did you check the config of the edge profiles to use the correct vlan id for that ?

Cheers,
p0wertje | VCIX6-NV | JNCIS-ENT
Please kudo helpful posts and mark the thread as solved if solved
0 Kudos
shank89
Expert
Expert

It sounds like the edges are wired incorrectly.

Edge vms vnic in vcenter should be trunking portgroups. The first vnic is your management portgroup, this doesn't have to be a trunk.

Within nsx create some vlan backed segments, tag them with your uplink vlans, then on the t0 uplink interfaces use these segments as the connected to segments when assigning the uplink interface ip  addresses to the edges.

 

Retest the ping after this.

Shashank Mohan

VCAP-NV 2020 | VCP-DCV2019 | CCNP Specialist

https://lab2prod.com.au

View solution in original post

0 Kudos
emistery
Enthusiast
Enthusiast

Yes, the ESXi hosts can connect to the physical router. I'm running a nested esxi cluster with a virtual pfsense appliance as the "physical" router

0 Kudos
shank89
Expert
Expert

If its nested,  make sure the security settings on the trunking pg are all set to accept.

Maybe also provide a diagram on how you have wired it all up?

Shashank Mohan

VCAP-NV 2020 | VCP-DCV2019 | CCNP Specialist

https://lab2prod.com.au
0 Kudos
emistery
Enthusiast
Enthusiast

I could try to make a diagram, but it's all gotten a bit complicated. I'm pretty new to ESXi and just started with this project. The image I added is the wiring of the vswitch on the physical esxi host.

0 Kudos
shank89
Expert
Expert

Ok, have you configured VLANs on the pfsense downlink interface?

Is routing to the pfsense working correctly otherwise?

Have you confirmed connectivity in the nested environment?

Shashank Mohan

VCAP-NV 2020 | VCP-DCV2019 | CCNP Specialist

https://lab2prod.com.au
0 Kudos
emistery
Enthusiast
Enthusiast

Yes. The other vlans on the pfsense work perfect. The for example, from my machine I can ping all the interfaces in the vlans on the pfsense router.

I'm pretty clueless on what I could do now.

Also, before I had the uplink interface of the T0 router on the same subnet as the TEP interfaces of the hosts and edge nodes, but I just changed that to seperate VLANs on the trunking interface.

0 Kudos
shank89
Expert
Expert

If you spin up a VM within the nested host and attach it to a portgroup with a VLAN tag, can you hit the gateway on the pfsense from it? 

 

You can use the same subnet as the uplink pg if you wish.

Shashank Mohan

VCAP-NV 2020 | VCP-DCV2019 | CCNP Specialist

https://lab2prod.com.au
0 Kudos
emistery
Enthusiast
Enthusiast

Yes, it can reach the gateway from that portgroup. But I think I found the problem. The edge node is connected to a different portgroup on a different dswitch in order to make the TEP tunnels work. The pfsense router is not connected to that dswitch. What is the best way to make that work?

0 Kudos
shank89
Expert
Expert

You need to define all the subinterfaces on the PFsense on that single trunked pg.

- 2 uplinks, edge tep, host tep .. these should be the networks you define / have sub interfaces for.

Then in NSX-T, when you prepare the hosts, in the uplink profiles for the host, define the VLAN and the same for the Edges uplink profiles.  This will set their TEP VLAN ID.

Then go to networking > segments > create VLAN backed segments, tagged as required for your uplink interfaces.

To > Interfaces > create your interfaces, assigned appropriate IP addresses, ensure the IP / mask you specify is being conected to the correct segment for that subnet.

As long as your Edge VM is wired correctly ( VM, 3 vnics, trunked PGs), and NSX-T is tagging the VLANs correctly, then the packets will egress the nested host and hit the PFsense if everything is wired correctly there aswell.

 

When you get the basics working, you can have a look at this https://www.lab2prod.com.au/2020/11/nsx-t-inter-tep.html for details on inter tep traffic (which is what it sounds like you were trying to achieve / single VLAN for edge and host TEPs using the same switch)

Shashank Mohan

VCAP-NV 2020 | VCP-DCV2019 | CCNP Specialist

https://lab2prod.com.au
0 Kudos
emistery
Enthusiast
Enthusiast

I can't choose my custom uplink profiles for the hosts and edge nodes.

0 Kudos
shank89
Expert
Expert

You'll need to create new profiles

Shashank Mohan

VCAP-NV 2020 | VCP-DCV2019 | CCNP Specialist

https://lab2prod.com.au
0 Kudos