VMware Networking Community
Rob80
Contributor
Contributor
Jump to solution

Static route issues on NSX-T

Hi all,

It seems that I can't find the way correctly connect Tier-0 to external network via static route on NSX-T 2.4. I have attached below my provisional network setup and I can't understand where the issue occur. Any help would be appreciated.

Thanks

Reply
0 Kudos
1 Solution

Accepted Solutions
daphnissov
Immortal
Immortal
Jump to solution

Your diagram shows you have two separate uplinks from your edges on VLAN overlays to two different networks. Not sure why you're doing this as it doesn't make too much sense. The proper way to do this is configure those uplinks on your T0 to be in the same subnet. One port per edge. Configure a HA VIP across those two T0 uplinks. Your static route on your Meraki has its next hop set to that HA VIP. However, in order to route outbound traffic from NSX-T land, you will need to configure a default route (0.0.0.0/0) on your T0 directed to the next hop upstream. That is usually on the same subnet as the T0 uplink in the VLAN overlay.

View solution in original post

Reply
0 Kudos
23 Replies
daphnissov
Immortal
Immortal
Jump to solution

Your diagram shows you have two separate uplinks from your edges on VLAN overlays to two different networks. Not sure why you're doing this as it doesn't make too much sense. The proper way to do this is configure those uplinks on your T0 to be in the same subnet. One port per edge. Configure a HA VIP across those two T0 uplinks. Your static route on your Meraki has its next hop set to that HA VIP. However, in order to route outbound traffic from NSX-T land, you will need to configure a default route (0.0.0.0/0) on your T0 directed to the next hop upstream. That is usually on the same subnet as the T0 uplink in the VLAN overlay.

Reply
0 Kudos
Rob80
Contributor
Contributor
Jump to solution

Daphnissov,

Currently I have setup each uplink on the same subnet and connected to one port per edge as described above but would appreciate guidance regarding HA VIP configuration.

Thanks

Reply
0 Kudos
daphnissov
Immortal
Immortal
Jump to solution

This should be covered in the official docs. You configure the HA VIP on your T0. After creating uplinks for each edge, you create the VIP but do not assign it to any interface. Leave the field blank and it will auto-assume both available uplinks. Goes without saying but the HA VIP needs to be in the same subnet as the individual uplink ports.

Reply
0 Kudos
Rob80
Contributor
Contributor
Jump to solution

I did search online and number of times on NSX-t 2.4 and can't find HA VIP configuration only router ports.

pastedImage_0.png

Reply
0 Kudos
daphnissov
Immortal
Immortal
Jump to solution

Configuration => HA VIP

pastedImage_0.png

pastedImage_1.png

Reply
0 Kudos
IPv4toIPv6
VMware Employee
VMware Employee
Jump to solution

Hello Rob80,

Base by you PDF you have 2 uplinks in 2 different vlans.

pastedImage_0.png

Did you check if you have L2 connectivity

VLAN 2011 from MX65 Meraki --- Edge Uplink1

Example:

MX65 Meraki  IP 192.20.11.1

Edge Uplink1 IP 192.20.11.2

There suppose to be connectivity.

Ping form 192.20.11.1 to 192.20.11.2 suppose to be ok

VLAN 2012 from MX65 Meraki --- Edge Uplink2

MX65 Meraki  IP 192.20.12.1

Edge Uplink1 IP 192.20.12.2

There suppose to be connectivity.

Ping form 192.20.12.1 to 192.20.12.2 suppose to be ok

Please before going to the Routing configuration check the steps from above to be sure you have L2 connectivity :smileygrin:

Then go on edge routing and add destination 0.0.0.0/0  GW 192.20.11.1,192.20.12.1

And if you have a default route just delete'it.

My recommendation is still to use BGP

Reply
0 Kudos
Rob80
Contributor
Contributor
Jump to solution

Hi,

HA VIP has now been configured with IP of 192.20.11.5 which I can ping from VM's.

I added the following addresses to nsx-t  static route

pastedImage_0.pngpastedImage_1.png

And another IP on MX

pastedImage_2.png

But still no connection to outside world.

Reply
0 Kudos
Rob80
Contributor
Contributor
Jump to solution

Hi,

In order to use bgp I need router capable in supporting bgp as one of the prerequisites is AS which seems can be only setup when mx in vpn concentrator mode, but then I have no vlans or network.

Thanks

Reply
0 Kudos
daphnissov
Immortal
Immortal
Jump to solution

Ok, I need to be able to see these screenshots you're posting as they're extremely small. Please also show your edge profiles and what interfaces are connected where. Also show your transport zone (overlay) profile and how your hosts are connected.

Reply
0 Kudos
Rob80
Contributor
Contributor
Jump to solution

Please find attached few screen shots of existing configuration.

Reply
0 Kudos
daphnissov
Immortal
Immortal
Jump to solution

Ok, this gives me a better idea. Here are questions for you to check out based on your images.

  1. Your overlay profile is specifying VLAN 1020. This means the actual vmnics you're defining on your ESXi hosts are connected to upstream ports in trunk mode with 1020 allowed. Are you certain this is the case? If yes => Ping something else on this VLAN. Remember to pass the -S flag to vmkping to use the VXLAN TCP/IP stack. If no => set to proper VLAN. Use VLAN 0 if connected to an access port, or if this transport node (ESXi host) is virtual and connected to a virtual switch (in which case the VLAN tag is stripped off).
  2. Your edge uplink profile profile is specifying an MTU of 1600. Normally this is left at 1500. Are you certain that you have MTU 1600 available on the VLANs in use by your edge uplinks to the physical infra? If yes => You're ok, but verify. If no => Change to 1500. To eliminate this variable, I'd change to 1500 regardless for testing.
  3. Your T0 uplinks on the edges are in 192.20.11.0/24 with addresses of .2 and .3 for the two edges. What's the HA VIP address? What is the gateway address on this segment?
  4. You are mixing configurations in the new policy-based API of 2.4 and the older format of 2.3. Anything you have under Networking => Tier-0 Gateways please remove. All configuration should be under Advanced Networking & Security.
  5. From Advanced Networking & Security => Routers => T0, show Routing => Static Routes
  6. Show from the same menu Route Redistribution
  7. Show your edge cluster
  8. Show your T1 => Configuration => Router Ports
  9. Show your T1 => Routing => Route Advertisement
Reply
0 Kudos
Rob80
Contributor
Contributor
Jump to solution

I have attached file with screens to some of the lines below.

In terms of Tier-0, do I need to remove all and recreate on Advanced networking? Shall I do the same with Tier-1 or keep as it is.

Gateway for HA VIP is subnet on meraki router 192.20.11.1

Reply
0 Kudos
daphnissov
Immortal
Immortal
Jump to solution

Your vmkping command in your screenshot is not what I wrote. Flags are case sensitive. From the ESXi host, do it again and check the result:

vmkping -S vxlan <TEP> -d -s 1572 -c 10

From your ESXi host, you should be able to do a vmkping from the TEP to the:

  • TEP of your edge transport nodes (both of them) on the same subnet

From a VM that is attached to a logical switch, you should be able to ping (do not alter MTU):

  • Gateway (T1 downlink for this segment)
  • T1 linked port to T0
  • T0 uplink HA VIP address
  • Gateway on the HA VIP network (which you say should be 192.20.11.1)

After you've corrected all these things, please indicate which of these ping tests pass and which fail.

Regarding your static route, it's wrong. The T0 static route needs to be the gateway for that segment which is the SVI on the Meraki. So it needs to be 192.20.11.1.

Your Meraki must have a static route setting the next hop for any networks you do not wish to NAT as the HA VIP address.

In terms of Tier-0, do I need to remove all and recreate on Advanced networking? Shall I do the same with Tier-1 or keep as it is.

Yes, to eliminate complexity, remove anything you have for routing and switching that's not under Advanced Networking & Security.

Reply
0 Kudos
Rob80
Contributor
Contributor
Jump to solution

NSX-T to edge node ping didn't succeed:

-- 192.20.11.5 ping statistics ---

10 packets transmitted, 0 packets received, 100% packet loss

[root@ESXi:~] vmkping -S vxlan 192.20.11.5 -d -s 1572 -c 10

PING 192.20.11.5 (192.20.11.5): 1572 data bytes

--- 192.20.11.5 ping statistics ---

10 packets transmitted, 0 packets received, 100% packet loss

[root@ESXi:~] vmkping -S vxlan 192.20.11.2 -d -s 1572 -c 10

PING 192.20.11.2 (192.20.11.2): 1572 data bytes

--- 192.20.11.2 ping statistics ---

10 packets transmitted, 0 packets received, 100% packet loss

[root@ESXi:~] vmkping -S vxlan 192.20.11.3 -d -s 1572 -c 10

PING 192.20.11.3 (192.20.11.3): 1572 data bytes

--- 192.20.11.3 ping statistics ---

10 packets transmitted, 0 packets received, 100% packet loss

Regarding below

  • Gateway (T1 downlink for this segment) - success
  • T1 linked port to T0 - success
  • T0 uplink HA VIP address - success
  • Gateway on the HA VIP network (which you say should be 192.20.11.1) - unsuccessful

pastedImage_0.png

Reply
0 Kudos
daphnissov
Immortal
Immortal
Jump to solution

If you only have one ESXi host and, for whatever reason, you can't ping the TEPs on your edges but a VM which is running on that host and connected to an NSX-T logical switch has that level of access then something isn't right with that vmkping command.


Based on what you say here

  • Gateway (T1 downlink for this segment) - success
  • T1 linked port to T0 - success
  • T0 uplink HA VIP address - success
  • Gateway on the HA VIP network (which you say should be 192.20.11.1) - unsuccessful

it sounds like you do not have your static routes configured correctly. So next show how you have configured your static route on your L3 switch.

Reply
0 Kudos
Rob80
Contributor
Contributor
Jump to solution

Basically I have mx65 doing layer 3 routing between vlans where esxi connected via layer 2 switch ms120-8lp as MTU of 1600 required as a minimum. I atatcehd below screens of current setup where I haven't added anything on switch for static routing.

MX 65

pastedImage_0.png

and for ms 120-8lp

pastedImage_2.png

Reply
0 Kudos
daphnissov
Immortal
Immortal
Jump to solution

Your static route appears to be wrong. If .5 is the HA VIP address, you need to direct any networks which you want to route into your T0. This would be any logical segments that exist behind a T1. From your diagram, that appeared to be 10.x.y.z/24 subnets. You will either need to summarize those routes into one, or set static routes for each network.

Reply
0 Kudos
Rob80
Contributor
Contributor
Jump to solution

I have made the following amendments on mx device

pastedImage_0.png

and wonder if I need to do similar on the switch.

Reply
0 Kudos
daphnissov
Immortal
Immortal
Jump to solution

Your App-Tier is really a 10.10.20.0/25 network?

Do a traceroute from a host external to any of these networks to a host that resides on one of them. What do you get?

Reply
0 Kudos