VMware Networking Community
sentania
Enthusiast
Enthusiast
Jump to solution

Routing with VMware NSX - no external access for vxLAN machines

All -

I'm trying to troubleshoot a problem I am encountering after setting up NSX in my lab.

First - here is a really bad drawing depicting a snippet of my environment

diagram.PNG

Basically I have a USG as the edge device, uplinked into my cable modem, and the LAN port on a Mikrotik CRS125; which then has a US16XG linked to it.

My VMware hosts are all uplinked into the US16XG, and connected to a single VDS.  "VLAN1" - 10.100.1.0/24 is untagged throughout the entire environment.

I also have VLAN1100 - 10.100.100.0/24, which is tagged and also available through the esx environment.

All of my VMs on the standard VLANs can communicate as expected, and has internet access, leveraging the USG as the router.  I am using OSPF to advertise routes between the Edge Router, Logical router, and USG.

The issues arises once I put a device on an NSX logical switch, which then has the needed logical router, and edge router in front of it.  vxLAN5001 - 10.100.101.0/24.

A VM now placed on the NSX vxLAN has full internal connectivity.  It can ping, RDP, etc other devices anywhere within my network.  For example: 10.100.101.100 on vxLAN5001  can ping a VM at 10.100.100.30 (VLAN1100). Through an NSX DHCP helper, VMs on vxLAN 5001 can get a DHCP address.

Additionally, anyother local resourece, physical or virtual can reach the devices on the vxLAN.

What isn't working is that the vxLAN hosted machine can't get out to the internet.

When doing a traceroute, I see traffic dying at/after the USG.  In the following picture, the cmd box on the left is from the VM on vxLAN one, the screen on the right is from my laptop.

tracert.PNG

I'm a server guy first, and in general pick up enough networking to do my job, so at this point I'm a little out of my league of where to look.

I feel like it could be a firewall issue or routing issue on the USG, but I'm not sure.  I've debated switching from OSPF to BGP to see if that makes a difference, but my hunch is no.

Any advice is appreciated.

ETA: It's not depicted in the diagram, but I have to site-to-site VPNs configured on the USG, with the far sides being 10.10.0.0/16 and 192.168.0.0/16.  The vxLAN machine is able to access those resources as expected.

Reply
0 Kudos
1 Solution

Accepted Solutions
bayupw
Leadership
Leadership
Jump to solution

As per your requirements, you want the network 10.100.101.0/24 (VXLAN) be able to traverse to the Internet.

Therefore, you will need to do a NAT for network 10.100.101.0/24 either

1. Directly to the public IP on your public facing NAT Device (USG)

2. or a NAT on your NSX Edge (so there will be double NAT, one at USG and one at NSX Edge), your NSX Edge must not have ECMP enabled

Bayu Wibowo | VCIX6-DCV/NV
Author of VMware NSX Cookbook http://bit.ly/NSXCookbook
https://github.com/bayupw/PowerNSX-Scripts
https://nz.linkedin.com/in/bayupw | twitter @bayupw

View solution in original post

Reply
0 Kudos
7 Replies
bayupw
Leadership
Leadership
Jump to solution

Hi

Where do you NAT/SNAT for access to the Internet? UBNT USG or other firewall?

Make sure:

1. VM on VXLAN 5001 - 10.100.101.100 can access UBNT USG 10.100.100.1 which I think this looks okay based on your explanation

2. Add SNAT or Outbound NAT for 10.100.101.0/24

Bayu Wibowo | VCIX6-DCV/NV
Author of VMware NSX Cookbook http://bit.ly/NSXCookbook
https://github.com/bayupw/PowerNSX-Scripts
https://nz.linkedin.com/in/bayupw | twitter @bayupw
Reply
0 Kudos
sentania
Enthusiast
Enthusiast
Jump to solution

The USG is the NAT device for the internet, and the vxLAN 5000 VM is indeed able to reach it successfully.

What does DNAT do for me in this situation?

Thanks.

Reply
0 Kudos
sentania
Enthusiast
Enthusiast
Jump to solution

In case it's helpful route tables:

sentania@usg1:~$ show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF,
I - ISIS, B - BGP, > - selected route, * - FIB route

S>* 0.0.0.0/0 [1/0] via ISPGW, eth0
O 10.100.1.0/24 [110/10] is directly connected, eth1, 16:18:34
C>* 10.100.1.0/24 is directly connected, eth1
C>* 10.100.2.0/24 is directly connected, eth1.2
C>* 10.100.3.0/24 is directly connected, eth1.3
C>* 10.100.4.0/24 is directly connected, eth1.4
C>* 10.100.5.0/24 is directly connected, eth1.5
C>* 10.100.6.0/24 is directly connected, eth1.6
C>* 10.100.15.0/24 is directly connected, eth1.15
C>* 10.100.16.0/24 is directly connected, eth1.16
O>* 10.100.50.0/24 [110/11] via 10.100.1.2, eth1, 15:49:19
C>* 10.100.99.0/24 is directly connected, eth1.1099
C>* 10.100.100.0/24 is directly connected, eth1.1100
O>* 10.100.101.0/24 [110/1] via 10.100.1.2, eth1, 15:46:59
C>* 10.100.200.0/24 is directly connected, eth1.1200
C>* 10.100.201.0/24 is directly connected, eth1.1201
C>* 65.30.192.0/20 is directly connected, eth0
C>* 127.0.0.0/8 is directly connected, lo
S>* 192.168.100.1/32 [1/0] is directly connected, eth0

++++++++++++++++++++++++++++++++

EGW01-0> show ip route

Codes: O - OSPF derived, i - IS-IS derived, B - BGP derived,
C - connected, S - static, L1 - IS-IS level-1, L2 - IS-IS level-2,
IA - OSPF inter area, E1 - OSPF external type 1, E2 - OSPF external type 2,
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

Total number of routes: 4

S 0.0.0.0/0 [1/1] via 10.100.1.1
C 10.100.1.0/24 [0/0] via 10.100.1.2
C 10.100.50.0/24 [0/0] via 10.100.50.1
O E2 10.100.101.0/24 [110/1] via 10.100.50.2

dlrroutes.PNG

Thanks.

I more or less used this as the guide to set things up:

http://blog.bertello.org/2015/01/nsx-for-newbies-series/

Reply
0 Kudos
bayupw
Leadership
Leadership
Jump to solution

Hi, I was referring to SNAT and not DNAT my mistake.

SNAT or Outbound NAT to translate internal network in VXLAN 5000 to a public external IP address.

Bayu Wibowo | VCIX6-DCV/NV
Author of VMware NSX Cookbook http://bit.ly/NSXCookbook
https://github.com/bayupw/PowerNSX-Scripts
https://nz.linkedin.com/in/bayupw | twitter @bayupw
Reply
0 Kudos
sentania
Enthusiast
Enthusiast
Jump to solution

OK, that makes more sense, but my public IP lies on the USG, so if I did a SNAT on the machines on the vxLAN, aren't I simply making life difficult for me from my other internal resources?

Reply
0 Kudos
bayupw
Leadership
Leadership
Jump to solution

As per your requirements, you want the network 10.100.101.0/24 (VXLAN) be able to traverse to the Internet.

Therefore, you will need to do a NAT for network 10.100.101.0/24 either

1. Directly to the public IP on your public facing NAT Device (USG)

2. or a NAT on your NSX Edge (so there will be double NAT, one at USG and one at NSX Edge), your NSX Edge must not have ECMP enabled

Bayu Wibowo | VCIX6-DCV/NV
Author of VMware NSX Cookbook http://bit.ly/NSXCookbook
https://github.com/bayupw/PowerNSX-Scripts
https://nz.linkedin.com/in/bayupw | twitter @bayupw
Reply
0 Kudos
sentania
Enthusiast
Enthusiast
Jump to solution

Thanks.  I added a source MASQ rule for the subnet(s) on the USG and internet access came up.

Thanks for the help.

Reply
0 Kudos