DLR Uplink and ESG Internal in same transit VXLAN cannot ping each other.

[Samsonite801]

To start with, I am running NSX 6.2.2, firewall rules set to 'allow all' from 'any' to 'any' 'all protocols', in other words disabled...

I have a transit VXLAN 5000, with a DLR Uplink interface attached to it, and an ESG Internal interface attached to it in which neither side can ping the other. So for troubleshooting, I added 2 Windows VM's attached to the same transit VXLAN 5000, one VM is on ESXi host 1 and the other is on ESXi host 4. They can ping each other fine, and both VMs can ping both the DLR Uplink and the ESG internal interfaces.

This issue has me perplexed as it makes no sense why the DLR and ESG can't ping each other but the 2 VMs in that VXLAN can ping all of the adjacent devices. I can even set gateways on these VMs along with a NAT rule on the ESG and these VMs can get internet through the ESG, but no matter what I try, the DLR can't ping the ESG, and ESG cannot ping DLR..

I need to set a static route between the DLR <-> ESG but if I can't even get the interfaces to reply to ping then I'm dead in the water.

If I install test VMs into a DLR LAN interface like WebApp and Database for example, I can ping all the way through the entire DLR right up to the DLR Uplink IP, but then it cannot ping the ESG Internal.

Does anyone have any troubleshooting suggestions? Test commands I can run? I have tried so many things and followed many websites with troubleshooting steps. Everything seems fine, All green checks in the installation stages.. All routes, MACs, ARP tables showing up as expected when I run test commands on the hosts and controllers. I have no idea what is causing this except a bug in code..

Any ideas are welcome... Thanks

[Samsonite801]

By the way, the other potential clue is when I am pinging from either the DLR > ESG, or from the ESG > DLR,  console to the adjacent interface I see this :

ping: sendmsg: Operation not permitted

ping: sendmsg: Operation not permitted

ping: sendmsg: Operation not permitted

ping: sendmsg: Operation not permitted

dlr-01-0>

--- 192.168.1.1 ping statistics ---

4 packets transmitted, 0 received, 100% packet loss, time 2999ms

ping: sendmsg: Operation not permitted

ping: sendmsg: Operation not permitted

ping: sendmsg: Operation not permitted

ping: sendmsg: Operation not permitted

esg-01-0>

--- 192.168.1.2 ping statistics ---

4 packets transmitted, 0 received, 100% packet loss, time 2999ms

And as I said before, the 2 Windows VM's I put in this transport zone can ping all adjacent interfaces, ESG, DLR, other VM, and itself of course..

[Samsonite801]

UPDATE:

So I enabled OSPF routing between the DLR 'Uplink' and ESG 'Internal' interfaces across the transit VXLAN 5000, and I can now ping out to the ESG's Uplink IP address, but cannot ping out to the physical network.

But both of my test Windows VMs I have attached to the transit VXLAN 5000 using custom NAT rules on secondary IP addresses for the same ESG 'Uplink' do route out the default gateway.

Now I'm really confused because I thought the OSPF is supposed to use the default route on the ESG Uplink, so not sure what is happening.

In the physical network I am also able to ping the IP of the ESG 'Uplink' interface, so it is there.

Going to play with static NAT and see what happens.. I think the routing should work but maybe NAT is where it is breaking?

[Samsonite801 · Accepted solution]

UPDATE:

Yeah, so it did need a static NAT rule on the ESG..

In my particular environment I added an SNAT rule on adapter: ESG_Uplink with src 0.0.0.0/24 to dst translation: 1.1.1.101  (my lab ESG Uplink IP).

It works now.. VM on tenant pod connected to WebApp portgroup (192.168.13.115) can now ping through DLR gateway, through OSPF routing to ESG and ping out to physical gateway..

I learned a lot on this one.. I'm not going to worry about why the static route I tried in first post didn't work, since I got the OSPF running instead (which is more appropriate for my lab applying it to real-world scenario anyway), and the foundation will now suffice for building the rest of this POC vRA / vRO lab..

Thanks anyway, sometimes it's just nice to have someone there to listen.