Can't ping VMs behind logical switches from external network

[zdingelit]

Hi,

I need some help please.

I have installed NSX on a nested environnment

now from the admin pc  I can ping the sophos router 192.168.100.254, I can ping 192.168.3.1 and 192.168.3.254

I can also ping the logical switch gateway for each Tier.

10.1.10.1

10.1.20.1

10.1.30.1

all Vms are Ups.

When I try to ping 10.1.10.11 and 10.1.10.12 (VMs) no response.

a ping from 10.1.10.11 to 10.1.10.12 works.

actually I have one static route on the NSX edge

Network 192.168.100.0/24

Next Hop 192.168.3.1

Interface Enterprise_Uplink 192.168.3.254

MTU 1600

Admin distance 1

Here are the static routes on the sophos router :

Kernel IP routing table

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface

10.1.10.0       0.0.0.0         255.255.255.0   U     5      0        0 eth1

10.1.20.0       0.0.0.0         255.255.255.0   U     5      0        0 eth1

10.1.30.0       0.0.0.0         255.255.255.0   U     5      0        0 eth1

127.0.0.0       0.0.0.0         255.0.0.0       U     0      0        0 lo

192.168.3.0     0.0.0.0         255.255.255.0   U     0      0        0 eth1

192.168.5.0     0.0.0.0         255.255.255.0   U     5      0        0 eth1

192.168.100.0   0.0.0.0         255.255.255.0   U     0      0        0 eth0

a ping from the Sophos router to 10.1.10.1 is ok but 10.1.10.11 is not !

a ping from the NSX Edge to all VM behind logical switches works !!

Thanks for your help !!!

[Sreec]

Can these VM's ping their own gateway ?

[zdingelit]

Yes, each VM can ping it's own gateway.

[haozch]

Can you check if config default gateway in the VM.

[haozch]

Can you check if config default gateway in the VM.

or Can you ping from 10.1.10.11 to 10.1.20.11, is it ok?

[Carlos_E]

Hi,

Maybe there´s something else to fix, but remember to enable ICMP at the firewall level on the Edge.

Regards,

Carlos.

[zdingelit]

Hi Sree,

all VMs behind the logical switches have the ip default gateway configured correctly

from 10.1.10.11 I can ping 10.1.20.11 and 10.1.30.11.

from 10.1.20.11 I can ping  10.1.10.11 and 10.1.30.11.

from 10.1.30.11 I can ping  10.1.10.11 and 10.1.20.11.

Thank you !

[Sreec]

Okay! Can you run a trace-route from VM to PC and vice-versa and update the output ?

[zdingelit]

Admin PC to vm 10.1.10.11

root@LinuxControlST:~# route -n

Table de routage IP du noyau

Destination     Passerelle      Genmask         Indic Metric Ref    Use Iface

0.0.0.0         192.168.100.254 0.0.0.0         UG    0      0        0 eth0

192.168.100.0   0.0.0.0         255.255.255.0   U     0      0        0 eth0

root@LinuxControlST:~# traceroute 10.1.10.11

traceroute to 10.1.10.11 (10.1.10.11), 30 hops max, 60 byte packets

1  192.168.100.254 (192.168.100.254)  0.436 ms  0.752 ms  0.935 ms

2  * * *

3  * * *

4  * * *

5  * * *

6  * * *

7  * * *

8  * * *

9  * * *

10  * * *

11  * * *

12  * * *

13  * * *

14  * * *

15  * * *

16  * * *

17  * * *

18  * * *

19  * * *

20  * * *

21  * * *

22  * * *

23  * * *

24  * * *

25  * * *

26  * * *

27  * * *

28  * * *

29  * * *

30  * * *

root@LinuxControlST:~#

VM behind logical switch to admin pc

root@web01:~# route -n

Table de routage IP du noyau

Destination     Passerelle      Genmask         Indic Metric Ref    Use Iface

0.0.0.0         10.1.10.1 0.0.0.0         UG    0      0        0 eth0

10.1.10.0   0.0.0.0         255.255.255.0   U     0      0        0 eth0

traceroute to 192.168.100.6 (192.168.100.6), 30 hops max, 60 byte packets

1 * * *

2 * * *

3 * * *

4 * * *

5 * * *

6 * * *

7 * * *

8 * * *

9 * * *

10 * * *

...

[Sreec]

Okay,so both ways it looks not working. Can you move one of the VM 10.1.10.x and Edge to same Host and do a test ? Also this being nested - were did you enable promiscuous mode in vSphere network ? Also do a test by directly connecting the VM -external network to your router. For sure one of the test will answer your problem

[zdingelit]

okay

the vm 10.1.10.11 and the NSX edge are now hosted on nestedesxi01 and it is the same behavior :

each nested VM have  4 network adapters :

net1 NET0-PROMISC   vmk0 management promiscuous mode & forget transmit enabled

Net2 Nested Storage2 v vmk1 Storage openfiler

Net3 Nested-vMotion2 vmk2 vMotion

Net4 Trunk (not use for the nsx lab)

Net 5 uplinkEdgetoVDS vmk3 vteps mtu 1600 promiscuous mode & forget transmit enabled

Main esxi host :

Thank you !

[tspit0001]

If it's not working while the ESG and vm's are on the same host that's fishy.

Can you ping from a VM to the external IP of the ESG while they're on the same host? I know you can ping the internal IP (the vms default gateway), but can you ping that 192.168.3.254 IP address from the web VM?

If the answer is yes while the VM is on the same host as the ESG then we're on to something!

Also, the uplink on your ESG shouldn't be 1600 MTU. The only requirement for the 1600 MTU is your underlay for your VTEPS, so in this case that looks like it would be your vsphere switches. It's unlikely that's breaking tour setup, just something I noticed. I'm assuming that your sophos router is 1500 MTU, so you don't really want an MTU mismatch between two routers in any scenario.

[Sreec]

vm 10.1.10.11 and the NSX edge are now hosted on nestedesxi01 and it is the same behavior

Are u saying results are same for connectivity test from - VM->ESG>Router connectivity & VM-ESG-Router-Admin PC ?

I'm bit confused with your promiscuous mode settings - is it enabled on the main host ?

[zdingelit]

Thank for trying to help.

I need to resolve this issue to improve my NSX knowledge.

I will give you more detail When I will access to my lab tonight.

[cvrich]

This almost sounds like the network outside of NSX doesn't know where the 10.1.x.x subnets are located. I think I might change this setup a little by adding a DLR. I would move the .1 uplinks of each logical switch from the Edge to the DLR and then add a internal link from the DLR to the transit and then add a internal link from the from the Edge to the transit.. I also disabled the firewall on the Edge... I would use dynamic routing if possible. I am using ospf in my lab between the DLR and the Edge and it works quite well.

[bayupw]

could you share the routing table of:

1. admin PC

2. sophos router

3. NSX Edge

4. one of the VM behind logical switch

[vXav]

My knowledge of NSX is limited as I started playing with it just a week ago. It might sound silly but have you tried disabling the firewall on the guests (10.1.10.11, ...) ?

Ping works only in one direction makes me think about that...

Also:

"then add a UPLINK link from the DLR to the transit and then add a internal link from the from the Edge to the transit"

[BASITKAHNPK]

Try to disable firewall from everywhere.

NSX Firewall

Edge Firewall

Guest OS Firewall

and off course Admin PC Firewall.

from Sophos Console you should be able to ping to all VMs in ESG Internal or external IPs if ESG's Firewall is disbaled.

all internal vms must have ESG as their Gateway

Try to add Sophos IP in ESG as default gateway

Enable Promiscuous mode in all nested esxi