VMware Networking Community
TryllZ
Expert
Expert
Jump to solution

VM Not reaching Internet via Edge ?!

Hi All,

My NSX network setup is as follows, there are no firewall restrictions anywhere.

TryllZ_0-1692393854208.png

Both VMs can ping each other which tells me T1 Gateway is working.

Both Edge nodes can ping firewall interface and internet as well.

TryllZ_0-1692394407588.png

TryllZ_2-1692394057283.png

TryllZ_1-1692394043349.png

Its the VMs that cannot reach the internet, the reply is coming from T0 interface.

TryllZ_3-1692394127207.png

Traceroute results with the following.

TryllZ_1-1692394464844.png

Any thoughts where the issue might be ?

Reply
0 Kudos
44 Replies
TryllZ
Expert
Expert
Jump to solution

I changed Security settings on both Uplink portgroups to Accept all the 3.

Added Prefix list and route maps on the router.

Strangely the router is showing the below in its routing table.

TryllZ_0-1692542290879.png

For the network 10.10.26.0 the next hop is 10.10.25.101 and 10.10.25.102, 65000 is the AS number for NSX BGP.

The firewall cannot ping the 2nd Uplinks on both Edge Nodes (10.10.26.101, 10.10.26.102), nor can the Edge Nodes ping the firewall interface 10.10.26.1.

I tested ping from within the Edge Node, of now is how the interface 10.10.26.101 responds compared to 10.10.26.102 with DUP ping responses.

On the 2nd Edge Node its the reverse, 10.10.26.101 responds with DUP while 10.10.26.101 responds normally.

Pretty sure I've messed up somewhere, could it be due to 1 router with multiple interfaces.

 

edge1(tier0_sr[2])> ping 10.10.26.1
PING 10.10.26.1 (10.10.26.1): 56 data bytes
36 bytes from 10.10.26.1: Destination Host Unreachable
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
 4  5  00 0054 0000   0 0000  40  01 3230 10.10.26.101  10.10.26.1

^C
--- 10.10.26.1 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss

edge1(tier0_sr[2])> ping 10.10.26.101
PING 10.10.26.101 (10.10.26.101): 56 data bytes
64 bytes from 10.10.26.101: icmp_seq=0 ttl=64 time=12.413 ms
^C
--- 10.10.26.101 ping statistics ---
2 packets transmitted, 1 packets received, 50.0% packet loss
round-trip min/avg/max/stddev = 12.413/12.413/12.413/0.000 ms

edge1(tier0_sr[2])> ping 10.10.26.102
PING 10.10.26.102 (10.10.26.102): 56 data bytes
64 bytes from 10.10.26.102: icmp_seq=0 ttl=64 time=21.513 ms
64 bytes from 10.10.26.102: icmp_seq=0 ttl=64 time=21.630 ms (DUP!)
64 bytes from 10.10.26.102: icmp_seq=0 ttl=64 time=21.726 ms (DUP!)
64 bytes from 10.10.26.102: icmp_seq=0 ttl=64 time=21.824 ms (DUP!)
64 bytes from 10.10.26.102: icmp_seq=0 ttl=64 time=21.919 ms (DUP!)
^C
--- 10.10.26.102 ping statistics ---
2 packets transmitted, 1 packets received, +35 duplicates, 50.0% packet loss
round-trip min/avg/max/stddev = 21.513/63.136/120.443/34.055 ms

 

 

Reply
0 Kudos
DanielKrieger
Enthusiast
Enthusiast
Jump to solution

dup packages can occur in nested environments. 

I don't understand your setup and the routing table of the firewall. you don't need the route map. i would simplify the setup first. remove the routing on the opnsense. create an interface in your two uplink vlans and see that you can get the edge node via ping.

10.10.26.0/24 over 10.10.25.101 is not allowed

The Edge node needs a point to point connection to your firewall. The OPNsense needs an ip in the 10.10.25.0/24 and 10.10.26.0/24 network. Then enter 2 neighbors per vlan in frr.

a network plan would help at this point. is your edge vm outside or inside your nested lab? both are possible and depending on that the setup is slightly different.

----------------------------------------------------------------------
My Blog: https://evoila.com/blog/author/danielkrieger/
Reply
0 Kudos
TryllZ
Expert
Expert
Jump to solution

The setup is as follows.

Firewall, vCenter, and NSX-T are all running as regular VMs on baremetal ESXi. 

4 Nested ESXi

2 VM's running on Nested ESXi

2 Edge Nodes running on Nested ESXi

The firewall has 2 individual interfaces for each VLAN25 (10.10.25.1) and VLAN26 (10.10.26.1).

Only 2 neighbours are added in NSX-T BGP settings, 10.10.25.1 and 10.10.26.1.

Currently I'm suspecting a routing loop due use of 1 firewall. The tutorial I was following used 2 dufferent routers, still didn't understand why 2nd uplinks are not reachable, both uplinks are exactly the same in configuration.

I'll setup a 2nd firewall and test this again.

Thanks again @DanielKrieger for all the help.

Reply
0 Kudos
DanielKrieger
Enthusiast
Enthusiast
Jump to solution

your setup is pretty similar to mine. this works with a firewall. a firewall also does not trigger a routing loop. You can get problems with async routing and TCP strict when using multiple VLAN interfaces for peering.

What are your dvPGs, how is your edge profile configured, how are your uplink segments and transport zone?

----------------------------------------------------------------------
My Blog: https://evoila.com/blog/author/danielkrieger/
Reply
0 Kudos
TryllZ
Expert
Expert
Jump to solution

Great to know I'm doing something not different from others, hoping the below makes sense.

In NSX-T the T0 HA is Active/Active, in dvSwitch the portgroups Edge_UL1 and Edge_UL2 are Active/Standby of each other.

DVSwitch Portgroups

TryllZ_0-1692608397475.png

Edge Node Profile

TryllZ_2-1692608511073.png

Uplink Profile

TryllZ_4-1692608552541.png

Transport Zone

TryllZ_3-1692608536417.png

On the Firewall Side

Interfaces

TryllZ_5-1692608659137.png

Neighbours

10.10.25.101 of Edge Node 1 is connected to FW interface 9_NSX_Edge_UL1 and Edge Node 2 10.10.26.101 to 10_NSX_Edge_UL2

TryllZ_6-1692608681605.png

BGP Neighbour Configuration Uplink 1 (Left), Uplink 2 (Right)

TryllZ_8-1692608718441.pngTryllZ_9-1692608749914.png

Prefix list

TryllZ_7-1692608693382.png

Reply
0 Kudos
DanielKrieger
Enthusiast
Enthusiast
Jump to solution

What I noticed is that you seem to have applied your VLAN transport zone to your ESX hosts as well - this is not necessary. The uplink VLAN segments only need to be known to the edge nodes. How is your teaming policy written? Are you using named teaming policy? What are all the prefixes and routemaps doing? Personally, I would keep the setup simple for now and deploy something like that later. But I don't see anything that triggers your problem. Can you check on the switch if it can see the mac address of your edge node? I still think this is a layer 2 problem. Also, check the OPNSense to see if it has a mac address of your edge nodes from the 26 network. Just make a ping and then look in the Arp Table.

The problem is not so easy to debug remotely. In my opinion it's not the NSX or the OPNSense settings, much more something is crooked in your nested environment. Can you put a testvm into the dvPG and give it the VLAN ID 26 and test if it can communicate? I also had problems at the beginning with my nested environment that MAC addresses were not learned cleanly. Also make sure that Promiscuous Mode, Mac Address changes and forged transmits are active on the port group of the OPNSense.

----------------------------------------------------------------------
My Blog: https://evoila.com/blog/author/danielkrieger/
Reply
0 Kudos
TryllZ
Expert
Expert
Jump to solution

Thanks for taking the time, really appreciate it.


What I noticed is that you seem to have applied your VLAN transport zone to your ESX hosts as well - this is not necessary.

Thanks for adding that.


How is your teaming policy written? Are you using named teaming policy?

Not sure what you mean by that.

What are all the prefixes and routemaps doing?

Learned to configure BGP in OPNSense from here as the documentation is not that great for OPNSense and BGP.


Can you check on the switch if it can see the mac address of your edge node?

I still think this is a layer 2 problem. Also, check the OPNSense to see if it has a mac address of your edge nodes from the 26 network. Just make a ping and then look in the Arp Table.

 

Can you put a testvm into the dvPG and give it the VLAN ID 26 and test if it can communicate?

I'll check these, just that the switch is just a smart unmanaged switch.


The problem is not so easy to debug remotely.

Definitely agreed.


Also make sure that Promiscuous Mode, Mac Address changes and forged transmits are active on the port group of the OPNSense.

The only confusion I have with this is where all is this applicable, everwhere ?, I have this configured on the vSwitch on Baremetal ESXi, is this to be configured on ALL dvSwitch Portgroups ?

Reply
0 Kudos
DanielKrieger
Enthusiast
Enthusiast
Jump to solution


The only confusion I have with this is where all is this applicable, everwhere ?, I have this configured on the vSwitch on Baremetal ESXi, is this to be configured on ALL dvSwitch Portgroups ?

It depends, I would check. I use the distributed switch on all my baremetal ESX servers. I have 2 ESX servers for management, running the vCenter, my firewall cluster, my NSX Manager, ALB Manager, Unraid and my backup server with Veeam. For my lab I have 3 Intel NUCs running ESX virtual servers which then have the NSX workload. These are only started when needed. If I want to test NSX, the ESX hosts for NSX come up, if I need my Tanzu or ALB Lab, I start other virtual ESX servers. because of the flexibility i use distributed switches everywhere and because i am lazy to type.

----------------------------------------------------------------------
My Blog: https://evoila.com/blog/author/danielkrieger/
Reply
0 Kudos
TryllZ
Expert
Expert
Jump to solution

Interesting setup, I use 1 server and 1 switch to keep cost down.

I added VM to Edge_UL2 (VLAN26) and made ping test as well, interestingly, the VM can reach the firewall interface.

Edge has nothing in ARP table for Edge 2nd Uplinks

TryllZ_0-1692646391585.png

However, I reset firewall table states and suddenly both Edge node's second interfaces (10.10.26.101, 10.10.26.102) were able to reach firewall interface, and even after this there was no entry in the ARP table for both of these interfaces.

And when the edge nodes' 2nd uplink was able to reach the firewall, the 1st interface (VLAN 25) which was pinging before was not pinging.

Not sure I understand why this is the case.

Could it be due to NSX-T and Edge versions, or the HA for Edge setup in T0 Gateway (point being only 1 interface is active at a time) ?

I'm using NSX-T 4.0.1, Edge versions are also 4.0 something. The reason I'm bringing this is because I read on several sites that the way uplinks and their relevant VLANs were configured were changed between version 2.x and 3.x, unsire if 3.2 and 4.0.1 have similar ways. The Design guide is still 3.2

Reply
0 Kudos
TryllZ
Expert
Expert
Jump to solution

And after all reboot, its back to the same issue of not pinging VLAN 26 interfaces.

Reply
0 Kudos
DanielKrieger
Enthusiast
Enthusiast
Jump to solution

Even if the T0 is active/passive, the second interface must be pingable. What you describe sounds like a Mac learning problem. Have you checked the security settings of all port groups?

Was your test VM able to reach the Edge VM?
Did you have the TestVM on the same ESX virtual host as the Edge Node?
What does the ARP table on the switch look like? Are the Mac addresses of the edge there?

The edge nodes should always have the appropriate NSX version to the manager and co. The way the uplinks are configured has not really changed from 3.2 to 4.X.

 

----------------------------------------------------------------------
My Blog: https://evoila.com/blog/author/danielkrieger/
Reply
0 Kudos
TryllZ
Expert
Expert
Jump to solution

What you describe sounds like a Mac learning problem

Likely a firewall problem then.

Have you checked the security settings of all port groups?

All settings are default, only Edge Uplink portgroup settings were changed to Allow, will change all portgroups settinga to Allow.

Was your test VM able to reach the Edge VM?

No

Did you have the TestVM on the same ESX virtual host as the Edge Node?

No, will test this and above.

I will add though that I did test a traceroute in Edge and traffic leaves through its Management interface which is in 10.10.15.0 network.

What does the ARP table on the switch look like? Are the Mac addresses of the edge there?

Its a very basic switch.

The edge nodes should always have the appropriate NSX version to the manager and co.

Where can this be checked, matching versions of NSX-T with Edge. To my knowledge Release Nodes are only for NSX-T and no matching Edge version information is there.

Reply
0 Kudos
TryllZ
Expert
Expert
Jump to solution


The edge nodes should always have the appropriate NSX version to the manager and co.

Found this list, I'm using the right versions, highlighted in Red.

VMware NSX-T Component Build Matrix | virten.net

Name Main Build Manager Controller Edge Kernel Module L2 VPN Client Standalone Edge

NSX 4.1.0.221761691
21761695
2176169921761693
21761699
NSX 4.1.021332672
21332677
2133268021332675
19300606
NSX 4.0.1.120598726
20598732
2059873520598730
19300606
NSX 4.0.0.120159689
20159694
2015969720159692
11197779
Reply
0 Kudos
TryllZ
Expert
Expert
Jump to solution

Looks like a firewall related issue, I did a traceroute

edge1> traceroute 10.10.26.102
traceroute to 10.10.26.102 (10.10.26.102), 30 hops max, 60 byte packets
 1  10.10.15.1  5.577 ms  5.385 ms  4.633 ms
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  *^C

The firewall is sending ARP Broadcast request trying to find the IP address.

Sender IP 10.10.26.1 is the firewall interface.

TryllZ_1-1692674453922.png

TryllZ_0-1692674265242.png

I'm going to check about this on the OPNSense forum, see what they have to say.

Appreciate all the help @DanielKrieger 

Reply
0 Kudos
DanielKrieger
Enthusiast
Enthusiast
Jump to solution

the question is, why would the edge node go through the firewall at all? You do the traceflow from your edge and the edge should have an IP in the 26 network. So there is no reason to go through the firewall. It also looks to me that you are not in the VRF context of the SR T0 and then it is clear that the Edge is trying to communicate with the 10.109.26.102 via the MGMT interface.

----------------------------------------------------------------------
My Blog: https://evoila.com/blog/author/danielkrieger/
Reply
0 Kudos
TryllZ
Expert
Expert
Jump to solution

Will run a Traceflow again.

Both Edge nodes' both interfaces (10.10.25.101, 10.10.25.102, 10.10.26.101, 10.10.26.102) are configured for BGP Peering with the both firewall's interfaces (10.10.25.1, 10.10.26.1) which is only not happening on the 26 interface.

True, I id not check any of this from the VRF in T0, will check this.

Reply
0 Kudos
TryllZ
Expert
Expert
Jump to solution

From within VRF I can ping all interfaces from 1 edge node to another, still can't ping firewall interface.

edge1(tier0_sr[2])> ping 10.10.26.1
PING 10.10.26.1 (10.10.26.1): 56 data bytes
36 bytes from 10.10.26.1: Destination Host Unreachable
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
 4  5  00 0054 0000   0 0000  40  01 3230 10.10.26.101  10.10.26.1

36 bytes from 10.10.26.1: Destination Host Unreachable
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
 4  5  00 0054 0000   0 0000  40  01 3230 10.10.26.101  10.10.26.1

Traceflow

TryllZ_0-1692697888214.png

TryllZ_1-1692697899997.png

TryllZ_2-1692697929938.png

Reply
0 Kudos
DanielKrieger
Enthusiast
Enthusiast
Jump to solution

Hi, sorry I'm just getting back to you, but I had to rewire my entire lab first. Anyway - I tried to reproduce your problem, but I didn't manage to do so. The whole thing works for me. You could also give your firewall virtual interfaces in the uplink networks and tag them via the portgroup. Then the traffic would be untagged and any problems with double tagging would be solved. But then you have to change your peering a bit and need 2 nvds on the edge VMs. But this works without problems in nested labs.

----------------------------------------------------------------------
My Blog: https://evoila.com/blog/author/danielkrieger/
Reply
0 Kudos
TryllZ
Expert
Expert
Jump to solution

No worries,

Appreciate you taking the time to reproduce the problem.

Currently I'm going through the Edge logs, and will make it simpler and see where the issue is.

Thanks again @DanielKrieger 

Reply
0 Kudos
TryllZ
Expert
Expert
Jump to solution

Thanks @DanielKrieger 

I set the Edge Uplink portgroups to trunking.

TryllZ_0-1692899265709.png

And firewall ARP table now has the interface attached.

TryllZ_1-1692899291325.png

Now both interfaces are in Established state, and BGP peering on all Edge Interfaces successfully.

edge1> vrf 2
edge1(tier0_sr[2])> get bgp neighbor summary
BFD States: NC - Not configured, DC - Disconnected
            AD - Admin down, DW - Down, IN - Init, UP - Up
BGP summary information for VRF default for address-family: ipv4Unicast
Router ID: 10.10.25.101  Local AS: 65000

Neighbor                            AS          State Up/DownTime  BFD InMsgs  OutMsgs InPfx  OutPfx

10.10.25.1                          65555       Estab 00:12:58     UP  46      20      12     4
10.10.26.1                          65555       Estab 00:12:58     UP  46      20      12     14

Thu Aug 24 2023 UTC 17:54:55.772
edge2> vrf 1
edge2(tier0_sr[1])> get bgp neighbor summary
BFD States: NC - Not configured, DC - Disconnected
            AD - Admin down, DW - Down, IN - Init, UP - Up
BGP summary information for VRF default for address-family: ipv4Unicast
Router ID: 10.10.25.102  Local AS: 65000

Neighbor                            AS          State Up/DownTime  BFD InMsgs  OutMsgs InPfx  OutPfx

10.10.25.1                          65555       Estab 00:15:18     UP  48      23      12     12
10.10.26.1                          65555       Estab 00:15:18     UP  51      23      12     6

Thu Aug 24 2023 UTC 17:57:02.232

The issue is OPNSense VLAN interfaces cannot be created without tags, or cannot be set as 0.

Reply
0 Kudos