VM Not reaching Internet via Edge ?!

[TryllZ]

Hi All,

My NSX network setup is as follows, there are no firewall restrictions anywhere.

Both VMs can ping each other which tells me T1 Gateway is working.

Both Edge nodes can ping firewall interface and internet as well.

Its the VMs that cannot reach the internet, the reply is coming from T0 interface.

Traceroute results with the following.

Any thoughts where the issue might be ?

[TryllZ]

I changed Security settings on both Uplink portgroups to Accept all the 3.

Added Prefix list and route maps on the router.

Strangely the router is showing the below in its routing table.

For the network 10.10.26.0 the next hop is 10.10.25.101 and 10.10.25.102, 65000 is the AS number for NSX BGP.

The firewall cannot ping the 2nd Uplinks on both Edge Nodes (10.10.26.101, 10.10.26.102), nor can the Edge Nodes ping the firewall interface 10.10.26.1.

I tested ping from within the Edge Node, of now is how the interface 10.10.26.101 responds compared to 10.10.26.102 with DUP ping responses.

On the 2nd Edge Node its the reverse, 10.10.26.101 responds with DUP while 10.10.26.101 responds normally.

Pretty sure I've messed up somewhere, could it be due to 1 router with multiple interfaces.

edge1(tier0_sr[2])> ping 10.10.26.1
PING 10.10.26.1 (10.10.26.1): 56 data bytes
36 bytes from 10.10.26.1: Destination Host Unreachable
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
 4  5  00 0054 0000   0 0000  40  01 3230 10.10.26.101  10.10.26.1

^C
--- 10.10.26.1 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss

edge1(tier0_sr[2])> ping 10.10.26.101
PING 10.10.26.101 (10.10.26.101): 56 data bytes
64 bytes from 10.10.26.101: icmp_seq=0 ttl=64 time=12.413 ms
^C
--- 10.10.26.101 ping statistics ---
2 packets transmitted, 1 packets received, 50.0% packet loss
round-trip min/avg/max/stddev = 12.413/12.413/12.413/0.000 ms

edge1(tier0_sr[2])> ping 10.10.26.102
PING 10.10.26.102 (10.10.26.102): 56 data bytes
64 bytes from 10.10.26.102: icmp_seq=0 ttl=64 time=21.513 ms
64 bytes from 10.10.26.102: icmp_seq=0 ttl=64 time=21.630 ms (DUP!)
64 bytes from 10.10.26.102: icmp_seq=0 ttl=64 time=21.726 ms (DUP!)
64 bytes from 10.10.26.102: icmp_seq=0 ttl=64 time=21.824 ms (DUP!)
64 bytes from 10.10.26.102: icmp_seq=0 ttl=64 time=21.919 ms (DUP!)
^C
--- 10.10.26.102 ping statistics ---
2 packets transmitted, 1 packets received, +35 duplicates, 50.0% packet loss
round-trip min/avg/max/stddev = 21.513/63.136/120.443/34.055 ms

[DanielKrieger]

dup packages can occur in nested environments. 

I don't understand your setup and the routing table of the firewall. you don't need the route map. i would simplify the setup first. remove the routing on the opnsense. create an interface in your two uplink vlans and see that you can get the edge node via ping.

10.10.26.0/24 over 10.10.25.101 is not allowed

The Edge node needs a point to point connection to your firewall. The OPNsense needs an ip in the 10.10.25.0/24 and 10.10.26.0/24 network. Then enter 2 neighbors per vlan in frr.

a network plan would help at this point. is your edge vm outside or inside your nested lab? both are possible and depending on that the setup is slightly different.

[TryllZ]

The setup is as follows.

Firewall, vCenter, and NSX-T are all running as regular VMs on baremetal ESXi. 

4 Nested ESXi

2 VM's running on Nested ESXi

2 Edge Nodes running on Nested ESXi

The firewall has 2 individual interfaces for each VLAN25 (10.10.25.1) and VLAN26 (10.10.26.1).

Only 2 neighbours are added in NSX-T BGP settings, 10.10.25.1 and 10.10.26.1.

Currently I'm suspecting a routing loop due use of 1 firewall. The tutorial I was following used 2 dufferent routers, still didn't understand why 2nd uplinks are not reachable, both uplinks are exactly the same in configuration.

I'll setup a 2nd firewall and test this again.

Thanks again @DanielKrieger for all the help.

[DanielKrieger]

your setup is pretty similar to mine. this works with a firewall. a firewall also does not trigger a routing loop. You can get problems with async routing and TCP strict when using multiple VLAN interfaces for peering.

What are your dvPGs, how is your edge profile configured, how are your uplink segments and transport zone?

[TryllZ]

Great to know I'm doing something not different from others, hoping the below makes sense.

In NSX-T the T0 HA is Active/Active, in dvSwitch the portgroups Edge_UL1 and Edge_UL2 are Active/Standby of each other.

DVSwitch Portgroups

Edge Node Profile

Uplink Profile

Transport Zone

On the Firewall Side

Interfaces

Neighbours

10.10.25.101 of Edge Node 1 is connected to FW interface 9_NSX_Edge_UL1 and Edge Node 2 10.10.26.101 to 10_NSX_Edge_UL2

BGP Neighbour Configuration Uplink 1 (Left), Uplink 2 (Right)

Prefix list

[DanielKrieger]

What I noticed is that you seem to have applied your VLAN transport zone to your ESX hosts as well - this is not necessary. The uplink VLAN segments only need to be known to the edge nodes. How is your teaming policy written? Are you using named teaming policy? What are all the prefixes and routemaps doing? Personally, I would keep the setup simple for now and deploy something like that later. But I don't see anything that triggers your problem. Can you check on the switch if it can see the mac address of your edge node? I still think this is a layer 2 problem. Also, check the OPNSense to see if it has a mac address of your edge nodes from the 26 network. Just make a ping and then look in the Arp Table.

The problem is not so easy to debug remotely. In my opinion it's not the NSX or the OPNSense settings, much more something is crooked in your nested environment. Can you put a testvm into the dvPG and give it the VLAN ID 26 and test if it can communicate? I also had problems at the beginning with my nested environment that MAC addresses were not learned cleanly. Also make sure that Promiscuous Mode, Mac Address changes and forged transmits are active on the port group of the OPNSense.

[TryllZ]

Thanks for taking the time, really appreciate it.

---

What I noticed is that you seem to have applied your VLAN transport zone to your ESX hosts as well - this is not necessary.

---

Thanks for adding that.

---

How is your teaming policy written? Are you using named teaming policy?

---

Not sure what you mean by that.

What are all the prefixes and routemaps doing?

---

Learned to configure BGP in OPNSense from here as the documentation is not that great for OPNSense and BGP.

---

Can you check on the switch if it can see the mac address of your edge node?

I still think this is a layer 2 problem. Also, check the OPNSense to see if it has a mac address of your edge nodes from the 26 network. Just make a ping and then look in the Arp Table.

Can you put a testvm into the dvPG and give it the VLAN ID 26 and test if it can communicate?

I'll check these, just that the switch is just a smart unmanaged switch.

---

The problem is not so easy to debug remotely.

Definitely agreed.

---

Also make sure that Promiscuous Mode, Mac Address changes and forged transmits are active on the port group of the OPNSense.

---

The only confusion I have with this is where all is this applicable, everwhere ?, I have this configured on the vSwitch on Baremetal ESXi, is this to be configured on ALL dvSwitch Portgroups ?

[DanielKrieger]

---

The only confusion I have with this is where all is this applicable, everwhere ?, I have this configured on the vSwitch on Baremetal ESXi, is this to be configured on ALL dvSwitch Portgroups ?

---

It depends, I would check. I use the distributed switch on all my baremetal ESX servers. I have 2 ESX servers for management, running the vCenter, my firewall cluster, my NSX Manager, ALB Manager, Unraid and my backup server with Veeam. For my lab I have 3 Intel NUCs running ESX virtual servers which then have the NSX workload. These are only started when needed. If I want to test NSX, the ESX hosts for NSX come up, if I need my Tanzu or ALB Lab, I start other virtual ESX servers. because of the flexibility i use distributed switches everywhere and because i am lazy to type.

[TryllZ]

Interesting setup, I use 1 server and 1 switch to keep cost down.

I added VM to Edge_UL2 (VLAN26) and made ping test as well, interestingly, the VM can reach the firewall interface.

Edge has nothing in ARP table for Edge 2nd Uplinks

However, I reset firewall table states and suddenly both Edge node's second interfaces (10.10.26.101, 10.10.26.102) were able to reach firewall interface, and even after this there was no entry in the ARP table for both of these interfaces.

And when the edge nodes' 2nd uplink was able to reach the firewall, the 1st interface (VLAN 25) which was pinging before was not pinging.

Not sure I understand why this is the case.

Could it be due to NSX-T and Edge versions, or the HA for Edge setup in T0 Gateway (point being only 1 interface is active at a time) ?

I'm using NSX-T 4.0.1, Edge versions are also 4.0 something. The reason I'm bringing this is because I read on several sites that the way uplinks and their relevant VLANs were configured were changed between version 2.x and 3.x, unsire if 3.2 and 4.0.1 have similar ways. The Design guide is still 3.2

[TryllZ]

And after all reboot, its back to the same issue of not pinging VLAN 26 interfaces.

[DanielKrieger]

Even if the T0 is active/passive, the second interface must be pingable. What you describe sounds like a Mac learning problem. Have you checked the security settings of all port groups?

Was your test VM able to reach the Edge VM?
Did you have the TestVM on the same ESX virtual host as the Edge Node?
What does the ARP table on the switch look like? Are the Mac addresses of the edge there?

The edge nodes should always have the appropriate NSX version to the manager and co. The way the uplinks are configured has not really changed from 3.2 to 4.X.

[TryllZ]

What you describe sounds like a Mac learning problem

Likely a firewall problem then.

Have you checked the security settings of all port groups?

All settings are default, only Edge Uplink portgroup settings were changed to Allow, will change all portgroups settinga to Allow.

Was your test VM able to reach the Edge VM?

No

Did you have the TestVM on the same ESX virtual host as the Edge Node?

No, will test this and above.

I will add though that I did test a traceroute in Edge and traffic leaves through its Management interface which is in 10.10.15.0 network.

What does the ARP table on the switch look like? Are the Mac addresses of the edge there?

Its a very basic switch.

The edge nodes should always have the appropriate NSX version to the manager and co.

Where can this be checked, matching versions of NSX-T with Edge. To my knowledge Release Nodes are only for NSX-T and no matching Edge version information is there.

[TryllZ]

---

The edge nodes should always have the appropriate NSX version to the manager and co.

---

Found this list, I'm using the right versions, highlighted in Red.

VMware NSX-T Component Build Matrix | virten.net

Name Main Build Manager Controller Edge Kernel Module L2 VPN Client Standalone Edge

NSX 4.1.0.2	21761691	21761695	21761699	21761693	21761699
NSX 4.1.0	21332672	21332677	21332680	21332675	19300606
NSX 4.0.1.1	20598726	20598732	20598735	20598730	19300606
NSX 4.0.0.1	20159689	20159694	20159697	20159692	11197779

[TryllZ]

Looks like a firewall related issue, I did a traceroute

edge1> traceroute 10.10.26.102
traceroute to 10.10.26.102 (10.10.26.102), 30 hops max, 60 byte packets
 1  10.10.15.1  5.577 ms  5.385 ms  4.633 ms
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  *^C

The firewall is sending ARP Broadcast request trying to find the IP address.

Sender IP 10.10.26.1 is the firewall interface.

I'm going to check about this on the OPNSense forum, see what they have to say.

Appreciate all the help @DanielKrieger 

[DanielKrieger]

the question is, why would the edge node go through the firewall at all? You do the traceflow from your edge and the edge should have an IP in the 26 network. So there is no reason to go through the firewall. It also looks to me that you are not in the VRF context of the SR T0 and then it is clear that the Edge is trying to communicate with the 10.109.26.102 via the MGMT interface.

[TryllZ]

Will run a Traceflow again.

Both Edge nodes' both interfaces (10.10.25.101, 10.10.25.102, 10.10.26.101, 10.10.26.102) are configured for BGP Peering with the both firewall's interfaces (10.10.25.1, 10.10.26.1) which is only not happening on the 26 interface.

True, I id not check any of this from the VRF in T0, will check this.

[TryllZ]

From within VRF I can ping all interfaces from 1 edge node to another, still can't ping firewall interface.

edge1(tier0_sr[2])> ping 10.10.26.1
PING 10.10.26.1 (10.10.26.1): 56 data bytes
36 bytes from 10.10.26.1: Destination Host Unreachable
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
 4  5  00 0054 0000   0 0000  40  01 3230 10.10.26.101  10.10.26.1

36 bytes from 10.10.26.1: Destination Host Unreachable
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
 4  5  00 0054 0000   0 0000  40  01 3230 10.10.26.101  10.10.26.1

Traceflow

[DanielKrieger]

Hi, sorry I'm just getting back to you, but I had to rewire my entire lab first. Anyway - I tried to reproduce your problem, but I didn't manage to do so. The whole thing works for me. You could also give your firewall virtual interfaces in the uplink networks and tag them via the portgroup. Then the traffic would be untagged and any problems with double tagging would be solved. But then you have to change your peering a bit and need 2 nvds on the edge VMs. But this works without problems in nested labs.

[TryllZ]

No worries,

Appreciate you taking the time to reproduce the problem.

Currently I'm going through the Edge logs, and will make it simpler and see where the issue is.

Thanks again @DanielKrieger 

[TryllZ]

Thanks @DanielKrieger 

I set the Edge Uplink portgroups to trunking.

And firewall ARP table now has the interface attached.

Now both interfaces are in Established state, and BGP peering on all Edge Interfaces successfully.

edge1> vrf 2
edge1(tier0_sr[2])> get bgp neighbor summary
BFD States: NC - Not configured, DC - Disconnected
            AD - Admin down, DW - Down, IN - Init, UP - Up
BGP summary information for VRF default for address-family: ipv4Unicast
Router ID: 10.10.25.101  Local AS: 65000

Neighbor                            AS          State Up/DownTime  BFD InMsgs  OutMsgs InPfx  OutPfx

10.10.25.1                          65555       Estab 00:12:58     UP  46      20      12     4
10.10.26.1                          65555       Estab 00:12:58     UP  46      20      12     14

Thu Aug 24 2023 UTC 17:54:55.772

edge2> vrf 1
edge2(tier0_sr[1])> get bgp neighbor summary
BFD States: NC - Not configured, DC - Disconnected
            AD - Admin down, DW - Down, IN - Init, UP - Up
BGP summary information for VRF default for address-family: ipv4Unicast
Router ID: 10.10.25.102  Local AS: 65000

Neighbor                            AS          State Up/DownTime  BFD InMsgs  OutMsgs InPfx  OutPfx

10.10.25.1                          65555       Estab 00:15:18     UP  48      23      12     12
10.10.26.1                          65555       Estab 00:15:18     UP  51      23      12     6

Thu Aug 24 2023 UTC 17:57:02.232

The issue is OPNSense VLAN interfaces cannot be created without tags, or cannot be set as 0.