VMware Networking Community
MasterWayZ
Enthusiast
Enthusiast
Jump to solution

NSX-T BGP neighbor down "There are some unknown runtime issues."

Recently I have switched from NSX-V to NSX-T.

After deploying the Tier 0 Router and setting up the BGP neighbor of the main firewall, they do not connect.

When I look at the BGP neighbor status in the NSX-T console, it reads DOWN with this error: "There are some unknown runtime issues."

The main firewall is pfSense and the NSX-T version is 2.5

Anything that I am doing wrong or did something bug out?

This is the configuration on the Tier 0 Router:

firefox_XOwV1BSpWR.png

firefox_CBFwfKZ1dM.png

And on the pfSense firewall (with frr):

5T555nW8SL.png

firefox_8HXjLZQSJO.png

firefox_SpMhBYjnXL.png

vExpert 2020
Tags (3)
1 Solution

Accepted Solutions
MasterWayZ
Enthusiast
Enthusiast
Jump to solution

Final update:

It now works. pfSense configuration was correct. The solution was to remove the ESXi host from the Uplink N-VDS switch it seems like.

vExpert 2020

View solution in original post

Reply
0 Kudos
12 Replies
Sreec
VMware Employee
VMware Employee
Jump to solution

Do we have point to point connectivity between Tier-O and pf Sense ?

Please do share respective VRF routing table and BGP neighbour summary from Tier-O and pfsense global routing table , i can see your are trying a EBGP connection , If you are unsure about BGP debug & validation , please do change the AS and make it same on both the sides(65950) for time being.

Cheers,
Sree | VCIX-5X| VCAP-5X| VExpert 6x|Cisco Certified Specialist
Please KUDO helpful posts and mark the thread as solved if answered
Reply
0 Kudos
MasterWayZ
Enthusiast
Enthusiast
Jump to solution

I can ping the Tier 0 IP address from the pfSense router, and I can ping from the Tier 0 Router to pfSense.

(I changed both AS numbers to 65950 but it did not change anything; do note that on the pfSense router, the State changes from Connect to Active and back every now and then)

This is what I see on the pfSense router:

IPv4 Unicast Summary:

BGP router identifier 192.168.20.1, local AS number 65950 vrf-id 0

BGP table version 6

RIB entries 9, using 1440 bytes of memory

Peers 1, using 13 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd

192.168.20.3 4 65950 0 0 0 0 0 never Active

Total number of neighbors 1

Routing table:

Codes: K - kernel route, C - connected, S - static, R - RIP,

  O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,

  T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,

  F - PBR,

  > - selected route, * - FIB route

K>* 0.0.0.0/0 [0/0] via 94.x.x.1, em0, 00:03:12

K>* 46.4.x.x/32 [0/0] via 94.x.x.1, em0, 00:03:12

C>* 94.x.x.0/23 is directly connected, em0, 00:03:12

S 172.27.224.0/20 [1/0] via 192.168.254.146, em1, 00:03:12

K>* 172.27.224.0/20 [0/0] via 192.168.254.146, em1, 00:03:12

C>* 192.168.20.0/24 is directly connected, em2, 00:03:12

C>* 192.168.254.0/24 is directly connected, em1, 00:03:12

About showing the table of the Tier-0 router, I don't know how I can access it and run the commands on that you are asking for.

Could you explain that to me?

vExpert 2020
Reply
0 Kudos
MasterWayZ
Enthusiast
Enthusiast
Jump to solution

I have done some additional troubleshooting.

I ran tcpdump on the pfSense interface that faces the Tier 0 Router, and I see that BGP packets are being send to the Tier 0 Router, however, no packets are being received.

I can however, ping the Tier 0 router.

vExpert 2020
Reply
0 Kudos
mauricioamorim
VMware Employee
VMware Employee
Jump to solution

Do you have a gateway firewall configured on the T0?

Reply
0 Kudos
MasterWayZ
Enthusiast
Enthusiast
Jump to solution

I'm not sure what you mean.

If you mean that if I have a default gateway set on the Tier 0 Router, then yes, I do have that set to the pfSense router IP of the interface facing it. (192.168.20.1)

vExpert 2020
Reply
0 Kudos
MasterWayZ
Enthusiast
Enthusiast
Jump to solution

Update:

It seems to be an issue with the config on the pfSense side. When I set up frr on a Ubuntu Server VM and use BGP on that, it works.

vExpert 2020
Reply
0 Kudos
MasterWayZ
Enthusiast
Enthusiast
Jump to solution

Final update:

It now works. pfSense configuration was correct. The solution was to remove the ESXi host from the Uplink N-VDS switch it seems like.

vExpert 2020
Reply
0 Kudos
LilleCarl
Contributor
Contributor
Jump to solution

Hi MasterWayZ​!

Could you share a bit more information please? I've got the same issue.

What do you mean by "ESXi host from the Uplink N-VDS switch", are you not running ESXi mgmt over n-vds anymore?

Thanks in advance!

// Carl

Reply
0 Kudos
MasterWayZ
Enthusiast
Enthusiast
Jump to solution

Hi,

I meant to only have the ESXi host on the Overlay N-VDS and not on the VLAN one.. That fixed it for me.

vExpert 2020
Reply
0 Kudos
LilleCarl
Contributor
Contributor
Jump to solution

I assume you're not running NSX-T2.5.1 then? Because my setup only has 2pnics (fully collapsed), both attached to the same n-vds, so i can't really take the hosts off any n-vds.

Reply
0 Kudos
MasterWayZ
Enthusiast
Enthusiast
Jump to solution

When I made this post I believe I was running NSX-T 2.5

What I had was I had multiple NICs on my server. One was attached to a regular port group with the normal management vmknics and some VMs on it. The other NICs I assigned to the Overlay N-VDS. I'm not sure if something important changed in the .1 release, I don't think it has. If you'd like I can deploy it in a lab and document setting it up so you can see how I did it if that helps. Or if you have any questions feel free to ask.

vExpert 2020
Reply
0 Kudos
LilleCarl
Contributor
Contributor
Jump to solution

Okay so I've done my fair share of messing about with this and has reached a conclusion:

This error message simply means that the BGP peer isn't established, as in that the BGP isn't configured on the other side/misconfigured in any way.

On any other router software this would simply be a "state connecting", but here it throws and error to set you down the wrong path.

If you want more debug information:

  1. ssh into the edge gateway (ssh admin@edgegateway)
  2. get logical-router
  3. look for the SERVICE_ROUTER_TIER0 VRF "id" (number displayed in second to left col)
  4. vrf 1 (where 1 is the vrf number of the T0)
  5. get debug bgp

There you'll see information about BGP state that'll be able to help you along your path, hopefully this error message is corrected in the future to not set people down the "fuck idk what i'm doing" path just because they're new working with NSX-T.

MasterWayZ​ Thanks for your assistance and support too! Smiley Happy