VMware Networking Community
DonalB
Enthusiast
Enthusiast

Only two of four BGP paths being populated in routing table on DLR

Hi ,

We have an NSX DLR configured in a BGP peering relationship with 4 x NSX ESGs (2 x ESGs in one Datacenter and 2 x ESGs in another datacenter), with ECMP mode enabled on the DLR and each ESG. The 4 x NSX ESGs are in peering relationships with upstream physical routers local to them in the their respective datacenters.
We are advertising routes from the physical network to the ESGs, including the default route , and these routes are being advertised to the DLR from the ESGs.
Our challenge is that we are expecting to see 4 x instances of a route, one from each ESG, at the DLR however we only see 2 x routes when running sh ip route, and these are the routes from one datacenter only. We do not have any specific preferences set for BGP and filtering is minimal and configured the same on all ESGs, also the AS-Path is the same length on each route when viewed in the BGP outputs on the DLR (i.e. sh ip bgp)

Thanks

DB

6 Replies
Sreec
VMware Employee
VMware Employee

Hello DB,

                 1)Each ESG is certainly showing 4x subnets , is that correct ?

                 2)Do you have reachability from missing subnets(2x) to workloads behind DLR or vice versa keeping the routing table issue aside ?

                 3) You have mentioned filtering is minimal - however I'm interested to know the actual configuration

Cheers,
Sree | VCIX-5X| VCAP-5X| VExpert 6x|Cisco Certified Specialist
Please KUDO helpful posts and mark the thread as solved if answered
Reply
0 Kudos
DonalB
Enthusiast
Enthusiast

Hi Sreec,

Thanks for replying,

                 1)Each ESG is certainly showing 4x subnets , is that correct ?

                    --> Each ESG shows 2 paths to networks north of the ESGs published from it's upstream physical peer as expected

                    --> the DLR shows in BGP a path to each Northbound subnet advertised from the 4 x ESGs

                 2)Do you have reachability from missing subnets(2x) to workloads behind DLR or vice versa keeping the routing table issue aside ?

                    --> yes, have tested this by overriding BGP with static routes

                 3) You have mentioned filtering is minimal - however I'm interested to know the actual configuration

                    --> on ESGs out direction; we deny for the subnets that the ESGs and physical routers peer over, and permit any

                    --> on DLR for in direction; we deny for the subnet behind the DLR (this is to prevent any routing loops), and permit any

I'm attaching a diagram in case it helps

BGP with ECMP - 180419.png

Tks

DB

Reply
0 Kudos
Abhishek_Soni
Contributor
Contributor

Even before you check physical routing, can you see if you are getting ECMP routes for all 4 Edge's connected interfaces (towards physical and redistributed to BGP).

Reply
0 Kudos
DonalB
Enthusiast
Enthusiast

I had checked with the networking team on this and they had confirmed that they did see this. I didn't think it would be something to worry about however as the DLR to the south of the edges is where I'm only getting 2 of the 4 routes pushed into the routing table

Reply
0 Kudos
Sreec
VMware Employee
VMware Employee

Thanks for sharing the topology . Since the issue is specific to DC2 Edge Routes not showing in DLR . I would also like to know underlying vSphere design

1. Are these stretched clusters by any chance ?

2. The workload subnets which are behind DLR , is it showing under both the ESG routing tables (DC1 & DC2) - In your case there is no outbound filtering for DLR - so I'm expecting , ESG should show those routes or you have a bidirectional routing problem with DC2

3. Run a bgp debug specific to DLR interfaces which is peered with DC2 ESG and please do share the results

Cheers,
Sree | VCIX-5X| VCAP-5X| VExpert 6x|Cisco Certified Specialist
Please KUDO helpful posts and mark the thread as solved if answered
DonalB
Enthusiast
Enthusiast

Apologies for the delay in replying, I had some PTO the last few days. I had raised an SR with support around the same time I posted last week and have gotten feedback yesterday to say that what we have currently setup will not work as BGP on the ESGs evaluates the ASN also for ECMP so if different ASNs are used only the paths with the lowest ASN will be used. Resolution is to use the same ASN , KB article here outlines this:

VMware Knowledge Base

Bit of a surprising one, need to see if I can get this configuration on the physical side to confirm this works for us.

Cheers

DB