6 Replies Latest reply on Apr 23, 2019 11:24 PM by DonalB

    Only two of four BGP paths being populated in routing table on DLR

    DonalB Hot Shot

      Hi ,

      We have an NSX DLR configured in a BGP peering relationship with 4 x NSX ESGs (2 x ESGs in one Datacenter and 2 x ESGs in another datacenter), with ECMP mode enabled on the DLR and each ESG. The 4 x NSX ESGs are in peering relationships with upstream physical routers local to them in the their respective datacenters.
      We are advertising routes from the physical network to the ESGs, including the default route , and these routes are being advertised to the DLR from the ESGs.
      Our challenge is that we are expecting to see 4 x instances of a route, one from each ESG, at the DLR however we only see 2 x routes when running sh ip route, and these are the routes from one datacenter only. We do not have any specific preferences set for BGP and filtering is minimal and configured the same on all ESGs, also the AS-Path is the same length on each route when viewed in the BGP outputs on the DLR (i.e. sh ip bgp)

       

       

       

      Thanks

      DB

        • 1. Re: Only two of four BGP paths being populated in routing table on DLR
          Sreec Master
          Community WarriorsvExpert

          Hello DB,

                           1)Each ESG is certainly showing 4x subnets , is that correct ?

                           2)Do you have reachability from missing subnets(2x) to workloads behind DLR or vice versa keeping the routing table issue aside ?

                           3) You have mentioned filtering is minimal - however I'm interested to know the actual configuration

          • 2. Re: Only two of four BGP paths being populated in routing table on DLR
            DonalB Hot Shot

            Hi Sreec,

             

            Thanks for replying,

             

                             1)Each ESG is certainly showing 4x subnets , is that correct ?

                                --> Each ESG shows 2 paths to networks north of the ESGs published from it's upstream physical peer as expected

                                --> the DLR shows in BGP a path to each Northbound subnet advertised from the 4 x ESGs

                             2)Do you have reachability from missing subnets(2x) to workloads behind DLR or vice versa keeping the routing table issue aside ?

                                --> yes, have tested this by overriding BGP with static routes

                             3) You have mentioned filtering is minimal - however I'm interested to know the actual configuration

                                --> on ESGs out direction; we deny for the subnets that the ESGs and physical routers peer over, and permit any

                                --> on DLR for in direction; we deny for the subnet behind the DLR (this is to prevent any routing loops), and permit any

             

            I'm attaching a diagram in case it helps

            BGP with ECMP - 180419.png

            Tks

            DB

            • 3. Re: Only two of four BGP paths being populated in routing table on DLR
              Abhishek_Soni Lurker
              VMware Employees

              Even before you check physical routing, can you see if you are getting ECMP routes for all 4 Edge's connected interfaces (towards physical and redistributed to BGP).

              • 4. Re: Only two of four BGP paths being populated in routing table on DLR
                DonalB Hot Shot

                I had checked with the networking team on this and they had confirmed that they did see this. I didn't think it would be something to worry about however as the DLR to the south of the edges is where I'm only getting 2 of the 4 routes pushed into the routing table

                • 5. Re: Only two of four BGP paths being populated in routing table on DLR
                  Sreec Master
                  vExpertCommunity Warriors

                  Thanks for sharing the topology . Since the issue is specific to DC2 Edge Routes not showing in DLR . I would also like to know underlying vSphere design

                   

                   

                  1. Are these stretched clusters by any chance ?

                  2. The workload subnets which are behind DLR , is it showing under both the ESG routing tables (DC1 & DC2) - In your case there is no outbound filtering for DLR - so I'm expecting , ESG should show those routes or you have a bidirectional routing problem with DC2

                  3. Run a bgp debug specific to DLR interfaces which is peered with DC2 ESG and please do share the results

                  1 person found this helpful
                  • 6. Re: Only two of four BGP paths being populated in routing table on DLR
                    DonalB Hot Shot

                    Apologies for the delay in replying, I had some PTO the last few days. I had raised an SR with support around the same time I posted last week and have gotten feedback yesterday to say that what we have currently setup will not work as BGP on the ESGs evaluates the ASN also for ECMP so if different ASNs are used only the paths with the lowest ASN will be used. Resolution is to use the same ASN , KB article here outlines this:

                    VMware Knowledge Base

                     

                    Bit of a surprising one, need to see if I can get this configuration on the physical side to confirm this works for us.

                     

                    Cheers

                    DB

                    1 person found this helpful