UDLR with multiple default routes using locale ID

[kurtisvm]

Using NSX 6.2.4.4292526 and vCenter 6.0.0.4632154, I have a cross vcenter setup and I want to use local egress through 2 different ESGs for the different sites.  I created the 2 ESGs at the different sites and I've created 2 transit networks named PrimaryTransit and SecondaryTransit the ESGs from each sites have an interface on the associated transit network.  I have a UDLR with local egress enabled and it has an interface on both Transit networks.  I want to setup static routing so that traffic from the locale ID of my primary site goes out via the ESG on my PrimaryTransit network and traffic from VMs on my secondary site goes through the ESG on my SecondaryTransit network.

I don't have default gateway defined on the "Global Configuration" tab of the UDLR.  Instead I've try to add 2 0.0.0.0/0 routes on the "static routes" tab and specify different locale IDs for the different static routes.  I'm able to add the first 0.0.0.0/0 static route and specify the primary ESG as next hop and the LocaleID from my primary site.  However when I add the second 0.0.0.0/0 gateway to use my secondary ESG as next hop, I see this error when I hit publish changes:

[Routing] Adding multiple routes for same network 0.0.0.0/0 is disallowed when ECMP is disabled.

Seems like I should be allowed to do this since each static route has a different Locale ID.

If I try to enabling ECMP and setting up the same two static routes, I'm given this error when creating the second 0.0.0.0/0 static route on my secondary transit interface:

[Routing] For configuring ECMP, vnic must not be specified in routes. For network 0.0.0.0/0, please remove the vnic 3 input.

If I try creating the second 0.0.0.0/0 static route and not specifying an interface (still specifying locale ID of second site), when I hit publish the 2 routes get combined into one route, with the IP addresses of both ESGs listed as the next hop, but only the locale ID which is that of the first site.  I then tried removing the locale ID and still having ECMP enabled with both Edge appliances as the next hop.  VMs on my primary site could route out through my Primary ESG.  But VMs on my secondary site are not routing out as expected.

Any input is much appreciated.  Thx.

[chuckbell]

Did you deploy 2 uDLR control VM's? One for each site/NSX Mgr?

Check out page 123 of the Multi-Site guide if not. Walks you thru it.

NSX-V Multi-site Options and Cross-VC NSX Design Guide

[kurtisvm]

Thx for a response.  I did try deploying a Universal Control VM on the secondary site, but that doesn't seem to help when using static routes.  That would probably work if using dynamic routing but I don't think it works for static routing.  In fact, from page 120 it's stated that "Figure 120 below shows where Locale ID can be changed at the static route level. Note, Locale ID at a static route level can only be set if there is no Universal Control VM deployed. If a Universal Control VM is deployed, the text box to set the Locale ID is grayed out.".  I've confirmed that I can't specify Locale ID if any Universal Control VM is deployed, so I'm still not certain how to achieve active-active locale egress using static routing ;(

[chuckbell]

Sorry. Missed the static route part.

Are you using a single vCenter?

Does your design look like this?

[kurtisvm]

No, I have 2 vcenters joined together in enhanced link mode.  With a primary and secondary NSX manager.  It looks very much like this diagram:

VMware NSX for vSphere 6.2 Documentation Center

But I don't have Universal Logical Router appliances (aka Universal Control VMs) because I'm trying to use static routing.

[chuckbell]

I know there was a workaround as on page 128 for static routes and single VC (using API to add static and default routes for each site).

Page 128:

- Deploy a UDLR instance with Local Egress enabled and no Edge Appliance

- Enable ECMP on the UDLR

- A NSX REST API call must be made to configure a default gateway via the first Locale ID

- A NSX REST API call must be made to configure a static route for 0.0.0.0/0 via the second Locale ID

I tried this with 2 VC's and was not able to get it to work.

I got this error: 'edge appliance needs to be deployed'

The guide does not state this can be done with 2 VC and static routes, only single VC and static routes.

Not 100% sure it is supported with static, maybe only dynamic.

[kurtisvm]

Seems like NSX should support this with static routing, but since neither of us could get it to work I guess I'm forced to use dynamic routing.  I'll give that a try.  Thank you very much for your input on this topic!