Accepted Solutions
The reason why not flushed a non technical explanation would be because it is better than nothing, because the DLR instances don't have control plane on themselves, but rely on Control VM for learning routes.(or use Control VM as the control plane). So it may be a better choice to continue using these routes than flushing them and dropping the packets for 15 seconds until new routes are learned from the secondary control VM. Also I I think there's no signalling mechanism so that the DLR instances are aware if Control VM is up or down.Instead of talking directly to the DLR instance, Control VM uses the netcpa process, which in turn uses the Controller Cluster to update, so it may not be possible to learn this.
Would be good if there is more technical explanation, but according to understanding it is by DLR design and not configurable or changeable.
Some routing protocols such as RIP may depend on continuous refresh of the routing entries on the routing table by the routing process, but other protocols such as OSPF and BGP needs an event that creates a topology change, so they may stay long time as days months if the network is stable: (For DLR, OSPF and BGP are used and both are event driven).
https://supportforums.cisco.com/discussion/11602701/routing-table-time-last-update
"You may notice that for event-driven protocols such as OSPF, IS-IS, EIGRP or BGP, the time may go up to days, weeks, even years if the network is that stable. For timer-driven protocols like RIP, this timer should always show a value lower than the Update interval (30 seconds by default for RIP), as in these protocols, the route is essentially reinstalled into the routing table each time an update arrives that confirms the existence of this network."
http://www.routetocloud.com/2014/06/nsx-distributed-logical-router/#DLR_Control_VM_communications
On this basis the routing update happens in the following manner:
- Step (1) DLR Control VM learn new route information (from the dynamic routing as an example) to update the NSX-v controller,
- Step (2) the DLR will use the internal channel inside the ESXi01 host called the “Virtual Machine Communication Interface” (VMCI). VMCI will open a socket to transfer learned routes as Routing Information Base (RIB) information to the netcpa service daemon.
- Step (3) The netcpa service demon will send the RIB information to the NSX-v controller. The flow of routing information passes through the Management VMkernel interface of the ESXi host, which means that the NSX-v controllers do not need a new interface to communicate to the DLR control VM. The protocol and port used for this communication is TCP/1234.
- Step (4) NSX Controller will forward the DLR RIB to all netcpa service daemons on the ESXi host.
- Step (5) netcpa will forward the FIB’s to the DLR route instance.
Also remember that if your next hop from the DLR upstream is an ESG in HA mode and your running dynamic routing between the ESG and the DLR, you need to increase your protocol timeout timers to be larger than the HA failover times. Doing this should allow you to tolerate a complete failover of the DLR control VM with no data plane outage. Remember to enable graceful restart on the DLR Control VM 
If you using ECMP between the Upstream ESG and DLR, then you need to decrease the routing protocol timers, but also remember that you need to keep the ESG advertising the networks so they can keep attracting traffic and forwarding to the protocol forwarding address. Usually this is a floating static supernet route which is then redistributed via the ESG.
Check out the Design Guide at the following link as this goes some way to explaining how to cater for DLR Control VM failures.
VMware® NSX for vSphere Network Virtualization Design Guide ver 3.0
Dale
If there is no static routes configured in DLR Control VM, then the DLR Instances on the ESXi hosts have no way of learning these static routes as the DLR Control VM has the role of updating the routes to DLR kernel module instances. Most of the designs a single default route pointing to the inside interface of the Edge Gateway is sufficient. Even Ospf is used, generally the next-hop woud still be this interface.
From the NSX 6.2 Troubleshooting Guide page 76 routes on any specific host may be observed with the following command (or net-dvr command on the ESXi host CLI as next link)
https://pubs.vmware.com/NSX-62/topic/com.vmware.ICbase/PDF/nsx_62_troubleshooting.pdf
http://chansblog.com/6-nsx-distributed-logical-router/
http://chansblog.com/tag/net-vdr/
nsxmgr# show logical-router host hostID dlr dlrID route
Page 90 of the Troubleshooting document explains the scenario if the DLR Control VM is lost, the DLR would start to use the static default route, (if tested the Number of Routes on the net-vdr command would decrease to 1 which is the default route)
while the Edge would start to use the Summary Static route with higher administrative distance even the ospf routes are withdrawed, and the upstream physical routers would use the redistributed static summary route pointing to the outisde interface of the Edge Router.
NSX Routing Subsystem Failure Modes and Effects:
This chapter reviews the typical failure scenarios that might affect components of NSX routing subsystem and outlines the effects of these failures.
DLR Control VM is lost or powered off n Create, update, and delete operations for this DLR’s LIFs and routes fail n Any dynamic route updates will not be sent to hosts (including withdrawal of prefixes received via now broken adjacencies) DLR Control VM loses connectivity with the NSX Manager and Controllers
Same effects as above,
Although best way is to avoid the loss of DLR Control VM with HA and Anti-Affinity mechanisms, if understood correctly the question is " what would happen to the routing tables on the DLR instances on ESXi hosts if b Both DLR VMs are down when OSPF is used?
If the Design is ECMP based for throughput increase instead of Active-Standby Edge which is single next-hop for DLR, then dynamic routing protocols are necessary.
The 2 default routes are inserted into the routing table of the DLR by the Edge through OSPF Routing Protocol, which is in turn distributed to the DLR Instances on the ESXi hosts. Since this is ECMP Scenario, different Flows may choose different next-hops pointing either to .2 or .3. Since there are 2 next-hops, not sure without testing 2 distinct static default routes with higher administrative distance would achieve a backup mechanism. (may not be recommended due to possibility of losing the next-hop edge which may create problem of losing half of the flows since there is no mechanism as ip sla of understanding the next-hop edge is lost ).
Worst case may be again configuring single static route for one the Edges, but this again Anti affinity rules are needed for minimizing the risk of losing the Control VM and this next-hop Edge at the same time. If the ESXi host is lost, even if the Edge would power on on another ESXi host through Vmware Vsphere HA Mechanism, it may be tested if this time is sufficient for Applications.
So for ECMP scenarios, best practices for increasing the Availability of DLR Control VM with HA(High Availability) and avoiding the possibility of losing Edge and Active Control VM at the same time would decrease OSPF failure to a very minimum, which may be sufficient for most of the cases. Affinity rule for DLR VM HA is defined automatically, but recommended seperation of Data Stores of the 2 DLR VMs with Storage DRS manually, because it is not automatic as of 6.2.
https://fojta.wordpress.com/tag/ecmp/
"The other consideration is placement of DLR Control VM. If it fails together with one of ECMP Provider Edges the ESXi host vmkernel routes are not updated until DLR Control VM functionality fails over to the passive instance and meanwhile route to the dead Provider Edge is black holing traffic. If we have enough hosts in the Edge Cluster we should deploy DLR Control VMs with anti-affinity to all ECMP Edges. Most likely we will not have enough hosts therefore we would deployed DLR Control VMs to one of the compute clusters. The VMs are very small (512 MB, 1 vCPU) therefore the cluster capacity impact is negligible"
"The design guide speaks to some of this in table 10 of the 3.0 guide. ESG and DLR Control HA enables anti-affinity automatically but ECMP needs enabled manually."
http://www.routetocloud.com/2014/06/nsx-distributed-logical-router/#DLR_High_Availability
The High Availability (HA) DLR Control VM allows redundancy at the VM level. The HA mode is Active/Passive where the active DLR Control VM holds the IP address, and if the active DLR Control VM fails the passive DLR Control VM will take ownership of the IP address (flip event). The DLR route-instance and the interface of the LIFs and IP address exists on the ESXi host as a kernel module and are not part of this Active/passive mode flip event.
The Active DLR Control VM sync-forwarding table to secondary DLR Control VM, if the active fails, the forwarding table will continue to run on the secondary unit until the secondary DLR will renew the adjacency with the upper router.
http://www.routetocloud.com/2014/12/nsx-edge-and-drs-rules/
An Edge and DLR that belong to the same tenant should not run in the same ESXi host:
The reason why not flushed a non technical explanation would be because it is better than nothing, because the DLR instances don't have control plane on themselves, but rely on Control VM for learning routes.(or use Control VM as the control plane). So it may be a better choice to continue using these routes than flushing them and dropping the packets for 15 seconds until new routes are learned from the secondary control VM. Also I I think there's no signalling mechanism so that the DLR instances are aware if Control VM is up or down.Instead of talking directly to the DLR instance, Control VM uses the netcpa process, which in turn uses the Controller Cluster to update, so it may not be possible to learn this.
Would be good if there is more technical explanation, but according to understanding it is by DLR design and not configurable or changeable.
Some routing protocols such as RIP may depend on continuous refresh of the routing entries on the routing table by the routing process, but other protocols such as OSPF and BGP needs an event that creates a topology change, so they may stay long time as days months if the network is stable: (For DLR, OSPF and BGP are used and both are event driven).
https://supportforums.cisco.com/discussion/11602701/routing-table-time-last-update
"You may notice that for event-driven protocols such as OSPF, IS-IS, EIGRP or BGP, the time may go up to days, weeks, even years if the network is that stable. For timer-driven protocols like RIP, this timer should always show a value lower than the Update interval (30 seconds by default for RIP), as in these protocols, the route is essentially reinstalled into the routing table each time an update arrives that confirms the existence of this network."
http://www.routetocloud.com/2014/06/nsx-distributed-logical-router/#DLR_Control_VM_communications
On this basis the routing update happens in the following manner:
- Step (1) DLR Control VM learn new route information (from the dynamic routing as an example) to update the NSX-v controller,
- Step (2) the DLR will use the internal channel inside the ESXi01 host called the “Virtual Machine Communication Interface” (VMCI). VMCI will open a socket to transfer learned routes as Routing Information Base (RIB) information to the netcpa service daemon.
- Step (3) The netcpa service demon will send the RIB information to the NSX-v controller. The flow of routing information passes through the Management VMkernel interface of the ESXi host, which means that the NSX-v controllers do not need a new interface to communicate to the DLR control VM. The protocol and port used for this communication is TCP/1234.
- Step (4) NSX Controller will forward the DLR RIB to all netcpa service daemons on the ESXi host.
- Step (5) netcpa will forward the FIB’s to the DLR route instance.