First and foremost, when I ran into issues that were somewhat similar to yours (NSX DLR routing and switching were screwy), the first thing I had to fix was the VTEPs and the controller VM IPs. I'm pretty sure someone will point out or correct me but I couldn't get my NSX environment to function properly when I had the controller IPs and the VTEP IPs on the same subnet as my management VMs (vCenter, ESXi mgmt interfaces, etc...).
The first thing I would check before doing anything would be the "logical switch" testing...if you go into the [network and security section > logical switches > monitor]. I would first test the VXLAN MTU size to ensure there aren't any issues. Then I would perform the broadcast test/s to ensure you get 100% successful results.
If your tests are successful, the next thing I would do is check your ESXi hosts by running the following commands:
(list DLR instances registered with the ESXi host)
*used to capture the "VDR name" for the commands that will follow
# net-vdr --instance -l
(list VDR routes)
# net-vdr --route -l <vdrname>
These commands should show you that your LDR is registered with each ESXi host as it should be and also show routing in case you've moved past the NSX base functionality and are troubleshooting network connectivity (L2/L3).
I tried that, but it did not fix the issue. Still the router connections fail.