We have a hosting environment wherein NSX is our network edge. Each tenant receives a Tier 0 and Tier 1 behind another Tier 0 we use for aggregation prior to reaching the physical network. ( Provider Tier 0 <> Tenant Tier 0 <> Tenant Tier 1 [BGP throughout])
Some of our tenants have requested access to each others environments (shared research and technologies), and so we setup IPsec tunnels between the Tenant Tier 0s. These secure and route the traffic very well.
The problem is stability. About every 3-4 days, the tunnels will drop and not restore. Looking through the logs doesn't provide any significant errors, only target/destination unreachable. When this occurs we simple reboot one of the Tenant Edges in the cluster and the tunnels immediately come up.
Anyone know why or where to look for the tunnels dropping out like this? It doesn't happen on Edge to physical or third party endpoint. Only on our Edge to Edge tunnels.
