When designing af stectched cluster vSAN, with the witness placed at a third site, I have some considerations regarding the network design for this.
Data sites are L2 adjacent, and witness site are reachable via routing.
Both data sites and witness sites are connected to ISP MPLS via BGP dynamic routing. So by default all traffic going to the witness, would go via the primary data site...
now the scenario I am concerned about, is what if my primary site goes down (that is, network, vSAN nodes etc.) - then my secondary site would need to wait for BGP convergence, before being able to reach the witness site...
How long would vSAN tolerate this witness outage? and what is the appropiate way of configuring this? maybe multiple vmk interfaces in different subnets, and then BGP traffic engineering?
Please help me choose the best solution to this or share any experiences with similar setups