Im dumb, somehow I thought it was OK to have the VSAN's configured as DHCP. I do not know why exactly resetting the fw/router broke VSAN 2 switches away, but thats the way she goes.
To make matters more difficult my Photon machine was in that VSAN.
I have since re-ip'd the VSAN's on each host. I can ping each vmnic, but they just are not communicating. I have looked for relevant logs but dont see anything.
This is a flat network, no vlans, fw's open for lan.
Attached are what I believe to be relevant pictures.
Thanks for your time
Is this a vSAN 6.5 0d or later cluster?
If so, check the Unicast agent list and repopulate it if necessary if the IPs have changed (or are incorrect)
# esxcli vsan cluster unicast list
This should only contain the information of all other hosts in the cluster, not the host you are checking this on.