What's your FTT setting for the vms that are shown as disconnected or inaccessible?
"when all hosts are shut down, we power on secondary fault domain host and witness the vms didnt restart. all vms are disconnected or inaccessible.
there ist no way to get the vms back online.
How did you shut down the hosts and were the VMs on the cluster powered-on when this was performed? (e.g. hard drop/MM with No Action etc.)
If the VMs were on, are you positive that the node on Secondary site didn't go down first thus making the data on this node technically stale? (as it missed the writes Primary site committed)
This could be very easily clarified by checking the state of the data under these conditions, e.g. with just Witness and Secondary available all/most of the DOM-Objects would likely have a Config-Status of 28 or some other non 7/15 state indicating stale. This can be checked using:
# cmmds-tool find -f python | grep CONFIG_STATUS -B 4 -A 6 | grep 'uuid\|content' | grep 'state\\\":' | sort | uniq -c
If the data was cold when this occurred then it is likely a case of Preferred site selection - if the data is not stale and you can reproduce this, try changing the Preferred Fault Domain to the Secondary site and test if you can power-on VMs.
I am assuming you have no site affinities or 'Must-run' rules applied here, do check this if not sure.
Thank you all and happy christmas!
After doing only one failover test after the other, and with patience, and delay between them, everything works fine (and automatically)