Background -
Have a Gemalto keysecure KMS server that was migrated from a single disk to a mirrored array. Didn't notice that it was not communicating with the cluster any more. Had a host get wonky, reinstalled esxi, preserved the exiting vmfs data store (I did not backup the host config this round however, stupid me!) with the intent of a quick reconfig and move on with life. Drives locked due to lack of comms with KMS.
Have since determined that the KMS wasn't too fond of being copied to the new array, despite booting and operating as what looked to be normal. KMS has now been recovered to it's original state and is confirmed for at least one way trust (vcenter trusts kms right now). I'm unable to get the kms to trust vcenter again though. Have another storage array that was able to communicate fine with the kms again and do it's thing without intervention on my part.
The vSAN cluster has 5 hosts running 6.7, all drives are unlocked with exception of the host that was rebuilt and reintroduced into the cluster. For that host, host encryption mode will not enable - it gives "general runtime error, cannot generate key" when trying. I'm unable to do much with it at this point.
I'm not super savvy in this vsan/encryption realm. My thought is that I were to remove that host and it's locked drives as gracefully as possible from the cluster, that may allow for the two way kms comms restoral with the remaining hosts since no other keys would have changed, at which point the removed host and drives could be reintroduced and encrypted again.
Am I way off base with this my thoughts? Anyone have any suggestions for another path to sort this mess? At this point, the devs using the vm's are working, so that's good.
I'm a bit terrified of restarting anything on the back end at this point until kms trusts are two way again, probably for good reason.
