We are running vSphere 7.0 U1. We planned to upgrade everything to U2.
We have guests that use VMware disk encryption. We have a KMS platform (Dell CloudLink). We have a Storage Policy that applies VMware VM Encryption. The policy is assigned to the VM Home and all disks located on standard datastores.
I upgraded vCenter to U2 first. Immediately after this, all VMs using disk encryption were showing the wrong storage policies, or showing warnings for policies that were on standard datastores.
I then had to shut down a host at one of our branch offices for unrelated maintenance. When the host came back up, all VMs that used disk encrypion were marked as Invalid and could not be unlocked.
I opened a support ticket but VMware has not been able to solve this after several weeks. I ended up manually rebuilding a bunch of servers meanwhile. But now, I just tested this at our primary production site, and I see the same behavior. I've got a huge problem that nobody can solve 😞
I believe the root cause is that the hosts lose the cached KEK. When I query the host using crypto-util, it returns 'NO' for iskeycached. On the KMS platform, we are seeing Key Retrieval Failure. There are no network issues. vCenter reports all green on KMS status. The keys identifed as missing are confirmed to be on the KMS platform, and they have not changed.
Hoping that somebody has an idea on this. Also, if anyone is using disk encryption and upgrades their vCenter, I would assume they would see the same behavior.
