Posting to document an issue I encountered while updating to VMware Workstation Pro v15.5.6 from v15.5.5 which was not trivial to track down.
Now that VMware Workstation is compatible with Hyper-V and features which rely on it (e.g. Device Guard, Credential Guard, etc ...) it's possible to have those features installed side-by-side and use them in tandem. One of those features is Windows Defender Application Guard (WDAG). The WDAG container (a lightweight Hyper-V VM) maintains open handles to many driver catalog files located at C:\Windows\System32\CatRoot. This includes catalog files for VMware drivers which need to be updated during the upgrade process.
The result is the upgrade process will fail with any of several possible messages. I personally witnessed errors referencing vsock (as a pop-up dialogue during upgrade), but after the upgrade failure and subsequent rollback, only saw generic failure messages without any pop-ups on subsequent attempts. In both cases the issue was due to driver catalog files being open by the WDAG container.
The workaround is fortunately simple but not obvious. Simply stop the Application Guard Container Service (hvsics) before attempting the upgrade. This will stop the WDAG VM process maintaining the open file handles. Once the upgrade process is complete the service can be safely restarted.
I'm not sure if the VMware Workstation Pro installer can better handle this case, but if not, it probably at least merits a reference as a Known Issue in the release notes to save others potentially a lot of trouble.
Thank you for posting the results of your thorough investigation!
Do you estimate that this problem may occur on a fresh installation of 15.5.6 too or would it happen only when updating from 15.5.5 (or even from an earlier version) to 15.5.6 or from 15.5.6 to a future version of WS?
Also, in what OS version(s)/build(s) did you detect the problem?
Finally, do you consider this a WS bug or a WDAG bug?
Regarding your Qs:
In all cases the answers above assume the underlying system has the WDAG feature enabled. If not, the circumstances to encounter the issue won't be present.
Thank you - while you were writing your reply, I was continuously editing my questions - please re-read them... I woke up early today and had inspiration
Regarding your additional two Qs:
I think that all the info you provided is invaluable. According to System requirements for Microsoft Defender Application Guard (Windows 10) - Windows security | Micro... , WDAG can also be installed to Win10 Pro and Education, so statistically it is expected to see reports from these users too.
Since WMware and MS are keeping an open channel of communication we expect they will eventually sort this out.
However, I expect that any program that installs its own drivers (e.g. a virtual DVD drive or an advanced audio software) as well as hardware driver installation programs themselves may be facing their own problems too, so maybe this thread in the future will appear in net searches, informing users that these may be related to a recent WDAG installation they did.