ralish
Enthusiast
Enthusiast

VMware Workstation upgrade issue with WDAG

Posting to document an issue I encountered while updating to VMware Workstation Pro v15.5.6 from v15.5.5 which was not trivial to track down.

Now that VMware Workstation is compatible with Hyper-V and features which rely on it (e.g. Device Guard, Credential Guard, etc ...) it's possible to have those features installed side-by-side and use them in tandem. One of those features is Windows Defender Application Guard (WDAG). The WDAG container (a lightweight Hyper-V VM) maintains open handles to many driver catalog files located at C:\Windows\System32\CatRoot. This includes catalog files for VMware drivers which need to be updated during the upgrade process.

The result is the upgrade process will fail with any of several possible messages. I personally witnessed errors referencing vsock (as a pop-up dialogue during upgrade), but after the upgrade failure and subsequent rollback, only saw generic failure messages without any pop-ups on subsequent attempts. In both cases the issue was due to driver catalog files being open by the WDAG container.

The workaround is fortunately simple but not obvious. Simply stop the Application Guard Container Service (hvsics) before attempting the upgrade. This will stop the WDAG VM process maintaining the open file handles. Once the upgrade process is complete the service can be safely restarted.

I'm not sure if the VMware Workstation Pro installer can better handle this case, but if not, it probably at least merits a reference as a Known Issue in the release notes to save others potentially a lot of trouble.

0 Kudos
5 Replies
Mits2020
Hot Shot
Hot Shot

Thank you for posting the results of your thorough investigation!

Do you estimate that this problem may occur on a fresh installation of 15.5.6 too or would it happen only when updating from 15.5.5 (or even from an earlier version) to 15.5.6 or from 15.5.6 to a future version of WS?

Also, in what OS version(s)/build(s) did you detect the problem?

Finally, do you consider this a WS bug or a WDAG bug?

0 Kudos
ralish
Enthusiast
Enthusiast

Regarding your Qs:

  1. I don't expect it would affect new installations as there's no existing driver catalog files at that point for the WDAG container to be locking.
  2. I'd definitely expect it to affect future updates, alongside the existing v15.5.6 update, unless there's some change in WDAG behaviour (via Microsoft) or the installer handling (via VMware).
  3. It could conceivably affect updates from older versions as well, but as the pre-v15.5.5 releases aren't compatible with Hyper-V (and thus WDAG), you'd be updating from a non-functional installation. So possible, but uncommon. Maybe if someone intentionally installed WDAG before updating to v15.5.5+ knowing the latter will fix the incompatibility (i.e. updated both, but in the "wrong" order).

In all cases the answers above assume the underlying system has the WDAG feature enabled. If not, the circumstances to encounter the issue won't be present.

Mits2020
Hot Shot
Hot Shot

Thank you - while you were writing your reply, I was continuously editing my questions - please re-read them... I woke up early today and had inspiration Smiley Happy

0 Kudos
ralish
Enthusiast
Enthusiast

Regarding your additional two Qs:

  1. Windows 10 v2004 Enterprise x64 (Final release; i.e. Build 19041). Only the 2004 release has the support required for Hyper-V interop (excluding Insider builds).
  2. Right now I'm not sure I'd call it a bug in either. I don't know enough about WDAG to say if keeping locks on those catalog files is expected behaviour, and the VMware Workstation Pro installer presumably just isn't expecting other processes to be maintaining references to those files. Updating the VMware Workstation Pro installer to at least check for this case would likely be the simplest path forward (and updating the release notes), but it's not necessarily a bug, just a system configuration that should be handled.
Mits2020
Hot Shot
Hot Shot

I think that all the info you provided is invaluable. According to System requirements for Microsoft Defender Application Guard (Windows 10) - Windows security | Micro... , WDAG can also be installed to Win10 Pro and Education, so statistically it is expected to see reports from these users too.

Since WMware and MS are keeping an open channel of communication we expect they will eventually sort this out.

However, I expect that any program that installs its own drivers (e.g. a virtual DVD drive or an advanced audio software) as well as hardware driver installation programs themselves may be facing their own problems too, so maybe this thread in the future will appear in net searches, informing users that these may be related to a recent WDAG installation they did.

0 Kudos