Nested hypervisor support under VBS

[ralish]

Are any VMware developers monitoring this forum able to comment on future support for nested hypervisors on systems with Hyper-V (or dependent features like Device Guard)? At the time host VBS support was introduced back in Workstation v15.5.5 my understanding was this feature was missing due to limitations in the Windows Hypervisor Platform API. Is this still the case? Are there any plans to add support and what are the roadblocks to doing so?

If at all helpful as background, my use case is testing VBS configurations in VMs on a host which itself uses VBS, as well as wanting to test ESXi configurations in a VM. Both of these scenarios require Intel VT-X/EPT virtualisation (or AMD-V/RVI for AMD CPUs), which isn't supported with host VBS.

Thanks in advance!

[46terherh]

It looks that Microsoft already provided nested virtualization feature using Hyper-V interface, but I'm not sure whether VMware implemented it.

https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/tlfs/nested-virtualization

[bluefirestorm]

Nested Hyper-V (Hyper-V running inside a Hyper-V VM) has been available for some time.

https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/user-guide/nested-virtualization

But it is one thing to have the feature for Microsoft Hyper-V and it is a different matter altogether for Microsoft to provide an API to let third parties like VMware to provide that capability.

From the link you provided, it looks like the call has to be made within the VM (it is referred to as the L1 hypervisor). I can't seem to find documentation about the Hypervisor APIs. The Hypercall is for the guest VM to call which may or may not be what VMware is using. If it is, effectively the VMware VM effectively is already a nested VM, where

the Hyper-V on host -> VMware Workstation VM monitor Hypervisor API is already a pseudo VM -> runs VMware VM.

[ralish]

I suppose the best way to confirm the current state of affairs is to see if a Hyper-V VM can be launched with VBS enabled in the guest while VBS is enabled on the host. If the answer is no, then it's almost certainly not possible under VMware either when using Hyper-V as the virtualisation backend. If the answer is yes, the question is around if the relevant support is exposed through documented APIs.

[ralish]

Some results from a very quick test using Hyper-V on a Windows 10 v21H1 x64 host w/ VBS enabled. All testing was performed in a Generation 2 VM with a fresh Windows 10 v21H1 x64 installation:

1. Enabling VBS (aka. Core Isolation) worked with no additional changes. All that was required was enabling Core Isolation via the Windows Security app and rebooting for the requisite Windows support to be installed and enabled. I've attached a screenshot from System Information post-reboot showing VBS enabled in the VM.
2. Nested virtualisation also works with a few extra steps. These are documented by Microsoft here. To summarise, you need to enable nested virtualisation for the (outer) VM, disable dynamic memory for the (outer) VM, and enable Hyper-V in the (inner) VM. I was then able to launch a Hyper-V VM inside the guest VM.

So to summarise, it clearly is possible under Hyper-V to use both VBS enabled VMs and nested virtualisation (inc. simultaneously), including on hosts which themselves have VBS enabled. It being technically possible, the next question is does Microsoft expose the necessary public APIs for 3rd-parties to leverage these configurations?

Is anyone from VMware able to comment if such support is on the development roadmap and if there are any major blockers to adding it?

[ralish]

Bumping one time for any input from a VMware employee. Another area this causes problems is network labs using tools like GNS3.