Highlighted
Contributor
Contributor

Fusion 12 Pro No Nested Virtualization

We were assured the GA version of Fusion 12 would support nested virtualization in Big Sur. It does not appear to.

13 Replies
Highlighted
User Moderator
User Moderator

...And Big Sur is still in beta form at the moment!  So I'd wait until it is GA.

0 Kudos
Highlighted
Community Manager
Community Manager

It's more complicated unfortunately.

It turns out the virtualization APIs that were provided to us to support this use case require a specific Intel CPU hardware feature to function. (VMCS Shadowing).

The unfortunate part is that this hardware feature is not commonly found on consumer oriented CPUs. (None of my MacBook Pros support this anymore... I'm staying on Catalina on most of my machines as a result).

We're working with Apple to address this regression, as it would need enhancements to macOS itself.

- Michael Roy - Product Line Manager: Fusion & Workstation
0 Kudos
Highlighted
VMware Employee
VMware Employee

VMware Knowledge Base has more details about nested limitations on Big Sur.

Highlighted
Enthusiast
Enthusiast

My Googling basically brought up stuff from 2013 and 2014 for VMCS Shadowing.  How does one confirm if their Mac's CPUs might have it?

0 Kudos
Highlighted
Community Manager
Community Manager

The easiest way I've found thus far is by looking at the vmware.log of a VM that has VT-x enabled. (this can be done when the VM is running)

From the VM Library, select the VM > right click > hold 'Command' > Click 'Open Latest Log File' (neat little trick...)

From Console.app, search 'vmcs' and you should see an entry like the following:

<timestamp> vmx| I005:   Use VMCS shadowing                       { 0 }

This is my 2019 MBP, and it does NOT have VMCS shadowing.

This is from my 2017 iMac Pro, which does have VMCS shadowing:

<timestamp> vmx| I005:   Use VMCS shadowing                       {0,1}

- Michael Roy - Product Line Manager: Fusion & Workstation
Highlighted
Expert
Expert

I tried this on a late 2013 MBP (Core i7 Haswell) with Catalina 10.15.6 and Fusion 12 Pro, and the output was this (with double entries: {0,1} and { 0 }😞

vmx| I005:   VMCS revision ID                  18

vmx| I005:   VMCS region length              1024

vmx| I005:   VMCS memory type                  WB

vmx| I005:   Use VMCS shadowing                       {0,1}

vmx| I005: VMCS Enumeration (0x000000000000002a)

vmx| I005:   VMCS revision ID                   1

vmx| I005:   VMCS region length              4096

vmx| I005:   VMCS memory type                  WB

vmx| I005:   Use VMCS shadowing                       { 0 }

vmx| I005: VMCS Enumeration (0x000000000000005a)

vmx| I005:    VMCS shadow

vmx| I005: OvhdMem OvhdUser_vhvCachedVMCS              :       2      2      - |      2      2      -

vmx| I005: OvhdMem OvhdUser_vhvCachedVMCS              :       2      2      - |      2      2      -

... So, does it support VMCS shadowing, or not...?

0 Kudos
Highlighted
Virtuoso
Virtuoso

If the VM has "Enable hypervisor applications in this virtual machine" checked, there should be two sections.

vmx| I005: Host VT-x Capabilities:

vmx| I005: Guest VT-x Capabilities:

So the one to look at is the "Use VMCS shadowing" under the Host VT-x Capabilities which is the one that comes first.

From the Intel paper referenced in the KB, it looks like vPro Eligibility "Yes" in the Intel ARK website would indicate whether VMCS Shadowing is present or not for Haswell and newer CPUs.

This link is a comparison of all 2013 MBP with i7 CPUs.

https://ark.intel.com/content/www/us/en/ark/compare.html?productIds=75992,76086,76087,76088

Highlighted
Enthusiast
Enthusiast

...I knew I got that iMac Pro for something...and now I know!  Thanks!  I was able to confirm the log entries.

0 Kudos
Highlighted
Expert
Expert

In nested virtualization, is Apple’s hypervisor perhaps still much slower than VMware’s...? That would explain some things in Big Sur; or maybe there are other reasons (side channel

mitigations? a Big Sur-only problem, thus?)...

0 Kudos
Highlighted
Expert
Expert

After some more experiments, at least on my machine (old, but apparently capable of doing nested virtualisation on Big Sur), nested virtualisation is essentially unusably slow on Big Sur, even with side channel mitigations disabled. Let's hope this situation will evolve...

0 Kudos
Highlighted
Community Manager
Community Manager

What hardware do you have?

- Michael Roy - Product Line Manager: Fusion & Workstation
0 Kudos
Highlighted
Expert
Expert

... So, it looks like these older Macs indeed support VMCS shadowing, as everything works without error messages; but while, for example, in a Catalina host booting a Windows 7 installer takes only a couple of minutes inside a nested VMware Player 16 in a Windows 10 Fusion 12 VM, in a Big Sur host it still doesn’t reach the “install” stage after an hour: so something is evidently wrong somewhere, even if formally it “works”; and turning off side channel mitigations only speeds things up a little at the beginning: afterwards, after the “Starting Windows” phase and reboot, it seems to hang indefinitely, as described before. A rather curious behaviour, anyway...