VMware Communities
Coltons
Contributor
Contributor

Attempting to do nested virtualization

I am running the public beta of Mac - Big Sur public beta 3. I see that VMWare Fusion has full Big Sur support based on the website. Currently, I'm trying to run nested virtualization on a Ubuntu-20.04 instance and i'm getting the error "VMware Fusion does not support nested virtualization on this host. Module 'HV' power on failed. Failed to start the virtual machine"

I have enabled hypervisor applications in the virtual machine under Processors and memory.

Is there anything I need to do in order to have this run successfully?

Thanks!

Screen Shot 2020-09-18 at 2.37.49 PM.png

Tags (1)
106 Replies
ReneLarsen
Contributor
Contributor

No one ask you to pay anything, if you are using it for Personal use - VMware has a free Fusion Player

Reply
0 Kudos
lmicali
Contributor
Contributor

I'm using it for professional use and I work with a regularly purchased license that I had to upgrade to run Fusion 12, what's the matter here?

 

Reply
0 Kudos
mhj_poulussen
Contributor
Contributor

Solved it with changing VMX file setting "vhv.enable" to False and the problem was solved.

Check also : vpmc.enable = "FALSE"

After changing this I could Start my Windows10 VM on Fusion Pro 12 with Big Sur 11.0.1

Reply
0 Kudos
lmicali
Contributor
Contributor

This solves nothing and the nested virtualization flag is exposed in the Fusion UI, so there's no need to fiddle with the vmx.

If disabling nested virtualization is good for you, you weren't using it.

babak1980
Contributor
Contributor

Can you advise how you did this? I recently bought MAC book and not familiar with the CLIs at MAC.

 

Thanks

Reply
0 Kudos
mlippold1
Enthusiast
Enthusiast

I agree - VMWare needs to modify their product to use their old hypervisor in Big Sur until Apple gets their act together.   I run 3 VMs for work that require Virtualization Based Security be enabled and they ran like absolute garbage in Big Sur.   After I was back to Catalina they run great.

Reply
0 Kudos
wila
Immortal
Immortal

Hi,

As mentioned.. this is because of missing features in apple Hypervisor Framework.
If you want to know if your CPU has VMCS shadowing then look up your CPU at ark.intel.com and search for the vPro label.
That -apparently- is intel's way of stating that the CPU has VMCS shadowing support.

For figuring out what CPU your machine is using run the following command in the terminal:
$ sysctl -a | grep brand

If you don't need nested virtualisation then make sure your VM is shut down.
If it is suspended, hold down the option key, then choose "Power Off" from the Virtual Machine menu.

Open the menu "Virtual Machine" -> settings -> Processors & Memory -> open Advanced options and then make sure the following checkboxes are unchecked:

  • Enable Hypervisor applications in this virtual machine
  • Enable code profiling applications in this virtual machine
 
 
 
 
 
 

See attached screenshot.


PS: You do _not_ need to edit the .vmx file manually.

--
Wil

 

| Author of Vimalin. The virtual machine Backup app for VMware Fusion, VMware Workstation and Player |
| More info at vimalin.com | Twitter @wilva
Reply
0 Kudos
mlippold1
Enthusiast
Enthusiast

My CPU:

machdep.cpu.brand_string: Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz
machdep.cpu.brand: 0

According to the website ark.intel.com:

"Intel vPro® Platform Eligibility ‡ Yes" for my CPU.

While my VMs did technically run in Big Sur with VBS enabled, which I require for these work VMs (they run like 20 times slower, if that) -  I can't remove the hypervisor settings because - "The "Enable IOMMU in this virtual machine" setting and the "Enable hypervisor applications in this virtual machine" setting cannot be changed unless the VBS option in Advanced setting page is unchecked."

Reply
0 Kudos
gringley
Hot Shot
Hot Shot

I noticed after Big Sur I have a "Disable Side Channel Mitigations" box in advanced settings for all my VMs (and really wish there was a global setting to disable them for all my VMs...) so I wonder if the default enabled side channel mitigations is the issue?

Reply
0 Kudos
mlippold1
Enthusiast
Enthusiast

No, unfortunately, it didn't affect my VM performance at all (none that I'd perceived) - They were still incredibly slow. 

Reply
0 Kudos
babak1980
Contributor
Contributor

In this case we can’t use ESXI, eve-ng and similar products on Big sur whiteout nested virtualisation which is just ridiculous. I just bought my Mac OS pro couple of months ago and wasn’t aware of those Apple nonsense.

Reply
0 Kudos
Mikero
Community Manager
Community Manager

As a note, we'll be shipping an update this week which has a workaround to support nested on hosts which don't have VMCS Shadowing in the CPU. There is a performance hit involved, but things work. ESXi performs best because it was designed to run well as a VM (because that's how we test it)

-
Michael Roy - Product Marketing Engineer: VCF
pfruth
Enthusiast
Enthusiast

Thanks @Mikero for trying to make things work, despite Apple's best effort to screw us over 😉

So what about those of us who have "VMCS shadowing" capable hardware?  Are we still going to be out in the cold, even after you release that update later this week?


I have a 2019 Macbook Pro with a core i9-9980HK processor.  According to Intel's product brief [*1] that processor has VMCS shadowing support.

EDIT: It turns out the core i9-9980HK processor does NOT support VMCS shadowing.  Despite Intel's product brief indicating that the core i9-9980HK supports VMCS shadowing, actual log data, collected from a vmware.log during VM startup, proves otherwise.  See the log file excerpt a couple posts below.

I wasted spent the weekend upgrading to MacOS BigSur V11.0.1 and VMware Fusion Pro V12, only to discover that VMs that require nested virtualization no longer work.  And those are VMs that work just fine on MacOS Catalina V10.15.7 and VMware Fusion Pro V11.5.6.

I've subsequently restored from a backup, so I'm back on Catalina 10.15.7 w/ Fusion V11.5.6.
But it looks like, at least for the moment, there's no path forward at this point.
Thanks alot Apple.  Going forward, I think I'll be looking into a non-Apple hardware platform.

[*1] - https://www.intel.com/content/dam/www/public/us/en/documents/product-briefs/9th-gen-core-mobile-brie...

wila
Immortal
Immortal

Hi pfruth,

I checked both your pdf, which seems to suggest your CPU has VMCS shadowing support as well as the vPro eligibility support on that processor against the intel ark site which suggests the opposite.
From the looks of it in macOS you don't get to see this feature in CPU features (sigh) and intel's website on this type of feature availability is poor at best.

There's another way to check if your CPU has VMCS support and that's by checking the vmware.log file.
See this post for the details:
https://communities.vmware.com/t5/VMware-Fusion-Discussions/Fusion-12-Pro-No-Nested-Virtualization/m...

--
Wil

| Author of Vimalin. The virtual machine Backup app for VMware Fusion, VMware Workstation and Player |
| More info at vimalin.com | Twitter @wilva
Reply
0 Kudos
mlippold1
Enthusiast
Enthusiast

What about the performance hit for those of us that do have VMCS Shadowing?   The speed when using nested vms (which I need in order to use Virtualization Based Security) is obscenely slow - I had to roll back to Catalina because it was at least 20 times slower in Big Sur.  

Has VMWare been assured by Apple that they are going to fix this in their hypervisor?  ... or is this a problem with VMWare's implementation?  

Why can Parallels get around this by still using their old hypervisor in Big Sur and Vmware can't?

Like another poster had mentioned it may be time for me to move away from Apple because this is nonsense and it'll probably only get worse with their own silicon.

Reply
0 Kudos
pfruth
Enthusiast
Enthusiast

Here are the salient bits of a vmware.log taken from the startup of my Windows 10 Pro VM

This is a 2019 Macbook Pro host running Catalina 10.15.7 and VMware Fusion 11.5.7

 

2020-11-16T15:50:35.970-07:00| vmx| I005: hostCPUID vendor: GenuineIntel
2020-11-16T15:50:35.970-07:00| vmx| I005: hostCPUID family: 0x6 model: 0x9e stepping: 0xd
2020-11-16T15:50:35.970-07:00| vmx| I005: hostCPUID codename: Kaby Lake-S/H QS
2020-11-16T15:50:35.970-07:00| vmx| I005: hostCPUID name: Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz


2020-11-16T15:50:36.052-07:00| vmx| I005: Host VT-x Capabilities:
...
2020-11-16T15:50:36.052-07:00| vmx| I005:   Use VMCS shadowing                       { 0 }



2020-11-16T15:50:36.053-07:00| vmx| I005: vmm64-modules: [vmm.vmm64, vmce-vmce.vmm64, viommu-none.vmm64, vprobe-none.vmm64, hv-vt.vmm64, gphys-ept.vmm64, callstack-none.vmm64, gi-none.vmm64, gmm-none.vmm64, ahci-ahci.vmm64, !ahciRegs=0x0, !ahciShared=0x1000, !e1000Shared=0x1c00, !vmSamples=0x2480, !theIOSpace=0x24c0, !ttGPPerVcpu=0x8840, {UseUnwind}=0x0, numVCPUsAsAddr=0x2, {SharedAreaReservations}=0x8880, {rodataSize}=0x21115, {textAddr}=0xfffffffffc000000, {textSize}=0x9137e, <MonSrcFile>]
2020-11-16T15:50:36.053-07:00| vmx| I005: vmm64-vcpus:   2



2020-11-16T15:50:36.058-07:00| vmx| I005: KHZEstimate 2399754
2020-11-16T15:50:36.058-07:00| vmx| I005: MHZEstimate 2400
2020-11-16T15:50:36.058-07:00| vmx| I005: NumVCPUs 2



2020-11-16T15:50:36.338-07:00| vmx| I005: Guest VT-x Capabilities:
...
2020-11-16T15:50:36.338-07:00| vmx| I005:   Use VMCS shadowing                       { 0 }

  

Reply
0 Kudos
aftab102
Contributor
Contributor

Hi Mike, is there any further update on this?

Reply
0 Kudos
wila
Immortal
Immortal

Hi pfruth

Thanks for the log excerpt.
Wow, that's a shock for such a flagship processor, I'm more than a bit disappointed by intel here.
It doesn't look like the processor has VMCS shadowing support contrary to what the pdf stated.

--
Wil

| Author of Vimalin. The virtual machine Backup app for VMware Fusion, VMware Workstation and Player |
| More info at vimalin.com | Twitter @wilva
Reply
0 Kudos
Mikero
Community Manager
Community Manager

I have the same MBP with the same CPU, it does not have VMCS shadowing. I couldn't believe it either.

I'm not sure if it's Intel or Apple leaving that feature out, but it seems to be not available on notebooks.

My iMac Pro has it, and I think my Mini does as well. Nested stuff runs really well on the iMac, but my Mini isn't at Big Sur yet. (daily machine is in for repair so I have to keep the Mini on Catalina for now).

 

 

-
Michael Roy - Product Marketing Engineer: VCF
Reply
0 Kudos
Mikero
Community Manager
Community Manager

We're shipping an update to Fusion on Thursday which will has includes a workaround, so MBPs and systems without VMCS Shadowing will be able to run Nested VMs, but there will be a performance hit due to the translation that we have to do.

ESXi VMs will run quite well as they're optimized specifically to run as a VM (it's how we build and test ESXi itself after all).

-
Michael Roy - Product Marketing Engineer: VCF