VMware Communities
Coltons
Contributor
Contributor

Attempting to do nested virtualization

I am running the public beta of Mac - Big Sur public beta 3. I see that VMWare Fusion has full Big Sur support based on the website. Currently, I'm trying to run nested virtualization on a Ubuntu-20.04 instance and i'm getting the error "VMware Fusion does not support nested virtualization on this host. Module 'HV' power on failed. Failed to start the virtual machine"

I have enabled hypervisor applications in the virtual machine under Processors and memory.

Is there anything I need to do in order to have this run successfully?

Thanks!

Screen Shot 2020-09-18 at 2.37.49 PM.png

Tags (1)
106 Replies
Mikero
Community Manager
Community Manager

Which Mac do you have?

It's likely that your Mac is lacking a critical CPU feature that is required by macOS virtualization APIs called 'VMCS Shadowing'.

This issue only affects macOS Big Sur hosts, Catalina hosts continue to run nested VMs without this issue on Fusion 12.

VMware Knowledge Base

We're working to address this is issue, as well as provide a list of Macs that we expect should work and ones that we expect won't.

-
Michael Roy - Product Marketing Engineer: VCF
Reply
0 Kudos
Coltons
Contributor
Contributor

Thanks for your reply. I have to assume it's not related to my mac as it's the newest one available, see below.

Screen Shot 2020-09-18 at 4.16.33 PM.png

Just for clarity, are you saying this is a bug with Big Sur or something that's not supported on my mac itself?

Reply
0 Kudos
Mikero
Community Manager
Community Manager

What I'm saying is that it's a combination of both Big Sur and your Mac.

I don't believe VMCS shadowing is included on non-Xeon CPUs, and that hardware feature is, at this time, required to do nested vms.

I have the same Mac but with the i9... Doesn't work for me either Smiley Sad

-
Michael Roy - Product Marketing Engineer: VCF
Reply
0 Kudos
Coltons
Contributor
Contributor

Mike, that makes sense. Admittedly though, that would be very frustrating if that's the case moving forward. As the Xeon CPUs are only in the Mac Pro (The very expensive desktop) and i cannot see this being an issue with the hardware not being powerful enough since people have been doing it on macs since 2013, I believe.

Thanks for your input. I'll still keep my fingers crossed.

Reply
0 Kudos
DotBat
Contributor
Contributor

I ran into the same issue when I upgraded to Big Sur but I realized that one out 5 VMs was still working.

My thought process was automatically to look in the settings of the working VM and compare it with the non-working ones.

A simple fix was to go to the VM settings select the processor and memory, and on the bottom in "advanced options" disable the HyperVisor, and voila your VM will work.

SvenGus
Expert
Expert

I tried with VMware Player 16 inside a Windows 10 guest (with the correct configuration) in VMware Fusion 12 running on a macOS Big Sur (latest beta) host, on a MacBook Pro Retina 15” from late 2013: nested virtualization works, but it is way too slow (several minutes to boot an installer): so, there must still be something not quite right, as nested virtualization has always worked with reasonable speed on previous macOSes...

Reply
0 Kudos
DotBat
Contributor
Contributor

Mine boots fairly quick and normal - you gotta look at your RAM, what's your physical size and how much are you allocating and virtualizing out of that to run your VM. If your Macbook has 4GB RAM and you are allocating 1GB for the Windows yes it is going to boot super slow.

You need a good 4GB allocation for the windows to boot quickly, also you may want to allocate 2 cor processors for it. 

Reply
0 Kudos
SvenGus
Expert
Expert

The machine (Core i7 quad-core) has 16 GB RAM, the Windows 10 VM has 4 GB and 2 cores allocated, the nested guest 1 GB and 1 core: so, this should be fine, probably. In Catalina, the speed of the nested VM is reasonable, in Big Sur definitely not: 10 times slower, approximately (depends on the respective hypervisor, perhaps?)...

Reply
0 Kudos
Coltons
Contributor
Contributor

Thank you for responding! So, my issue is not with virtualization as a whole but nested virtualization. Essentially, building a VM then building a VM inside of that. This is what used to work fine on older versions of macOS and is broken. When you disable the 'Enabled hypervisor applications in this virtual machine' you are no longer able to do nested virtualizations.

If you are in-fact doing nested virtualizations the way i've discribed above and you are successful. Can you please share the setup you're using?

DotBat
Contributor
Contributor

I haven't done nested my post was initially in response to the first VM that's going directly on the top of Big Sure OS.

Now lets say you build a Windows VM on the top of Big Sur in that the windows VM you install Vmware fusion and you have the HyerV option enable  see no reason why it shouldn't work

Reply
0 Kudos
Coltons
Contributor
Contributor

Gotcha, thank you.

If you look at the original post you'll see the issue. With the host machine is Big Sur you cannot do nested virtualization, no matter how you do it. I would love to be wrong on this if you get it working.

Essentially what i'm doing is:

Host - Big Sur

VM - Ubunutu - VMware Fusion

Nested VM - Ubunutu - Virtual box.

You get the error that you must enable `VT-x/EPT` to run this (see OP screenshot). Once you enable this the VM will no longer start back up.

i have not tested with a windows VM, but i would not expect different results since that setting still must be set in order for nested virtualization to work.

SvenGus
Expert
Expert

After some more experiments, at least on my machine (old, but apparently capable of doing nested virtualisation on Big Sur), nested virtualisation is essentially unusably slow on Big Sur, even with side channel mitigations disabled. Let's hope this situation will evolve...

Reply
0 Kudos
DotBat
Contributor
Contributor

Coltons you are correct - I initially didn't try to nest but after your comment I decided to try:

Big Sur --> VM Fusion WIndows 10

VM Fusion Windows 10 --> Work Station (attempting to run) Ubuntu VM and I am getting the same error that you ar getting.

Granted if you enable the HyperV option in the settings of the first VM in my case is windows, and since it is on the top of Big Sur then the first VM will not run.

At it this point it definitely an issue with Big Sur Stack for processing the memory and core processor, and I would assume this issue is created since they trying to switch to Arm.

Reply
0 Kudos
bwyunker
Contributor
Contributor

I would actually be more inclined to say this one is on VMWare to resolve, as it seems (I haven't directly attempted, but through anecdotal accounts) that Parallels already has VT-x passthrough implemented.Though Apple's documentation leaves something to be desired as far as correct implementation, at least at first glance...

(EDIT: I just tried it out, Parallels can indeed do nested virtualization in Big Sur - the difference seems to be Parallels is still loading its own hypervisor kext to make it happen. Passthru is unavailable when the Apple Hypervisor is selected.

So it appears to be two issues: #1) Apple has problems they need to fix with their Hypervisor API to allow VT-x passthru/nested virtualization - and in that case, it's likely the VMWare team has already implemented what's needed for when Apple fixes things  (a totally understandable issue on Apple's part, seeing as how Big Sur is still in beta); and #2) VMWare no longer loads/uses its own hypervisor kext in Big Sur - which presumably is how passthru was happening in earlier iterations.

I really hope that if Apple drops the ball on fixing their Hypervisor API issues, that VMWare takes a note from Parallels and continues to allow the selection of their own kext to be able to continue using passthru in VMs)

Reply
0 Kudos
lmicali
Contributor
Contributor

So, let me set this straight:

  • top of the line MacBook Pros 
  • with the latest OS

cannot

  • run your product until it's upgraded to version 12 ($$$)
  • and when it's upgraded ($$$) it cannot run fully featured

 

I'm going for a refund as fast as I can.

 

Reply
0 Kudos
arunkn82
Contributor
Contributor

thank you Dotbat...it worked fine for me when i disabled the hypervisor

Reply
0 Kudos
erikread
Enthusiast
Enthusiast

To say I'm disappointed in this is an understatement. Big Sur has been in beta for ages and for this to not have been addressed by RTM is almost inexcusable. It throws my whole development workflow in the trash. I am using a 2019 MacBook Pro i9 8 core w/32GB RAM. I was evaluating VMWare and Parallels deciding whether to switch off of Fusion. I decided to stay with Fusion at the time as my compiles were roughly twice as fast. It now seems that might have been a mistake. I will give VMWare the benefit of the doubt for a while but it looks like I may need to move over if Parallels does indeed support nested virtualization on Big Sur

Reply
0 Kudos
ColoradoMarmot
Champion
Champion

From what I understand, this is a limitation of Big Sur's hypervisor framework itself.  If so, it's on Apple to fix it.

Reply
0 Kudos
VMUser71
Contributor
Contributor

Ditto! I need to be refunded - 3 hours wasted in trying again and again.... this is ridiculous

Reply
0 Kudos