Attempting to do nested virtualization

[Coltons]

I am running the public beta of Mac - Big Sur public beta 3. I see that VMWare Fusion has full Big Sur support based on the website. Currently, I'm trying to run nested virtualization on a Ubuntu-20.04 instance and i'm getting the error "VMware Fusion does not support nested virtualization on this host. Module 'HV' power on failed. Failed to start the virtual machine"

I have enabled hypervisor applications in the virtual machine under Processors and memory.

Is there anything I need to do in order to have this run successfully?

Thanks!

[dempson]

---

@Mikero wrote:

I have the same MBP with the same CPU, it does not have VMCS shadowing. I couldn't believe it either.

I assume you are talking about the Core i9-9980HK? (Upgrade option in the Late 2019 16-inch MacBook Pro.)

Based on a comparison at ark.intel.com: of the three processor options in the 16-inch MacBook Pro (Core i7-9750H, Core i9-9880H and Core i9-9980HK) only the middle one of the three is listed as having vPro support (which implies nested virtualisation among other features). That particular chip is the 2.3 GHz 8-core one included in a standard configuration of the higher model 16-inch MacBook Pro.

The extra cost configuration option of the Core i9-9980HK increases the speed but removes the vPro feature.

That looks like Intel's choice rather than Apple's, but excluding a feature from the fastest variant seems an odd decision.

https://ark.intel.com/content/www/us/en/ark/compare.html?productIds=191045,192987,192990

Before I found out about this detail, I had already bought my 16-inch MacBook Pro, and by a stroke of luck happened to choose not to spend extra money on the fastest processor, so I do have vPro support, but haven't needed it yet. (I haven't checked this in VMware Fusion yet but can do so later.)

[lacierdias]

Good afternoon Mike. Do you have a time to update the update? Thanks.

[m0leX]

hi,

first post here - so "hello world"

i'm having the same situation with my MBP 13" 2020 with i7-1068NG7...

here's friday already... what time can we expect above-mentioned  update?

dunno, i wonder if should just go with 10.15.7 and leave that 11.0.1 behind, meh

Fusion 12 Pro works on 10.15.7 (from what 've seen)

all best,

xx

[bscully]

Hi @Mikero,

I was eager to try 12.1 and relieved to be able to run Cisco's CML with nested virtualization on Big Sur.

You were right, there's definitely a performance hit - about 5x slower than 12. on Catalina and almost 2x more CPU utilization for me.

Is there any hope for substantial improvement in the not too distant future?

Thanks!

[m0leX]

@bscully 

i was chatting with Apple for over hour on this issue, they've sent me "Capture data" tool and will look into my logs

i've asked them questions regarding their cooperation with VMware and another one was about changes in their framework and how are they going to solve this. that being said i've wasted whole 3rd night in a row on Big Sur, great

dunno how is it on M1 with nested virtualisation 

[stolowski]

Was is through a regular customer service of Apple? If so then I wouldn't hold my breath... I think something like that can only be effectively pushed forward by VMWare via other communication channels (hopefully they have some). Of course, it won't hurt to try.

I'm glad there is a workaround for now at least with latest Fusion update; this was honestly a big bummer after I upgraded last weekend and discovered my ~6 months old MBP 16 with i9-9980HK CPU @ 2.40GHz is not "good enough" for nested virtualization anymore.

[Grennith]

Doubt that will change much - if anything...

Does anyone else have issues with qemu-kvm crashing with the nested virtualization in Fusion 12.1? 

qemu-system-x86_64: error: failed to set MSR 0x48d to 0x5600000016
qemu-system-x86_64: /build/qemu-2AfuBA/qemu-4.2/target/i386/kvm.c:2691: kvm_buf_set_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.

[m0leX]

yeah. but im kind “famous” for escalating everything to the highest possible level lol

anyway, my windows server 2019 nested theough hyper-v in windows 10 “works”. im stunned, it is unusable. great

[erikread]

12.1 with nested virtualization turned on makes the whole VM virtually unusable let alone attempting to run docker. 

Not happy. Luckily, this is my personal mac and I know to not upgrade my work mac for some time.

[lacierdias]

Good Morning. Had the update but it didn't solve my problem with EVE-NG at all. He fuses an image of Ubuntu in Fusion and uses qemu inside this Ubuntu to virtualize the routers. It's virtualization within virtualization. This is not happening. On the same machine Parallels, even if slow, can make everything work. who can save us ... ????

[bennykusman]

Had upgraded to 12.1, but nested virtualization didnt work for KVM (redhat) 😞

[bscully]

Thanks.  I am interested to know how they respond.

[ColoradoMarmot]

They posted in another thread that a workaround is on its way.

[bscully]

@ColoradoMarmot, any chance you can share where that post is?  Wondering if that post was referring to the 12.1 update that just came out and is, unfortunately, unusable for our needs:

Nested virtualization "works" under 12.1, however, CPU utilization is literally double what it is under Catalina for the same VM and about 5x slower.

I am looking for a ray of hope!  Thanks

[ColoradoMarmot]

I wrote that just before I saw the 12.1 drop - that is the workaround.  It's a limitation in what apple supports in their hypervisor framework, and Mike noted that it'd have a performance impact.

[bscully]

I figured.  Unfortunately, it is so underperforming as to not be a workaround.

I am not going to pretend to understand that technical challenges that VMware and others are facing with Apple's changes in their new OS, but I wonder if VMware is going to have to bundle their own hypervisor like Parallels in the meantime.  Some how, it appears that Parallels is able to achieve nested virtualization and performance in Big Sur.

Still hanging on...

[SvenGus]

In Catalina and Fusion 12, my dated but perfectly capable MBP ran Windows 10 and WSL 2 without any problems; in Big Sur and Fusion 12.1, Windows 10 is very sluggish with nested virtualization turned on, and WSL 2 is almost unusable (and download speeds are drastically reduced, in both): so, if nothing changes (improved Apple hypervisor and/or VMware hypervisor also in Big Sur, à la Parallels), the only way to have acceptable performance for now seems to be to turn off nested virtualization (so, no WSL 2 and other nice things) - which isn’t exactly what one would have expected...

BTW, disabling side channel mitigations has little to no effect; and WSL 2 takes about 2 minutes to open to the command prompt, and over 5 minutes to perform a simple sudo apt update - completely unusable!

[bscully]

Unfortunately, Cisco CML and others need nested virtualization to work.

[m0leX]

i've reinstalled catalina, won't update to Big Sur

offtopic: in MacOS 11 tMachine creates backups in APFS, so if you'd like to restore your "Applications" after successful downgrade 10.15.7 - you won't be able to do that as its has to be HFS+

[mlippold1]

Yeah getting back to Catalina for me was not the usual normal Time Machine restore - I had to create a bootable USB with the full Catalina install (Which was fun because Apples instructions in their web article don't work) and let it wipe the drives and then I was able to restore as usual with Time Machine.