Attempting to do nested virtualization

[Coltons]

I am running the public beta of Mac - Big Sur public beta 3. I see that VMWare Fusion has full Big Sur support based on the website. Currently, I'm trying to run nested virtualization on a Ubuntu-20.04 instance and i'm getting the error "VMware Fusion does not support nested virtualization on this host. Module 'HV' power on failed. Failed to start the virtual machine"

I have enabled hypervisor applications in the virtual machine under Processors and memory.

Is there anything I need to do in order to have this run successfully?

Thanks!

[m0leX]

is there actually some other solution? except parallels (which imho isn't much better)

[ReneLarsen]

Anyone tried VirtualBox ?? - I haven't, but maybe some of you have ??

[mlippold1]

I love Apple and VMware products and how VMWare has performed (Catalina and before) but with Apple switching to their new chips going forward and VMWare seemingly having to dumb down/gimp the performance of their product on machines that still have Intel chips by using the Apple hypervisor I think my solution is ultimately going to be to dump Apple and VMWare and just go to Windows and Hyper-V.   

I've used Hyper-V for a few years for work and it is "OK", nothing flashy, but I just need it to work.   I bought VMWare Fusion Pro because it let me have my VBS/Bitlockered Windows work machines on my Mac (they performed very well in fact).  I would just like a VMware representative tell me if they, Apple or both know about this issue and are working to address it or not.... or are they resigned to the fact that the performance is going to be terrible going forward on Big Sur and beyond?

[m0leX]

well, im debating with myself if i should contact Apple (again) and try to escalate it but to be honest i don't think i have enough energy to do that. i've already had my machine replaced this year and it took almost 2,5 months and endless numbers of calls. so, i guess i'm going to stick with 10.15.7

no word from VMware either 

[gaiush]

qemu isn't working in vbox either on big sur

[bscully]

@dempson, wow, your post may be just what I need.  I appreciate the details on which 16” MBP processor supports nested virtualization and it sounds like it’s the standard i9 option, the one that you bought.  I am curious to know if you were able to confirm that nested VMs work fine on your MBP, without the horrendous lag that others and I have been experiencing?

Thanks for your post and any confirmation you can give.

[bscully]

Hi, I just wanted to report back that a 16" MacBook Pro with the following CPU was able to run nested VMs (nested virtualization) and solve the issue I was experiencing with a 15" 2018 MBP and an HK processor (detail in this thread).

PROCESSOR                 065-C87C  2.3GHz 8-core Intel Core i9

Thank you @dempson !

[dempson]

Great, thanks for reporting back. As I've not yet had a need to do nested virtualisation, I had started researching what would be needed to test it, but due to being rather busy at the moment I hadn't got very far.

[bscully]

I am so grateful for you posting the detail that you did.  Thank you!

I tested using Cisco's CML 2, which relies heavily on nested virtualization.  On a 16" MBP with the H proc., it worked like a 15" MBP running Catalina with an HK.  I was so relieved, especially since there is a strong possibility these may be the last, new Intel MBPs we see from Apple again!

Many thanks.

[pfruth]

@bscully 

I'm having trouble connecting this;

PROCESSOR                 065-C87C  2.3GHz 8-core Intel Core i9

To the Intel processor model number.

Could you please confirm a couple things...

1 - The Intel processor model number you're running on.
You can find this in the vmware-x.log file of a VMware virtual machine (where x is 0, 1, or 2)
Look for a string similar to; "hostCPUID name: Intel(R) Core(TM) i9-XXXX"

2 - You are running VMWare Fusion V12.1 on MacOS V11 BigSur

For example, on my system (where nested virtualization performance is terrible), my processor model number is "Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz".
I'm curious to know what your Intel processor model number is.

[dempson]

The processor in my 16-inch MacBook Pro, which is the same one that @bscully confirmed was working with nested virtualisation and good performance, is an "Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz".

This is the processor you get if you start with the higher spec base model of the 16-inch MacBook Pro (US$2799) and do not configure the processor to get the fastest option.

In Apple's 16-inch MacBook Pro technical specs, it the first processor listed in the right column.

The cheaper base model (US$2399) doesn't work for nested virtualisation with either processor option, as neither its standard "Intel(R) Core(TM) i7-9750H @ 2.60 GHz" nor its configurable "Intel(R) Core(TM) i9-9980HK @ 2.40 GHz" has the necessary hardware feature (I'm not sure if it is part of vPro or Trusted Execution Technology, both of which are only on the i9-9880H out of the three possible processors in the 16-inch MacBook Pro).

My 16-inch MacBook Pro is still running VMware Fusion 11.5.x on Catalina and I am in no hurry to upgrade it to Big Sur or deal with other complications of VMware Fusion 12.x and Apple's hypervisor.

[pramet]

Thanks for your reply.
But, of course, everything worked perfectly on my side also under Catalina with the previous version of Fusion. My nested-virtualization problem appeared under BigSur with the Fusion update, and it's still a problem after the latest Windows 21H1 update.

[bscully]

@pfruth ,

Before with 2018 15" MBP

vmx| I005: hostCPUID name: Intel(R) Core(TM) i9-8950HK CPU @ 2.90GHz

After with 2019 16" MBP

vmx| I005: hostCPUID name: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz

@dempson  was spot on in identifying exactly what CPU is needed and why.  The Intel comparison chart was extremely helpful in identifying which processors support vPro, which apparently is needed for nested virtualization.

This thread is long, but helpful to look through and @dempson provides all the information needed to figure out how to identify which CPU is needed.

Please keep in mind that this discussion has to do with nested virtualization or running VMs within VMs.  I had no problem running simple VMs with the HK CPU, it was just those that used nested virtualization.

[AntonioMeireles]

I'm lost and utterly confused

according to https://kb.vmware.com/s/article/80785 and quoting "Starting with Fusion 12.1.0 VMCS shadowing is no longer required to use nested virtualization on macOS 11.0 (...)"(sic). 

I'm on stock Big Sur (11.2.2) on a MacBook Pro (15-inch, 2019) which Fusion (Professional Version 12.1.0 (17195230)) report as "hostCPUID name: Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz", and anytime i try to load something with KVM (via libvirt) i get ..

Call to virDomainCreateWithFlags failed: internal error: qemu unexpectedly closed the monitor: 2021-03-05T10:48:22.408760Z qemu-system-x86_64: error: failed to set MSR 0x48d to 0x5600000016
qemu-system-x86_64: ../target/i386/kvm.c:2701: kvm_buf_set_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.

so, what kind of nested virtualization if ANY is actually supported in the currently shipping Fusion Pro release over linux guests ?

Thanks in advance

[bscully]

You left out the rest of the sentence in your quote, "...however performance of nested virtualization without VMCS shadowing may be degraded."  VMware's workaround allowed the nested VM to run, however, it performed so poorly as to be unusable.

Looks like you've got the non-nested VM like I originally had, ending in HK.

Personally, and due to the poor performance I experienced with the Intel-HK CPUs and Fusion on Big Sur, I would have to say that nested virtualization does not work on these processors.  See previously in this thread, but an Intel-H processor running nested VMs is working fine for me.

[AntonioMeireles]

@bscully 

FWIW i left that part out of the sentence on purpose. i wasn't complaining about performance or slowness. VMware stated (unless my understanding of plain english really low these days) that nested virtualization would be again possible on BigSur without requiring a CPU that supports VMCS shadowing, so it should work again in HK CPUs as mine. AFAICT it doesn't (fast or slow) at all. So that's why i asked for a clarification.

have a great weekend.

António

P.S. FWIW other than this (and the loss of ability to reserve static IPs on private nets) performance and stability on big sur top notch so far in my i9980HK macbook pro.  

[bscully]

@AntonioMeireles, are you running nested VMs (VMs within VMs) and getting that performance?

[AntonioMeireles]

@bscully wrote:

@AntonioMeireles, are you running nested VMs (VMs within VMs) and getting that performance?

---

on a plain linux vm, running libvirt/kvm - anytime i try to have qemu/kvm load anything (so yes nesting) it just fails with (either if i use libvirt or if i try to load it by hand)... 

qemu-system-x86_64: error: failed to set MSR 0x48d to 0x5600000016
qemu-system-x86_64: ../target/i386/kvm.c:2701: kvm_buf_set_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.

This error was supposed happen afaict in pre 12.1 due to lack of VMCS shadowing in some CPUs... OTOH VMCS shadowing isn't supposed to be an hard requirement anymore (per https://kb.vmware.com/s/article/80785) which is why i asked for a clarification... 

Thanks in advance and have a great weekend!

[pfruth]

This whole thing has really soured my MacBook experience.
I paid an extra $200 premium, thinking I was getting a "more capable" processor.
Instead, I got a processor that has a nominally faster (0.1 Ghz) base clock, and fewer (not more) features.  No vPro support.
I don't know who I'm more disgusted with - Intel for not including vPro in a top-of-the-line mobile i9 processor.  Or Apple for not disclosing this little detail in it's marketing material.  I mean c'mon Apple, even a footnote calling out the fact that the 2.4 Ghz i9 is faster but lacks vPro would have been nice.
Shame on me for not consulting the Intel datasheets before ordering.

I am so done with Apple.

[m0leX]

imho it's not your fault and i suppose that VMware didn't consulted enough with Apple. i wouldn't blame solely Apple on that one (i'm affected as well, MBP 13" 2020 i7 and i can't do nested virt LOL). also reaction from VMware isn't "appropriate" to say it moderately. afaik it works quite well in Parallels.

x