ppan7
Contributor
Contributor

VMware Workstation does not support nested virtualization on this host.

Jump to solution

Screenshot_2.jpg

After Windows update 2020 may build 2004, vmvare doesnt support Virtualize intel Vt-x/EPT. How fix this problem. I update vmvare to 15.5.5, But it didn`t help

Tags (1)
68 Replies
bleys
Enthusiast
Enthusiast

While troubleshooting a different issue I wound up going down the rabbit-hole of disabling Hyper-V on a currently up-to-date Windows10Pro host. Even the Microsoft support articles for this are not complete, so I'll add what I needed to do here. Also I think VMware needs to make it more obvious when your host is running Hyper-V...

 

I had an issue with snapshots of running systems taking 20+minutes that I initially thought might be related to the performance degradations that come from using Hyper-V/"Host VBS Mode", so I followed the steps outlined on these two pages (the first one is not enough):

  1. Disable Hyper-V to run virtualization software - Windows Client | Microsoft Docs
  2. How do I disable Virtualization based Security in order to use the Ryzen Master software? - Microsof...

This didn't effect the slow snapshotting issue, but it did make the Linux guest performance much better in other ways. The most significant improvement is that guest audio was poppy/crackly/interrupted (to the point of being nearly unusable) prior and now it comes through perfectly clear.

(Note: I'm using an X570 motherboard with a Ryzen 5950X and have the virtualization extensions on in UEFI and VMware Workstation is configured with "Virtualize Intel VT-x/EPT or AMD-V/RVI" selected in the Processors configuration)

 

Quick summary to disable Hyper-V/VBS

Disable the following Windows components (they all cause Hyper-V to run, preventing the VMware Hypervisor from running instead):
Hyper-V
Device Guard
Credential Guard
Virtual Machine Platform
Windows Hypervisor Platform

Microsoft Defender Application Guard

 

A hint to VMware devs - There really ought to be an obvious indicator in VMware Workstation of whether it is running with the VMware Hypervisor ("Traditional Mode") or with Hyper-V ("Host VBS Mode"). Even combing the log for HyperV only gives an obtuse clue about whether it's running. I'd prefer if the second line of the log says something to the effect of "Host is running Hyper-V, switching to Host VBS Mode. Performance and feature-set are degraded in this mode. To use the traditional mode, see: <shortlink to currently non-existent VMware support article about disabling Hyper-V on your Windows Host>".

bleys
Enthusiast
Enthusiast

Another hint to VMware devs:

The error message screenshotted in the original post is misleading ("VMware Workstation does not support nested virtualization on this host.").

 

This would be a better way of handling it...

If Hyper-V is running:

 

Your Windows host is running Hyper-V.
This degrades performance and disables some virtualization features.
See: <shortlink to VMware support article about disabling Hyper-V on Windows hosts>

 

 

If a CPU with support for VT-x/EPT or AMD-V/RVI is detected, but the feature is disabled in BIOS/UEFI:

 

Your processor supports hardware virtualization extensions, but VMware Workstation cannot use them currently.
This is likely because the feature is disabled in your system configuration.
Please reference your motherboard or OEM system support documentation or
support instructions to enable Intel VT-x/EPT or AMD-V/RVI support in your
BIOS or UEFI settings.

Guest performance and features are degraded without access to hardware virtualization extensions.

 

 

These could be additional checks and tips that pop up when a nested virtualization error box is triggered, along with other circumstances of limited feature sets triggered by Hyper-V being enabled or hardware virtualization being disabled.

 

 

bleys
Enthusiast
Enthusiast

Another thing that would be nice to see in the VMware Workstation UI...

A new item in the "VM" dropdown of the Menubar: "VM Status..."

It would pop up some information like:

  • Location and current size of VM disk file(s).
  • Location and a link to open the VM log file.
  • RAM currently in use by the VM (maybe some other guest performance counters)
  • Whether the VM is running with the VMware Hypervisor ("Traditional Mode") or with Hyper-V ("Host VBS Mode")
  • Whether hardware-assisted virtualization (Intel VT-x / AMD-V, etc.) is available and/or enabled
  • A box of hardware-supported features which are currently supported and available, similar to what CPU-z and the like shows for processor extensions (SSE2, AVX, etc.). In this case it would be features like: Nested VMs, HLE, RTM, PMCs, PKU.
0 Kudos
bleys
Enthusiast
Enthusiast

Hardware-assisted virtualization is the biggest factor for VM performance. The closer your VM can get to hardware, the better it will perform. In general we want to be able to see what VM features are currently hardware accelerated.

A table in the VM status popup I mentioned wishing for which shows details similar to the table here would be amazing:

AWS EC2 Virtualization 2017: Introducing Nitro (brendangregg.com)

(table attached as well)

0 Kudos
bleys
Enthusiast
Enthusiast

More generally, I think it would be in VMware's interest to have more of a focus on highlighting when, where, and how it might be able to configure your system and the software to achieve better performance.

Opaquely running in a crippled Hypervisor environment is bad for user experience. Any way you can nudge users to get the most out of their hardware and your software is a win for everyone.

 

Some more practical advice: In the Help page for the "VM Status..." popup I'm wishing for, make and link to a very detailed and exhaustive summary of virtualized desktop performance and security factors which runs through the advantages and compromises of the hypervisor and host hardware-assisted virtualization settings

 

Another practical example of something VMware can do both to improve user experience and as a marketing/sales/good-faith-credibility-to-power-users play...
Make a blog post with a title like:

"An Ode to Our VMware Workstation Power Users."

Include some blurbs in this spirit::

 

 

"Like many of our power users, we like to run the latest and greatest hardware and push it to its limits. We are all about getting the best experience out of desktop virtualization. You could say that's the DNA of what motivates us to make VMware Workstation stand out with best-in-class performance and support. Every year we're pushing the limits of what we can achieve.

This year is no different with the release of VMware Workstation 17. We've made our hypervisor leaner and meaner and we're using it to get closer to bare-metal performance on the latest high-end desktop, workstation, and server systems. Intel Core i9 Alder Lake? Intel Xeon Sapphire Rapids? AMD Ryzen 9, Threadripper, and EPYC in the latest Zens? Yeah, we're excited too.

Today we'd like to take you on a tour of the performance advancements we've made in this update and what that looks like in the real world with real use cases from us and our beloved power users...

We've done a lot in this update based on feedback and our own testing to focus on improving the user experience and making it as easy as possible to get the best performance possible out of our software..."

 

 

One can dream, right?

bleys
Enthusiast
Enthusiast

This document is a great example of how un-obvious performance settings are currently in VMware Workstation:
Configuring Virtual Machine Processor Settings (vmware.com)

That covers this section of the Virtual Machine Settings > Hardware > Processors page:

Example from a running guest.Example from a running guest.

It also highlights that the documentation isn't very thoroughly written. Examples:

  1. Some of the virtualization engine features listed in the document only show up under certain conditions (e.g. the Linux guest I'm looking at now has no setting for IOMMU, see above). This isn't documented.
  2. Again with IOMMU, only VT-d is mentioned. Nothing about AMD-Vi. One could easily be led to believe that AMD-Vi support is not available. Is this the case? It's hard to find any information and others seem to have similar questions, unanswered: Virtualize IOMMU (IO memory management unit) optio... - VMware Technology Network VMTN
  3. In particular, the "Virtualize Intel VT-x/EPT or AMD-V/RVI" feature is ambiguous and unclear. I'll go into depth below...

Currently the article states this about the feature:

Workstation Pro forces the virtual machine execution mode to VT-x/EPT or AMD-RVI. Physical Address Extension (PAE) mode must be enabled to use virtualized AMD-V/RVI.

If the execution mode is not supported by the host system, virtualized VT-x/EPT or AMD/RVI is not available. If you migrate the virtual machine to another VMware product, virtualized VT-x/EPT or AMD-V/RVI might not be available.

I'd like to point out:

  1. Nothing in the settings UI implies anything about the (potentially drastic) performance implications. Nor does the documentation. Having closer hardware access to the CPU as allowed by VT-x or AMD-V is great for performance. This should be spelled out. (Same goes for IOMMU, by the way)
  2. It's unclear whether ticking the box for this setting enables or disables these features. What does "Virtualize VT-x" mean? One could easily be led to believe by this phrasing that it's a workaround for when VT-x or AMD-V are unavailable. This is certainly not the case though. The generally expected phrase for this class of technologies is "Hardware-assisted virtualization", and should be used as it is more self-apparent then that this gives you closer access to the hardware.

 

Instead of what's currently in the "Virtualization engine" settings box, I'd like to see this:

  1. Have a preamble for the section which says, "Hardware-assisted virtualization: These settings allow you to enable performance features made available by your computer hardware to accelerate virtual systems."
  2. Replace the "Virtualize Intel VT-x/EPT or AMD-V/RVI" tickbox with: "Accelerate CPU access with VT-x or AMD-V"
  3. Replace the "Virtualize IOMMU (IO memory management unit)" tickbox with: "Accelerate IO access with IOMMU technology (Intel VT-d or AMD-Vi)"

The Help for this page should have more detailed information about what hardware-assisted virtualization is, how to make sure it's enabled on your system if available, and why you would generally want to use it.

bleys
Enthusiast
Enthusiast

It's unclear anywhere I can find what IOMMU acceleration (VT-d or AMD-Vi) might get you in VMware Workstation anyway.

Example: virtualization - Does VMware Workstation support PCI Passthrough with the VT-d technology? - Super U...

A few unanswered questions out there. People mentioning GPU passthrough definitely doesn't work.

Will enabling it improve USB3 speeds at all? Reduce CPU overhead at all? Can I use it to passthrough a 10GbE card to a VM? How would I?

Even if the answer is "No. Use ESXi." It would be nice to HAVE an answer of any sort published.

This is the promise as pitched by VMware 13 years ago: VMware VMDirectPath and Intel VT-d explained in 3 minutes | TinkerTry IT @ Home

I can't find anything about a real world use of it in VMware Workstation currently. Is there a point to having the feature exposed at all? In ESXi/vSphere they have lots to say (pages 12, 40, 41).

bleys
Enthusiast
Enthusiast

We're in an era now where you can get 2.5GbE on a Raspberry Pi, and people are setting up cheap home lab clusters. It seems reasonable to expect VMware Workstation to have some PCI passthrough support. Does it?

addohm1
Contributor
Contributor

There are many band-aid solutions to this problem, and I will add to it.  NONE of the aforementioned band-aids worked for me.

Note: Disabling the "Virtualize Intel VT-x/EPT or AMD-V/RVI" option did allow me to boot into my VMs, however I could not boot into any VMs I already had suspended with the option turned on.

Here is what I did that worked:

addohm1_0-1638126110397.png

Uncheck Virtual Machine Platform

0 Kudos
milindng
Enthusiast
Enthusiast

Yes, this will work but it will like 32 bit and not 64 bit . So VM performance will be very slow. Some of existing VM will not run sometimes.

Better to uninstall / disable Hyper V from Windows 10 and disable Device Guard
Credential Guard

 

0 Kudos
addohm1
Contributor
Contributor

I don't have it installed.

 

As I said, I was contributing to the list of band-aid fixes.  Nothing else worked for me.

 

Also: 15.5.7 build-17171714

0 Kudos
raghuvamshi
Contributor
Contributor

disabling Virtualize Intel VT-x/EPT or AMD-V-RI and Virtualize CPU performance counters under VM Settings is what worked for me! 

 

Even if hypervisor is off, this setting must be disabled. At least on windows 10.

0 Kudos
charlielewis
Enthusiast
Enthusiast

@bleys wrote:

 

 

Quick summary to disable Hyper-V/VBS

Disable the following Windows components (they all cause Hyper-V to run, preventing the VMware Hypervisor from running instead):
Hyper-V
Device Guard
Credential Guard
Virtual Machine Platform
Windows Hypervisor Platform

Microsoft Defender Application Guard

 


@bleys that is the conflict here. If we disable the components you are listing here, then Docker will not be able to run on the host using Windows Subsystem for Linux and for that matter we would not be able to run WSL standalone either. 

That is a productivity killer. Yes we will have VMware Workstation running "legit" as in it will use its own hypervisor and yes you will be able to run nested virtualization inside VMs. But loosing those two aforementioned features is not an acceptable compromise I believe may software developers are willing to make. 

 

 

alexander09431
Contributor
Contributor

Hello,  Thank you very much for the information about fixing this very annoying problem.

I have tried all that I can think of and I'm still unable to resolved the problem.

do you have any other subjection.

I'm trying to run GNS3 and just won't start.

new Windows 11 machine new installation of everything and there is no Hyper-V or any other virtualization software running on the machine.

Thank you much!!!

 

0 Kudos
raghuvamshi
Contributor
Contributor

please try downloading the latest version.

 

for me i downloaded 15.5.6 version of workstation pro and it resolved this issue.

0 Kudos
addohm1
Contributor
Contributor

Well, no fixes work for me anymore.  I have a VM I am trying to move over to an upgraded laptop and I can't get it to start up with virtualization enabled.  Two of the VMs are currently suspended and I cannot turn off virtualization.  Not sure what else to do.  Sure does seem weird that the VM works on one system and not the other when the two systems are pretty much the same.  In addition, some other VMs I have run with virtualization ticked.  In fact, a clone of this virtual machine, runs with virtualization.  The only difference between the two is a few extra pieces of software installed.  I made sure the bios was the same, the "features" were the same, the bcdedit was the same.  New VMs I create with virtualization ticked do not work.  

0 Kudos
addohm1
Contributor
Contributor

Regardless of the above (HV not enabled, `bcdedit /set hypervisorlaunchtype off` ran, system rebooted)...  systeminfo.exe still shows... 

 

addohm1_0-1645284646540.png

 

0 Kudos
addohm1
Contributor
Contributor

I was able to get this working by the following (make sure to read through to the bottom):

After rebooting my PC, and starting up my Ubuntu terminal I didn't know Windows (by default) utilizes the Hyper-V process to enable virtualization for the Ubuntu terminal. What I also didn't know is Win10 CANNOT simultaneously run 2x different virtualization software; in my case, both Hyper-V and VMware Workstation Pro.

 

As I mentioned before, uninstalling the Ubuntu application, disabling WSL on my Win10 PC within "Turn Windows features on or off", and rebooting my computer DID NOT resolve the GNS3 VM startup issues within VMware Workstation. What I found, which was very odd, is even though ANY/ALL references to Hyper-V were NOT checked (enabled) within "Turn Windows features on or off", running the following system utility command within a terminal window ---> systeminfo.exe, showed Hyper-V was STILL running on my system due to the following output displayed:

 

Hyper-V Requirements: A hypervisor has been detected. Features required for Hyper-V will not be displayed 

 

 

The moment I saw that, I used the following powershell command (run as admin) to disable Hyper-V in Boot Manager, and rebooted my Win10 PC:

 

bcdedit /set hypervisorlaunchtype off

 

 

After my Win10 PC rebooted successfully, I ran the systeminfo.exe, system utility command once again to verify Hyper-V was truly disabled. Thankfully, it was as I saw the following output displayed this time around:

 

Hyper-V Requirements:     VM Monitor Mode Extensions: Yes

                                             Virtualization Enabled In Firmware: Yes

                                             Second Level Address Translation: Yes

                                             Data Execution Prevention Available: Yes

 

 

I am pleased to say, once everything mentioned above was done... I started the GNS3 application, which successfully powered on the GNS3 VM in VMware Workstation Pro, and I was able to open my projects once again!

 

NOTE: All devices and their associated configurations within my GNS3 projects remained intact. None of my GNS3 projects were negatively impacted by this virtualization fiasco. Also, the following sites helped me identify the issue and provided the correct solution for my GNS3 VM startup problems:


After following this, I still couldnt get HyperVisor fully turned off, which prompted me to google "fully disable hypervisor" which lead me to the following link, which is what finally worked: Disable Hyper-V to run virtualization software - Windows Client | Microsoft Docs

Keep in mind I had still already gone through all the other suggested fixes.

 

0 Kudos
luhe7876
Contributor
Contributor

I have closed hyper-v and memory integrity. However, Vmware Workstation still tell me  does not support nested virtualization on this host. I need to use nested virtualization. How can I solve it?

0 Kudos
NSX007
Contributor
Contributor

As simple as doing this. Thanks a lot! you save me a lot of troubles!.

0 Kudos