The phrase "running nested VMs" refers to running a VM inside another VM. The outer guest is the VM that runs on physical hardware. The inner guest (or nested guest) is the VM that runs within another VM. The host hypervisor is the hypervisor that runs on the physical hardware. The guest hypervisor is the hypervisor that runs within a VM. To avoid confusion, we will not consider deeper levels of nesting here.
Note that no Type 1 hypervisors are supported as guest operating systems under any VMware product. In particular, ESXi is not supported as a guest operating system, and VMware discourages running ESXi as a guest hypervisor in production environments. (See http://kb.vmware.com/kb/2009916 for VMware's official policy on running ESXi as a guest hypervisor.)
Most hypervisors require hardware-assisted virtualization (HV). VMware products require hardware-assisted virtualization for 64-bit guests on Intel hardware. When running as a guest hypervisor, VMware products also require hardware-assisted virtualization for 64-bit guests on AMD hardware. The hardware-assisted virtualization features of the physical CPU are not typically available in a VM, because most hypervisors (from VMware or others) do not virtualize HV. However, Workstation 8, Player 4, Fusion 4, and ESXi 5.0 (or later) offer virtualized HV, so that you can run guest hypervisors which require hardware-assisted virtualization.
With virtualized HV enabled for the outer guest, you should be able to run any guest hypervisor that requires hardware-assisted virtualization. In particular, this means that you will be able to run 64-bit nested guests under VMware guest hypervisors.
Virtualized HV is fully supported for virtual hardware version 9 or later VMs on hosts that support Intel VT-x and EPT or AMD-V and RVI. To enable virtualized HV, use the web client and navigate to the processor settings screen. Check the box next to "Expose hardware-assisted virtualization to the guest operating system." This setting is not available under the traditional C# client.
Virtualized HV is fully supported for virtual hardware version 9 or later VMs on hosts that support Intel VT-x and EPT or AMD-V and RVI. To enable virtualized HV, select VM->Settings and navigate to the processor settings screen. Check the box next to "Virtualize Intel VT-x/EPT or AMD-V/RVI."
Virtualized HV is fully supported for virtual hardware version 9 or 10 VMs on hosts that support Intel VT-x and EPT or AMD-V and RVI. To enable virtualized HV, select VM->Settings and navigate to the processor settings screen. Check the box next to "Virtualize Intel VT-x/EPT or AMD-V/RVI."
Virtualized HV is fully supported for virtual hardware version 9 or 10 VMs on hosts that support Intel VT-x and EPT or AMD-V and RVI. To enable virtualized HV, use the web client and navigate to the processor settings screen. Check the box next to "Expose hardware-assisted virtualization to the guest operating system." This setting is not available under the traditional C# client.
Virtualized HV is fully supported for virtual hardware version 9 VMs on hosts that support Intel VT-x and EPT or AMD-V and RVI. To enable virtualized HV, select VM->Settings and navigate to the processor settings screen. Check the box next to "Virtualize Intel VT-x/EPT or AMD-V/RVI."
Virtualized HV is fully supported for virtual hardware version 9 VMs on hosts that support Intel VT-x and EPT or AMD-V and RVI. To enable virtualized HV, use the web client and navigate to the processor settings screen. Check the box next to "Expose hardware-assisted virtualization to the guest operating system." This setting is not available under the traditional C# client.
Virtualized HV is available for virtual hardware version 8 VMs on hosts that support Intel VT-x and EPT or AMD-V and RVI. To enable virtualized HV, select VM->Settings and navigate to the processor settings screen. Check the box next to "Virtualize Intel VT-x/EPT or AMD-V/RVI." You will see a warning that virtualized HV will make this VM incompatible with other VMware products. In particular, if this VM is moved to an ESXi 5.0 host, virtualized HV will not be available without additional tweaking.
Virtualized HV is available for virtual hardware version 8 VMs on hosts that support Intel VT-x and EPT or AMD-V and RVI, but it cannot be selected through the user interface. To enable virtualized HV, edit the outer guest's configuration file by hand and add the following line:
vhv.enable = TRUE
On ESXi 5.0, virtualized HV is prohibited by default. This feature is used internally within VMware for testing purposes, but it is not recommended for production systems. It is available on hosts that support Intel VT-x or AMD-V, but it is not recommended for systems without second level address translation (EPT or RVI), because of its poor performance without SLAT. Unfortunately, this feature does not work on AMD "Bulldozer" CPUs running ESXi 5.0.
To allow the use of this feature, the ESXi administrator must add the following configuration option to the /etc/vmware/config file on the physical host:
vhv.allow = TRUE
Once allowed by the ESXi administrator, virtualized HV will be enabled by default for all hardware version 8 VMs with guest OS type "ESX Server 4" and "ESX Server 5." To enable virtualized HV for other guests, add the following lines to the outer guest's configuration file:
VMware products prior to ESXi 5.0, Workstation 8 or Fusion 4 do not virtualize the hardware-assisted virtualization capabilities of the physical processor (or 64-bit segment limit checks on AMD processors). If the host hypervisor is an older VMware product, or if your host does not support EPT or RVI (but it does support Intel VT-x or AMD-V) you can still run nested guests without virtualized HV. However, you will only be able to run 32-bit nested guests using binary translation under a VMware guest hypervisor. You can also run 32-bit nested guests using the binary translation features of Windows Virtual PC or VPC2007.
Even in this configuration, running nested VMs requires host support for hardware-assisted virtualization. Furthermore, the outer guest must be configured to use hardware-assisted virtualization. For instructions on enabling hardware-assisted virtualization for the outer guest, see VMware Products and Hardware-Assisted Virtualization (VT-x/AMD-V).
For networking to work in a nested guest, the network interfaces of the outer guest must be put into promiscuous mode. For details, see KB 287 (Linux hosts) or KB 1004099 (ESX hosts). No special configuration is required for Windows hosts.
Virtualized HV may not work on hosts that are part of an EVC cluster, due to the feature masking that is performed to make all of the CPUs in the cluster look alike.
Beginning with ESXi 5.0, Workstation 8 and Fusion 4, VMware hypervisors implement a handshake between the host hypervisor and the guest hypervisor which allows you to run VMware Tools in both the inner guest and the outer guest. Note that both hypervisors must be of this recent vintage for the handshake to work.
If either hypervisor is older than this, you will only be able to run VMware Tools in the inner guest. To ensure that VMware Tools work properly, you must either set the outer guest OS type to "ESX Server 4" or "ESX Server 5," or you must add the following line to the outer guest's configuration file:
monitor_control.restrict_backdoor = TRUE
Note that this setting is not available when your host hypervisor is ESX 3.0. You cannot use VMware Tools in the inner guest if your host hypervisor is ESX 3.0.
ESX(i) may be used as a guest hypervisor to learn about VMware's server products, experiment with server setup, conduct training, show demos, and test configurations. VMware does not support running nested ESX(i) servers in production environments. Issues running ESX(i) in a nested configuration fall outside of VMware's Support and Service Level Agreements. If you experience issues, VMware is under no obligation to acknowledge or investigate immediately or to provide a resolution. However, VMware is interested in obtaining an easily reproducible scenario for our engineers to investigate through discussions in the Nested Virtualization community forum.
The files required to run an XP mode VM under Windows 7 are available from http://www.microsoft.com/windows/virtual-pc/download.aspx. To use Windows Virtual PC with binary translation, all three downloads are needed. Without the binary translator, Windows Virtual PC requires hardware-assisted virtualization.
Windows Virtual PC has a bug related to power-saving states when running with Intel VT-x. If you try to sleep or hibernate the outer Windows VM while a nested Windows Virtual PC VM is running, you may run into a variety of issues when the outer guest is awakened. To solve this problem, you will have to install the hotfix from http://support.microsoft.com/?kbid=977632. Even if the description does not sound like it is applicable, this hotfix should help with any sleep/hibernate issues.
Hyper-V requires hardware-assisted virtualization, so it can only be run under ESXi 5.0, Workstation 8, Player 4 or Fusion 4 (or later). Hyper-V performs relatively poorly as a guest hypervisor under ESXi 5.0, but it performs reasonably well under Workstation 8, Player 4 or Fusion 4 (or later).
Under Workstation 9, Player 5 or Fusion 5, you should set the guest OS type to "Hyper-V."
Under older products, Hyper-V requires the following additional configuration option for the outer guest:
hypervisor.cpuid.v0 = FALSE
Without this option, launching a nested VM under Hyper-V R2 will fail with the following error:
Failed to create partition: Unspecified error (0x80004005)
Without this option, Hyper-V R3 will refuse to install, claiming:
Hyper-V cannot be installed: A hypervisor is already running.
Hyper-V R4 requires virtual hardware version 11 or greater. Under older virtual hardware versions, Hyper-V R4 will fail with the following error:
Hypervisor launch failed; The hypervisor was unable to initialize successfully (phase 0x2), and was not started. This initialization failure may be the result of a platform configuration or firmware issue. Contact your system vendor for more information or updated firmware.>
Xen requires hardware-assisted virtualization, so it can only be run under ESXi 5.0, Workstation 8, Player 4 or Fusion 4 (or later). The version of Xen provided with most Linux distributions performs relatively poorly as a guest hypervisor. Citrix XenServer 5.6 performs reasonably well.
KVM requires hardware-assisted virtualization, so it can only be run under ESXi 5.0, Workstation 8, Player 4 or Fusion 4 (or later). KVM performs relatively poorly as a guest hypervisor on Intel CPUs using virtualized Intel VT-x. Performance is improved with Linux kernel versions 3.0 and greater.
Under some VMware products, libvirtd hangs awaiting completion of a dmidecode child process. If this happens, you will be unable to connect to the local hypervisor. A workaround is to disable CPU hot-add for the outer VM with the following configuration option:
vcpu.hotadd = FALSE
If we detect that a VMware product is running under a foreign hypervisor, we will refuse to power on a nested guest. To bypass this constraint, add the following option to your nested guest's configuration file:
vmx.allowNested = TRUE
You can run the VMware binary translator (for 32-bit guests) under Hyper-V if your host has SLAT support. Without SLAT support, if you attempt to run a nested VM under VMware Workstation, Hyper-V will synthesize a machine check and BSOD the host.
The Workstation installer won't allow you to install Workstation while the Hyper-V role is active, so you will have to disable Hyper-V to do the installation. See http://blogs.msdn.com/b/virtual_pc_guy/archive/2008/04/14/creating-a-no-hypervisor-boot-entry.aspx for instructions on how to boot Windows 2008 with Hyper-V disabled, without actually having to remove the Hyper-V role.
Nice piece of work,
If you found this or any other answer useful please consider the use of the Helpful or correct buttons to award points
VMware Communities User Moderator
Where this document did provide information that was good (vital), a couple of things to add are that the "configuration" file is the .vmx file found with the rest of the vm files. Also, the lval's shown in the example should have quotes around them (as do the other entries in the vmx file).
The vmtools part is ugly (oh, so ugly) ... at least in the matrix point I tried.
outer vm XP installed on fusion 3.0.2
inner vm linux installed on Player 3.0.1
I had installed vmtools first on XP; then put the vm (XP) in hardware mode as outlined by the article. I then successfully installed vm player on the outer vm. Then I installed linux as the inner vm. I got indications that vmtools needed to be installed on the outer vm (vmtools not installed ...)
I was not sure if vmtools needed to be reinstalled with the change to the configuration file ... so I tried installing.
I got messages including these: "the vmware tools should only be installed inside a virtual machine"; then "the wizard was interrupted before vmware tools could be completely installed" ... and a message encouraging the tools installation to be done again -- "please run the installation again". First I thought it might be a result of some setting or software in the (xp) vm ... so I opened it wide up ... same result. I thought maybe I would need to uninstall the tools to reinstall again. After uninstalling, the tools will still not install and now I don't have the pretty tools features available.
As a bonus, after tools was uninstalled player would no longer find my linux vm. This was because the shared files (shared with the host) are no longer available when tools are not installed.
I suppose I should have been wary and backed up my stuff before embarking on this route. ... and apparently I should not have trusted the tools install.
Be careful with the tools installs and don't trust the messages.
On with VMware support, I'm told community documentation isn't official and in fact ESXi isn't supported.. Nothing like the company conflicting with itself.
I should have been more clear regarding the genesis of my post... Using Fusion 3.1.1 to run ESXi due to lack of support for Apple hardware RAID card. Working with VMware support on kernel panics while running ESXi, they say ESXi isn't officially supported in Fusion, only ESX. Trying ESX now and I'm receiving same kernel panics.
I'm using an AMD Athlon Neo X2 Dual Core Processor L335, does it support for running nested VMs?
I believe that the L335 is a family 0fH processor. If so, then the answer is no. Family 10h or later is required for AMD64 processors.
How can you tell that L335 is a family 0fH processor? Do you have some links for me to learn deeper about it..some of the new microservers use L36 and L335, will that be impacted if we plan to install ESXi 4.0 hypervizor (or later) directly on top of that particular machine (without VMWare Workstation)?
I googled for 'L335 family model stepping.' I couldn't really find a definitive source.
Since you have the processor, you should be able to determine the answer. Under Linux, cat /proc/cpuinfo. For Windows, you can use a third-party tool like CPU-Z. Both of these approaches report the family in decimal, so 15 is 0fH and 16 is 10H.
As long as your system is on the ESXi HCL, you should not have any problems due to this particular processor.
First of, thanks for this documentation. I do have a question regarding nesting OS' when having a guest hypervisor.
I currently have a computer with VT-capable functionality where I installed an ESXi system. The following sketch is what I'm trying to resolve.
"Host system" Windows 7 Ultimate x64 with an Intel Core i5 450M with 4 GB of RAM
.........-> "Host Hypervisor" VMware Workstation 7.1.3
...................-> "Guest Hypervisor" VMware ESXi 4.1
.........................-> "Nested OS" Windows Server 2008 R2
I installed the VMware workstation (running at x64) and ESXi server (running at x64 as well) without any inconveniences within the host machine, yet, when launching the nested OS I encounter the error message that the virtual machine I'm trying to run is not capable of Longmode (i.e. x64).
What I've done:
1. I'd placed inside the ESXi VMX file the parameters given here for allocation the VT capability
2. I'd placed inside the nested OS' VMX file the necessary parameters to use both, the VT capability as well as the nested argument.
My question is now, is there a way the nested OS can reach the VT of the host machine and be capable of using the longmode procedures enabled through the host machine?
I'd tested my equipment with the VT.iso file given and it reflects it can indeed assume VT.
Thanks for all help provided.
The problem is that currently VT is not passed up to the first guest OS (ESXi). Thus it only sees a 64 bit CPU which allows you to install ESXi, but without VT you can't run 64 bit VM. You can still run nested 32 bit VMs on the ESXi VM, but in that case you would have to go with Win2008 x86 or earlier.
VMware Communities User Moderator
Forum Upgrade Notice - the VMware Communities forums will be upgraded the weekend of December 12th. The forum will be in read-only mode from Friday, December 10th 6 PM PST until Sunday, December 12th 2 AM PST.
Now available - VMware ESXi: Planning, Implementation, and Security
Also available - vSphere Quick Start Guide
Do you have a system or PCI card working with VMDirectPath? Submit your specs to the Unofficial VMDirectPath HCL .
No current VMware product virtualizes VT-x (or AMD-V), so the virtual CPU on which you are running your guest hypervisor does not have VT-x support. As a result, you can only run 32-bit nested guests under the guest hypervisor (using binary translation). There is no workaround for this limitation.
Thanks for such a prompt reply! I was just wondering, will this limitation be implemented in a future version?
Once again thanks a lot for all your assistance. I'd foresee this answer and installed W2K3 x86 as a nested VM.
Sorry. VMware policy is to not comment on unannounced features, products, timelines, etc.
Thanks Dave! I thought that was the issue even though I was hoping to do some sort of VMX additions in order to pass the VT-x capabilities to the nested VM.
I had a hunch on this matter when I entered the virtual BIOS of the nested VM. Yet it would be great if this limitation could be placed in a newer version of either VMware Workstation, VMware ESX or ESXi so that nested VM's could benefit of VT-x and not only the guest VM.
Once again thanks for such a prompt reply.
On ESXi 4.1.0, if I edit the vESXi vmx and set guestOS = "vmkernel" it appears in the inventory as Other (32-bit). Is there any benefit to doing this? -- I have a few other test vESXi set to RHEL 5 (64-bit) and these seem to be working just as well as the one where I set "vmkernel"
It should save you the step of adding "monitor_control.restrict_backdoor = TRUE" to the configuration file. (This is necessary for VMware Tools to run properly in nested guests.)
So just to confirm, as in my situation, I am running a host computer with Windows 2008 R2 64-bit. I have installed VMPlayer and created a VM running ESXi 4.1. After accessing the ESXi server through the vSphere client, I will NOT be able to create a virtual machine running a 64-bit OS like Windows 7 Enterprise or Windows Server 2008?
At this point you won't be able to run a 64 bit VM on ESXi that is nested on VMPlayer.
ESXi 5.0 is the first product that I'm aware of that passes through the CPU capabilities to a nested hypervisor to allow it to run 64 bit guests.
Is ESXi 5.0 downloadable yet? If so, could you share the link?
I have installed the 5.0 vSphere client and ESXi 5.0 inside VMPlayer as a Virtual Machine.
When I try to add a virtual machine for Windows Server 2008 R2 64-bit, I get "Running VMware ESX in a virtual machine requires the outer virtual machine to be configured for running a VMware ESX guest operating system. You may not power on a virtual machine until the outer virtual machine is reconfigured."
The host itself has to be running ESXi 5.0. The guest VM can run ESXi 4/5 or another hypervisor. The VMs you then run on the nested hypervisor can be 64 bit.
Oh, I guess I can't do it the way I have it set up then. Windows Server 2008 R2 running as the host OS. VMPlayer running with a VM running ESXi 5.0. I wanted to created a VM from within vSphere on the ESXi 5.0 host for Windows 7 Enterprise 64-bit.
Also, I can't pull down a license key for 5.0 for some reason and 4.1 doesn't work with it.
I believe license keys will be available by the end of the week.
Is the license key available now? I just hope we can get it this week
About nested XP Mode...
Indeed, running Windows XP Mode with VMware Player 4 inside a Fusion 4 (or Workstation 😎 Windows 7 VM now works correctly, with the Tools running both in the outer and inner guest: everything seems to be OK, and the speed is certainly acceptable for light uses.
Very good! :smileycool:
BTW, until Player 4 will be available for direct download, it can be obtained as a part of the Workstation 8 (trial) package.
P.S.: It would be great if the Player XP Mode applications were visible also from the host (for Windows VPC, they are added to All Programs in the Start menu): for example, in Mac OS X, from the Fusion 4 Applications menu...
Hello Guys, I have Intel Core 2 Quad Q9400 2.66Ghz Cpu. I also enabled VT technology from BIOS. Installed windows 7 64-bit Enterprise edition to my PC. Shotly like that Windows 7 64-bit--->VMWare Workstation 8---> ESXI 5. Althought when I try open ESXI it says VT is not enabled on this PC also it shows message like I could not install 64-bit OS inside ESX 5. I researched everywhere but could not find an aswer. Please help me for this issue....
Thanks in advance...
Core 2 systems do not support EPT, so you will not be able to run nested 64-bit VMs.
Talking about foreign hypervisors, nested XP Mode (within a Windows 7 VM) now works properly also in Parallels Desktop 7 with VMware Player 4; in this case, one must first add
vmx.allowNested = TRUE
to the configuration file.
VMware Player 4 now imports XP Mode through the vCenter Converter, which one must first install, and in client-server mode.
BTW, Apple's Boot Camp Services are still incompatible with the importer, so if present they must be uninstalled and then reinstalled after the XP Mode import.
The classic Windows Virtual PC XP Mode still doesn't work in Parallels Desktop 7: configuring fails shortly before the end (don't know why, however).
Edit: In VirtualBox 4, I had similar results, after some quick experiments, on the same Mac OS X Lion host (so, YMMV): Windows VPC XP Mode doesn't work (exactly as in Parallels, failing at the end of the initial configuration), while Player 4 XP Mode works, albeit quite slowly; and it wasn't necessary to edit the .vmx file (so, VirtualBox isn't a "foreign" hypervisor?), while it was necessary to install the VMware Tools manually inside the inner guest: but they don't seem to work so well, inside the VB Player XP Mode VM.
So, in summary, currently the best nested XP Mode situation is clearly VMware's (Fusion 4, in my case): both Windows Virtual PC and VMware Player 4 work perfectly, and with reasonable speed.
When I install a new CentOS VM, I do not see vmx or svm support in /proc/cpuinfo, so I'm unable to use KVM. I also do not see any ESX Server versions as guest OS options when installing a new VM. Does this mean that my processor isn't compatible?
I'm running ESXi on a Dell 2950 and have added the vhv.allow = TRUE to /etc/vmware/config and added all of the lines to the CentOS guest's VMX file.
What kind of processor(s) do you have in your Dell 2950? Virtualized HV requires Nehalem or later Intel processors and Barcelona or later AMD processors.
Oops, it turns out that you can't choose ESX/i Server as a guest OS in the setup wizard and I had to actually just go back and edit the VM's settings after it was created. Changed the guest OS to ESX Server and now I'm showing the right flags in CentOS.
This got me upside down for a few days, it turns out on WS 8 I needed to upgrade virtual hardware to 8, as my VM's were at 7.x - then added the above..
Is the license key available right now?
Hyper-V performs relatively poorly as a guest hypervisor under ESXi 5.0, but it performs reasonably well under Workstation 8 or Fusion 4.
Could we know the reason ?
Thanks for the paper.
Workstation 8 and Fusion 4 have improved management of shadow nested page tables.
Is this planned to be added to ESXi too ?
VMware policy is to not comment on unannounced time-lines, products, features, etc.
Is it improved in Player too ?
Yes. Player 4 and Workstation 8 share the same core hypervisor.
Please clarify if I can run the free version of ESXi 5.0 on physical server and then create VMs to run the full eval version of ESXi 5.0 to test vMotion, etc? If so, then I'd save myself from having to purchase Workstation.
Would ESXi work better with nesting compared to Workstation? I have the option to install Workstation, but it looks like that will take up more resources since it runs on top of an existing OS. I only have 16 GB of RAM so I want to leave as much of that to the nested VMs as possible. Thank you.
Virtualized HV is more mature in Workstation 8 than in ESXi 5.0, but you should be able to do anything in ESXi 5.0 that you can do in Workstation 8. Some features of ESXi 5.0 will not work in a nested environment.
I upgraded my ESXi 5.0 to 5.1 several days ago, and somehow the ESXi does not recognize the VT-x/EPT support on the hardware anymore. I used to run 2 levels of nested virtualization using ESXis but now the the 2nd level cannot run anymore. I am running IBM x3850 A22 with Intel Xeon X7450, when I put vhv.enable = TRUE on the /etc/vmware/config, all VM startup complained with Virtualized Intel VT-x/EPT is not supported on this platform. Is there a way to 'debug' this as this was working in 5.0.
I can't find specs for an X7450, but the X7460 (which I presume is similar) does not have EPT support. Virtualized VT-x/EPT under ESXi 5.1 is not supported on your hardware.
We decided that virtualized VT-x without EPT was not product quality due to extremely poor performance, so we dropped support for it when virtualized VT-x/EPT became an official ESXi feature.
Thanks for the quick reply Jim.. It was a typo, should be E7450.
However, based on http://ark.intel.com/products/36941/Intel-Xeon-Processor-E7450-%2812M-Cache-2_40-GHz-1066-MHz-FSB%29 the last line indicates that the processor supports VT-x with EPT...
(I also checked http://ark.intel.com/products/36947/Intel-Xeon-Processor-X7460-%2816M-Cache-2_66-GHz-1066-MHz-FSB%29 and it shows that it supports VT-x/EPT).
Do I missed something here?
ark.intel.com is notoriously bad about reporting which processors have EPT support. The E7450 does not support EPT. The first CPUs to support EPT have a Nehalem core.
So where should I checked for an Intel processor features?