Terminology
The phrase "running nested VMs" refers to running a VM inside another VM. The outer guest is the VM that runs on physical hardware. The inner guest (or nested guest) is the VM that runs within another VM. The host hypervisor is the hypervisor that runs on the physical hardware. The guest hypervisor is the hypervisor that runs within a VM. To avoid confusion, we will not consider deeper levels of nesting here.
Note that no Type 1 hypervisors are supported as guest operating systems under any VMware product. In particular, ESXi is not supported as a guest operating system, and VMware discourages running ESXi as a guest hypervisor in production environments. (See http://kb.vmware.com/kb/2009916 for VMware's official policy on running ESXi as a guest hypervisor.)
Running Guest Hypervisors with Virtualized HV
Most hypervisors require hardware-assisted virtualization (HV). VMware products require hardware-assisted virtualization for 64-bit guests on Intel hardware. When running as a guest hypervisor, VMware products also require hardware-assisted virtualization for 64-bit guests on AMD hardware. The hardware-assisted virtualization features of the physical CPU are not typically available in a VM, because most hypervisors (from VMware or others) do not virtualize HV. However, Workstation 8, Player 4, Fusion 4, and ESXi 5.0 (or later) offer virtualized HV, so that you can run guest hypervisors which require hardware-assisted virtualization.
With virtualized HV enabled for the outer guest, you should be able to run any guest hypervisor that requires hardware-assisted virtualization. In particular, this means that you will be able to run 64-bit nested guests under VMware guest hypervisors.
ESXi 6.0
Virtualized HV is fully supported for virtual hardware version 9 or later VMs on hosts that support Intel VT-x and EPT or AMD-V and RVI. To enable virtualized HV, use the web client and navigate to the processor settings screen. Check the box next to "Expose hardware-assisted virtualization to the guest operating system." This setting is not available under the traditional C# client.
Workstation 11 / Player 7 / Fusion 7
Virtualized HV is fully supported for virtual hardware version 9 or later VMs on hosts that support Intel VT-x and EPT or AMD-V and RVI. To enable virtualized HV, select VM->Settings and navigate to the processor settings screen. Check the box next to "Virtualize Intel VT-x/EPT or AMD-V/RVI."
Workstation 10 / Player 6 / Fusion 6
Virtualized HV is fully supported for virtual hardware version 9 or 10 VMs on hosts that support Intel VT-x and EPT or AMD-V and RVI. To enable virtualized HV, select VM->Settings and navigate to the processor settings screen. Check the box next to "Virtualize Intel VT-x/EPT or AMD-V/RVI."
ESXi 5.5
Virtualized HV is fully supported for virtual hardware version 9 or 10 VMs on hosts that support Intel VT-x and EPT or AMD-V and RVI. To enable virtualized HV, use the web client and navigate to the processor settings screen. Check the box next to "Expose hardware-assisted virtualization to the guest operating system." This setting is not available under the traditional C# client.
Workstation 9 / Player 5 / Fusion 5
Virtualized HV is fully supported for virtual hardware version 9 VMs on hosts that support Intel VT-x and EPT or AMD-V and RVI. To enable virtualized HV, select VM->Settings and navigate to the processor settings screen. Check the box next to "Virtualize Intel VT-x/EPT or AMD-V/RVI."
ESXi 5.1
Virtualized HV is fully supported for virtual hardware version 9 VMs on hosts that support Intel VT-x and EPT or AMD-V and RVI. To enable virtualized HV, use the web client and navigate to the processor settings screen. Check the box next to "Expose hardware-assisted virtualization to the guest operating system." This setting is not available under the traditional C# client.
Workstation 8 / Player 4
Virtualized HV is available for virtual hardware version 8 VMs on hosts that support Intel VT-x and EPT or AMD-V and RVI. To enable virtualized HV, select VM->Settings and navigate to the processor settings screen. Check the box next to "Virtualize Intel VT-x/EPT or AMD-V/RVI." You will see a warning that virtualized HV will make this VM incompatible with other VMware products. In particular, if this VM is moved to an ESXi 5.0 host, virtualized HV will not be available without additional tweaking.
Fusion 4
Virtualized HV is available for virtual hardware version 8 VMs on hosts that support Intel VT-x and EPT or AMD-V and RVI, but it cannot be selected through the user interface. To enable virtualized HV, edit the outer guest's configuration file by hand and add the following line:
vhv.enable = TRUE
ESXi 5.0
On ESXi 5.0, virtualized HV is prohibited by default. This feature is used internally within VMware for testing purposes, but it is not recommended for production systems. It is available on hosts that support Intel VT-x or AMD-V, but it is not recommended for systems without second level address translation (EPT or RVI), because of its poor performance without SLAT. Unfortunately, this feature does not work on AMD "Bulldozer" CPUs running ESXi 5.0.
To allow the use of this feature, the ESXi administrator must add the following configuration option to the /etc/vmware/config file on the physical host:
vhv.allow = TRUE
Once allowed by the ESXi administrator, virtualized HV will be enabled by default for all hardware version 8 VMs with guest OS type "ESX Server 4" and "ESX Server 5." To enable virtualized HV for other guests, add the following lines to the outer guest's configuration file:
cpuid.1.ecx="----:----:----:----:----:----:--h-:----"
cpuid.80000001.ecx.amd="----:----:----:----:----:----:----:-h--"
cpuid.8000000a.eax.amd="hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh"
cpuid.8000000a.ebx.amd="hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh"
cpuid.8000000a.edx.amd="hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh"
Running Guest Hypervisors without Virtualized HV
VMware products prior to ESXi 5.0, Workstation 8 or Fusion 4 do not virtualize the hardware-assisted virtualization capabilities of the physical processor (or 64-bit segment limit checks on AMD processors). If the host hypervisor is an older VMware product, or if your host does not support EPT or RVI (but it does support Intel VT-x or AMD-V) you can still run nested guests without virtualized HV. However, you will only be able to run 32-bit nested guests using binary translation under a VMware guest hypervisor. You can also run 32-bit nested guests using the binary translation features of Windows Virtual PC or VPC2007.
Even in this configuration, running nested VMs requires host support for hardware-assisted virtualization. Furthermore, the outer guest must be configured to use hardware-assisted virtualization. For instructions on enabling hardware-assisted virtualization for the outer guest, see VMware Products and Hardware-Assisted Virtualization (VT-x/AMD-V).
Special Considerations
Networking in Nested Guests
For networking to work in a nested guest, the network interfaces of the outer guest must be put into promiscuous mode. For details, see KB 287 (Linux hosts) or KB 1004099 (ESX hosts). No special configuration is required for Windows hosts.
EVC Clusters
Virtualized HV may not work on hosts that are part of an EVC cluster, due to the feature masking that is performed to make all of the CPUs in the cluster look alike.
VMware Tools
Beginning with ESXi 5.0, Workstation 8 and Fusion 4, VMware hypervisors implement a handshake between the host hypervisor and the guest hypervisor which allows you to run VMware Tools in both the inner guest and the outer guest. Note that both hypervisors must be of this recent vintage for the handshake to work.
If either hypervisor is older than this, you will only be able to run VMware Tools in the inner guest. To ensure that VMware Tools work properly, you must either set the outer guest OS type to "ESX Server 4" or "ESX Server 5," or you must add the following line to the outer guest's configuration file:
monitor_control.restrict_backdoor = TRUE
Note that this setting is not available when your host hypervisor is ESX 3.0. You cannot use VMware Tools in the inner guest if your host hypervisor is ESX 3.0.
Guest Hypervisors
ESX(i)
ESX(i) may be used as a guest hypervisor to learn about VMware's server products, experiment with server setup, conduct training, show demos, and test configurations. VMware does not support running nested ESX(i) servers in production environments. Issues running ESX(i) in a nested configuration fall outside of VMware's Support and Service Level Agreements. If you experience issues, VMware is under no obligation to acknowledge or investigate immediately or to provide a resolution. However, VMware is interested in obtaining an easily reproducible scenario for our engineers to investigate through discussions in the Nested Virtualization community forum.
Windows Virtual PC (XP mode)
The files required to run an XP mode VM under Windows 7 are available from http://www.microsoft.com/windows/virtual-pc/download.aspx. To use Windows Virtual PC with binary translation, all three downloads are needed. Without the binary translator, Windows Virtual PC requires hardware-assisted virtualization.
Windows Virtual PC has a bug related to power-saving states when running with Intel VT-x. If you try to sleep or hibernate the outer Windows VM while a nested Windows Virtual PC VM is running, you may run into a variety of issues when the outer guest is awakened. To solve this problem, you will have to install the hotfix from http://support.microsoft.com/?kbid=977632. Even if the description does not sound like it is applicable, this hotfix should help with any sleep/hibernate issues.
Hyper-V
Hyper-V requires hardware-assisted virtualization, so it can only be run under ESXi 5.0, Workstation 8, Player 4 or Fusion 4 (or later). Hyper-V performs relatively poorly as a guest hypervisor under ESXi 5.0, but it performs reasonably well under Workstation 8, Player 4 or Fusion 4 (or later).
Under Workstation 9, Player 5 or Fusion 5, you should set the guest OS type to "Hyper-V."
Under older products, Hyper-V requires the following additional configuration option for the outer guest:
hypervisor.cpuid.v0 = FALSE
Without this option, launching a nested VM under Hyper-V R2 will fail with the following error:
Failed to create partition: Unspecified error (0x80004005)
Without this option, Hyper-V R3 will refuse to install, claiming:
Hyper-V cannot be installed: A hypervisor is already running.
Hyper-V R4 requires virtual hardware version 11 or greater. Under older virtual hardware versions, Hyper-V R4 will fail with the following error:
Hypervisor launch failed; The hypervisor was unable to initialize successfully (phase 0x2), and was not started. This initialization failure may be the result of a platform configuration or firmware issue. Contact your system vendor for more information or updated firmware.>
Xen
Xen requires hardware-assisted virtualization, so it can only be run under ESXi 5.0, Workstation 8, Player 4 or Fusion 4 (or later). The version of Xen provided with most Linux distributions performs relatively poorly as a guest hypervisor. Citrix XenServer 5.6 performs reasonably well.
KVM
KVM requires hardware-assisted virtualization, so it can only be run under ESXi 5.0, Workstation 8, Player 4 or Fusion 4 (or later). KVM performs relatively poorly as a guest hypervisor on Intel CPUs using virtualized Intel VT-x. Performance is improved with Linux kernel versions 3.0 and greater.
Under some VMware products, libvirtd hangs awaiting completion of a dmidecode child process. If this happens, you will be unable to connect to the local hypervisor. A workaround is to disable CPU hot-add for the outer VM with the following configuration option:
vcpu.hotadd = FALSE
VMware as a Guest Hypervisor Under Foreign Hypervisors
If we detect that a VMware product is running under a foreign hypervisor, we will refuse to power on a nested guest. To bypass this constraint, add the following option to your nested guest's configuration file:
vmx.allowNested = TRUE
VMware Workstation under Hyper-V
You can run the VMware binary translator (for 32-bit guests) under Hyper-V if your host has SLAT support. Without SLAT support, if you attempt to run a nested VM under VMware Workstation, Hyper-V will synthesize a machine check and BSOD the host.
The Workstation installer won't allow you to install Workstation while the Hyper-V role is active, so you will have to disable Hyper-V to do the installation. See http://blogs.msdn.com/b/virtual_pc_guy/archive/2008/04/14/creating-a-no-hypervisor-boot-entry.aspx for instructions on how to boot Windows 2008 with Hyper-V disabled, without actually having to remove the Hyper-V role.
.