VMware Communities
deuberger
Contributor
Contributor
Jump to solution

Nested Virtualization Degraded Performance

I'm trying to get nested virtualization working / set up properly and am having some issues apparently. My ultimate goal is the following stack

0. Bare Metal w/ virtualization extensions

1. Windows 7 Enterprise

2. VMWare Workstation 10

3. Ubuntu 13.04 64 bit

4. Virtual Box (ideally) or VMWare Workstation 10 (alternatively)

5. Various guest OSs (mainly RHEL, CentOS, and Ubuntu 64 bit)

Everything is working great up through Ubuntu and I've verified that Intel VT-x/EPT is showing up as enabled in Ubuntu (#4), meaning it should be possible for the inner hypervisor to make use of them.

Using VMWare Workstation as the inner Hypervisor works it seems, but I get a question on startup saying "Running VMWare Workstation in a virtual machine will result in degraded performance. Do you want to continue?" Will this warning always show up or does it indicate that I've got something set up wrong for proper nested virtualization?

Using Virtual Box, things are very slow and sometimes fail completely (kernel panic, etc). Since VMWare virtualizes the hardware extensions, I would think that Virtual Box should work fine, but the article - Running Nested VMs - doesn't say for sure one way or the other. Is it possible to use Virtual Box as the inner hypervisor? If not, why? I know that virtual box can't be the outer hypervisor (because of #4032 ([feature-request] Nested Virtualization: VT-in-VT) – Oracle VM VirtualBox ), but I see know reason why it couldn't be the inner.

What am I missing?

Reply
0 Kudos
1 Solution

Accepted Solutions
admin
Immortal
Immortal
Jump to solution

deuberger wrote:

I'm trying to get nested virtualization working / set up properly and am having some issues apparently. My ultimate goal is the following stack

0. Bare Metal w/ virtualization extensions

1. Windows 7 Enterprise

2. VMWare Workstation 10

3. Ubuntu 13.04 64 bit

4. Virtual Box (ideally) or VMWare Workstation 10 (alternatively)

5. Various guest OSs (mainly RHEL, CentOS, and Ubuntu 64 bit)

Everything is working great up through Ubuntu and I've verified that Intel VT-x/EPT is showing up as enabled in Ubuntu (#4), meaning it should be possible for the inner hypervisor to make use of them.

Using VMWare Workstation as the inner Hypervisor works it seems, but I get a question on startup saying "Running VMWare Workstation in a virtual machine will result in degraded performance. Do you want to continue?" Will this warning always show up or does it indicate that I've got something set up wrong for proper nested virtualization?

It will always say that.

Using Virtual Box, things are very slow and sometimes fail completely (kernel panic, etc). Since VMWare virtualizes the hardware extensions, I would think that Virtual Box should work fine, but the article - Running Nested VMs - doesn't say for sure one way or the other. Is it possible to use Virtual Box as the inner hypervisor? If not, why? I know that virtual box can't be the outer hypervisor (because of #4032 ([feature-request] Nested Virtualization: VT-in-VT) – Oracle VM VirtualBox ), but I see know reason why it couldn't be the inner.

It could be a bug in our virtualization of VT-x, or it could just be because things are too slow with VirtualBox as the inner hypervisor. VirtualBox is perhaps the most challenging guest hypervisor for virtualized VT-x, because it performs the most VMREADs and VMWRITEs in response to a (nested) VM-exit, and the placement of these instructions in VirtualBox defeats our clustering strategy for reducing VM-exits.  Performance should be improved with Workstation 11 as the outer hypervisor if you have a Haswell CPU with support for VMCS shadowing.

What sort of kernel panics, etc. are you seeing?

View solution in original post

Reply
0 Kudos
2 Replies
admin
Immortal
Immortal
Jump to solution

deuberger wrote:

I'm trying to get nested virtualization working / set up properly and am having some issues apparently. My ultimate goal is the following stack

0. Bare Metal w/ virtualization extensions

1. Windows 7 Enterprise

2. VMWare Workstation 10

3. Ubuntu 13.04 64 bit

4. Virtual Box (ideally) or VMWare Workstation 10 (alternatively)

5. Various guest OSs (mainly RHEL, CentOS, and Ubuntu 64 bit)

Everything is working great up through Ubuntu and I've verified that Intel VT-x/EPT is showing up as enabled in Ubuntu (#4), meaning it should be possible for the inner hypervisor to make use of them.

Using VMWare Workstation as the inner Hypervisor works it seems, but I get a question on startup saying "Running VMWare Workstation in a virtual machine will result in degraded performance. Do you want to continue?" Will this warning always show up or does it indicate that I've got something set up wrong for proper nested virtualization?

It will always say that.

Using Virtual Box, things are very slow and sometimes fail completely (kernel panic, etc). Since VMWare virtualizes the hardware extensions, I would think that Virtual Box should work fine, but the article - Running Nested VMs - doesn't say for sure one way or the other. Is it possible to use Virtual Box as the inner hypervisor? If not, why? I know that virtual box can't be the outer hypervisor (because of #4032 ([feature-request] Nested Virtualization: VT-in-VT) – Oracle VM VirtualBox ), but I see know reason why it couldn't be the inner.

It could be a bug in our virtualization of VT-x, or it could just be because things are too slow with VirtualBox as the inner hypervisor. VirtualBox is perhaps the most challenging guest hypervisor for virtualized VT-x, because it performs the most VMREADs and VMWRITEs in response to a (nested) VM-exit, and the placement of these instructions in VirtualBox defeats our clustering strategy for reducing VM-exits.  Performance should be improved with Workstation 11 as the outer hypervisor if you have a Haswell CPU with support for VMCS shadowing.

What sort of kernel panics, etc. are you seeing?

Reply
0 Kudos
deuberger
Contributor
Contributor
Jump to solution

jmattson wrote:

It could be a bug in our virtualization of VT-x, or it could just be because things are too slow with VirtualBox as the inner hypervisor. VirtualBox is perhaps the most challenging guest hypervisor for virtualized VT-x, because it performs the most VMREADs and VMWRITEs in response to a (nested) VM-exit, and the placement of these instructions in VirtualBox defeats our clustering strategy for reducing VM-exits.  Performance should be improved with Workstation 11 as the outer hypervisor if you have a Haswell CPU with support for VMCS shadowing.

What sort of kernel panics, etc. are you seeing?

That's very interesting/enlightening. Thanks for the info. Unfortunately, I don't currenlty have a Haswell CPU, but I'll look forward to the improvements whenever they come.

Sorry for the slow reply, but I haven't been able to reproduce consistent results and have mostly given up doing so because the nested Virtual Box guest is too slow. I can't remember off hand what the exact messages were that I got before, but my gut reaction was that they were caused by a kernel panic. I could have been wrong. Now the VMs are booting and sometimes are even responsive, but sometimes aren't. For now, I'm happy knowing that this is expected and will wait till I have a configuration that is expected to work before pursuing it any farther. If for some reason, I do play with it some more and am able to reproduce anything useful, I'll post back here.

Thanks for the help.

Reply
0 Kudos