Nested virtualization not available in Solaris 11.2

[mortengh]

Hello!

Anybody know why virtual hardware virtualization is not available for my Solaris 11.2 guest?

I want to be able to use Solaris kernel zones.

Host environment:

   SuperMicro A1SRM-2758F, 32GB RAM

   ESXi-5.5 (build 2302651)

I know the CPU (Atom C2758) supports both VT-x and EPT that is required for 64bit virtualization nesting

And browsing to:

   https://myip/mob/?moid=ha-host&doPath=capability

Shows me that the (unsupported) feature is available:

nestedHVSupported

boolean

true

I have created a VM with vHardware vmx-10, selected 'Expose hardware assisted virtualization to the guest OS' and installed Solaris-11.2 as a guest.

I have also checked that 'vhv.enable = "TRUE" ' is to be found in the vmx file.

But from the Solaris guest Kernel zone are not available:

# virtinfo

NAME            CLASS

vmware          current

non-global-zone supported

# uname -a

SunOS global1 5.11 11.2 i86pc i386 i86pc

Anybody have any ideas why this is not available. Worked fine in VMware Fusion.

Best regards,

- Morten Green Hermansen, Fanitas

[admin]

Please post the vmware.log file.

[mortengh]

Ups sorry! Here it comes!

In the meanwhile I upgraded ESXi-5.5 to build 2456374 and upgraded to the the newest vmtools (9.4.11). Still not working for me.

Best regards,

- Morten Green Hermansen, Fanitas

[admin]

Virtualized hardware-assisted virtualization is being exposed to the guest, so the next question is why Solaris is unwilling to use it.

When you say that it works under Fusion, is that with virtual hardware version 10 or 11?

[mortengh]

Hello jmattson!

WOW! That was a fast reply! 😮

But ok! This was on Fusion-7(.1.1) so that would be vHW-11.

But I think I have read that vhv was available from vHW-9?

I see that ESXi-6 is using vHW-11.

Best regards,

- Morten Green Hermansen, Fanitas

[admin]

mortengh wrote:

Hello jmattson!

WOW! That was a fast reply! 😮

But ok! This was on Fusion-7(.1.1) so that would be vHW-11.

But I think I have read that vhv was available from vHW-9?

I see that ESXi-6 is using vHW-11.

Virtualized HV has been available from virtual hardware version 9, but new functionality has been added incrementally in virtual hardware version 10 and virtual hardware version 11.  Since we know that Solaris likes vVT from virtual hardware version 11, I would suggest trying this advanced configuration option with your virtual hardware version 10 VM on ESXi 5.5:

vvt.etmt.uc = TRUE

I think that's the only vVT new feature in virtual hardware version 11 that is likely to be relevant.  It was added so that Intel's HAXM emulator would be happy with our virtual VT-x implementation.  Maybe it's the missing piece that Solaris is looking for as well.

[mortengh]

Hello jmattson!

I tried to add:

  vvt.etmt.uc = "TRUE"

to the vmx file and according to the log it got picked up:

...

2015-03-31T20:03:40.790Z| vmx| I120: DICT	featMask.vm.hv.capable = Min:1
2015-03-31T20:03:40.790Z| vmx| I120: DICT	vhv.enable = TRUE
2015-03-31T20:03:40.790Z| vmx| I120: DICT	vvt.etmt.uc = TRUE

2015-03-31T20:03:40.790Z| vmx| I120: DICT sched.scsi0:4.throughputCap = off

2015-03-31T20:03:40.790Z| vmx| I120: DICT sched.scsi1:3.throughputCap = off

...

But Solaris will still not support kernel zones:

# virtinfo

NAME            CLASS

vmware          current

non-global-zone supported

Will I then need to upgrade to ESXi-6 - and downgrade to the Web Client :'( for this to work?

Best regards and thank you SO much for your help!

- Morten Green Hermansen, Fanitas

[admin]

I would be surprised if ESXi 6.0 would help in this regard.

Are there any log files in the Solaris guest that might indicate why it is unhappy with the VT-x implementation?

[admin · Accepted solution]

I think I found it.  Solaris 11.2 apparently does not like the model and stepping of your Avoton CPU.  If I change the model and stepping of my Haswell system to match your Avoton, I also lose support for guest kernel zones in my Solaris 11.2 guest.

Try these configuration options:

featMask.vm.cpuid.MODEL = Val:0x3c

featMask.vm.cpuid.STEPPING = Val:0x3

That will make it look like you have a Haswell CPU (at least insofar as the model and stepping are concerned).

[mortengh]

Weee! jmattson!

You are a GENIUS!! After adding those settings see what I got:

# virtinfo

NAME            CLASS

vmware          current

non-global-zone supported

kernel-zone     supported

PERFECT! Thanks!

So does this mean that I still also need the 'vvt.etmt.uc' setting?

Best regards and thank you SO much for all your help!

- Morten Green Hermansen, Fanitas

[admin]

mortengh wrote:

Weee! jmattson!

You are a GENIUS!! After adding those settings see what I got:

# virtinfo

NAME            CLASS

vmware          current

non-global-zone supported

kernel-zone     supported

PERFECT! Thanks!

Great!  Note, however, that this means that Solaris does not support guest kernel zones on Avoton CPUs.  I can't imagine why, but it appears to simply blacklist them based on model number.  You are bypassing this restriction by lying about your CPU's model number.

So does this mean that I still also need the 'vvt.etmt.uc' setting?

No; that setting appears to be irrelevant.

[mortengh]

Super!

The Solaris-11.2 installation will not even boot on this motherboard! (SuperMicro A1SRM-2758F) :-S

Now I will play around with it and see if it kernel-panics.

I will post my findings below in a couple of days.

Thanks again!

- Morten Green Hermansen, Fanitas