ESXi inside KVM

[LesykM]

Hello!

I wonder, if it is possible to run ESXi 5.0/5.1 inside KVM (proxmox) ?

I need to run few for testing purposes. I cannot setup any other virtualization software.

I need only ESXi itself, even without virtual machines inside it:)

Thanks!

[admin]

The only thing I can suggest is to switch to ESXi 6.0 and an e1000 vNIC.  That may be the only combination that works under kvm right now.

[WScott1966]

Where to start...

I am looking to get ESXi (6.0 Preferably) running in my KVM environment.

I have 3 nodes running CentOS 7 (1503) with KVM as the Hypervisor.

I have started to deploy CentOS domains(VM's) for a OpenStack instance.

But I cannot get ESXi 4,5,6 (any of them) to even show a boot screen!

I have a CentOS Desktop VM that I have installed Virt Manager so I can see the console and it doesn't boot!

I can boot any of the Linux VMs without a hitch.

I even have a WDS/PXE server and have tried to boot to PXE to grab the ESXi and I get nothing.

Can anyone assist?

Versions:

virsh -c qemu:///system version --daemon [Output]:

Compiled against library: libvirt 1.2.8

Using library: libvirt 1.2.8

Using API: QEMU 1.2.8

Running hypervisor: QEMU 1.5.3

Running against daemon: 1.2.8

[admin]

The installer disks won't even boot?  Are you using the UEFI firmware by any chance?  (And these are x86 machines, I assume.)

[WScott1966]

Ok, So I wokred past the non-bootig.

and I now can see the ESXi media load, but I am seeing the error:unsupported configuration: host doesn't support invariant TSC

Below is the XML for the VM:

(As you can see I am using the host-model processor to expose the VT functionality of the L5520 processor to th VM.)

<domain type='kvm'>

  <name>gsf-kESXi01</name>

  <uuid>5d3a2a39-8937-4aa3-885f-5b48ef6d86a4</uuid>

  <memory unit='KiB'>4194304</memory>

  <currentMemory unit='KiB'>4194304</currentMemory>

  <vcpu placement='static'>4</vcpu>

  <os>

    <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>

    <boot dev='hd'/>

  </os>

  <features>

    <acpi/>

    <apic/>

    <pae/>

  </features>

  <cpu mode='custom' match='exact'>

    <model fallback='allow'>Nehalem</model>

    <vendor>Intel</vendor>

    <feature policy='require' name='tm2'/>

    <feature policy='require' name='est'/>

    <feature policy='require' name='monitor'/>

    <feature policy='require' name='ds'/>

    <feature policy='require' name='ss'/>

    <feature policy='require' name='vme'/>

    <feature policy='require' name='dtes64'/>

    <feature policy='require' name='rdtscp'/>

    <feature policy='require' name='ht'/>

    <feature policy='require' name='dca'/>

    <feature policy='require' name='pbe'/>

    <feature policy='require' name='tm'/>

    <feature policy='require' name='pdcm'/>

    <feature policy='require' name='vmx'/>

    <feature policy='require' name='ds_cpl'/>

    <feature policy='require' name='xtpr'/>

    <feature policy='require' name='acpi'/>

    <feature policy='require' name='invtsc'/>

  </cpu>

  <clock offset='utc'>

    <timer name='rtc' tickpolicy='catchup'/>

    <timer name='pit' tickpolicy='delay'/>

    <timer name='hpet' present='no'/>

  </clock>

  <on_poweroff>destroy</on_poweroff>

  <on_reboot>restart</on_reboot>

  <on_crash>restart</on_crash>

  <pm>

    <suspend-to-mem enabled='no'/>

    <suspend-to-disk enabled='no'/>

  </pm>

  <devices>

    <emulator>/usr/libexec/qemu-kvm</emulator>

    <disk type='file' device='disk'>

      <driver name='qemu' type='qcow2'/>

      <source file='/var/lib/libvirt/images/vm1.qcow2'/>

      <target dev='hda' bus='ide'/>

      <address type='drive' controller='0' bus='0' target='0' unit='0'/>

    </disk>

    <disk type='block' device='cdrom'>

      <driver name='qemu' type='raw'/>

      <target dev='hdb' bus='ide'/>

      <readonly/>

      <address type='drive' controller='0' bus='0' target='0' unit='1'/>

      <target dev='hdb' bus='ide'/>

      <readonly/>

      <address type='drive' controller='0' bus='0' target='0' unit='1'/>

    </disk>

    <controller type='usb' index='0' model='ich9-ehci1'>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>

    </controller>

    <controller type='usb' index='0' model='ich9-uhci1'>

      <master startport='0'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>

    </controller>

    <controller type='usb' index='0' model='ich9-uhci2'>

      <master startport='2'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>

    </controller>

    <controller type='usb' index='0' model='ich9-uhci3'>

      <master startport='4'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>

    </controller>

    <controller type='pci' index='0' model='pci-root'/>

    <controller type='ide' index='0'>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>

    </controller>

    <controller type='virtio-serial' index='0'>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>

    </controller>

    <interface type='direct'>

      <mac address='52:54:00:2c:56:cf'/>

      <source dev='p55p1.10' mode='passthrough'/>

      <model type='e1000'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>

    </interface>

    <serial type='pty'>

      <target port='0'/>

    </serial>

    <console type='pty'>

      <target type='serial' port='0'/>

    </console>

    <channel type='spicevmc'>

      <target type='virtio' name='com.redhat.spice.0'/>

      <address type='virtio-serial' controller='0' bus='0' port='1'/>

    </channel>

    <input type='mouse' bus='ps2'/>

    <input type='keyboard' bus='ps2'/>

    <graphics type='spice' autoport='yes'/>

    <sound model='ich6'>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>

    </sound>

    <video>

      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>

    </video>

    <redirdev bus='usb' type='spicevmc'>

    </redirdev>

    <redirdev bus='usb' type='spicevmc'>

    </redirdev>

    <memballoon model='virtio'>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>

    </memballoon>

  </devices>

</domain>

Its my understanding that the libvirt has been upstream patched so that the earlier posts about patching one of the file is no longer necessary. Is that correct?

[WScott1966]

the version I am running has also change a bit...

virsh -c qemu:///system version --daemon [Output]:

Compiled against library: libvirt 1.2.8

Using library: libvirt 1.2.8

Using API: QEMU 1.2.8

Running hypervisor: QEMU 2.0.0

Running against daemon: 1.2.8

[admin]

I think that ESX should assume an invariant TSC if it detects that it is running in a VM.  Is your virtual CPU configured to report the HYPERVISOR bit in CPUID.1.ECX?

[WScott1966]

OK....

Made it past the boot, Getting past the TSC issue.

Now getting a halt screen:

Anyone understand why I am halting?

XML configuration for VM that has got me this far:

Anything to do with using the e1000 NIC as everything I have read says to use the vmxnet3, but its not an option/accepted.

<domain type='kvm'>

  <name>gsf-kESXi01</name>

  <uuid>5b36e691-d421-433b-8f8d-acd62e8ed103</uuid>

  <memory unit='KiB'>4194304</memory>

  <currentMemory unit='KiB'>4194304</currentMemory>

  <vcpu placement='static'>4</vcpu>

  <os>

    <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>

    <bootmenu enable='yes'/>

  </os>

  <features>

    <acpi/>

    <apic/>

    <pae/>

  </features>

  <cpu mode='custom' match='exact'>

    <model fallback='allow'>Nehalem</model>

    <feature policy='require' name='vmx'/>

  </cpu>

  <clock offset='utc'>

    <timer name='rtc' tickpolicy='catchup'/>

    <timer name='pit' tickpolicy='delay'/>

    <timer name='hpet' present='no'/>

  </clock>

  <on_poweroff>destroy</on_poweroff>

  <on_reboot>restart</on_reboot>

  <on_crash>restart</on_crash>

  <pm>

    <suspend-to-mem enabled='no'/>

    <suspend-to-disk enabled='no'/>

  </pm>

  <devices>

    <emulator>/usr/libexec/qemu-kvm</emulator>

    <disk type='file' device='disk'>

      <driver name='qemu' type='qcow2'/>

      <source file='/var/lib/libvirt/images/vm1.qcow2'/>

      <target dev='hda' bus='ide'/>

      <boot order='2'/>

      <address type='drive' controller='0' bus='0' target='0' unit='0'/>

    </disk>

    <disk type='file' device='cdrom'>

      <driver name='qemu' type='raw'/>

      <source file='/home/administrator/ESXi_6.0.iso'/>

      <target dev='hdb' bus='ide'/>

      <readonly/>

      <boot order='1'/>

      <address type='drive' controller='0' bus='0' target='0' unit='1'/>

    </disk>

    <controller type='usb' index='0' model='ich9-ehci1'>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>

    </controller>

    <controller type='usb' index='0' model='ich9-uhci1'>

      <master startport='0'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>

    </controller>

    <controller type='usb' index='0' model='ich9-uhci2'>

      <master startport='2'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>

    </controller>

    <controller type='usb' index='0' model='ich9-uhci3'>

      <master startport='4'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>

      <boot order='1'/>

      <address type='drive' controller='0' bus='0' target='0' unit='1'/>

    </disk>

    <controller type='usb' index='0' model='ich9-ehci1'>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>

    </controller>

    <controller type='usb' index='0' model='ich9-uhci1'>

      <master startport='0'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>

    </controller>

    <controller type='usb' index='0' model='ich9-uhci2'>

      <master startport='2'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>

    </controller>

    <controller type='usb' index='0' model='ich9-uhci3'>

      <master startport='4'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>

    </controller>

    <controller type='pci' index='0' model='pci-root'/>

    <controller type='ide' index='0'>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>

    </controller>

    <controller type='virtio-serial' index='0'>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>

    </controller>

    <interface type='direct'>

      <mac address='52:54:00:3e:c3:bb'/>

      <source dev='p55p1.10' mode='passthrough'/>

      <model type='e1000'/>

      <boot order='3'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>

    </interface>

    <serial type='pty'>

      <target port='0'/>

    </serial>

    <console type='pty'>

      <target type='serial' port='0'/>

    </console>

    <channel type='spicevmc'>

      <target type='virtio' name='com.redhat.spice.0'/>

      <address type='virtio-serial' controller='0' bus='0' port='1'/>

    </channel>

    <input type='mouse' bus='ps2'/>

    <input type='keyboard' bus='ps2'/>

    <graphics type='spice' autoport='yes'/>

    <sound model='ich6'>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>

    </sound>

    <video>

      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>

    </video>

    <redirdev bus='usb' type='spicevmc'>

    </redirdev>

    <redirdev bus='usb' type='spicevmc'>

    </redirdev>

    <memballoon model='virtio'>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>

    </memballoon>

  </devices>

</domain>

[admin]

RCX is 0xCE, which is MSR_PLATFORM_INFO.  I'm guessing that ESXi is trying to read this MSR, but kvm doesn't implement it.  Based on the family and model of the CPU, this MSR should exist, but all hypervisors have shortcomings when it comes to MSRs.  Unfortunately, the default kvm behavior is to raise a #GP on unimplemented MSRs, even when the MSR is implemented by physical CPUs of the same family and model.  You probably need to tell kvm to ignore unimplemented MSRs.

[WScott1966]

So the odd thing is that I have had the entry: modprobe kvm ignore_msrs=1

In my deployment documentation forever, but I just checked the state of the parameter and its actually still No?

[root@ocompute01 ~]# cat /sys/module/kvm/parameters/ignore_msrs

N

Why is this parameter not setting? Am I passing it an incorrect value?

[admin]

I think the recommended way of doing this is to add "options kvm ignore_msrs=1" to /etc/modprobe.d/kvm.conf.

[WScott1966]

It actually is...

#  ## Enable kvm/kvm-intel options (Immediate)

modprobe kvm-intel nested=1

modprobe kvm_intel ept=y

modprobe kvm ignore_msrs=1

#  ## Enable kvm/kvm-intel options (persistent)

cat >/etc/modprobe.d/kvm.conf <<EOF

options kvm ignore_msrs=y

EOF

cat >/etc/modprobe.d/kvm-intel.conf <<EOF

options kvm_intel nested=y

options kvm_intel ept=y

EOF

The issue was that I had =1 instead of =y for the option.

It now reports as expected.

[admin · Accepted solution]

Odd.  Try this:

rmmod kvm_intel

rmmod kvm

modprobe kvm ignore_msrs=y

cat /sys/module/kvm/parameters/ignore_msrs

[WScott1966]

With that corrected I can now install ESXi 6.

But! (You knew it was coming...) I am alerted that Hardware Virtualization is not enabled/available on my CPU.

What am I missing?

[admin]

cat /sys/module/kvm_intel/parameters/nested

It probably says 'N'.

Try the same approach to fix that:

rmmod kvm_intel

modprobe kvm_intel nested=y ept=y

[WScott1966]

First, thank you for all your time and effort in helping me get this working...

It is greatly appreciated!

All enabled already:

[root@gsf-ocompute01 modprobe.d]# cat /sys/module/kvm_intel/parameters/nested

Y

[root@gsf-ocompute01 modprobe.d]# cat /sys/module/kvm_intel/parameters/ept

Y

[root@gsf-ocompute01 modprobe.d]# cat /sys/module/kvm/parameters/ignore_msrs

Y

[admin]

Which Linux kernel are you running?  As of 3.17, this patch was still necessary for kvm to provide all of the VT-x features that ESXi requires:

   https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/commit/?id=a7c0b07d570848e50fce4d31...

[WScott1966]

Latest release...

CentOS Linux release 7.1.1503 (Core)

Linux gsf-ocompute01.gsf.home 3.10.0-229.7.2.el7.x86_64 #1 SMP Tue Jun 23 22:06:11 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

virsh -c qemu:///system version --daemon

Compiled against library: libvirt 1.2.8

Using library: libvirt 1.2.8

Using API: QEMU 1.2.8

Running hypervisor: QEMU 1.5.3

Running against daemon: 1.2.8

[admin]

The best thing to do is to continue without hardware assisted virtualization.  Set up a 64-bit nested VM and try to run it.  Then, we should get a more informative message about what's wrong.  At the very least, we can look at the vmware.log file for the inner guest and figure out why ESXi is unhappy with the virtual CPU.

[WScott1966]

Still Trying...

Switched to OpenSUSE...

Have QEMU/KVM installed and Running with the following versions:

virsh -c qemu:///system version --daemon [Output]:

Compiled against library: libvirt 1.2.9

Using library: libvirt 1.2.9

Using API: QEMU 1.2.9

Running hypervisor: QEMU 2.1.3

Running against daemon: 1.2.9

Check to see if everything is configured to support Nested Hypervisors:

# cat /sys/module/kvm_intel/parameters/nested

Y

# cat /sys/module/kvm_intel/parameters/ept

Y

# cat /sys/module/kvm/parameters/ignore_msrs

Y

And...

Still receiving Hardware Virtualization Warning:

For a fixed platform there still seems to be issues with QEMU/KVM in exposing VT technologies of the CPU's...

Are there any specific CPU's that this will work with?

I am presently using the Nahaiem Model from the list which matches the L5520 CPU's in the physical server, but It is detected as a Intel i7 by ESXi.

Any thoughts?

[admin]

Go ahead and install a nested 64-bit guest and try to power it on.  Though it will fail, the vmware.log file will give us more information about the deficiencies of the virtual CPU.