ESXi inside KVM

[LesykM]

Hello!

I wonder, if it is possible to run ESXi 5.0/5.1 inside KVM (proxmox) ?

I need to run few for testing purposes. I cannot setup any other virtualization software.

I need only ESXi itself, even without virtual machines inside it:)

Thanks!

[michelemase]

Here is attached the otput of cpuid

[admin]

That's the CPUID output from the host, which doesn't really tell me what I need to know.  Thank you, though, for providing the CPUID output from a kvm guest in a private thread.

It appears that kvm reports inconsistent CPUID information with '-cpu host' if the host supports XSAVE.  Though kvm reports XSAVE support, it hides CPUID leaf 0xD, which contains information on how to use XSAVE.  As a result, ESXi tries to use XSAVE incorrectly, and the CPU raises a #GP fault.  This leads to a PSOD.

The workaround, for the present, is to hide XSAVE from the ESXi guest.  This can be done with '-cpu host,-xsave'.

The full command line is:

qemu-system-x86_64 -enable-kvm -cpu host,-xsave -machine vmport=off -m 4096 -smp 2 -hda esxi00.img -netdev tap,helper=/usr/lib/qemu-bridge-helper,id=hostnet0 -device vmxnet3,netdev=hostnet0,id=net0 -cdrom VMware-VMvisor-Installer-6.0.0.update01-3029758.x86_64.iso

[JonathonReinhar]

I wrote a blog post with step-by-step directions for getting this to work. Note that the latest QEMU doesn't need the patch anymore, as they've added a "-machine vmport=off" option.

These directions specifically cover setting up ESXi in a guest managed by libvirt / virt-manager. Hope it helps.

The Perils and Triumphs of Being a Geek: Installing ESXi in a QEMU-KVM virtual machine, under libvir...‌

[admin]

I've discovered the following additional issues with ESXi on kvm:

1. At present, kvm does not appear to support XSAVE, AVX, AVX2, or BMI2.  On my (now fairly ancient) Haswell system, I find I need to invoke qemu-system-x86_64 with "-cpu host,-xsave,-avx,-avx2,-bmi2"

2. There appear to be some issues regarding the communication of CPU topology.  For an 8-vCPU ESXi VM, I find that I need to invoke qemu-system-x86_64 with "-smp cores=8,sockets=1"

3. When running under kvm, ESXi 6.0 cannot run VMs with debugging enabled, because it tries to verify the value of a VMCS field that kvm does not implement.

This configuration is still not very stable, but some things do work.

[Liksx]

hello all

read this topic and thanks for info

in my case

qemu 2.5

ubuntu 14.04

after pre installing esxi , I see last message kernel up

then first yellow screen and stuck initializing chipset

using -cpu host

what can be wong

[admin]

Please connect a virtual serial device and post the serial device log.

[Liksx]

hello!

Compiled against library: libvirt 1.3.2

Using library: libvirt 1.3.2

Using API: QEMU 1.3.2

Running hypervisor: QEMU 2.5.50

Running against daemon: 1.3.2

cat /sys/module/kvm_intel/parameters/nested

Y

cat /sys/module/kvm_intel/parameters/ept

Y

cat /sys/module/kvm/parameters/ignore_msrs

Y

qemu-system-x86_64 -enable-kvm -cpu host,-xsave,-avx,-avx2,-bmi2 -m 4096 -smp cores=2,sockets=1 \

-machine vmport=off \

-boot d -cdrom ESXI5.iso \

-drive file=esxi.qcow2,format=qcow2,if=virtio,bus=0,unit=0,cache=none \

-serial mon:stdio \

-vnc :smileylaugh:

no network, for testing not need.

sorry for stupid questiong how i can connect and take this log?

like this ? -serial mon:stdio \

[Liksx]

hmm in my case, when i use cpu host all time stuck on initializing chipset, and HOST very unstable state or stuck too...

i change to core2duo and ESXI installing

why? i am use 2630L 2.4Ghz

[Liksx]

hello againg) dont know get i luck or not for answer

now i can run ESXI under KVM

but lol

when VM start long LOGO in BIOS VMWARE and then ( in my case ) virtual machine start boot up

initrd -> kernel up -> stuck

install windows....first loading succesfull then stuck....

i try local hard ( ide kvm )

and iscsi

all time VM stuck when i starting install or start boot up VM machine

[covelli]

Nice to see this sort of working.  Can you power off the VM and add the following settings to the VMs .vmx file:

vmx.buildType=debug

monitor_control.log_vmsample = "TRUE"

this will enable additional debugging info that will help us to root cause.

[Liksx]

Thanks for answer

first step

run vm

qemu-system-x86_64 -enable-kvm -cpu IvyBridge,+vmx,-xsave,-avx,-avx2,-bmi2,-erms,-xsaveopt -m 8192 -smp cores=8,sockets=1 \

-machine q35,vmport=off \

-drive file=esxi.raw,format=raw,if=ide,bus=0,unit=0,cache=none \

-device vmxnet3,netdev=pnet0,mac=00:AA:AA:AA:AA:AB -netdev tap,id=pnet0 \

-vnc :smileylaugh:

why i not use -cpu host or -cpu pass-throught, cuz dont know why, all time ESXI stuck on initializing chipset

work only if i change to other CPU ( core2duo for example )

i am using Xeon 2630L v2 2.4GHz ( i am try VSA on nested ESXI and all work correctly on those cpus )

from cpu_map.xml i see my IvyBridge E3-12xx ( not my, but good too )

now ESXI run correctly, and installing correctly, no warrning see

i can run 64 bit Guest without error or someing else

add to .vmx

vmx.buildType= "debug"

monitor_control.log_vmsample = "TRUE"

cat >> /vmfs/volumes/VMFS/VM/VM.vmx << EOF

vmx.buildType=debug

monitor_control.log_vmsample = "TRUE"

EOF

cat /vmfs/volumes/VMFS/VM/VM.vmx

1. Black Screen 20-30 sec

2. Logo VMWARE 20-30 sec

3. boot up and stuck

cannot reset machine, only power off

[Liksx]

this is part of log can be helpfully

ubuntu in CD

boot up -> Try ubuntu without install

VM power off

2001-01-01T00:04:02.261Z| vcpu-0| I120: GetMSR: MSR[0x11e] (read as 0x708101): rip=0x4a13

2001-01-01T00:04:02.262Z| vcpu-0| I120: SetMSR: unknown MSR[0x11e]:=0x708001 (sinking): rip=0x4a18 count=9

2001-01-01T00:04:02.267Z| vcpu-0| I120: GetMSR: MSR[0x11e] (read as 0x708101): rip=0x4a13

2001-01-01T00:04:02.268Z| vcpu-0| I120: SetMSR: unknown MSR[0x11e]:=0x708001 (sinking): rip=0x4a18 count=10

2001-01-01T00:04:02.273Z| vcpu-0| I120: GetMSR: MSR[0x11e] (read as 0x708101): rip=0x4a13

2001-01-01T00:04:02.274Z| vcpu-0| I120: SetMSR: unknown MSR[0x11e]:=0x708001 (sinking): rip=0x4a18 count=11

2001-01-01T00:04:02.277Z| vcpu-0| I120: GetMSR: MSR[0x11e] (read as 0x708101): rip=0x4a13

2001-01-01T00:04:02.277Z| vcpu-0| I120: SetMSR: unknown MSR[0x11e]:=0x708001 (sinking): rip=0x4a18 count=12

2001-01-01T00:04:02.585Z| vcpu-0| I120: GetMSR: MSR[0x11e] (read as 0x708101): rip=0x4a13

2001-01-01T00:04:02.585Z| vcpu-0| I120: SetMSR: unknown MSR[0x11e]:=0x708001 (sinking): rip=0x4a18 count=13

2001-01-01T00:04:02.758Z| vcpu-0| I120: GetMSR: MSR[0x11e] (read as 0x708101): rip=0x4a13

2001-01-01T00:04:02.759Z| vcpu-0| I120: SetMSR: unknown MSR[0x11e]:=0x708001 (sinking): rip=0x4a18 count=14

2001-01-01T00:04:02.765Z| vcpu-0| I120: GetMSR: MSR[0x11e] (read as 0x708101): rip=0x4a13

2001-01-01T00:04:02.765Z| vcpu-0| I120: SetMSR: unknown MSR[0x11e]:=0x708001 (sinking): rip=0x4a18 count=15

2001-01-01T00:04:02.769Z| vcpu-0| I120: GetMSR: MSR[0x11e] (read as 0x708101): rip=0x4a13

2001-01-01T00:04:02.769Z| vcpu-0| I120: SetMSR: unknown MSR[0x11e]:=0x708001 (sinking): rip=0x4a18 count=16

2001-01-01T00:04:02.935Z| vcpu-0| I120: GetMSR: MSR[0x11e] (read as 0x708101): rip=0x4a13

2001-01-01T00:04:02.936Z| vcpu-0| I120: SetMSR: unknown MSR[0x11e]:=0x708001 (sinking): rip=0x4a18 count=17

2001-01-01T00:04:03.056Z| vcpu-0| I120: GetMSR: MSR[0x11e] (read as 0x708101): rip=0x4a13

2001-01-01T00:04:03.057Z| vcpu-0| I120: SetMSR: unknown MSR[0x11e]:=0x708001 (sinking): rip=0x4a18 count=18

2001-01-01T00:04:03.062Z| vcpu-0| I120: GetMSR: MSR[0x11e] (read as 0x708101): rip=0x4a13

2001-01-01T00:04:03.062Z| vcpu-0| I120: SetMSR: unknown MSR[0x11e]:=0x708001 (sinking): rip=0x4a18 count=19

2001-01-01T00:04:03.064Z| vcpu-0| I120: GetMSR: MSR[0x11e] (read as 0x708101): rip=0x4a13

2001-01-01T00:04:03.065Z| vcpu-0| I120: SetMSR: unknown MSR[0x11e]:=0x708001 (sinking): rip=0x4a18 count=20

2001-01-01T00:04:03.067Z| vcpu-0| I120: GetMSR: MSR[0x11e] (read as 0x708101): rip=0x4a13

2001-01-01T00:04:03.067Z| vcpu-0| I120: SetMSR: unknown MSR[0x11e]:=0x708001 (sinking): rip=0x4a18 count=21

2001-01-01T00:04:03.069Z| vcpu-0| I120: GetMSR: MSR[0x11e] (read as 0x708101): rip=0x4a13

2001-01-01T00:04:03.069Z| vcpu-0| I120: SetMSR: unknown MSR[0x11e]:=0x708001 (sinking): rip=0x4a18 count=22

2001-01-01T00:04:03.071Z| vcpu-0| I120: GetMSR: MSR[0x11e] (read as 0x708101): rip=0x4a13

2001-01-01T00:04:03.072Z| vcpu-0| I120: SetMSR: unknown MSR[0x11e]:=0x708001 (sinking): rip=0x4a18 count=23

2001-01-01T00:04:03.075Z| vcpu-0| I120: GetMSR: MSR[0x11e] (read as 0x708101): rip=0x4a13

2001-01-01T00:04:03.075Z| vcpu-0| I120: SetMSR: unknown MSR[0x11e]:=0x708001 (sinking): rip=0x4a18 count=24

2001-01-01T00:04:03.086Z| vcpu-0| I120: GetMSR: MSR[0x11e] (read as 0x708101): rip=0x4af4

2001-01-01T00:04:03.086Z| vcpu-0| I120: SetMSR: unknown MSR[0x11e]:=0x8101 (sinking): rip=0x4aff count=25

2001-01-01T00:04:03.897Z| vcpu-0| I120: BIOS-UUID is 56 4d 46 3f e6 75 79 d9-e0 67 11 90 f6 d2 2e 5a

2001-01-01T00:04:04.797Z| vcpu-0| I120: GetMSR: MSR[0x11e] (read as 0x708101): rip=0x4a13

2001-01-01T00:04:04.798Z| vcpu-0| I120: SetMSR: unknown MSR[0x11e]:=0x708001 (sinking): rip=0x4a18 count=26

2001-01-01T00:04:04.812Z| vcpu-0| I120: GetMSR: MSR[0x11e] (read as 0x708101): rip=0x4a13

2001-01-01T00:04:04.812Z| vcpu-0| I120: SetMSR: unknown MSR[0x11e]:=0x708001 (sinking): rip=0x4a18 count=27

2001-01-01T00:04:06.119Z| vcpu-0| I120: SVGA enabling SVGA

2001-01-01T00:04:17.919Z| mks| I120: SVGA disabling SVGA

2001-01-01T00:04:18.498Z| vcpu-0| I120: Unknown int 10h func 0x0000

2001-01-01T00:04:18.500Z| vcpu-0| I120: GetMSR: unknown MSR[0x1a0] (read as zero): rip=0x73526a count=1

2001-01-01T00:04:18.502Z| vcpu-0| I120: Intel VT enabled.

2001-01-01T00:04:18.582Z| vcpu-0| W110: MONITOR PANIC: vcpu-0:ASSERT public/x86vtinstr.h:252

2001-01-01T00:04:18.582Z| vcpu-0| I120: Core dump with build build-2068190

2001-01-01T00:04:18.582Z| vcpu-0| W110: Writing monitor corefile "/vmfs/volumes/3a4fc99d-f6437147-561e-00aaaaaaaaab/123/vmmcores.gz"

2001-01-01T00:04:18.628Z| vcpu-0| I120: Counting amount of anonymous memory

2001-01-01T00:04:18.632Z| vcpu-0| I120: Total Count of Anon Pages and CR3 pages 6976

2001-01-01T00:04:18.632Z| vcpu-0| W110: Dumping core for vcpu-0

2001-01-01T00:04:18.632Z| vcpu-0| I120: CoreDump: dumping core with superuser privileges

2001-01-01T00:04:18.632Z| vcpu-0| I120: VMK Stack for vcpu 0 is at 0x4123a7bd5000

2001-01-01T00:04:18.632Z| vcpu-0| I120: Beginning monitor coredump

2001-01-01T00:04:19.362Z| vcpu-0| I120: End monitor coredump

2001-01-01T00:04:19.363Z| vcpu-0| W110: Dumping extended monitor data

2001-01-01T00:04:19.695Z| vcpu-0| I120: CoreDump: ei->size 30707712 : len = 30707712

2001-01-01T00:04:31.781Z| vcpu-0| W110: A core file is available in "/vmfs/volumes/3a4fc99d-f6437147-561e-00aaaaaaaaab/123/vmx-debug-zdump.001"

2001-01-01T00:04:31.793Z| vcpu-0| I120: Msg_Post: Error

2001-01-01T00:04:31.793Z| vcpu-0| I120: [msg.log.error.unrecoverable] VMware ESX unrecoverable error: (vcpu-0)

2001-01-01T00:04:31.793Z| vcpu-0| I120+ vcpu-0:ASSERT public/x86vtinstr.h:252

2001-01-01T00:04:31.793Z| vcpu-0| I120: [msg.panic.haveLog] A log file is available in "/vmfs/volumes/3a4fc99d-f6437147-561e-00aaaaaaaaab/123/vmware.log". 

2001-01-01T00:04:31.793Z| vcpu-0| I120: [msg.panic.requestSupport.withoutLog] You can request support. 

2001-01-01T00:04:31.793Z| vcpu-0| I120: [msg.panic.requestSupport.vmSupport.vmx86]

2001-01-01T00:04:31.793Z| vcpu-0| I120+ To collect data to submit to VMware technical support, run "vm-support".

2001-01-01T00:04:31.793Z| vcpu-0| I120: [msg.panic.response] We will respond on the basis of your support entitlement.

2001-01-01T00:04:31.793Z| vcpu-0| I120: ----------------------------------------

[Liksx]

clean install ubuntu 14.04

this try with ESXI6

cat /sys/module/kvm_intel/parameters/nested

cat /sys/module/kvm_intel/parameters/ept

cat /sys/module/kvm/parameters/ignore_msrs

all Y

latest update for

Compiled against library: libvirt 1.3.2

Using library: libvirt 1.3.2

Using API: QEMU 1.3.2

Running hypervisor: QEMU 2.5.50

Running against daemon: 1.3.2

cat /etc/modprobe.d/kvm.conf

options kvm ignore_msrs=1

cat /etc/modprobe.d/kvm-intel.conf

options kvm-intel nested=y ept=y

i am testing all supported CPU

Broadwell

Broadwell-noTSX

Haswell

Haswell-noTSX

qemu32

kvm32

cpu64-rhel5

cpu64-rhel6

kvm64

qemu64

Conroe

Penryn

Nehalem

Westmere

SandyBridge

IvyBridge

no one work normal....

cpu host ( not work )

cpu-passthrough ( via virt-manger not work too )

all time stuck in chipset initzialization

1. Clean VM witout HDD minuim hardware, only CD\ROM with Ubuntu 14.04

UNREAL SLOW WORK, really 20-30 min for  "TRY UBUNTU" in normal case, this work instantly ( for this type installation all work in RAM )

2. External .OVF template ASAv ( cisco virtual machine ) all time stuck, all night wait, no work at all all time stuck after initrd\kernel initsialization

3. Windows load from CD, after frist menu stuck

really dont know, what i doing wrong, or why other ppl can work with nested esxi under kvm ))

in my case i am build lab, and very important run ESXI in single machine with QEMU

perfomance dont worry, but it this case or stuck or unreal slow work....

i am have vmmcores but dont know how i can read this dump

[covelli]

The following error:

2001-01-01T00:04:31.793Z| vcpu-0| I120: [msg.log.error.unrecoverable] VMware ESX unrecoverable error: (vcpu-0)

2001-01-01T00:04:31.793Z| vcpu-0| I120+ vcpu-0:ASSERT public/x86vtinstr.h:252

indicates that there is a problem with our software using KVMs virtual VT HW.  I asked around and was told by the folks who have tried it that this does not quite work yet.  Can you upload the vmmcores.gz file and the corresponding vmware.log?  For now you can run 32-biit guests using software virtualization if you add the following  options to your 32-bit VMs config file:

monitor.virtual_exec = "software"

monitor.virtual_mmu = "software"

[Liksx]

hello!

i am many time reinstall and vmware.log i think it's two logs in my post upper

but vmmcores i save

and i am try this thanks

monitor.virtual_exec = "software"

monitor.virtual_mmu = "software"

[Liksx]

this options not work at all

i am try on

Ubuntu 14.04 ( IvyBridge Xeon )

RHEL7 ( IvyBridge Xeon )

Ubuntu 14.04 ( Core i7 4770 )

cpu host not work at all

nested ESXi not work at all

same problem on both CPU

don't know why

mb need use older QEMU with patch.... but... dont know really where i have problem

[covelli]

I looked into the following error and found it was fixed last November by Jim:

2001-01-01T00:04:18.502Z| vcpu-0| I120: Intel VT enabled.

2001-01-01T00:04:18.582Z| vcpu-0| W110: MONITOR PANIC: vcpu-0:ASSERT public/x86vtinstr.h:252

2001-01-01T00:04:18.582Z| vcpu-0| I120: Core dump with build build-2068190

I have requested the fix to be backported to the next patch of ESX 6.0.

Thanks,

Doug

[Liksx]

hello

latest 6u2

same problem

Physical Server -> ESXi 6u2 -> KVM ( Ubuntu 14.04 )  ->

Compiled against library: libvirt 1.3.2

Using library: libvirt 1.3.2

Using API: QEMU 1.3.2

Running hypervisor: QEMU 2.5.50

Running against daemon: 1.3.2

-> ESXi 6u2 on KVM

[AlexanderB1]

Hi,

trying to set up ESXi inside of proxmox. And got tha attached screenshot.

Wich features I need to enable/disable?

Is it real? As I know Proxmox is actually guy for KVM.

[AlexanderB1]

Here is the answer to my question.

For the all who wants to build this: http://www.marksei.com/install-esxi-nested-inside-proxmox-ve/