10 Replies Latest reply on May 27, 2016 12:30 PM by KrishSri

    Running ESXi under Ubuntu

    rawsik Lurker

      Hi guys,

        could you help me please run ESXi 6 under Ubuntu 14.04.04  with kvm-qemu.

       

      I have   Ubuntu 14.04.4 LTS

                   ESX  6.0   vmware-201601001-3380124-iso

                   Running hypervisor: QEMU 2.4.0

       

      I try to run VM on EXSi  which running under KVM.  

      So, ESXi was successfull installed,  the command in ps :

      qemu-system-x86_64 -enable-kvm -name esx -S -machine pc-i440fx-2.4,accel=kvm,usb=off -cpu Nehalem,+invpcid,+vmx -m 8096 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid b2c3b37b-9596-98ab-8e0d-f05df89d2920 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/esx.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/libvirt/images/esx-20.img,if=none,id=drive-ide0-0-0,format=raw -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=26,id=hostnet0 -device e1000,netdev=hostnet0,id=net0,mac=52:54:00:96:74:1c,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 0.0.0.0:0 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4

       

      But I always get error like :

      Virtualized INtel VT-x/EPT is not supported on this platform. Continure without virtualized msg.intel.hvhwmmu? during starting VM on ESXi

       

      cat /proc/cpuinfo  on 0 host machine
      flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer xsave avx f16c rdrand lahf_lm abm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid

       

       

      what is wrong ?   what kind of flags I should add for run VM on  1 host machine ( esxi) ?

        • 1. Re: Running ESXi under Ubuntu
          covelli Enthusiast
          VMware Employees

          > what kind of flags I should add for run VM on  1 host machine ( esxi) ?

           

          ESXi is the inner hypervisor so you need to configure QEMU to expose VT to ESXi.  Can you check with the qemu folks on how to do this?  ALso you may need to add  "vmx.allowNested" to the ESXi VMs .vmx file.  

          • 2. Re: Running ESXi under Ubuntu
            covelli Enthusiast
            VMware Employees

            Sorry the option should be "vmx.allowNested=TRUE"

            • 3. Re: Running ESXi under Ubuntu
              rawsik Lurker

              I added these options in  /etc/vmware/config

               

              vhv.allow = "TRUE"

              hv.assumeEnabled="TRUE"

              vhv.enable = "TRUE"

              vmx.allowNested = "TRUE"

               

              But, it didn't help me.  ( I rebooted ESXi also)

               

              It affected only for 32-bit VM, which failed with new error .

               

               

              I read many articles which describes that people successfully run  ESXi under KVM.  But I can't reproduce it.  So, I try to understand what I doing wrong.

              • 4. Re: Running ESXi under Ubuntu
                covelli Enthusiast
                VMware Employees

                I don't think the problem is with how you are configuring ESX - you need to configure QEMU to expose VT to ESXi.  Can you check with the qemu folks on how to do this?

                • 5. Re: Running ESXi under Ubuntu
                  rawsik Lurker

                  for clarify:

                   

                  cat /sys/module/kvm_intel/parameters/nested

                  Y

                  cat /sys/module/kvm/parameters/ignore_msrs

                  Y

                  cat /sys/module/kvm_intel/parameters/ept

                  Y

                   

                   

                  my xml:

                   

                  <domain type='kvm' id='41'>

                    <name>esx</name>

                    <uuid>b2c3b37b-9596-98ab-8e0d-f05df89d2920</uuid>

                    <memory unit='KiB'>8290304</memory>

                    <currentMemory unit='KiB'>8290304</currentMemory>

                    <vcpu placement='static'>4</vcpu>

                    <resource>

                      <partition>/machine</partition>

                    </resource>

                    <os>

                      <type arch='x86_64' machine='pc-i440fx-2.4'>hvm</type>

                      <boot dev='hd'/>

                    </os>

                    <features>

                      <acpi/>

                      <apic/>

                      <pae/>

                    </features>

                    <cpu mode='custom' match='exact'>

                      <model fallback='allow'>Nehalem</model>

                      <vendor>Intel</vendor>

                      <feature policy='require' name='vmx'/>

                      <feature policy='require' name='lm'/>

                      <feature policy='require' name='lahf_lm'/>

                      <feature policy='require' name='nx'/>

                      <feature policy='require' name='fxsr'/>

                      <feature policy='require' name='invpcid'/>

                    </cpu>

                    <clock offset='utc'/>

                    <on_poweroff>destroy</on_poweroff>

                    <on_reboot>restart</on_reboot>

                    <on_crash>restart</on_crash>

                    <devices>

                      <emulator>/usr/bin/kvm-spice</emulator>

                      <disk type='file' device='disk'>

                        <driver name='qemu' type='raw'/>

                        <source file='/var/lib/libvirt/images/esx-20.img'/>

                        <target dev='hda' bus='ide'/>

                        <alias name='ide0-0-0'/>

                        <address type='drive' controller='0' bus='0' target='0' unit='0'/>

                      </disk>

                      <disk type='block' device='cdrom'>

                        <driver name='qemu' type='raw'/>

                        <target dev='hdc' bus='ide'/>

                        <readonly/>

                        <alias name='ide0-1-0'/>

                        <address type='drive' controller='0' bus='1' target='0' unit='0'/>

                      </disk>

                      <controller type='usb' index='0'>

                        <alias name='usb0'/>

                        <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>

                      </controller>

                      <controller type='pci' index='0' model='pci-root'>

                        <alias name='pci.0'/>

                      </controller>

                      <controller type='ide' index='0'>

                        <alias name='ide0'/>

                        <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>

                      </controller>

                      <interface type='bridge'>

                        <mac address='52:54:00:96:74:1c'/>

                        <source bridge='virbr0'/>

                        <target dev='vnet0'/>

                        <model type='e1000'/>

                        <alias name='net0'/>

                        <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>

                      </interface>

                      <serial type='pty'>

                        <source path='/dev/pts/2'/>

                        <target port='0'/>

                        <alias name='serial0'/>

                      </serial>

                      <console type='pty' tty='/dev/pts/2'>

                        <source path='/dev/pts/2'/>

                        <target type='serial' port='0'/>

                        <alias name='serial0'/>

                      </console>

                      <input type='mouse' bus='ps2'/>

                      <input type='keyboard' bus='ps2'/>

                      <graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0'>

                        <listen type='address' address='0.0.0.0'/>

                      </graphics>

                      <video>

                        <model type='cirrus' vram='9216' heads='1'/>

                        <alias name='video0'/>

                        <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>

                      </video>

                      <memballoon model='virtio'>

                        <alias name='balloon0'/>

                        <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>

                      </memballoon>

                    </devices>

                    <seclabel type='none'/>

                  </domain>

                   

                   

                  If you run ESXi on KVM successful,  please  write your OS and KVM version, also please provide your XML config

                  • 6. Re: Running ESXi under Ubuntu
                    covelli Enthusiast
                    VMware Employees

                    The error :


                    Virtualized INtel VT-x/EPT is not supported on this platform. Continure without virtualized msg.intel.hvhwmmu? during starting VM on ESXi


                    indicates that you won't be able to use virtualized VT-X under ESXi but you still should be able to run VMs using VT-x and EPT if you remove the line vhv.enable = "TRUE".  You won't be able to run guest hypervisors like Hyper-V under the ESXi VM but normal guests that don't require VT-X should run fine.

                    • 7. Re: Running ESXi under Ubuntu
                      rawsik Lurker

                      I'll try to ask folks from KVM.

                       

                       

                      my /etc/vmware/config now:

                      hv.assumeEnabled = "TRUE"

                      vmx.allowNested = "TRUE"

                      vhv.allow = "TRUE"

                       

                      error which I got in GUI:

                       

                      Failed to start the virtual machine.

                      This virtual machine is configured for 64-bit guest operating systems. However, 64-bit operation is not possible.

                      This host supports Intel VT-x, but the Intel VT-x implementation is incompatible with VMware ESX.

                      For more detailed information, see http://vmware.com/info?id=152.

                       

                      from vmware.log

                       

                      2016-05-17T13:31:46.976Z| vmx| I120: guestCPUID level 8000001e, 0: 0x00000000 0x00000000 0x00000000 0x00000000

                      2016-05-17T13:31:46.976Z| vmx| A115: ConfigDB: Setting monitor.phys_bits_used = "42"

                      2016-05-17T13:31:46.980Z| vmx| I120: Msg_Post: Error

                      2016-05-17T13:31:46.980Z| vmx| I120: [msg.cpuid.noLongmode2] This virtual machine is configured for 64-bit guest operating systems. However, 64-bit operation is not possible.

                      2016-05-17T13:31:46.980Z| vmx| I120+ This host supports Intel VT-x, but the Intel VT-x implementation is incompatible with VMware ESX.

                      2016-05-17T13:31:46.980Z| vmx| I120+ For more detailed information, see http://vmware.com/info?id=152.

                      2016-05-17T13:31:46.980Z| vmx| I120: ----------------------------------------

                      2016-05-17T13:31:46.984Z| vmx| I120: Vigor_MessageRevoke: message 'msg.cpuid.noLongmode2' (seq 2445) is revoked

                      2016-05-17T13:31:46.984Z| vmx| I120: Module CPUID power on failed.

                      2016-05-17T13:31:46.984Z| vmx| I120: VMX_PowerOn: ModuleTable_PowerOn = 0

                      • 8. Re: Running ESXi under Ubuntu
                        rawsik Lurker

                        I tryed to use ubuntu 16.04 with new module version.

                         

                        At the all VM runs on the ESX but without  VT-x ( VMX flags),  so I have performance issues.  ( more than 4 times )

                           ( I chosed   answer.msg.cpuid.noVHVQuestion = "Yes"  )

                        I tryed different CPU option for KVM but none of them didn't delivery  VT-x into ESX.

                         

                        for example,  every linux distributive has vmx, when i run it into KVM.   ( cat /proc/cpuinfo )

                         

                        esxcfg-info |grep "HV Support"

                                 |----HV Support............................................3

                                    |----World Command Line.................................grep HV Support

                         

                        So, I'm dissapointed with ability ESX use VMX from KVM.

                         

                        If you have any advice, you are welcome

                        • 9. Re: Running ESXi under Ubuntu
                          covelli Enthusiast
                          VMware Employees

                          I'm sorry to hear you are still having trouble. 

                           

                          > I tryed different CPU option for KVM but none of them didn't delivery  VT-x into ESX.


                          It is unfortunate that KVM makes it so hard to enable support for nested VMs.  I wish they could make it as easy as VMware does.


                          > So, I'm dissapointed with ability ESX use VMX from KVM.

                           

                          I'm quite disappointed that KVM makes it so difficult to expose VT-x to ESX as well.

                           

                          > If you have any advice, you are welcome

                           

                          Unfortunately we can not control what features the underlying HW or Hypervisor chooses to expose to us.  My advice is to contact KVM.

                          • 10. Re: Running ESXi under Ubuntu
                            KrishSri Lurker
                            VMware Employees

                            It looks like you are running on a Nehalem CPU. Have you tried this on an Haswell ?