VMware Cloud Community
balajibandi
Contributor
Contributor

Running nested Esxi under Esxi-- ubuntu(qemu)

Hi All

Just learning  Virtual world and trying install nested esxi. here is my setup.

I have ESXi 6.0 running

Under ESXI i have Debian as VM.

Under Debian iam running qemu.

Install Esxi inside Esxi 6.0 and export OVA to Qemu as qcow2

Running under Qemu i get CPU Error ( same Esxi vm working under Esxi ok)

I have attached image and config for reference, any help will be appreciated

BB.

Reply
0 Kudos
18 Replies
covelli
VMware Employee
VMware Employee

It seems QEMU is exposing a very old unsupported CPU to the nested ESX.  Can you please upload the vmware.log file from the Debian VM?

Reply
0 Kudos
balajibandi
Contributor
Contributor

Hi

i did not see vmware.log in my debian box where iam running qemu

here is the find out put..any assistance will be great. am i missing anything here ?

# find / -name vmware* -print

/usr/lib/vmware-tools

/usr/lib/vmware-tools/moduleScripts/thinprint/vmware-db.pl

/usr/lib/vmware-tools/sbin32/vmware-hgfsmounter

/usr/lib/vmware-tools/sbin32/vmware-checkvm

/usr/lib/vmware-tools/sbin32/vmware-rpctool

/usr/lib/vmware-tools/bin64/vmware-xferlogs

/usr/lib/vmware-tools/bin64/vmware-user-suid-wrapper

/usr/lib/vmware-tools/configurator/XOrg/7.3_64/vmware_drv.so

/usr/lib/vmware-tools/configurator/XOrg/6.8.x_64/vmware_drv.o

/usr/lib/vmware-tools/configurator/XOrg/7.5/vmware_drv.so

/usr/lib/vmware-tools/configurator/XOrg/7.0_64/vmware_drv.so

/usr/lib/vmware-tools/configurator/XOrg/7.4_64/vmware_drv.so

/usr/lib/vmware-tools/configurator/XOrg/7.1/vmware_drv.so

/usr/lib/vmware-tools/configurator/XOrg/7.4/vmware_drv.so

/usr/lib/vmware-tools/configurator/XOrg/7.5_64/vmware_drv.so

/usr/lib/vmware-tools/configurator/XOrg/7.3/vmware_drv.so

/usr/lib/vmware-tools/configurator/XOrg/6.8.x/vmware_drv.o

/usr/lib/vmware-tools/configurator/XOrg/6.7.x/vmware_drv.o

/usr/lib/vmware-tools/configurator/XOrg/7.3.99_64/vmware_drv.so

/usr/lib/vmware-tools/configurator/XOrg/6.7.x_64/vmware_drv.o

/usr/lib/vmware-tools/configurator/XOrg/7.3.99/vmware_drv.so

/usr/lib/vmware-tools/configurator/XOrg/7.1_64/vmware_drv.so

/usr/lib/vmware-tools/configurator/XOrg/7.0/vmware_drv.so

/usr/lib/vmware-tools/configurator/XOrg/7.6/vmware_drv.so

/usr/lib/vmware-tools/configurator/XOrg/7.6_64/vmware_drv.so

/usr/lib/vmware-tools/configurator/XFree86-4/4.3.x_64/vmware_drv.o

/usr/lib/vmware-tools/configurator/XFree86-4/4.3.x/vmware_drv.o

/usr/lib/vmware-tools/configurator/XFree86-4/4.x/vmware_drv.o

/usr/lib/vmware-tools/configurator/XFree86-4/4.2.x/vmware_drv.o

/usr/lib/vmware-tools/sbin64/vmware-hgfsmounter

/usr/lib/vmware-tools/sbin64/vmware-checkvm

/usr/lib/vmware-tools/sbin64/vmware-modconfig-console

/usr/lib/vmware-tools/sbin64/vmware-rpctool

/usr/lib/vmware-tools/bin32/vmware-xferlogs

/usr/lib/vmware-tools/bin32/vmware-xdg-detect-de

/usr/lib/vmware-tools/bin32/vmware-user-suid-wrapper

/usr/share/doc/vmware-tools

/usr/sbin/vmware-checkvm

/usr/sbin/vmware-vmblock-fuse

/usr/sbin/vmware-rpctool

/usr/bin/vmware-xferlogs

/usr/bin/vmware-toolbox-cmd

/usr/bin/vmware-xdg-detect-de

/usr/bin/vmware-user

/usr/bin/vmware-user-wrapper

/usr/bin/vmware-uninstall-tools.pl

/usr/bin/vmware-config-tools.pl

/usr/bin/vmware-hgfsclient

/usr/src/linux-headers-3.16.7-ckt8-unetlab/include/config/vmware

/vmware-root

/vmware-root/vmware-apploader-1371.log

/vmware-root/vmware-apploader-1433.log

/vmware-root/vmware-apploader-1428.log

/vmware-root/vmware-apploader-1366.log

/etc/init/vmware-tools-thinprint.conf

/etc/init/vmware-tools.conf

/etc/vmware-tools

/etc/vmware-tools/vmware-tools

/etc/vmware-tools/scripts/vmware

/etc/vmware-tools/vmware-tools-prelink.conf

/etc/vmware-tools/vmware-tools-libraries.conf

/etc/vmware-tools/vmware-user.desktop

/etc/vmware-tools/vmware-user.Xresources

/etc/ld.so.conf.d/vmware-tools-libraries.conf

/run/lock/subsys/vmware-tools

/tmp/vmware-root

/tmp/vmware-root/vmware-apploader-3788.log

/tmp/vmware-root/vmware-apploader-3783.log

/var/log/upstart/vmware-tools.log

Reply
0 Kudos
covelli
VMware Employee
VMware Employee

Sorry.  I was looking for the vmware.log for the Debian VM.  It will be on the outer ESX host in the same directory as the Debian VMs .vmx file and vmdks.

Reply
0 Kudos
balajibandi
Contributor
Contributor

Hi

Thank for the quick reply i have attached the vmware.log for your reference.

Let me know what is wrong here ?

BB

Reply
0 Kudos
balajibandi
Contributor
Contributor

Hi VMWARE support

any support on this or is this request not allowed to post here ?

BB

Reply
0 Kudos
covelli
VMware Employee
VMware Employee

Sorry I missed this.  It doesn't look like the problem is on our end - we are exposing a Haswell CPU to Ubuntu but then QEMU turns around and exposes something that looks like a Celeron to us.  Is there any way to configure the CPU family/model/stepping that QEMU exposes to guests?  

Reply
0 Kudos
balajibandi
Contributor
Contributor

Hi

I have other OS running Like under qemu, Linux and windows 2012

but iam looking to have nested LABS inside qemu, so i have number of esxi and can be done some LABS like vmotion and OTV.

Below option iam passing when iam running qemu

$p['type'] = 'qemu';

$p['name'] = 'esxi';

$p['icon'] = 'Server.png';

$p['cpu'] = 4;

$p['ram'] = 4096;

$p['ethernet'] = 1;

$p['console'] = 'vnc';

$p['qemu_arch'] = 'x86_64';

$p['qemu_options'] = '-machine type=pc-1.0,accel=kvm -vga std -usbdevice tablet -boot order=dc';

and when i run i can see its started.

/opt/qemu/bin/qemu-system-x86_64 -d 0 -x -- -device e1000,netdev=net0,mac=50:00:00:01:00:00 -netdev tap,id=net0,ifname=vunl0_1_0,script=no -device e1000,netdev=net1,mac=50:00:00:01:00:01 -netdev tap,id=net1,ifname=vunl0_1_1,script=no -device e1000,netdev=net2,mac=50:00:00:01:00:02 -netdev tap,id=net2,ifname=vunl0_1_2,script=no -device e1000,netdev=net3,mac=50:00:00:01:00:03 -netdev tap,id=net3,ifname=vunl0_1_3,script=no -smp 2 -m 4096 -name esxi -uuid 4d55c191-13e2-4c6d-949a-5ef86ac6664d -vnc :26869 -hda hda.qcow2 -machine type=pc-1.0,accel=kvm -vga std -usbdevice tablet -boot order=dc

any other suggestion ?

BB

Reply
0 Kudos
covelli
VMware Employee
VMware Employee

I think you should probably ask the QEMU folks but it looks like "-cpu host" might expose the host CPUIC which should make the ESX VM happy

Reply
0 Kudos
balajibandi
Contributor
Contributor

Thank you for the quick help i have posted the same qemu also.


below one should set in options ?


"-cpu host"


BB

Reply
0 Kudos
balajibandi
Contributor
Contributor

Hi

Thank you for the option with that option i could able to move next level

Now exit stuck at intializing memory, i have allocated 4GB and tried 8GB still no luck.. any help will be great.

Reply
0 Kudos
covelli
VMware Employee
VMware Employee

I think this stage can take a while.  How long did you wait?  Can you add a serial port to the QEMU VM?  I think if you do this the ESX VM will log to it during boot.

Reply
0 Kudos
balajibandi
Contributor
Contributor

As soon as i see this message in 60 seconds Debian(qemu whole VM Crashing) and  lost connection, even trying to use vspeher with console access no response, i have to force to shutdown in vpshere and bring back.

Let me add serial to qemu and post you the vmware.log

BB

Reply
0 Kudos
balajibandi
Contributor
Contributor

here is the logs of vmware.log and debian syslog ( when i crashed i took this logs )

VMWARE Logs

2016-04-14T07:48:14.789Z| vcpu-1| I120: Intel VT: FlexPriority disabled.

2016-04-14T07:48:14.789Z| vcpu-0| I120: Intel VT: FlexPriority disabled.

2016-04-14T07:48:14.789Z| vcpu-2| I120: Intel VT: FlexPriority disabled.

2016-04-14T07:48:14.789Z| vcpu-3| I120: Intel VT: FlexPriority disabled.

2016-04-14T07:52:27.214Z| vmx| I120: GuestRpcSendTimedOut: message to toolbox timed out.

2016-04-14T07:52:35.344Z| vcpu-0| I120: Tools: Tools heartbeat timeout.

2016-04-14T07:52:42.217Z| vmx| I120: GuestRpcSendTimedOut: message to toolbox timed out.

2016-04-14T07:52:42.217Z| vmx| I120: GuestRpc: app toolbox's second ping timeout; assuming app is down

2016-04-14T07:52:42.217Z| vmx| I120: GuestRpc: Reinitializing Channel 0(toolbox)

2016-04-14T07:52:42.217Z| vmx| I120: GuestMsg: Channel 0, Cannot unpost because the previous post is already completed

Debian Syslogs.

Apr 14 07:51:15 unl01 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl del-port vunl0_1_0

Apr 14 07:51:15 unl01 ovs-vsctl: ovs|00002|vsctl|ERR|no port named vunl0_1_0

Apr 14 07:51:15 unl01 kernel: [  422.019172] IPv6: ADDRCONF(NETDEV_UP): vunl0_1_0: link is not ready

Apr 14 07:51:15 unl01 kernel: [  422.032695] IPv6: ADDRCONF(NETDEV_CHANGE): vunl0_1_0: link becomes ready

Apr 14 07:51:15 unl01 kernel: [  422.225533] kvm [5401]: vcpu0 ignored wrmsr: 0x38d data 0

Apr 14 07:51:15 unl01 kernel: [  422.225771] kvm [5401]: vcpu0 ignored wrmsr: 0x38f data 0

Apr 14 07:51:15 unl01 kernel: [  422.225970] kvm [5401]: vcpu0 ignored wrmsr: 0x309 data 0

Apr 14 07:51:15 unl01 kernel: [  422.226163] kvm [5401]: vcpu0 ignored wrmsr: 0x30a data 0

Apr 14 07:51:15 unl01 kernel: [  422.226346] kvm [5401]: vcpu0 ignored wrmsr: 0x30b data 0

Apr 14 07:51:15 unl01 kernel: [  422.226538] kvm [5401]: vcpu0 ignored wrmsr: 0x38e data 0

Apr 14 07:51:15 unl01 kernel: [  422.226715] kvm [5401]: vcpu0 ignored wrmsr: 0x390 data 0

Apr 14 07:51:15 unl01 kernel: [  422.226904] kvm [5401]: vcpu0 ignored wrmsr: 0x38d data 0

Apr 14 07:51:15 unl01 kernel: [  422.227083] kvm [5401]: vcpu0 ignored wrmsr: 0x38f data 0

Apr 14 07:51:15 unl01 kernel: [  422.228724] kvm [5401]: vcpu1 ignored wrmsr: 0x38d data 0

Apr 14 07:51:16 unl01 kernel: [  422.567229] kvm: zapping shadow pages for mmio generation wraparound

Apr 14 07:52:14 unl01 kernel: [  481.324925] kvm [5401]: vcpu0 ignored rdmsr: 0xce

Apr 14 07:52:14 unl01 kernel: [  481.325149] kvm [5401]: vcpu0 ignored rdmsr: 0x1fc

Apr 14 07:52:14 unl01 kernel: [  481.325521] kvm_set_msr_common: 26 callbacks suppressed

Apr 14 07:52:14 unl01 kernel: [  481.325522] kvm [5401]: vcpu0 ignored wrmsr: 0x1a8 data 0

Apr 14 07:52:14 unl01 kernel: [  481.387720] kvm [5401]: vcpu0 ignored rdmsr: 0x38f

Apr 14 07:52:14 unl01 kernel: [  481.388062] kvm [5401]: vcpu0 ignored wrmsr: 0x38f data 1

Apr 14 07:52:14 unl01 kernel: [  481.388243] general protection fault: 0000 [#1] SMP

Apr 14 07:52:14 unl01 kernel: [  481.388381] Modules linked in: vmw_vsock_vmci_transport(E) vsock(E) openvswitch(E) gre(E) vxlan(E) libcrc32c(E) ppdev(E) vmw_balloon(E) bridge(E) stp(E) llc(E) coretemp(E) kvm_intel(E) kvm(E) crct10dif_pclmul(E) crc32_pclmul(E) ghash_clmulni_intel(E) aesni_intel(E) aes_x86_64(E) lrw(E) gf128mul(E) glue_helper(E) ablk_helper(E) cryptd(E) serio_raw(E) vmw_vmci(E) i2c_piix4(E) shpchp(E) parport_pc(E) mac_hid(E) lp(E) parport(E) vmw_pvscsi(E) mptspi(E) mptscsih(E) vmwgfx(E) ttm(E) psmouse(E) drm_kms_helper(E) drm(E) mptbase(E) vmxnet3(E) e1000(E) scsi_transport_spi(E) pata_acpi(E) floppy(E)

Apr 14 07:52:14 unl01 kernel: [  481.389302] CPU: 0 PID: 5462 Comm: qemu-system-x86 Tainted: G            E 3.16.7-ckt8-unetlab #1

Apr 14 07:52:14 unl01 kernel: [  481.389443] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/30/2014

Apr 14 07:52:14 unl01 kernel: [  481.389736] task: ffff88061a5b0a30 ti: ffff880619f78000 task.ti: ffff880619f78000

Apr 14 07:52:14 unl01 kernel: [  481.389882] RIP: 0010:[<ffffffff81055056>]  [<ffffffff81055056>] native_read_pmc+0x6/0x20

Apr 14 07:52:14 unl01 kernel: [  481.390027] RSP: 0018:ffff880619f7b9d0  EFLAGS: 00010087

Apr 14 07:52:14 unl01 kernel: [  481.390141] RAX: ffffffff81c20cc0 RBX: 0000000000000000 RCX: 0000000040000000

Apr 14 07:52:14 unl01 kernel: [  481.390267] RDX: 000000000002803e RSI: ffff88061aa3d500 RDI: 0000000040000000

BB

Reply
0 Kudos
balajibandi
Contributor
Contributor

Hi support

any update on this issue ? or advice ?

BB

Reply
0 Kudos
covelli
VMware Employee
VMware Employee

> As soon as i see this message in 60 seconds Debian(qemu whole VM Crashing) and  lost connection, even trying to use vspeher with console access no response, i have to force to shutdown in vpshere and bring back.

This does not look good:

Apr 14 07:52:14 unl01 kernel: [  481.388243] general protection fault: 0000 [#1] SMP

...

Apr 14 07:52:14 unl01 kernel: [  481.389882] RIP: 0010:[<ffffffff81055056>]  [<ffffffff81055056>] native_read_pmc+0x6/0x20


Can you try adding "vpmc.enable=TRUE" to the Debian VMs .vmx file?


Thanks,

Doug

Reply
0 Kudos
balajibandi
Contributor
Contributor

Hi

I have added the this line in that file as recomended adding "vpmc.enable=TRUE" to the Debian VMs .vmx

now i see diffrent screen

after all loading, any further advice ?

Reply
0 Kudos
balajibandi
Contributor
Contributor

added this line

cat /sys/module/kvm/parameters/ignore_msrs

Y

Later observed that debian still crashing.

BB

Reply
0 Kudos
balajibandi
Contributor
Contributor

Support

iam still waiting for the response..any one have solution for this ?

BB

Reply
0 Kudos