Very Slow Performance

[Emanoel_Krieger]

Hi All,

I am experiencing an extreme slow performance. The nested virtual machine stay stuck up to 02 minutes to power on. (VMware Screen).

My hardware is a dual 6 core 2.10 GHz Intel. (Total 24 lcpus)

I am using latest stable build from ESXi 5.1.

There is no vms or load in nested ESXi, the first virtual machine windows 2003 server 32 bit takes up to 10 minutos to start installation. The installation completes with no errors.

Any ideias for this extreme slow performance??

I attached the screenshot of VMware screen that nested vms stay delayed up to 02 minutes.

Thanks in advance, Emanoel (VCP-DCV)

[TommyFreddy]

Here is some troubleshoot for slow performance on your ESXi.

Have a look ..

VMware KB: Troubleshooting ESX/ESXi virtual machine performance issues

Take care!

[admin]

WHat hypervisor are you running on the physical host?  Can you upload the vmware.log file for the outer VM?

[Emanoel_Krieger]

Hi Jmattson! This machine will run VSICM courses. I am a VCI preparing the labs for the students.

This is a very weird problem. When I click to power on the Windows 2003 Guest OS VM inside the vESXi, its freeze 01 minute with a black screen, and then appears the VMware screen and takes more 01 minute to start to boot. The boot process is very slow, it takes 05 minutes, after the Guest OS is booted, the performance appears to be normal/acceptable.

I already tried both builds at physical ESXi and vESXi: VMware-VMvisor-Installer-5.1.0.update01-1065491.x86_64.iso, VMware-VMvisor-Installer-5.1.0-799733.x86_64.iso

Since nest is not supported, any kernel developer can assist us since this machine will serve vmware educational purposes?

I attach the vmware.log of the vESXi, and from the guest OS vm running inside vESXi.

Note that when I use:

monitor.virtual_exec = hardware

monitor.virtual_mmu = hardware

I got this output:

/vmfs/volumes/52586770-dd0f6ad5-edfd-001e677b2c69/esxi02_1 # cat vmware.log |grep HV

2013-10-19T14:29:17.524Z| vmx| I120: MONITOR MODE: allowed modes          : BT32 HV HWMMU

2013-10-19T14:29:17.524Z| vmx| I120: MONITOR MODE: guestOS preferred modes: HWMMU HV BT32

2013-10-19T14:29:17.524Z| vmx| I120: HV Settings: virtual exec = 'hardware'; virtual mmu = 'hardware'

Bellow is the .vmx of the vESXi:

/vmfs/volumes/52586770-dd0f6ad5-edfd-001e677b2c69/esxi02_1 # cat esxi02.vmx

.encoding = "UTF-8"

config.version = "8"

virtualHW.version = "9"

pciBridge0.present = "true"

pciBridge4.present = "true"

pciBridge4.virtualDev = "pcieRootPort"

pciBridge4.functions = "8"

pciBridge5.present = "true"

pciBridge5.virtualDev = "pcieRootPort"

pciBridge5.functions = "8"

pciBridge6.present = "true"

pciBridge6.virtualDev = "pcieRootPort"

pciBridge6.functions = "8"

pciBridge7.present = "true"

pciBridge7.virtualDev = "pcieRootPort"

pciBridge7.functions = "8"

monitor.virtual_exec = hardware

monitor.virtual_mmu = hardware

vmci0.present = "true"

hpet0.present = "true"

nvram = "esxi02.nvram"

virtualHW.productCompatibility = "hosted"

powerType.powerOff = "soft"

powerType.powerOn = "hard"

powerType.suspend = "hard"

powerType.reset = "soft"

displayName = "esxi02"

extendedConfigFile = "esxi02.vmxf"

floppy0.present = "true"

featMask.vm.hv.capable = "Min:1"

numvcpus = "6"

scsi0.present = "true"

scsi0.sharedBus = "none"

scsi0.virtualDev = "lsilogic"

scsi0.pciSlotNumber = "16"

memsize = "6144"

scsi0:0.present = "true"

scsi0:0.fileName = "esxi02.vmdk"

scsi0:0.deviceType = "scsi-hardDisk"

sched.scsi0:0.shares = "normal"

sched.scsi0:0.throughputCap = "off"

ide1:0.present = "true"

ide1:0.fileName = "/vmfs/volumes/52586770-dd0f6ad5-edfd-001e677b2c69/ISOS/VMware-VMvisor-Installer-5.1.0-799733.x86_64.iso"

ide1:0.deviceType = "cdrom-image"

floppy0.startConnected = "false"

floppy0.fileName = ""

floppy0.clientDevice = "true"

ethernet0.present = "true"

ethernet0.pciSlotNumber = "32"

ethernet0.virtualDev = "e1000"

ethernet0.networkName = "vPOD-1-Network"

ethernet0.addressType = "vpx"

ethernet0.generatedAddress = "00:50:56:98:a0:0d"

ethernet1.present = "true"

ethernet1.pciSlotNumber = "33"

ethernet1.virtualDev = "e1000"

ethernet1.wakeOnPcktRcv = "false"

ethernet1.networkName = "vPOD-1-Network"

ethernet1.addressType = "vpx"

ethernet1.generatedAddress = "00:50:56:98:ea:36"

ethernet2.present = "true"

ethernet2.pciSlotNumber = "34"

ethernet2.virtualDev = "e1000"

ethernet2.wakeOnPcktRcv = "false"

ethernet2.networkName = "vPOD-1-Network"

ethernet2.addressType = "vpx"

ethernet2.generatedAddress = "00:50:56:98:0a:47"

ethernet3.present = "true"

ethernet3.pciSlotNumber = "35"

ethernet3.virtualDev = "e1000"

ethernet3.wakeOnPcktRcv = "false"

ethernet3.networkName = "vPOD-1-Network"

ethernet3.addressType = "vpx"

ethernet3.generatedAddress = "00:50:56:98:e9:06"

ethernet4.present = "true"

ethernet4.pciSlotNumber = "36"

ethernet4.virtualDev = "e1000"

ethernet4.wakeOnPcktRcv = "false"

ethernet4.networkName = "vPOD-1-Network"

ethernet4.addressType = "vpx"

ethernet4.generatedAddress = "00:50:56:98:3f:96"

vmci0.pciSlotNumber = "37"

chipset.onlineStandby = "false"

guestOS = "other-64"

uuid.bios = "42 18 3f 6f f3 fb ca 5b-17 36 a8 76 ef f7 ad 6f"

vc.uuid = "50 18 d9 28 ae 68 d8 84-6c ba 98 c8 7d 29 58 d9"

virtualHW.scheduledUpgrade.state = "none"

virtualHW.scheduledUpgrade.when = "never"

snapshot.action = "keep"

vhv.enable = "true"

sched.cpu.min = "0"

sched.cpu.units = "mhz"

sched.cpu.shares = "normal"

sched.cpu.latencySensitivity = "normal"

sched.mem.min = "6144"

sched.mem.shares = "normal"

tools.upgrade.policy = "manual"

replay.supported = "FALSE"

replay.filename = ""

scsi0:0.redo = ""

pciBridge0.pciSlotNumber = "17"

pciBridge4.pciSlotNumber = "21"

pciBridge5.pciSlotNumber = "22"

pciBridge6.pciSlotNumber = "23"

pciBridge7.pciSlotNumber = "24"

vmotion.checkpointFBSize = "4194304"

softPowerOff = "FALSE"

ethernet0.startConnected = "TRUE"

ethernet1.startConnected = "TRUE"

ethernet2.startConnected = "TRUE"

ethernet3.startConnected = "TRUE"

ethernet4.startConnected = "TRUE"

vmci0.id = "-268980881"

tools.syncTime = "FALSE"

uuid.location = "56 4d 9b cd 1e 10 81 5f-2b d6 c9 34 cf 52 6b 7d"

cleanShutdown = "FALSE"

sched.swap.derivedName = "/vmfs/volumes/52586770-dd0f6ad5-edfd-001e677b2c69/esxi02_1/esxi02-d3e94363.vswp"

[a_p_]

It looks like you accidentally posted the log for the nested Windows Server twice. Please attach the vmware.log for the virtual ESXi host.

If you don't need the virtual CD-ROM connected to the VMs, consider to disconnect it.

André

[Emanoel_Krieger]

Sorry, here is the vmware.log of vESXi.

This is bizarre, using a very poor hardware I never experienced this behaviour. But in this very expensive Intel Duan Xeon with ECC memory this is happening.

It stay stuck 02 minutes at Vmware screen porwer on at console. Its like a delay or hold or something.

[admin]

It isn't likely to help, but try changing the guestOS type to "VMware ESX 5" (or vmkernel5, if you edit the .vmx file by hand).

[Emanoel_Krieger]

Thanks for your reply!

I had power off the vESXi edit .vmx and try vmkernel5 at Guest OS type. unfortunately nothing changed.

It is unbelievable, any vm that I try to poweron, it stay stuck up to 02 minutes hold into the screenshot attached.

[Datto]

Have you tried an update to the BIOS of your physical machine to see if that would make the startup of the nested VM work better?

Datto

[Emanoel_Krieger]

This is the first thing that I tryed!

I dont made an update, because, the BIOS of physical machine is very new (2013) from an Intel Board Motherboard S2600CP2 Canoe Pass.

I already tried change settings related to VT-Technology, Power Control, Performance.. Load BIOS defaults.

Do you really believe that an BIOS update from an BIOS dated of 2013 will correct this issue?

[Emanoel_Krieger · Accepted solution]

[SOLVED] and works like a charm..

I dont know why was using software mmu, this was causing the delay and poor performance.

2013-10-16T09:20:53.305Z| vmx| I120: HV Settings: virtual exec = 'dynamic'; virtual mmu = 'software'

The way is force it for use hardware exec inserting the bellow parameters at .vmx from guest OS.

monitor.virtual_exec = hardware

monitor.virtual_mmu = hardware

Anyone know why Did I must specify it manually if mode available are BT HV HWMMU ??

[Datto]

Emanoel_Krieger wrote:

Do you really believe that an BIOS update from an BIOS dated of 2013 will correct this issue?

There's no way to know for sure. When I run into these kinds of situations similar to what you've described (ie, quirky unexpected results), updating the BIOS of a physical machine is something I consider doing.

I don't see where you describe what type of physical disks you're using where the slow nested VM is hosted -- what type (5400 spinning, 7200 spinning, USB drive, SSD drive, iSCSI SAN, FC SAN) physical disks are you using?

Datto

[Datto]

Glad to hear you've solved the problem (and, were nice enough to post the solution you found so others can benefit).

Datto

[admin]

32-bit Windows 2003 defaults to binary translation, because of its heavy use of TPR.  It probably should not do so when running nested.