pstorkoh
Contributor
Contributor

VMWare Workstation 11 poor host performance

I've used VMware Workstation for years since I'm frequently called upon to run multiple Server VMs as part of my consulting practice.  After using Workstation 10 for the last couple years I recently upgraded to Workstation 11.  Within a few days after the upgrade I began to notice hesitation while typing, mousing or scrolling in both the host and guest environments.  System activity is not higher than normal during these pauses and nothing ever hangs permanently.  But it makes the system very difficult to use.  Today I downgraded from Workstation 11 back to Workstation 10 and the problems immediately went away after rebooting.  I'm running on a Lenovo W-520 with 3 internal SSD drives and 32 GB RAM on Windows 8.1 Pro, so its not a lack of resources.  I've been running 6 VMs concurrently using about 24 GB total memory in Workstation 10 for several hours this afternoon with no hesitations.  So it really seems to be something in Workstation 11 causing the issue.  Anyone else seen similar issues?

Tags (1)
59 Replies
jhorsman
Contributor
Contributor

Cloning out the NFS Store back to the SMB Store now. I tore it down, but easy enough to get back up. (I ❤️ NAS) 1 hour ETA.

0 Kudos
jhorsman
Contributor
Contributor

Reporting back. Answer is Negative. It does not resolve the issue. So I proceeded to go ahead and try it one line at a time in each VM.. The biggest one to give me a performance gain so far is

cpuid.1.ecx = "----:----:0---:----:----:----:----:----". Attaching Logs of both run by themselves, same reference VM used (2012R2)


I am going to try the other one:

  1. hypervisor.cpuid.v3 = FALSE - Tested - Negative
  2. monitor.phys_bits = 40
  3. cpuid.1.ecx = "----:----:0---:----:----:----:----:----" - Tested - Positive
  4. cpuid.7.ebx = "----:----:----:----:----:----:----:--0-"
0 Kudos
jhorsman
Contributor
Contributor

Okay. I finished. Here are the results.

  1. hypervisor.cpuid.v3 = FALSE - Tested - Negative
  2. monitor.phys_bits = 40 - Tested - Positive
  3. cpuid.1.ecx = "----:----:0---:----:----:----:----:----" - Tested - Positive
  4. cpuid.7.ebx = "----:----:----:----:----:----:----:--0-" - Tested - Negative



So I proceeded to alter all VMs with:


monitor.phys_bits = 40

cpuid.1.ecx = "----:----:0---:----:----:----:----:----"


That netted me the best performance.


Attaching Logs.  I will keep the enviroment around in-case you wish for me to try something else.

0 Kudos
admin
Immortal
Immortal

Thanks for those tests.  I'm not sure what to make of the results, but I will look into it.

Meanwhile, I notice that your CPU microcode is a bit dated.  You have version 12H, and the latest microcode for your CPU is version 1BH.  It may be completely irrelevant, but can you check to see if your system vendor has a BIOS update that may include a microcode patch?  There is an erratum (BV111) for your CPU that can lead to memory corruption, and I suspect it's not fixed until after version 12H.  I wouldn't expect that erratum to result in performance issues, but it's worth a shot.

(The microcode level is reported in the vmware.log file as "Minimum ucode level."  On Intel CPUs, the revision number is followed by eight zeroes.)

0 Kudos
jhorsman
Contributor
Contributor

Yes. I know where there is a newer BIOS Version for my motherboard.. I had to roll it back due to it causing problems with it turning off my VM Extensions on my CPU randomly on reboot. (Its Main "Feature" was secureboot functionality) I would not be surprised if there is a microcode update in there. Will try it out. Worse case I will roll it back again.

0 Kudos
admin
Immortal
Immortal

If you don't want the new BIOS features, you can get the microcode patch from https://downloadcenter.intel.com/Detail_Desc.aspx?DwnldID=24661&lang=eng (which is current as of 19 Feb 2015) and install it with the VMware Microcode Update Driver (https://labs.vmware.com/flings/vmware-cpu-microcode-update-driver).

0 Kudos
p3sullivan
Contributor
Contributor

Dell 6430u i7 16GB - upgraded from 10 to 11 and the exact same load in the guest that was no issue at all with 10 is now killing host with 11. I read the thread but I don't see a root cause just tweaking and down grading... is this going be resolve for all without having to jump through hoops.

0 Kudos
jhorsman
Contributor
Contributor

My best advice is if your going to use WS11 on that laptop, to keep HW 10 on the VMs (or use the HW11 Work arounds on the VMX, though for Prod and interoperability I recommend against.) active until some sort of resolution can be found. I decided to mix it up a little on my end and I introduced in a Lenovo TS-440 with a E3-1225 v3 into my enviroment (Cheap, does the job *extremely well*) .. WS11/HW11 is showing no problems with the same VM's at all. So it is the older CPU's that are showing it the most issue with HW11. Thats the only thing I really recommend at this point.

0 Kudos
harry4516
Contributor
Contributor

after 3 days of trying I finally gave up with WS11 and have reinstalled WS10.

WS10 does not have USB problems in shared VMs and runs much better than 11.1.

With 11.1 I also tried all the tweaks here and also set hardware to 10, but nothing helped,

the hosts KDE desktop got very slow after a while (takes more than 10 seconds to react on a mouse click).

I also tried with Gnome, but it has the same problem.

The PC itself is still fast, ssh-ing into the system is no problem and also the VMs are running fast, just

the host's GUI is sleeping.

I am using the i7-3770k, and I see that many people here are using this CPU, maybe thats the problem.

What is the best CPU for WS11.1 ? Which CPU does support all the new features ?

0 Kudos
jhorsman
Contributor
Contributor

From my tests, seems the haswell-e is the best with WS11/HW11. I use it headless on a ubuntu server attached to NFS space for VM's (Admin through ssh-x11 redirect).. But I didn't have problems with the Ivory Bridge having slow downs like that either on a linux host which is set up with the has-e and is running same exact style. (Running WS11/HW10) Though I have run the IB host with kde, cinnamon or XFCE with little problems.

I guess I would ask the following. with gnome or kde, compositing is on no? (ATI or nvidia?) Guest is windows? (If yes, try reinstalling tools since it is GUI Response?) Accelerate 3d on or off for the VM properties?

0 Kudos
jhorsman
Contributor
Contributor

Only other thing I can say is because I do this by reflex is look at this and see if it also improves things.  Using system GTK+ with VMware Workstation on Linux | @jgoldschrafe

0 Kudos
harry4516
Contributor
Contributor

thank for advice, but the libgtkmm is already installed.

Its really strange, I reinstalled VMware WS 10, but now I have exactly the same problem as with WS11. I have no idea what happens.

My setup:

CPU: i7-3770k, 16 GB ram, Nvidia graphics, 3D driver working fine.

Host: Opensuse 13.2 (KDE or Gnome)

Guest: Win7 and/or Ubuntu (HW10 or 11, same problem)

* I start the Win7 VM, everything works fine.

* then, after about 2 or 3 minutes everything looks fine. But, as soon as I pick a Window and try moving it over the screen the GUI blocks for 5 or 10 seconds.

* a console running htop runs in normal speed. htop shows significant (100%) CPU usage on 1 or 2 cores during this blocking.

* the only chance to fix it is to shutdown the VM. So I shutdown Win7 and this immediately fixes the problem.

The same happens with Ubuntu as Guest

When I restart the VM then everything works fine for some minutes and then the GUI blocking continous.

Regarding the blocking:

When I am working in i.e. text editor (in the host) then I can work without any problem. It only happens as soon as I try to move a window on the screen, and also if I switch from one window to another (moving it from background to front).

So it looks like that the host has a problem drawing on the screen while a VM is running. When I shutdown all VMs then everything runs fine.

Does it make sense to try an Ubuntu host ? Maybe I should test it.

0 Kudos
jhorsman
Contributor
Contributor

Okay. I got an good idea what is going on. I used OPENSUSE 13.2 as a host as well. (I spent weeks looking for an alternative OS to house VM work, was fun.) The FS in it is something else. You mentioned multiple VM's and a couple of minutes. So I suspect that it is IO related with vmem files. (You can confirm with iotop, losf, nmon)

This is what I recommend to try it out..

go into your /etc/vmware/config

Add the following line.

mainMem.backing = "swap"

Restart host so it restarts services, etc. It will eliminate the vmem files which write the ram of the VM's to the File System (Used to cut ram use.) when VMs start. You can verify the functions by looking in the VM folder, will see no *.vem files. (Except a vmem.lck) which is okay. Instead it will use your swap partition. Should speed things up. Also if you want, you might want to check out iotop and see if it is thrashing your disk. (SSD? Mechanical?) before doing it if you do not want to tweak your config file. But I did see that as well. (Got leery when I deployed a WSUS and saw it wrote in one day .7TB, sent me off investigating.)

0 Kudos
harry4516
Contributor
Contributor

that looks good !

In addition to Opensuse I installed a Kubuntu Host.

It was a bit better on Kubuntu, but after a while it blocked the GUI too.

Then I followed your advice and added mainMem.backing = "swap" to the config file.

Since then the GUI is no longer blocked. And the CPU load is  lower than before.

iotop shows the vmware processes writing about 20 to 60K/s.

I am running 3 VMs (Win7, Ubuntu-64bit and Kubuntu-32bit) simultaneously as shared VMs.

Even if the CPU load is high the GUI does not block and is responsive.

It looks like the problem is fixed with mainMem.backing = "swap".

Is there anything I can do regarding the hardware ?

I am running all VMs from an SSD and have 16GB ram is the system.

Where is the bottleneck that causes the problem without above fix ?

Or other question: why is mainMem.backing = "swap" not set by default ?

0 Kudos
jhorsman
Contributor
Contributor

Problem is vmem files are the size of your VMs ram and are constantly written to, so if you say use 10 Gigs of RAM for VM's.. You got 10 Gigs of Files be written/read all the time that is not a vmdk. The vmem choking the drive is usually non-issue until you start scaling with more VMs. As for the main purpose for the vmem files it is to save VMs RAM in use against host RAM.( More Density) thats why I think it is default..  But I am not VMware employee or make the design descisions so I can only speculate. I even ran into problems with my Samsung 850 Pro's in RAID0 starting to choke at points when I was going into the range of 24-28 Gigs of RAM on with 14 or so VMs. So, 28 Gigs of vmem files being written to constantly and ontop of it all the vmdk's for the VM's being constantly read and write operations can simply kill a sata connection, so CPU ends up backlogged with IO which pegs your processor and the VM is trying to make a memory call from a file, which causes the lag on the UI in this case due to saturated sata connections.. Also it hammers the writes on a SSD so it shorten's its life out. So when I found the tweaks to make the vmems go away. I shipped all off to my VMs to NFS space since now a LACP gigabit connection is now suffice to keep them running and reserve the SSD's for high I/O VM's. As for the Hardware Version I am in wait and see mode, maybe a future patch will make it better for Ivory Bridge. If not, HW 10 gets the job done for now.

0 Kudos
harry4516
Contributor
Contributor

thanks a lot for your help and explanation !

That finally solves my problems.

VMware should pay you some commission for your deep knowlegde about this product.

0 Kudos
WhiteKnight
Hot Shot
Hot Shot

I'm exposed to the same WS11 flaw. I have opened a similar thread, too

I do have similar hardware to what you have:

C: drive - 1 * SSD (Samsung 850 PRO) [System]

😧 drive - 2 * SSD (Samsung 850 PRO), RAID 1 [Data, VM guests]

E: drive - 2 * HDD, RAID 1 [Archive]

Intel Core i7-5960X

32 GB RAM

Now I'm very baffled about jhorseman's reply stating that all VM RAM gets permanently backed up to disk. Why is this happening? I've got plenty of RAM, bought to have Workstation use it. I don't need any RAM disk backup.

How can I switch this off permanently?



[VMware]: Workstation 16 Pro; --
[host]: Windows 10x64 host; --
[guests]: Windows 10x64, Windows 8x64.
0 Kudos
jhorsman
Contributor
Contributor

Windows or Linux?

0 Kudos
jhorsman
Contributor
Contributor

never mind.. See that there is a C:\..

mainMem.useNamedFile = "FALSE"

location at c:\ProgramData\vmware\workstation\config.ini

Editing for clarification: Add the line to the config.ini and then reboot host and start them back up. Look inside your VM folders and they will not show the vmem files except the lck file when running.

jhorsman
Contributor
Contributor

White, Quick question, your RAID. What are you using? Windows Softraid or RST? (Is RST throwing errors in the Eventlog?)

0 Kudos