pstorkoh
Contributor
Contributor

VMWare Workstation 11 poor host performance

I've used VMware Workstation for years since I'm frequently called upon to run multiple Server VMs as part of my consulting practice.  After using Workstation 10 for the last couple years I recently upgraded to Workstation 11.  Within a few days after the upgrade I began to notice hesitation while typing, mousing or scrolling in both the host and guest environments.  System activity is not higher than normal during these pauses and nothing ever hangs permanently.  But it makes the system very difficult to use.  Today I downgraded from Workstation 11 back to Workstation 10 and the problems immediately went away after rebooting.  I'm running on a Lenovo W-520 with 3 internal SSD drives and 32 GB RAM on Windows 8.1 Pro, so its not a lack of resources.  I've been running 6 VMs concurrently using about 24 GB total memory in Workstation 10 for several hours this afternoon with no hesitations.  So it really seems to be something in Workstation 11 causing the issue.  Anyone else seen similar issues?

Tags (1)
59 Replies
harry4516
Contributor
Contributor

I am also surprised that VMware writes the mem to disk by default.

Even after the fix from jhorsman I can see that only 3GB of my 16GB are in use, and I am running 3 VMs in parallel.

So it should be no problem to set these options.

I found this page:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=100888...

the last hint says:

for Windows hosts:

mainMem.useNamedFile = "FALSE"


for Linux hosts:

mainmem.backing = "swap"


exactly what jhorsman told us.

This will safe the life of my SSDs.

0 Kudos
jessepool
VMware Employee
VMware Employee

Hi harry4516.

Or other question: why is mainMem.backing = "swap" not set by default ?

Sorry for the confusion, but mainMem.backing = "swap" is the default on Linux. However! We switch from the default to creating a vmem file if your system does not have enough swap space to support the amount of pageable memory needed by the VM. If you want to make sure that mainMem.backing = "swap" is always used by default (without manually editing the configuration files) then go to Edit->Preferences->Memory and choose "Fit all virtual machine memory into reserved host RAM".

jhorsman
Contributor
Contributor

Thats nice to know. Now I am curious. Whats the threshold VM RAM Vs Host Swap for that behavior to kick in? Example: 100 Percent on Running VM RAM to 100 Percent Host swap? Windows has a similar mechanism? Or is it easier just to make sure swap/page file reflects host ram?

0 Kudos
harry4516
Contributor
Contributor

what is the difference between: swap and reserved host ram ?

I have 16GB physical RAM,

if I run 3 VMs then top shows a ram usage of 12GB.

Does it make sense to add additional 16GB ? Will this additional RAM be used ?

(I have set: "Fit all virtual machine memory into reserved host RAM")

0 Kudos
munrobasher
Enthusiast
Enthusiast

I've got it set to "some RAM can be swapped" but I'm sure v11 is doing more than v10. Had just one W7 VM running just now - using a mere 1.37GB of RAM. And VM still froze from about 2 minutes a short while back with lots of host activity to the VMEM file. There was no reason for it do be doing this. There is over 10GB free in my host PC at the moment. As always, the VM was still running as I could hear the occasional noise from it (Facebook notification) - it's just the user interface that's mainly not responsive.

0 Kudos
WhiteKnight
Hot Shot
Hot Shot

I apoligize for not having replied yet. I didn't get automatically subscribed to this thread so I thought it got stalled.

Thanks, J, for providing me with this setting. I'll try it right away. Ah, and thank you, Jesse, for pointing to the appropriate setting in Workstation settings dialog. I prefer this approach, because I don't have to recall the setting name when I'm going to setup my host machine anew.

I'm actually utilizing Intel RST on all my dedicated machines for establishing RAID.

Three things are disturbing me since I have installed W11 (given default settings):

  • Guests hang occasionally when I'm working in Fullscreen mode. No keyoard or mouse input is processed then. This happens about four times a day. The freeze never takes longer than a minute. All I can do when the guest freezes is to move the mouse to the top of the screen to have the Workstation toolbar to appear. There I can switch-off fullscreen. Then, after a few seconds, the guest continues working.

  • Occasional latency. When I'm typing text in a word processor or Visual Studio within a guest, the cursor appears to be delayed. Holding down one of the cursor keys, for example, and releasing that key again will have the cursor continue to run for about one second. This doesn't always happen. Just occasionally. But it's still disturbing, though. Particularly as I never had this happening on any of my machines with W10. Even the slower ones.

  • Drag & Drop from guest to host is impaired. Occasionally (= most of the time, but not always) I can't drag files from guest to host OS. The mouse pointer just leaves the guest but I won't get the Plus sign added to the mouse pointer in the host OS; so nothing gets dragged or dropped. [EDIT: Drag & Drop is delayed by about 5-10 seconds. The Plus sign seems to appear after holding down the mouse button within the host OS for an unexpectedly long period.]

Host: Windows 7/64

Guest: Windows Server 2008 R2



[VMware]: Workstation 16 Pro; --
[host]: Windows 10x64 host; --
[guests]: Windows 10x64, Windows 8x64.
0 Kudos
piggyz
Enthusiast
Enthusiast

Windows 8.1 64 bit Host

VMware Workstation 11.1

I7-3770

32 GB RAM

7200 rpm HDD reserved to VMs

Usually seven or eight concurrent mixed guests running fine in version 10. Never had particular slowness problems in the past with Worstation 10. Eleven with hardware 11 was/is a pain. Downgraded to hardware revision 10 everything is fine but slowness to access the disk never ever experienced in the past still persist.

mainMem.useNamedFile = "FALSE"

instruction globally put in config.ini file wasn't enough to restore old performance to guests. Then obviously it consumes a lot of main memory RAM. Someone here said he was still in doubt related with the above settings real effect on host RAM; I can say I can see the host RAM gone for every single virtual guest machine I do run, so no doubt it has an impact.

Still, before Workstation 11, I was able to run everything quietly speedy without the need to disable VMEM files for guest machines, so I still think Workstation 11 wasn't probably tested enough before realese.

WhiteKnight
Hot Shot
Hot Shot

Just as a quick reply to Jesse:

Nope, the GUI setting does not disable utilizing VMEM files:

Memory.png

Config.png

Explorer.png



[VMware]: Workstation 16 Pro; --
[host]: Windows 10x64 host; --
[guests]: Windows 10x64, Windows 8x64.
0 Kudos
jessepool
VMware Employee
VMware Employee

Hi WhiteKnight,

Nope, the GUI setting does not disable utilizing VMEM files:

That only works on Linux where we have the mainMem.backing = "swap" option. If you're on Windows then you need to use mainMem.useNamedFile = "FALSE", and then Workstation will use the system pagefile instead.

Sorry for the confusion!

0WayneH0
Hot Shot
Hot Shot

Throwing in my anecdotal $0.02.

Since updating to WS 11.0 (now 11.1) I have noticed some guest performance issues (Win 7) running Microsoft Visual Studio 2012. Scrolling code windows sometimes slows right down to the point it makes working within Visual Studio quite painful. As per the comments in this thread I have tried (just now) downgrading to HW10 and will see if that resolves the issue. I'll report back in a few days.

My host:

Intel core i7-3930K (Sandy Bridge-E).

32GB RAM

ASUS HD7950

Win7 x64 Ultimate

c:\ (500GB Samsung SSD) - Contains master VMs (for linked clones).

d:\, e:\ 2 x 600GB Enterprise Velociraptors. (Linked clone and full VMs live on these two drives).

f:\ 2TB WD Enterprise HDD

g:\, h:\ 2 x 4TB WD Black drives.

The host is very stable and otherwise has no issues.

I've also noticed not super performance in a Debian 7.x guest (whilst running HW10), but this is on a Windows 8.1 host (HP EliteBook 8560w, i7-2820QM (Sandy Bridge)). I'm not entirely convinced that it's not a Windows 8.1 issue since that system is far from problem free. (Was fine when I had Windows 7 on it though). Smiley Sad

0 Kudos
harry4516
Contributor
Contributor

the absolute worst VM: Ubuntu running the Unity desktop.

If you ever plan running Ubuntu as a virtual machine, avoid Unity like the plague.

Unity kills even machines with lot of resources and makes it slow.


I have another VM running Kubuntu (Ubuntu with KDE) needing less RAM and runs much smoother.

0 Kudos
harry4516
Contributor
Contributor

I finally solved the problem with poor Host perfomance:

all the setting discussed here help a bit, but did not solve the blocking GUI.

Under some conditions KDE or Gnome was still badly blocking.

I have 16GB DDR3 ram and usually a Win7 and a Ubuntu VM running simultaneously.

Because I ran out of ideas I extended the ram to 32GB.

Wowww, what a difference, now everything runs smooth, no more blocking. Now I am very happy with the system.

My theory (is that true ???) is a problem with the "reserved" host ram and VMware:

Linux takes all of the available RAM for buffering and caching (i.e. if we copy files). This ram is not really occupied because as soon as an application requests more ram the system will release this cache and pass it to the application. It is just temporarily used by Linux. (htop shows this ram in yellow color).

Now if VMware needs ram it DOES NOT get it from this temporary used ram (as other applications are doing), instead it takes additionally free ram.

What I can see:

1) ram used by applications (htop: green)

2) ram used by the linux caches and buffers (htop: yellow)

3) ram used by vmware (htop: also yellow)

the problem is, if VMware needs more ram, it will not take it from the linux buffers as other applications do. This results in the problem that we need more and more ram. As longer as the system runs as more ram we need. When copying huge files linux takes ram for caching. VMware is not able to get this portion of temporarily used ram as other applications do.

There is a simple test for this issue: copy many huge files and watch ram usage. As soon as the temporary buffer is full, VMware will cause blocking instead of getting this temporary ram.

It runs fine with 32GB, but I am sure in a few days also this ram will be fully occupied.

VMware must chance its way to request host ram, I believe this is the only solution.

Now I found a way to solve this problem:

I made a cronjob which is executed every 10 minutes:

sh -c "sync; echo 3 > /proc/sys/vm/drop_caches"

this releases the linux cache. I hope this will finally solve all the blocking issues, so far it runs good.

Harry

0 Kudos
munrobasher
Enthusiast
Enthusiast

harry4516 wrote:


VMware must change its way to request host ram, I believe this is the only solution.

Something has definitely changed between WS10 and WS11 in this area and only VMware know what. It's definitely around (overcommitted?) memory management. The observed effect is that periodically WS carries out a lot of writes to a VMEM file. During this time, the WS GUI is in a "not responding state". I've come over here as I've just been waiting 10+ minutes for it to wake up again. It's currently writing ~5,000,000 bytes per second to a VMEM file on the SATA-3 E: drive. That's not a particularly high write rate for SATA-3 - about 5MB/s; SATA-3 can easily handle way more than that. In fact, I managed to grab a task manager window whilst waiting. Most of the time, disk throughput is quite low, compared to a 100MB/s peak at one point - that's getting towards SATA-3 saturation. Which shows that my system can handle full SATA-3 speed. So I've got relatively low throughput but the drive is 100% active - what does that suggest? Lots of random I/O?

I'm going back to WS10 today as it's hurting productivity.


Cheers, Rob.

0 Kudos
munrobasher
Enthusiast
Enthusiast

One more observation - recent versions of WS have always caused very high disk I/O when you pause a VM. I've always had to go and make a cup of tea (well not actually but lots of disk I/O) after pausing a VM on a busy host. If you look at the memory tab in resource meter, there is block called "Modified" in orange. When you pause a VM, the RAM being used goes instantly from "in use" (green) to modified (orange). Lots of references to Windows memory on the net but in summary, modified RAM is waiting to be written somewhere - I think in the WS case, it's writing the RAM snapshot VMEM file. However, whilst one has any significant amount of modified memory on a disk, you'll see lots of disk activity. Watch as the modified memory orange bar get smaller and when it hits zero, disk activity will stop. You'll see lots of references about modified memory saying it's a staging area for writing to the page file. I'm not sure that's 100% true in all cases. When you copy a large file with Windows, you'll also see modified memory increase and that's because it's using modified RAM as a big disk buffer. That's why if you copy a file that's around the size of free memory, it appears to copy almost instantly - but it hasn't actually finished copying; it'll be sat in the modified RAM waiting to write to disk. So the file I/O routines in Windows must use modified memory as a big disk buffer.

NOTE: in the original problem of this thread, modified memory flushing to disk doesn't appear to be the problem. I don't typically see big orange/modified buffers - just 100% disk activity but relatively low throughput.

Cheers, Rob.

PS. Just copied a large VMDK file from my C: drive (SSD) to E: drive (SATA-3) and it's managing 116MB/s which is totally normal for a non-RAID 7200RPM SATA-3 drive.

0 Kudos
harry4516
Contributor
Contributor

after 4 days of testing at least my problem with hanging GUI is solved.

Two settings helped:

1. Preferences, Memory: "fit all mom... into reserved host ram".

2. the cronjob: sh -c "sync; echo 3 > /proc/sys/vm/drop_caches" every 10 minutes.

Since then I never had any blocking GUI, everything runs fine.

0 Kudos
shbelay
Contributor
Contributor

Thank you jhorsman!! That did it. So far it is running for longer than a few minutes so that is a good sign. haha. I really didn't want to go through the process of downgrading to v10.

0 Kudos
WhiteKnight
Hot Shot
Hot Shot

Another strange observation:

On my notebook, which is equipped almost exactly as my desktop machine (except for the HDD RAID, but SSD RAID configuration is the same), I didn't have added

mainMem.useNamedFile = "FALSE"


yet.


However, no VMEM file is created.


How is this possible?



[VMware]: Workstation 16 Pro; --
[host]: Windows 10x64 host; --
[guests]: Windows 10x64, Windows 8x64.
0 Kudos
Source3
Contributor
Contributor

Thanks everyone to have posted solutions.  I have been running two guests all day without a single freeze/hang - finally!  Below is a summary of system specs and what I did.  It may help someone out there that was just as confused as me?  I was losing productivity and had to find a fix fast.

VMWare Workstation 11

Host -

CPU: Intel i7 5960 3Ghz

RAM: 32 GB

OS: Windows 7 64 Bit

Guest 1 -

OS: Windows 7 64-Bit

RAM: 8 GB

Processors: 2

3D graphics: Off (for now)

Guest 2 -

OS: Windows 7 64-Bit

RAM: 8GB

Processors: 2

3D graphics: Off (for now)

Solutions:

  1. Enabled Intel VT-x on both guests
  2. Per: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=100888...
  3. Open the .vmx file for the affected virtual machine while it is powered off. Add the following lines to the file using a text editor. For more information, see Editing the .vmx file of a VMware Workstation and VMware Player virtual machine (2057902)

    Note: If you are using VMware Server, you may need to restart the VMware Authorization Service (vmware-authd) for changes to take effect.

    MemTrimRate = "0"

    mainMem.useNamedFile = "FALSE"

    sched.mem.pshare.enable = "FALSE"

    prefvmx.useRecommendedLockedMemSize = "TRUE"

  4. In Workstation menu bar, Edit->Preferences->Memory and choose "Fit all virtual machine memory into reserved host RAM".  Then set memory to 26GB.  I set to 16GB, the amount of memory used by the guests, but Workstation gave me a warning of not enough memory when I tried to start Guest 2.  I have memory to spare so I set it to 26GB.


I am going to leave these settings for a week or so then start increasing guest processors and memory, one change at a time.  My goal is to run the two guests with 3 processors and 12 GB of ram with 3D graphics. 

0 Kudos
trboomer
Contributor
Contributor

Thanks for posting but in my case I tried all these things and promptly found VMWare hanging again.

Hey VMWare? Do you guys even care about the Workstation product?

0 Kudos
LaminarCS
Contributor
Contributor

I think you are onto something here.

Instead of periodically flushing the disk cache, I simply changed some of the kernel memory settings.

Here are the settings I added to the sysctl.conf:

vm.swappiness=40
vm.vfs_cache_pressure=400
vm.min_free_kbytes=4096000


You will need to reboot for the settings to take effect, or you can set them on the live system individually with sysctl -w var=whatever.

These settings could probably be fine tuned a lot more, as now almost nothing ends up in the buffer.  My system has 16GB of RAM and I typically have a 4GB Windows 7 VM and a 2GB FreeBSD VM running.  I am not noticing any major loss in performance with the much reduced cache since my machine is on an SSD.


I also switched the IO scheduler to CFQ, but I am not sure if that helped much ... it may have helped with the random freezing if the memory situation wasn't sorted with the above settings.


This issue with VMWare Workstation not being able to use memory that is in the Linux disk cache is definitely new.   I didn't have any issues with VMWare Workstation 10.  Something has changed in 11 to seriously break things on Linux.

0 Kudos