VMware Workstation 16 running on Linux Mint Cinnamon - SLOW

[vmwaresec_rs]

Hello,

I have a Dell XPS 15 9550 laptop that has been upgraded to 32GB RAM, has a 2TB Sabrent Rocket NVMe drive, running Linux Mint Cinnamon 20.2 and I have been struggling with VMware Workstation 16 on this system for months and months now.

I have numerous VM's installed on the system and I can run them but I find the behavior and responsiveness to be really, really slow and it regularly demonstrates a very jerky lack of responsiveness. While running a VM the entire OS will become unresponsive for 30 seconds to minutes at a time and then I will regain some level of control for a little while and then the system reverts to an unresponsive state ... rinse, wash, repeat ...

I have tried uninstalling and reinstalling the software (VMware Workstation 16) several times and this has not made any difference at all.

So ... I tried an experiment ... I installed Oracle VirtualBox to see how that would perform.

Well ... what a difference!!

I have 2 Windows 10 VM's, each with 8GB or RAM, and a Windows 2008 server, a Windows XP system, and 3 Ubuntu VM's (all the old systems are because I am doing Kali / Metaspolit training) and with ALL of these running at the same time the entire OS and each of the VM's hums along without a care in the world.

The system is actually behaving the way I would expect when running all of these VM's. Bottom line here is that I have been struggling with extremely frustrating poor performance using VMware Workstation 16 for months, and months and VirtualBox is blowing it away.

In general I love VMware, I have a Raspberry Pi ESXi cluster and an Intel NUC running ESXi all reporting into a vCenter instance, we use VMware at work, etc., etc. but I just cannot bear to use VMware Workstation when experiencing this abysmal performance.

Edit / adding another data point that I think is pertinent: I see the same super slow jerky performance when I use the "Launch Remote Console" option from vCenter (which appears to just launch VMware Player). So this isn't even really using any of the local computers resources (no vmdk disk access, no memory to run the VM, etc.) and it still performs poorly.

Anyone have any ideas what the issue might be? Any suggested solutions?

Thanks,

Robert

[S_D]

Corrected Post:  I'm not sure how I misread the KB, but Linux Mint is noted in the VMware Knowledge base as a supported Host OS for Workstation 16.

[continuum]

> but perhaps it's worth testing on Ubuntu, 

You may want to read the question again .... OP  is using an Ubuntu already.
Ulli

[S_D]

Correction #2 / clarification:  Ubuntu is a supported Guest OS for sure, ref: https://www.vmware.com/resources/compatibility/search.php?deviceCategory=software&details=1&releases....

[continuum]

Linux Mint is a derivative of Ubuntu - see https://linuxmint.com/about.php

[vmwaresec_rs]

The host OS is Linux Mint Cinnamon which is an Ubuntu derivative

https://en.wikipedia.org/wiki/Linux_Mint

"Linux Mint is a community-driven Linux distribution based on Ubuntu (in turn based on Debian), bundled with a variety of free and open-source applications.[6][7] It can provide full out-of-the-box multimedia support for those who choose (by ticking one box during its installation process) to include proprietary software such as multimedia codecs.[8]"

[vmwaresec_rs]

I don't see anything about Linux Mint 19. I've never even run that OS (only Linux Mint 18.04 and 20.x)

Host OS is Linux Mint 20.2

(base) rstrom@rstrom-XPS-15-9550:~/vmware$ lsb_release -a
No LSB modules are available.
Distributor ID: Linuxmint
Description: Linux Mint 20.2
Release: 20.2
Codename: uma

This behavior / poor performance has also been happening long before I recently upgraded to from 20.04 to 20.2

The host OS is Linux Mint Cinnamon which is an Ubuntu derivative

https://en.wikipedia.org/wiki/Linux_Mint

"Linux Mint is a community-driven Linux distribution based on Ubuntu (in turn based on Debian), bundled with a variety of free and open-source applications.[6][7] It can provide full out-of-the-box multimedia support for those who choose (by ticking one box during its installation process) to include proprietary software such as multimedia codecs.[8]"

[vmwaresec_rs]

Please see this https://kb.vmware.com/s/article/80807

Unless I need to relearn how to read English I see Linux Mint listed as a supported host OS

Robert

[S_D]

Robert have you opened an SR with VMware?  If so can you message me the #?  I'd like to look it up.  In case you haven't involved Support yet, here's how: https://kb.vmware.com/s/article/2006985.

[vmwaresec_rs]

S_D,

The software that I am using is licensed via the VMUG (VMware Users Group) and, as far as I can tell, will only be covered via pay per incident support.

I have never done pay per incident support before. While I am not necessarily opposed to this I do not want to pay for support and then have no resolution.

If I pay, I want my software to work. I want to have some screaming performance like I would / should expect.

Is that a reasonable expectation? Will my pay per incident support be guaranteed to be resolved satisfactorily or money back?

Robert

[S_D]

Sorry for the mis-information yesterday... my eyes aren't what they once were and I've corrected those posts.  I'm not a lawyer and can't speak for VMware, but based on what I found, if you haven't registered Workstation yet in Customer Connect, here's how: https://kb.vmware.com/s/article/2011177.  After registering, you'll have 30 days of no-cost Complimentary Support:  https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/support/vmware-complimentary-suppo...).  If that scenario doesn't apply then per-incident is the correct route: https://kb.vmware.com/s/article/2014035.  I believe your question about guarantees is addressed by the Ts & Cs linked at the bottom left side of Page 2 of this document: https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/support/vmware-per-incident-suppor....

Yep, that's a lot of references.  Back to the issue for which you created the post, have you tried these steps already (those that apply)?  https://www.makeuseof.com/tag/virtualization-issues-simple-solutions/.  Do you have VMware Tools installed in the guest VMs?

[vmwaresec_rs]

Thanks for the links ... as far as the makeuseof.com link ... rudimentary ... I have been using VMware Workstation since appropriately 2000 ... 20+ years experience.

... Do you have VMware Tools installed in the guest VMs?

Come on now ... I think that my original post should have eluded to the fact that I am no where near a VMware / Virtualization newbie

- Raspberry Pi ESXi cluster

- Intel NUC with ESXi installed

- vCenter installed and managing both of those sets of ESxi servers

Mulling over the idea of opening a paid support case.

Thanks,

Robert

[calc76]

I noted in another thread 'VMware Workstation Pro 15.5.2 Freezes - Ubuntu 18.04 LTS Host and Any Guest OS' this has been happening for years, at least since 2019, and does not matter which distribution is being used, and still happens with current Workstation. From reading the bug reports on LKML/Red Hat bugzilla it appears Workstation is probably trying to allocate too large a memory block or something similar which causes the kcompactd kernel thread to stall using 100% cpu along with the vmware-vmx threads. It doesn't happen when not running Workstation, even with other VMs like VirtualBox.

Its very surprising that VMware has not fixed this issue yet as it is very easy to reproduce and causes severe problems until the host is rebooted once it starts happening.

[louyo]

This is kind of out of left field, but worked for me. I  run WS on an LMDE 4 host, mate desktop. I have several W10 VM's, a Windows server VM, a Ubuntu, another LMDE and a Centos VM. It seemed like certain combinations of VM's active at the same time would cause what seemed like a freeze in the VM I was using. Lasted several seconds at a time. Sometimes switching to another VM would "free" it sometimes not. It seemed to never happen unless one of about 3 different Windows VMs was active. The same VMs copied over to ESXi never have seemed to do it, but that is different environment. So, perhaps these fixed it, perhaps it just went away.

1. Even though I never have all memory allocated, the host sometimes showed Swap in use. Seemed like this happened when doing heavy disk activity, like running a backup. I set swappiness to 10, default is 60. Does not prevent swapping, just delays it. I have run it at 0, with no problems. Not sure this is necessary any more but I have done it for years.

2. Found an old, old (2017) post I had used sometime back.

https://communities.vmware.com/t5/VMware-Workstation-Pro/vmware-workstation-100-disk-usage/m-p/27048...

Makes little sense for the Linux VMs, but I haven't seen any freezes in a quite a while and VM performance is good. I had not seen 100% disk activity but the Task Manager had frozen as well.

FWIIW.

Lou

[vmwaresec_rs]

Lou,

Thanks for the post. I will mess around with the swappiness setting but I looks like I already modified the setting (I honestly do not remember doing it but it looks like I have already made a change).

My swappiness is currently set to 5 so I do not have very much farther to go.

Thanks again,

Robert

[Mikero]

Hmmm.... Not sure if we test Cinnamon... Doest standard Gnome have the same issue?

What kernel version is in play here?

[calc76]

Mikero,

Its definitely not just Mint/Cinnamon.

I've seen it happen on multiple boxes with different distributions etc for several years. Looking through the forum it appears people have been complaining about this issue since at least version 15 and its still happening on 16.1.1. And the solved ones appear to have been solved by moving to VirtualBox instead. There are also posts with similar issues (below) that were on Windows so maybe this issue isn't even Linux specific. The current system I am seeing it on is Fedora 34 with 5.13.5, but I had problems with it for years on Ubuntu. I probably should have commented on the forum before now.

The easiest way I have found to reproduce it consistently is to run 2-3 VMs along with disk i/o, that should trigger the problem pretty quickly even if you have plenty of free ram and extra cores. I have seen it happen with 20GB used ( /proc/meminfo MemTotal-MemAvailable) on a 64GB system, so it should have another 44GB available to use. My current system has a Ryzen 3900x (12/24) with only 4 threads configured total between VMs when it happens to me. Also when it happens vmware-vmx threads run at 100% and the kcompactd0 kernel thread is at 100%. The system becomes nearly completely unresponsive and you can't even type. Nothing else on the system is using noticeable amount of cpu. And once I manage to shutdown the VMs the system is fine.

There was a similar Linux kernel issue with hugepages a few years ago that was fixed, but even with hugepages completely disabled this problem still happens with VMware. The following thread has some info on how to capture kernel data related to the issue. I'm not sure if this would be helpful for VMware or not.

Re: [regression -next0117] What is kcompactd and why is he eating 100% of my cpu? 

This Fedora BZ was closed without fix but was pointing at VMware doing something wrong way back in March 2019.

Bug 1694305 - kcompactd0 using 100% cpu

This Windows issue sounds very similar and is marked solved because they switched to VirtualBox.

VMWare Workstation 14 (vmware-vmx.exe) high CPU and VM unresponsive 

[vmwaresec_rs]

Linux Mint CInnamon

Kernel version as per uname -r

5.4.0-80-generic

[vmwaresec_rs]

I found posts about the hugepages settings also. I played with those settings and found that they really didn't make any difference.

It seems super ironic to me that I have VM's that run better on a Raspberry Pi ESXi cluster (and this isn't even software that is considered ready for prime time) that has 4 8GB Raspberry Pi's with a total of 32GB of RAM between all 4 of them than on my laptop with 32GB of RAM. I have 15 VM's running on that Raspberry Pi cluster and they run amazingly well.

VMware ... please, please leverage this thread to reach out to those that want to solve this issue. You seem to have a group that all agrees that it is an issue and we want to fix it. If you'll work with us, we'll work with you.

Robert

[vmwaresec_rs]

So I decided to do some testing.

I removed the Linux Mint Cinnamon SSD and installed a different SSD and tried installing Kubuntu to see how VMware Workstation would perform on another OS.

I got Kubuntu and VMware WS 16.1.2 build-17966106 up and running pretty quickly and copied over a number of existing VM's that were being using on the Linux Mint OS install and fired them up. I launched 6 VM's, several configured with 4 CPU's and 8GB of RAM and some with lesser resources.

Everything ran smooth as silk. No slowness, not system freezes, no jerkiness. What an amazing difference! Performs exactly as I would expect.

Since I've been looking at getting a new Linux laptop sometime in the near future I thought that I would see how VMware ran on the OS that I would probably have installed on the new laptop, Pop!_OS (System76.com)

I installed Pop!_OS and VMware Workstation 16.1.2 build-17966106 and copied over all the same VM's and fired them up. Everything is running smooth as can be ... I am typing this message on that OS install right now.

Then I thought that I would try a different flavor of Linux Mint (there are 3 - Cinnamon, MATE, and XFCE). I did an install of Linux Mint MATE and tried to install VMware Workstation 16.1.2 build-17966106 and was unsuccessful. VMware Workstation installed but would not launch any VM's. It could not load the VMMON and VMNET modules. I spent a bit of time trying to fix it but was unsuccessful (kernel headers were installed).

So it looks like I have found my new Linux OS and have VMware Workstation performing as expected. Have no idea why the performance was so poor on my install of Linux Mint Cinnamon, but I am very pleased that all is working well. Is it Linux Cinnamon itself or was it something that I had installed. We may never know.

Robert