VMware 16.2.2 results in hanging virtual machines

[zyxxel]

I regularly gets hanging virtual machines since I updated to version 16.2.2

Only resolution seems to be to kill the process for the virtual machine, restart the GUI and then restart the virtual machine.

Seems to affect any virtual machine independent of what is installed in it.

I never had any hangs with older VMware Workstation versions.

[Arcanan]

So you have a problem with host PC.

Either small memory, problem with graphics, antivirus, some new software is blocking, etc.

Tray put here a log file to see more details.

[continuum]

Switch back to Workstation 16.1 - WS 16.2 is work in progress ....

Ulli

[zyxxel]

Just that I don't seem to be able to find any links to older versions.

I would most definitely want to test 16.1 and see if that helps.

I have maybe one VM hang every week or two since I installed 16.2.1 and 16.2.2. And always requiring me to kill the VM process and then restart the GUI. At least 1 hour time lost each time it happens.

But the problem is that this machine didn't run 16.1 before, so I have no downloaded 16.1.x version. I'm pretty sure that VMware earlier allowed download of not just latest release, but seems I can now only download versions I have already downloaded before.

[zyxxel]

The machine has run VMware Workstation for about 5 years with zero VM crashes.

Then I update to latest version and have a crash/week. And then google and see lots of other people having the same issue with latest VMware Workstation release. There seem to be a pattern there, that does not point at the host machine suddenly becoming unstable. Especially since no other programs have started to magically fail.

This is the sad last log lines before hang:

Spoiler

2022-03-04T03:45:10.037Z In(05) vmx MemSched: caller 0 numvm 1 locked pages: num 437212 max 6327946
2022-03-04T03:45:10.038Z In(05) vmx MemSched: locked Page Limit: host 6336138 config 7309568
2022-03-04T03:45:10.038Z In(05) vmx MemSched: minmempct 50 timestamp 2054824
2022-03-04T03:45:10.038Z In(05) vmx MemSched: VM 0 min 437212 max 833500 shares 792576 paged 421154 nonpaged 36578 anonymous 4346 locked 437212 touchedPct 3 dirtiedPct 1 timestamp 2054824 vmResponsive is 1
2022-03-04T03:45:10.038Z In(05) vmx MemSched: locked 437212 target 833500 balloon 0 0 490589 swapped 371949 0 allocd 0 512 state 0 100
2022-03-04T03:45:10.038Z In(05) vmx MemSched: states: 0 676801 : 1 0 : 2 0 : 3 0
2022-03-04T03:45:10.038Z In(05) vmx MemSched: Balloon enabled 1 guestType 1 maxSize 754753
2022-03-04T04:15:10.605Z In(05) vmx MemSched: caller 0 numvm 1 locked pages: num 437212 max 6321332
2022-03-04T04:15:10.605Z In(05) vmx MemSched: locked Page Limit: host 6329524 config 7309568
2022-03-04T04:15:10.605Z In(05) vmx MemSched: minmempct 50 timestamp 2056625
2022-03-04T04:15:10.605Z In(05) vmx MemSched: VM 0 min 437212 max 833500 shares 792576 paged 421154 nonpaged 36578 anonymous 4346 locked 437212 touchedPct 3 dirtiedPct 0 timestamp 2056625 vmResponsive is 1
2022-03-04T04:15:10.605Z In(05) vmx MemSched: locked 437212 target 833500 balloon 0 0 490589 swapped 371949 0 allocd 0 512 state 0 100
2022-03-04T04:15:10.605Z In(05) vmx MemSched: states: 0 678601 : 1 0 : 2 0 : 3 0
2022-03-04T04:15:10.605Z In(05) vmx MemSched: Balloon enabled 1 guestType 1 maxSize 754753
2022-03-04T04:45:11.706Z In(05) vmx MemSched: caller 0 numvm 1 locked pages: num 437212 max 5920924
2022-03-04T04:45:11.706Z In(05) vmx MemSched: locked Page Limit: host 5929116 config 7309568
2022-03-04T04:45:11.706Z In(05) vmx MemSched: minmempct 50 timestamp 2058426
2022-03-04T04:45:11.706Z In(05) vmx MemSched: VM 0 min 437212 max 833500 shares 792576 paged 421154 nonpaged 36578 anonymous 4346 locked 437212 touchedPct 2 dirtiedPct 0 timestamp 2058426 vmResponsive is 1
2022-03-04T04:45:11.706Z In(05) vmx MemSched: locked 437212 target 833500 balloon 0 0 490589 swapped 371949 0 allocd 0 512 state 0 100
2022-03-04T04:45:11.706Z In(05) vmx MemSched: states: 0 680401 : 1 0 : 2 0 : 3 0
2022-03-04T04:45:11.706Z In(05) vmx MemSched: Balloon enabled 1 guestType 1 maxSize 754753
2022-03-04T05:15:12.711Z In(05) vmx MemSched: caller 0 numvm 1 locked pages: num 437212 max 6213243
2022-03-04T05:15:12.711Z In(05) vmx MemSched: locked Page Limit: host 6221435 config 7309568
2022-03-04T05:15:12.711Z In(05) vmx MemSched: minmempct 50 timestamp 2060227
2022-03-04T05:15:12.711Z In(05) vmx MemSched: VM 0 min 437212 max 833500 shares 792576 paged 421154 nonpaged 36578 anonymous 4346 locked 437212 touchedPct 2 dirtiedPct 0 timestamp 2060227 vmResponsive is 1
2022-03-04T05:15:12.711Z In(05) vmx MemSched: locked 437212 target 833500 balloon 0 0 490589 swapped 371949 0 allocd 0 512 state 0 100
2022-03-04T05:15:12.711Z In(05) vmx MemSched: states: 0 682201 : 1 0 : 2 0 : 3 0
2022-03-04T05:15:12.711Z In(05) vmx MemSched: Balloon enabled 1 guestType 1 maxSize 754753
2022-03-04T05:45:13.794Z In(05) vmx MemSched: caller 0 numvm 1 locked pages: num 437212 max 6210365
2022-03-04T05:45:13.794Z In(05) vmx MemSched: locked Page Limit: host 6218557 config 7309568
2022-03-04T05:45:13.794Z In(05) vmx MemSched: minmempct 50 timestamp 2062028
2022-03-04T05:45:13.794Z In(05) vmx MemSched: VM 0 min 437212 max 833500 shares 792576 paged 421154 nonpaged 36578 anonymous 4346 locked 437212 touchedPct 5 dirtiedPct 1 timestamp 2062028 vmResponsive is 1
2022-03-04T05:45:13.794Z In(05) vmx MemSched: locked 437212 target 833500 balloon 0 0 490589 swapped 371949 0 allocd 0 512 state 0 100
2022-03-04T05:45:13.794Z In(05) vmx MemSched: states: 0 684001 : 1 0 : 2 0 : 3 0
2022-03-04T05:45:13.794Z In(05) vmx MemSched: Balloon enabled 1 guestType 1 maxSize 754753
2022-03-04T06:15:14.884Z In(05) vmx MemSched: caller 0 numvm 1 locked pages: num 437212 max 6293944
2022-03-04T06:15:14.884Z In(05) vmx MemSched: locked Page Limit: host 6302136 config 7309568
2022-03-04T06:15:14.884Z In(05) vmx MemSched: minmempct 50 timestamp 2063829
2022-03-04T06:15:14.884Z In(05) vmx MemSched: VM 0 min 437212 max 833500 shares 792576 paged 421154 nonpaged 36578 anonymous 4346 locked 437212 touchedPct 4 dirtiedPct 0 timestamp 2063829 vmResponsive is 1
2022-03-04T06:15:14.884Z In(05) vmx MemSched: locked 437212 target 833500 balloon 0 0 490589 swapped 371949 0 allocd 0 512 state 0 100
2022-03-04T06:15:14.884Z In(05) vmx MemSched: states: 0 685801 : 1 0 : 2 0 : 3 0
2022-03-04T06:15:14.884Z In(05) vmx MemSched: Balloon enabled 1 guestType 1 maxSize 754753
2022-03-04T07:23:40.743Z Wa(03) mouse Poll timeout: Something may be hung... ungrabbing
2022-03-04T07:23:40.749Z Wa(03) mouse Poll timeout: Something may be hung... ungrabbing

So somewhere down the line I would probably need to add a network ping to spot the exact time it hung.

Moderator edit by wila: moved the log lines into a spoiler.

[wila]

Hi,

---

@zyxxel wrote:

Just that I don't seem to be able to find any links to older versions.

---

Here's a direct link for Workstation 16.1.2 on Windows

https://customerconnect.vmware.com/en/downloads/details?downloadGroup=WKST-1612-WIN&productId=1038&r...

Please note that you probably have to change the hardware version of your VMs while still in Workstation 16.2 to a hardware version compatible with Workstation 16.1 as otherwise you won't be able to start the VMs.

--
Wil

[zyxxel]

Still at zero VM hangs with 16.1.2.

A tiny bit early to say the problem is gone but 2.5 weeks is longer than the average fail rate I saw with 16.2.x.

[zyxxel]

Still no crashes after I went back fro 16.2.2 to 16.1.2.

Since then, version 16.2.3 and 16.2.4 have been relesaed - but it isn't obvious from the release notes if the hang issue has been resolved so I can move forward to the most recent version again.

16.2.3 had this fix "Workstation hangs while copying files from the host to the guest operating system", but my virtual machines didn't need me to copy files to hang - they could hang even without interaction. So very unclear if this fix is related to the issues I saw with earlier 16.2.x releases.

[mboekhold]

I've been having the same issue where my Ubuntu MATE 22.04 guest VM constantly freezes. This is on a Windows 21H2 (OS Build 19044.2006) host, with VM Workstation 16.2.4 build-20089737. This happens regardless of the activity of the Guest VM, e.g. it even freezes when left idle after boot. This started a few weeks back after I had a few Windows updates forced onto me (corporate laptop...).

I let the guest sit idle except for a Clock running, to try to identify when the VM started to hang. And after about 20 minutes, I was not able to interact with the VM anymore. The VMWare log file contains around the time of the hang:

2022-10-10T06:49:57.489Z In(05) vmx Tools_SetGuestResolution: Sending rpcMsg = Resolution_Set 2479 1363
2022-10-10T06:51:05.220Z In(05) vmx VNET: MACVNetLinkStateEventHandler: event, up:0, adapter:1
2022-10-10T06:51:05.220Z In(05) vmx VNET: MACVNetLinkStateEventHandler: 'ethernet0' state from 5 to 7.
2022-10-10T06:51:05.220Z In(05) vmx VNET: MACVNetLinkStateEventHandler: event, up:1, adapter:0
2022-10-10T06:51:05.220Z In(05) vmx VNET: MACVNetLinkStateEventHandler: 'ethernet0' state from 7 to 2.
2022-10-10T06:51:10.230Z In(05) vmx VNET: MACVNetLinkStateTimerHandler: 'ethernet0' state from 2 to 6.
2022-10-10T06:51:10.331Z In(05) vmx VNET: MACVNetLinkStateTimerHandler: 'ethernet0' state from 6 to 1.
2022-10-10T06:51:15.364Z In(05) vmx VNET: MACVNetLinkStateTimerHandler: 'ethernet0' state from 1 to 5.
2022-10-10T07:07:25.411Z In(05) vmx GuestRpcSendTimedOut: message to toolbox timed out.
2022-10-10T07:07:25.411Z In(05) vmx Tools: [AppStatus] Last heartbeat value 1095 (last received 14s ago)
2022-10-10T07:07:25.411Z In(05) vmx TOOLS: appName=toolbox, oldStatus=1, status=2, guestInitiated=0.
2022-10-10T07:07:29.775Z In(05) vmx GuestRpcSendTimedOut: message to toolbox-dnd timed out.
2022-10-10T07:07:29.775Z In(05) vmx TOOLS: appName=toolbox-dnd, oldStatus=1, status=2, guestInitiated=0.
2022-10-10T07:07:45.413Z In(05) vmx GuestRpcSendTimedOut: message to toolbox timed out.
2022-10-10T07:07:45.413Z In(05) vmx GuestRpc: app toolbox's second ping timeout; assuming app is down
2022-10-10T07:07:45.413Z In(05) vmx Tools: [AppStatus] Last heartbeat value 1098 (last received 13s ago)
2022-10-10T07:07:45.413Z In(05) vmx TOOLS: appName=toolbox, oldStatus=2, status=0, guestInitiated=0.
2022-10-10T07:07:45.420Z In(05) vmx GuestRpc: Reinitializing Channel 2(toolbox)

Something more weird is that there does seem to be some activity inside the guest VM. I have seen it update the time in the Clock application that I have open at least once. But I can't interact with the VM at all. Unfortunately, I only have a NAT network interface configured on the VM, so I can't check if it responds to network traffic. I doubt it though...

FYI the openvm-tools are running inside the VM... 

[mboekhold]

Following up on myself, here are 2 screenshots taken from the Clock window inside my frozen VM, taken 10 minutes apart:

As you can see, the clock has jumped. But it is not regularly updating, it only happens in big jumps. And I still cannot interact with the VM.

When the VM hangs, I can see vmware-vmx.exe suddenly taking 25% CPU, on an 8-core/16-thread system:

[wila]

Hi,

There's a similar thread where the following has been assigned as the solution:

With the VM shutdown - Under Edit, Preference, Memory - the setting for Additional Memory was set to "Allow some virtual machine memory to be swapped" I changed this to "Fit all virtual machine memory into reserved host RAM"

--
Wil

[mboekhold]

Hi @wila, thanks for the suggestion. I tried this, and unfortunately it does not help. My VM froze again after 15 minutes...

I checked my Windows host update history, and it is possible that this issue started with the installation of the 21H2 update (coming from 21H1).

More guest details, in case this can help:

boekhold@boekhold /etc $ uname -a
Linux boekhold 5.15.0-48-generic #54-Ubuntu SMP Fri Aug 26 13:26:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
boekhold@boekhold /etc $ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.1 LTS"
boekhold@boekhold /etc $ cat /etc/os-release 
PRETTY_NAME="Ubuntu 22.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.1 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

[mboekhold]

As many people recommended in this thread, I downgraded to 16.1.2... I'm at over 5 hours of stable uptime now, no hangs, no weird behavior. Seems there is some issue in 16.2.x....

[mboekhold]

Unfortunately, I celebrated too early. This morning I resumed my laptop from sleep, started VMWare Workstation again and launched my VM. It hang again after a short while.

I'll just monitor upcoming updates to Workstation and retest periodically. As it is, VMWare Workstation is useless to me.

[mboekhold]

Updated to 16.2.5 this morning, and my main Ubuntu MATE 22.04 VM has been running stable for 3 hours now. Since this issue started, I've never been able to keep it running for more than 20 minutes, so it seems the issue is solved. Strangely, the Release Notes do not mention this issue at all...