VMWare Workstation Slows Down When Saving State

[nadrojisk]

I've got VMWare Workstation 16 Pro (16.0.0 build-16894299) and recently whenever I try to take a snapshot my computer will get extremely laggy. My mouse will jitter around if I try to move it and if I am listening to music it will get glitchy. Note this is outside of the VM on the host. Additionally, if I click the suspend button for the guest it will stop lagging and take the snapshot. It seems that while the snapshot process is in the background and not prioritized it slows down? If I open task manager there doesn't seem to be any bottle necking of my CPU / RAM.

[arangaran]

Hi, i have the same problem, with VMWare Workstation 16 with a host running Windows 10 Pro with a Ryzen 9 3900x and 64GB RAM with a x570 motherboard and VMs running in a 1TB NVMe SSD, i have had it so bad that sometimes taking snapshots crashes the app and also corrupted a few VMs, worse is that i keep seeing errors of VSS (shadow copies or something in Windows 10) appearing in the event viewer, it must be a bug but it would be interesting to know if it is hardware related, i kind of solved it removing hyper-V support and platform and reinstalling the app, it readded that support back and i haven't had any more crashes but it gets very sluggish at times i thought it was something else but discovered that the VMs were the culprit as about why I don't know.

It would be good if you shared info about your devices and VMs maybe there's something in common.

[nadrojisk]

Sure. My host is running Windows 10 Pro with a Intel i7 7700k and 32 GB of RAM with an ASUS Z270-A motherboard. VMs are running on a 2TB NVMe SSD. I did not try disabling Hyper-V but I can attempt that. Currently I haven't had the application crash or corrupt any VMs thankfully. Usually it only lasts between a few minutes to four minutes.

[Robbie70]

I am experiencing the exact issue!  

AMD 3950x x570 chipset, 32GB Ram etc.

No such issues with HyperV on windows 10. I switched to Workstation 16 to have better USB support within VM's, but now taking snap shots are so slow I'm starting to regret my decision.

Just wanted to add, taking a snapshot when the VM is off is fast ... it's while its on I experience these issues.

[Catalyph]

Same issue, taking a snapshot while the VM is running takes like 15 minutes.
this is with Inter 8700k 32GB ram and 2TB NVME Samsung, snapshot while vm is off, instant.

[hunger]

I am having the same problem. VMware Workstation completely freezes as soon as it starts Saving State for about a minute. The worst part is that it seems to kill all of my TCP sockets which is really disruptive. The symptoms started after Microsoft crammed the Windows 10 Pro 20H2 update down my throat. I did not have the problem under the previous version.

Dell Latitude 5591

Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz

32 GB RAM

What's really weird is that I don't have Hyper-V enabled so that can't be the  problem.

[WTig3ner]

I have encountered a problem that may be related to this thread.  I'm running Kubuntu 20.04, Workstation 16, and a Windows 10 VM.  A couple of times in the last couple of weeks, my Windows VM has frozen up, and in the VM frame I see "saving state" with a progress bar that takes on the order of half an hour to finish, unfreezing the VM.  I did not ask Workstation to save the state.  Does anyone know (a) what causes this behavior and (b) whether there is anything I can do to stop it?

Thanks in advance.

[bleys]

Just want to chime in and see if anyone has gotten any traction on this issue.

I'm having the exact same problem (15+ mins to snapshot a running VM).

Hardware:
Ryzen 5950X
Corsair MP600 2TB PCIe 4.0 NVMe SSD (rated around 5GB/s read and write)
64GB DDR4 3200

Host: Win10, fully updated
Software: VMware Workstation 16.1.2, fully updated
Guest: Linux 5.13.12 x86_64, 12GB Memory (less than 1GB resident at time of the last snapshot I ran just now)

Throughout the snapshot process the system is idle and Windows Resource Monitor shows the snapshot file as the top activity, at around 10MB/s pretty consistently. Host memory consumption is always below 50%, CPU stays below 5%. I haven't had any stability issues but there's obviously something pathological in the slowness experienced. Even the absolute worst case of constant tiny random reads on a single thread gets 50+ MB/s with this drive.

[bleys]

Ok, I've dug into this some. I have fixed the issue of snapshots of running machines taking ages to complete.

It comes down to a default configuration issue. Live snapshots use something called "Lazy CheckPointing" which has a baked-in delay between reads, and only read tiny amounts of memory at once. This feature looks like it was coded in LOOOONG before the age of DDR4 and NVMe drives, and no one has looked at it since.

I found the solution here: Snapshots Take a Long Time When “Keep Memory” is Enabled (76687) (vmware.com)

I added these lines to my .vmx file:

mainMem.ioBlockPages = "4096" 
mainMem.iowait = "0"

Snapshotting my running VM took 25 minutes with the default configuration. After making this change, and taking another snapshot with the exact same system state it took less than 0.7 second! (It was barely more than a blink. I had to check the log to see fast it was.)

It looks like the snapshotting defaults are overdue for some rethinking. 

Another option might be to disable background snapshotting: Enable Background Snapshots (vmware.com)

Edit: If you're not sure which ioBlockPages setting would be most performant for you, try running a disk benchmark software like ATTO on the disk where your snapshots are getting saved. Spoiler: most fast NVMe drives get peak write performance when your IO chunks are around 8MB or maybe 16MB, which is 2048 or 4096. Hint to VMware devs who might look at this: if it were me updating the software, I'd make the Lazy CheckPointing default write settings aware of different storage media (e.g. it would have different default settings depending on whether it was writing to a traditional HDD, a SATA SSD, or an NVMe SSD).

[bleys]

I should add that I initially thought it might be related to the performance degradations that come from using Hyper-V/"Host VBS Mode", so I followed the steps outlined on these two pages (the first one is not enough):

1. Disable Hyper-V to run virtualization software - Windows Client | Microsoft Docs
2. How do I disable Virtualization based Security in order to use the Ryzen Master software? - Microsof...

This didn't effect the slow snapshotting issue, but it did make the Linux guest performance much better in other ways. The most significant improvement is that guest audio was poppy/crackly/interrupted (to the point of being nearly unusable) prior and now it comes through perfectly clear.

Quick summary to disable Hyper-V/VBS

Disable the following Windows components (they all cause Hyper-V to run, preventing the VMware Hypervisor from running instead):
Hyper-V
Device Guard
Credential Guard
Virtual Machine Platform
Windows Hypervisor Platform
Microsoft Defender Application Guard

Edit: Another hint to VMware devs - There really ought to be an obvious indicator in VMware Workstation of whether it is running with the VMware Hypervisor ("Traditional Mode") or with Hyper-V ("Host VBS Mode"). Even combing the log for HyperV only gives an obtuse clue about whether it's running. I'd prefer if the second line of the log says something to the effect of "Host is running Hyper-V, switching to Host VBS Mode. Performance and feature-set are degraded in this mode. To use the traditional mode, see: <shortlink to currently non-existent VMware support article about disabling Hyper-V on your Windows Host>".

[maxoptimus]

how you found that value - looks like 4096 is not enough for my system or old vmplayer do not read it;

I have used adaptec raid card with old harddrives 4k sectors or 512;

it's strange vmware bug; first I think it stupid zip memory but not; rewrite with wrong settings looks like;

simple save must take few seconfs stupid vm developers and testers.....

[Catalyph]

This is the answer.

[maxoptimus]

maybe someone known how disable backround snapshoot option in wmplayer settings or in vmf settings; (don't want to lookup what page size is needed for adaptec and old hdds);

other option disable snapshoots at all but I really like that vmoption

[Catalyph]

I found this video of this issue on youtube:
https://youtu.be/FK56wQYi7Po

[munrobasher]

>There really ought to be an obvious indicator in VMware Workstation of whether it is running with the VMware Hypervisor ("Traditional Mode") or with Hyper-V ("Host VBS Mode").

I wondered why I got warnings I'd not seen before about Hyper-V when I rebuilt my main PC or that the above was even a thing. I had no idea that VMware Workstation was working with Hyper-V at all when Hyper-V feature wasn't installed. I'd skim read articles and news about Device/Credential Guard but it didn't sink in what it all actually meant.

There appears to be two issues here - firstly on my system, it sits "Saving state 0%" for a very long time. Not specifically the iowait setting. Just about to try it though.

[munrobasher]

>There appears to be two issues here - firstly on my system, it sits "Saving state 0%" for a very long time. Not specifically the iowait setting. Just about to try it though.

No, using the following doesn't help with saving state stuck at 0%

mainMem.ioBlockPages = "4096"
mainMem.iowait = "0"

Although when it finally clears 0%, it certainly speeds up the rest of the snapshot. Interestingly, it's not every VM that has this problem. My Windows 11 test VM doesn't get stuck at 0%. Just my Windows 10 test VM.

In terms of knowing whether you're running native VMware Workstation hypervisor or Host VBS, I wonder if the presence of this side channel checkbox shows this?