VMware Communities
Luminatio
Contributor
Contributor
Jump to solution

VM became extremely slow after upgraded to macOS 10.14.6 [Officially Solved]

------------20190827--------------------------

This issue is officially solved with Apple’s macOS 10.14.6 supplemental update.

Thanks to all of you.

A thumb down to Apple’s software team...

————————

After I upgraded my 2 MBP to 10.14.6, all of the Windows VMs became extremely slow and stuck at the login screen...

The VMs are set to use TPM, I don't know if it's relevant.

Anyone came across the same situation?

Now I cannot continue my work...really frustrating...

—————20190727————-

I’m updating this for convenient reference.

Status: This issue has not been solved yet.

Many users encountered this issue which leads to completely unusable of VMs.

And we got some replies from Fusion developers, you could help developers identify issues by replying your hardware models/is disk encrypted or have the VM encrypted etc.

We have some reports that some users found out that if set the memory of VM to 2G could help.

Re: VM became extremely slow after upgraded to macOS 10.14.6

If you don’t need to use much memory and can work with it, you might try set the memory to 2G, or maybe less than 4G is also okay?

Still we are waiting for an official solution.

------------20190809--------------------------

Thanks to ksc

Re: VM became extremely slow after upgraded to macOS 10.14.6

I updated to Fusion 11.1.1, confirmed that I can login to the system, and desktop shows normally.

As ksc says, it is a workaround which I'm not quite sure if I can trust it to do my work in it....

At least, I'm able to get my works out and change to other options now.

I'll mark this issue as resolved for now, but keep the [Workaround Released] tag in the subject.

Looks like it would take quite some time for a real fix, I'll keep an eye on this.

消息编辑者为:Liam

227 Replies
ColoradoMarmot
Champion
Champion
Jump to solution

In memory of Jerry Pournelle's annual awards, Orchids to the Fusion devs for troubleshooting this, but an onion to Apple to drop something pretty fundamental into a minor OS upgrade.

MacSheep
Contributor
Contributor
Jump to solution

Count me in.

2017 MBP 14,3

Mojave 10.4.6

Fusion 11.1.0 Pro

Win 10 VM with 4096MB and 2 cores

No encryption on either the host or the VM

Starts out fine but then performance degrades over time -- sometimes quickly, sometimes not. Any activities in the VM become very laggy and when it's at it's worst, there's about a 10-15 second delay between a mouse click in the VM and the action being executed. Will try dropping the memory allocation to 2GB.

Thanks

Reply
0 Kudos
ndachev
VMware Employee
VMware Employee
Jump to solution

Before time i have a issues with freezing vm's with fusion and macbook pro 2018,

After I remove Accelerated 3D Graphics  I did not notice any issues in the last months.

For Windows vm's please  don't use avast antivirus .. at the moment i-net is full with complains related to avast and vmware workstation .. I'm not sure if Fusion have similar issues with it.

Model Name:    MacBook Pro

Model Identifier:    MacBookPro15,1

Processor Name:    Intel Core i7

Processor Speed:    2,2 GHz

Number of Processors:    1

Total Number of Cores:    6

L2 Cache (per Core):    256 KB

L3 Cache:    9 MB

Hyper-Threading Technology:    Enabled

Memory:    32 GB

Current Fusion:

Professional Version 11.1.0 (13668589)

Reply
0 Kudos
0f2g
Contributor
Contributor
Jump to solution

Same issue here:

MBP 2016; using solitary encrypted VM.  Previous daily settings was 3 proc & 6GB RAM; now it will only function with 1 proc, 2GB.

Reply
0 Kudos
pauljacobevans
Contributor
Contributor
Jump to solution

Not to alarm everyone... but Apple did just release another supplemental update this morning.

Does it fix it? Does it make it worse?  Who knows?! 

I’m Installing now...

Reply
0 Kudos
dwimmreuter
Contributor
Contributor
Jump to solution

any luck with the apple update? is it improving anything?

tonyfortpedroco
Contributor
Contributor
Jump to solution

I was having the symptoms (periodic freezing) after installing 10.14.6. The only thing that eliminated the symptoms for me was reducing the Win10 guest to 2GB of memory.

Installed the OSX 10.14.6 Supplemental Update just now and restarted the Win10 guest after bumping its config back up to 4GB memory, and symptoms returned.

Reduced the Win10 guest back to 2GB and restarted and symptoms are gone again.

I don't think the supplemental update (all 950MB of it, which is clearly doing something more than correcting an issue where certain models have trouble sleeping) did anything to alleviate this issue we're having with VMWare Fusion.

Reply
0 Kudos
pauljacobevans
Contributor
Contributor
Jump to solution

After a while and several changes to the vm config... nothing really changed with it. Smiley Sad  sigh.

Getting super old.

Reply
0 Kudos
ksc
VMware Employee
VMware Employee
Jump to solution

Official update... root cause identified and reduced to a small sample program. Still investigating workarounds; until we ship an update, smaller VM sizes or decrypting is going to be the best option. No ETA yet.

(I just reported this root cause to Apple about 15 minutes ago, so I'm almost certain it's not related to their supplemental update.)

The root cause, if curious, is the behavior of paging memory from an unlinked file. Posix reference counts file handles independently from the on-disk entry, so it's  possible to create and open a file, make it a certain size, unlink it, then mmap the file handle to a chunk of memory. The operating system will still page to and from the file; when the file handle is closed, the operating system will release the disk blocks too. It's an old-school UNIX trick to get shared memory that automatically gets cleaned up after the last process using it exits. On 10.14.6, accesses to the mmapped parts of the file get slower ... and ... slower ... over time. If the exact same file is not unlinked, accesses slow down some but are still very usable.

Fusion uses this unlinked-file technique for encrypted VMs. (Normal VMs are memory-mapped to the .vmem file, but that obviously doesn't work when the file is encrypted).

Long-time VMware users who have used Workstation may remember the "mainMem.useNamedFile" option. On Linux and Windows, we had the opposite problem: a memory-mapped file was slow, but switching to an unlinked file (sometimes) made things much faster, especially when Linux re-tuned the memory manager. This is the first time we've seen MacOS re-tune the memory manager, and having the first time be in a dot release was quite a surprise. Alas, that option won't do any good here: encrypted VMs cannot use a named file so ignore it.

Wegener22
Contributor
Contributor
Jump to solution

Guys, hurry up, please! Other "Virtualisators" work flawlessly under 10.14.6 or Catalina.

A machine equipped with only 2 GB RAM isn't the perfect solution under some

circumstances.

Sincerly,

Steve

Reply
0 Kudos
Sven1802
Enthusiast
Enthusiast
Jump to solution

Hi,

thanks for the detailed answer. Just to be sure I've tested the supplemental update. It did not change anything to the slow down issue with VMware.

Reply
0 Kudos
DaveP
Commander
Commander
Jump to solution

How does this affect unencrypted guests? I have problems wirh macOS and Windows guests since the 10.14.6 upgrade. I can sort of work for awhile by dropping memory and cores. For example 16GB 6 cores MacBook Pro 15" 2018 dropping 8GB to 6GB for VMs works with a drop of 2 to 1 cores.

Reply
0 Kudos
athurdent
Contributor
Contributor
Jump to solution

+1

Same here, also posted my config a few pages back.

FileVault is on, VM is not encrypted.

Still having problems, also sluggish screen when changing fullscreen desktops between VM and OSX.

Reply
0 Kudos
lfreitag
Contributor
Contributor
Jump to solution

I have exactly the same issue on a mid-2017 13 inch Macbook Pro with 16G of memory and not encrypted VM that runs the bootcamp partition.  VM is unusable now and I am just running Windows by launching Bootcamp. I have tried to reduce the 8GB allocated memory, which used to work well, to 2Gb.  It makes my machine unusable because the software I run requires the additional memory.  Please issue a fix asap.

Reply
0 Kudos
ksc
VMware Employee
VMware Employee
Jump to solution

Another official update...

Apple has confirmed reproduction and identified the bug on their end. No timeline for a fix shared yet.

In light of that, we're also working on a workaround. "Days but not weeks", can't be more specific because the workaround is not yet written.

A few more things we know now:

  • Affects hosts with >=4GB and <32GB of memory. (Yes, really).
  • Triggers after 2GB of modified data is cached, regardless of file. Once triggered, it stays triggered as long as the file is open.
  • So why does this affect encrypted VMs more than unencrypted?
    • Due to different code paths, this cache bug only affects emulated device I/O. The virtual CPU is not affected.
    • Encrypted VMs are much more sensitive to additional I/O costs, and "touch" memory many extra times (due to crypto internals).
    • Some benchmarking I did suggests we get ~4MBps touching memory backed by an affected file in normal usage, but crypto's high-touch access pattern drops this to ~0.2MBps or worse. In comparison, an SSD on random-read workloads is 75-100MBps, a HDD is 1-2MBps. So an unencrypted VM "feels" like HDD speed, and an encrypted VM "feels" 10x worse than an HDD.
  • In theory, this should affect everybody who is writing >2GB to a file. In practice, most applications writing that much data to a file do so sequentially, which makes it easy to write out modified data fast enough that the bug never triggers. Virtual machines ... have to emulate RAM, which has a very different access pattern.

Really gory details, for the technically inclined ... we know where the actual bug is. This is a link to an (older, 10.14.1) Apple source code drop.

https://github.com/apple/darwin-xnu/blob/a449c6a3b8014d9406c2ddbdc81795da24aa7443/bsd/vfs/vfs_cluste...

This is a function that keeps a hash table of blocks used from a file; the hash table automatically grows once it reaches a certain size. Apple appears to have "fixed" the XXX comment above the function and added a 32GB case (in addition to the 4GB case already present) ... but their new case has a bug that re-allocates the entire hash table every time.

Once they release sources, I expect the new fix will be a 1-liner to add a return statement in their new code, just like line 6643. Oops.

pauljacobevans
Contributor
Contributor
Jump to solution

Hey ksc,  really appreciate your updates and full explanations. And I know I can’t be the only one on here that really appreciates them. Outstanding fanatical support and communication Of what’s going on. Thank you.

Reply
0 Kudos
zendnez
Contributor
Contributor
Jump to solution

Thanks for the quick effort and for your transparency around the issue.

I'm impacted by the issue because I use an encrypted VM. I use an encrypted VM because encryption is required when using an emulated TPM. I use an emulated TPM because I have a specific work-related scenario which requires one.

I understand the tradeoffs around a virtualized TPM. What I don't understand is why I'm required to encrypt my virtual disk in order to use a virtual TPM. I understand the need for TPM contents to be encrypted, and it makes sense that you force this. But why do you force encryption of the disk? Would you consider making disk encryption optional when a TPM is in use?

I'll add that, in my case, my virtual disk sits on a FileVault encrypted volume and the machine itself has Apple's access controls enabled. If someone gains physical control of my machine and manages to access the host file system (in order to get at the virtual disk), the situation is already bad and gaining access to the virtual disk won't make it measurably worse. So in my case, I don't consider the second layer of virtual disk encryption to be helpful and I would strongly prefer the performance and battery-life benefits of not encrypting the volume.

I doubt that this is the "workaround" you're looking for in the near term, but the ask here is to remove the requirement for volume encryption when using a TPM. Perhaps put up a warning dialog that forces me to click "OK" to a prompt which reads "I understand that I have chosen to add a virtual TPM without encrypting my drive and I understand and accept responsibility for the consequences of this decision"

ColoradoMarmot
Champion
Champion
Jump to solution

ksc

First, thanks for the transparent update - that's very much appreciated, and kudos to you and the team for tracking it down.

One question, if we have a 32GB machine, and a VM that only uses 4 or 8GB, why does it open a swap file at all? Or am I misunderstanding?

Aside:   I have a sinking feeling that we'll only see a fix in Catalina....

Reply
0 Kudos
lfreitag
Contributor
Contributor
Jump to solution

I hope your "sinking feeling" about no fix until Catalina is wrong!  I need this fix asap, as we all do.

Reply
0 Kudos
Luminatio
Contributor
Contributor
Jump to solution

Thanks for official updates.

No doubt that Apple’s software team is going worse day by day,

but still, VMware didn’t realize this issue during the developer beta is also unbelievable.....

It not only happens in 10.4.6, but also in 10.5 betas!

makes me wonder is fusion still developing or not?

On the other hand, Parallels and Virtual Box don’t have such fatal issue....

I downloaded PD, and trying it for almost 2 week but here the problem still not solved....

Look like I have to buy a license otherwise I cannot work....

Reply
0 Kudos