VMWare Workstation Consumming Swap

[Dryne]

I have noticed this with the latest v15 and v16 Workstation versions.  Within my guests Windows VMs, my host's Linux swap space gets consumed even though I have plenty of memory.  This happens interestingly enough when I am performing lots of disk activity within these guests.  For example, when I was downloading many gigabytes from the Internet, I noticed my swap space going up on my host.  I am currently running Ultradefrag against my Windows 10 guest, and this is what my host is currently showing:

$ free -h
              total        used        free      shared  buff/cache   available
Mem:          125Gi       5.2Gi       651Mi        24Gi       120Gi        95Gi
Swap:         488Gi       517Mi       487Gi

What is it with VMWare that results in so much host swap usage?  As I mentioned, this happens with high disk activity.

[Dryne]

Swap is steadily increasing while this defragmentation is running.  Since I posted the above, these are results now:

$ free -h
              total        used        free      shared  buff/cache   available
Mem:          125Gi       5.0Gi       621Mi        24Gi       120Gi        95Gi
Swap:         488Gi       522Mi       487Gi

[scott28tt]

Seen these settings?

https://docs.vmware.com/en/VMware-Workstation-Pro/16.0/com.vmware.ws.using.doc/GUID-2D539D2A-B76A-43...

[Dryne]

I set that to Fit all virtual machine memory into reserved host RAM after installation and leave it set that way.

The difference I am seeing could simply be the way Linux works.  Linux is taking all available memory for buffer/cache and leaving none unused.  I do not ever see this memory being released to other applications to use.  Even though I do not have the options for swap set, it seems some swapping is happening via VMWare Workstation anyway.  It is a small amount of memory, but it continues to grow while disk activity is happening.  This is the part that was not clear to me.  Why would disk activity increase swap usage?

All of my defragmentation is finally done.  This is how swap looks now:

$ free -h
              total        used        free      shared  buff/cache   available
Mem:          125Gi       4.3Gi        57Gi       226Mi        64Gi       120Gi
Swap:         488Gi       683Mi       487Gi

This 683MiB of swap currently used will remain in use until I either reboot or drop swap.  If I was not either downloading in my guest or running a defrag, swap would remain at zero bytes used.  I am not understanding why disk access would use swap.  This part does not make sense to me.

[Dryne]

After this post above, I shutdown my machine followed by powering it back up.  My machine has been powered on ever since with two guests running simultaneously all day, and various other applications also running.  Today I spent minimal time with disk activity in my guests:

$ uptime
 16:35:54 up 1 day, 13:59,  1 user,  load average: 5.16, 4.63, 4.49
$ free -h
              total        used        free      shared  buff/cache   available
Mem:          125Gi       5.3Gi        45Gi        48Gi        74Gi        71Gi
Swap:         488Gi          0B       488Gi

I am still puzzled by guests' disk activity stealing host swap.

[Dryne]

I am again performing lots of downloads from the Internet in my Windows 7 guest.  I am downloading many gigabytes of data.  While the download is taking place, my free memory under Ubuntu LTS 20.04.1 is constantly going down.  No swap is yet being used, but it will eventually.  This looks like a classic memory leak.

[wila]

humm .. sounds like modern OS file caching in action to me.
Aka.. working as intended.
Your host has a lot of RAM, unused RAM is wasted RAM and thus your host OS keeps files in memory in case you ask for it again. It is much faster to query from RAM as from disk.
Even a host with a lot of RAM can use swap. I know that sounds a bit strange, but the kernel normally does know best what to write out to swap. If you think your host does not need swap... then don't setup swap.

I'm not saying that a memory leak is impossible, but from what I am reading here I would not be able to draw the conclusion that there is a memory leak.

If VMware was leaking memory then the most likely suspect would be vmware-vmx process which then would constantly be growing in size. (in command line check: ps -ax )

edit: re. swap usage: check your swappiness config:
https://help.ubuntu.com/community/SwapFaq/#What_is_swappiness_and_how_do_I_change_it.3F

I know it talks about a relatively old version of ubuntu, but I don't think this has changed.

--
Wil

[Dryne]

It's only VMware where I experience this, and it happens with excessive disk access.  It looks like an oversight in code related to disk to me.  The memory never gets returned, and I must reboot to get it back.  If I turned off swap, my memory would still be consumed as is happening right now and continuing to do so. I went from over 50GiB free memory to now 9.2GiB only from downloading.  Swap will start getting used very soon.  If I turned off swap, my machine would crash next.

[wila]

How much is in cache?

Your last free report looked like this:

$ free -h
              total        used        free      shared  buff/cache   available
Mem:          125Gi       4.3Gi        57Gi       226Mi        64Gi       120Gi
Swap:         488Gi       683Mi       487Gi

That's less than 1G in swap? With 487G in swap unused/free.

Here you had 64G in cache with 120G available for use...
Sorry, but I do not see anything wrong with that picture.

--
Wil

[Dryne]

The longer it runs, the worse it gets.  I am writing from experience.  When this continues, it slows my machine to a crawl.  If I let this run for a longer time, it will look much worse.  I am stopping it when I see this behavior.  I could certain drop swap and show the result of that.  I have already experienced it and seen first hand what happens.

[wila]

It's not that I don't believe you and it is possible that there's a memory leak in your setup somewhere.

But I need to see numbers for being able to say something sensible that supports your claim and so far I have not seen that.
When you say "I have only 9G free" ... I do not know which column in the free you talk about, there's several options.

How much memory is vmware-vmx using?
What's the output of ps -ax?

If you look in top what process is gobbling up your memory?

Running out of RAM without swap is indeed never a good idea. How about changing the swappiness as I suggested earlier.

--
Wil

[Dryne]

I just lowered vm.swappiness = 10 and I am seeing a difference now.  Let me try this out for a bit to see how this changes what I am seeing.

It looks like the default value for this is bad for me and part of the issue.  It's weird that it defaults to 60, but that site states they recommend a value of 10.  Why doesn't it just default to 10?

[Dryne]

Lowering swappiness to 10 helped, but VMware continues to use memory during guest disk access that eventually leads to swap usage.

I still think this is a memory leak specific to VMware.  As far as top is concerned, vmware is using the most memory than anything else being run.  This actually is not surprising at all, since two guests run simultaneously use 24GB each.

Additionally, I asked about this on askubuntu to get more opinions on what I am seeing:

https://askubuntu.com/questions/1292427/ubuntu-swappiness-default-confusion