BUG: workstation calls CreateCursor repeatedly with memory leak, when using relative or gaming mouse

[behealed]

I encountered this bug while running a virtual machine that was playing a modern 3D game, and while using relative cursor positioning.

This bug can get triggered in certain cases when using a relative positioning mouse. Does not matter if it is a USB or PS/2 mouse. Doesn't matter if the relative positioning is because of using "gaming mode" mouse, or by various options available to switch to relative mouse in the VMX config file.

While the cursor is grabbed, the MKS thread is repeatedly calling the same functions over and over, in the code where it "hides" the cursor on the host. The cursor is successfully hidden, however, the repeated allocations needed for CreateCursor, has a memory leak which gradually causes the host to run slower and slower. Shutting down vmware-vmx.exe, and restarting the guest, does not clear up the leaked resources. Only thing I can find to get the resources back (and get the frame rate back) is to fully reboot the host machine.

The exact location where these calls to CreateCursor are made, is in the VMWARE-VMX.EXE, in the same function and same general location where it checks the value of mks.noColorHostCursor. This is the function where it hides the cursor on the host. Without this function, the host's cursor would still be visible and stationary on the host, even while grabbed by the guest, which looks ugly.

NOTE: There are actually two different spots where vmware-vmx.exe makes calls to "CreateCursor". The spot that gets called 1k times per second, is NOT the spot that's located in the same function where it checks the value of mks.noColorHostCursor. But rather, it's the other call, located in the other spot. Trust me.

Rather than call CreateCursor repeatedly, it might be better to simply move the host's cursor off the edge of the screen. That would hide it from view, without the need to call CreateCursor to hide the cursor. This would solve the memory leak. But, whatever.

Also, you're welcome. This took me days to find. If vmware would like me to find more bugs like this, gimme the pdb file. 😛

I'm subscribed to this thread, so any questions, just reply.

[JustOneQuestion]

I am having the same issue with VMware Workstation 16 Player(16.2.0 build-18760230).  This recently started happening to me after updating the windows host to 21H1...  What is the fix?  How can we get this fixed asap?

[behealed]

I fixed this on my end, but I can’t share my fix because it involved making modifications to vmware-vmx.exe. I did look for any config settings that could fix this, but I couldn’t find any that worked. So until vmware fixes this, I don’t know of any official workaround. And this thread is dead, so I wouldn’t hold my breath for a fix. Only option in the mean time is to simply not use relative mouse mode, and only use absolute mouse mode.

[JustOneQuestion]

Hello, thank you for all your work.  I changed to absolute mouse mode and it still happens.  I also rolledback window updates and it still happens.  From what you said... this really smells like regression.  Im pretty sure this was not in the dx11 beta player.  What version of the vmplayer are you using?

[behealed]

Are you sure you are on absolute mouse? Easiest way to tell, on relative mode the guest and the host have different opinions about where the cursor is, but on absolute mode the guest always keeps the position in sync with the host’s cursor position.

if you are sure that you have it on absolute mode, yet you still see a resource leak, then it probably is a different leak than mine, unless you’ve done your own debugging and you know for sure that it’s making lots of calls to createcursor, on your end.

[behealed]

You might be able to track down the leak using window’s performance recorder / analyzer (google it), running those programs on the host side. But if you’ve never used those programs, be aware that it took me two days to hunt down my leak, and I’m pretty good at it.

[JustOneQuestion]

in my .vmx file I added the following line: 

mks.gamingMouse.policy = "absolute"

Im not sure what "absolute mode the guest always keeps the position in sync with the host’s cursor position." means.  My vm will capture the host mouse and I have to ctrl+alt to break out.

Im pretty sure this is the same bug because it will slowly happen and whenever I move the mouse FPS drops a lot.  Also, huge amount of memory leaking.

[behealed]

There are multiple places in the vmx where it could be set to relative mode. And there’s also an ini file that has settings which can also activate relative mouse mode. But I’ll explain again how to tell if it is in relative or absolute. When you switch to the vm window, if the mouse is in absolute mode, the cursor will not change its position. But if you are in relative mouse mode, when you switch to the vm window, the cursor will snap to a different position, which might be the center of the vm’s screen, or could be whatever was the previous position where the cursor was on the vm, when you were previously there. Point is, if the cursor changes position suddenly when you switch to the vm, then you are probably in relative mouse mode. But if the cursor doesn’t change position when you switch to the vm, then you are probably in absolute mouse mode. 

[JustOneQuestion]

okay seems like that setting didnt put in absolute mode. where else can I set this?

[behealed]

By default, it’s supposed to be in absolute mode. The fact that it is in relative mode means you changed something, I think. Disabling USB on the vm, will definitely put it in relative mode, because disabling usb will force it to use its PS/2 mouse on the vm, which only supports relative mouse mode. Activating gaming mouse mode will do it too, but that happens in the ini file. You can do a google search to find where the ini file is, but I think its under %appdata%\vmware

[behealed]

You can check the device manager on the vm (assuming its a windows vm), to see if the mouse is a ps/2 mouse, or if it is a HID / USB mouse. It needs to be the usb/hid mouse, for absolute mode to even be an option. But assuming its the usb mouse, then the only way it would NOT be in the default absolute mode, is if there is a line in the vmx or in the ini that contains the word “mouse”, which tells it to not be in absolute mode, and there are quite a few that do it, but pretty much all contain the word “mouse”.

[JustOneQuestion]

Adding USB and the config fixed it.

Thank you.