Converted physical PC->VM will not uncapture mouse

[keithruck]

Currently using VMWare workstation 8.05 on Win7 x64.

I've been using VMWare workstation since about version 6 with few problems. All of my machines have been created in workstation (OS install, etc.).

A few months ago, I had a separate XP x32 box that had a PSU failure, and I had always wanted to virtualize that box anyway, so I hooked up a spare PSU and used vCenter Converter standalone client to convert it. Unlike all my other VMs, which seemlessly capture and uncapture the mouse, this VM will only release the mouse using Ctrl-Alt, otherwise the mouse stops at the screen border.

I've messed with the VM settings (although nothing has really made a difference). I did set the floppy to automatically detect, I installed VMWare enhanced keyboard (or whatever it is called), and uninstalled the mouse drivers that were on the machine (didn't find the folders containing those drivers, just uninstalled them from the mouse properies window).

I'd really, really like to get this VM working like the others, as I use it often and my productivity would go up if I didn't have to deal with this mouse uncapture issue.

Thank you for any help!

Keith

[a_p_]

Did you install VMware Tools after conversion? This installs all required services and drivers.

André

[keithruck]

Yes, I've installed VMWare tools (starting with repair, then modify to make sure nothing was missing). The only components not installed are Paravirtual SCSI, and LSI. All the components are 'installed locally'.

What else can I check?

Thanks,

Keith

[a_p_]

All I can think of is that there are some left over drivers from the physical systems still in place. Please compare the drivers for the mouse and display with those of a working VM. The mouse driver should be a VMware pointing device.

André

[dariusd · Accepted solution]

From the Department of Utterly Random Ideas: Does your converted VM by any chance have mouse "pointer trails" enabled?  If so, try turning that option off.  We have had a few users report that enabling that option somehow causes the problem you are encountering.  Credit to rob_oli for describing this solution (and providing a screenshot too) in this post.

Cheers,

--

Darius

[keithruck]

Wow, that was exactly it- mouse trails turned off, and the problem was completely and immediately resolved!

Thankyouthankyouthankyou

and a <facepalm> at VMWare

Thanks,

Keith

[dariusd]

I'm glad that got things working for you!

Here's an update from the Department of Possibly More Information Than You Wanted...

I couldn't leave this mystery un-investigated... the WTF factor was waaaaay too high.  So, I collated a bit of information from some internal resources to describe what is actually happening here:

Workstation has a feature called "Gaming mouse", which is designed to try to figure out when you are running a game inside the VM and to "optimize" the mouse for that situation.  What does that mean?  There are many (many) games where, if we were to track only the absolute mouse movement, we'd end up auto-ungrabbing from the VM at inopportune times.  For instance, if you're using a pointing device to turn/move leftwards a lot in a first-person shooter, the effective position of the mouse pointer would soon depart the left edge of the VM's window, we'd ungrab while you're mid-game, and you'd say unpublishable things at your computer.  So the Gaming mouse feature tries to figure out if you're playing a game in the VM, at which time we disable auto-ungrab.  You can control this behavior in Workstation, Edit > Preferences, Input, Optimize mouse for games.

Unfortunately, there is no single magic thing that a VM does that tells Workstation that you're playing a game.  We have to try to guess, based upon a number of factors of the VM's behavior.  One of those factors is the visibility of the mouse pointer: Usually, if the hardware-supported mouse cursor is explicitly hidden by the guest OS, it means that the absolute position of the pointer is not relevant to the user, which is a h-u-g-e clue that we should enter gaming mode and disable auto-ungrab.

When you enable mouse pointer trails in Windows XP, it does a funny thing: Unless the video hardware has some bizarre support for pointer trails (which many don't, including Workstation's virtual SVGA), Windows turns off the SVGA hardware mouse pointer and starts drawing the pointer entirely in software.  As far as Workstation is concerned, the guest mouse pointer is now hidden (we can't tell that those black-and-white pixels drawn by the guest OS are actually the mouse pointer), and bam, we go into gaming mode, disabling auto-ungrab.

Two workarounds are available: Disabling pointer trails in the guest or disabling "gaming mode" in Workstation's preferences.

Thanks for prompting me to investigate this... I learned a lot and can finally understand a bizarre facepalm-worthy quirk.  Hope this information is useful!

Cheers,

--

Darius

[keithruck]

In hindsight, that makes great sense.

To the user (who didn't admit how many months he's been cursing that one VM), there was no intuitive way to figure this out, and nothing obvious in any of my web searches. Hopefully your post will help others find the answer in less time than I did 😉

Best,

Keith