Windows XP guest stuck in 640x480 4-bit

[Rattle189]

Hello,

I'm using VMWare Workstation Pro 16.2.1 build-18811642 on Debian 11 (Bullseye) 64-bit and I have a Windows XP SP3 guest for developing programs in Visual Basic 6. Installing VMware Tools' SVGA driver causes the virtual machine to reboot in 640x480 4-bit. While stuck in this mode, I'm unable to change the mode to 800x600 4-bit and trying to do so does nothing.

Before installing VMWare Tools.After installing VMWare Tools.An 800x600 4-bit mode is reported, but is unusable.Display Properties report 800x600 4-bit but is still in 640x480 4-bit.

I have tried these:

• Tried the solution in this thread but to no avail.
• Uninstalled VMWare Tools and re-installed it.
• Uninstalled the SVGA driver and repaired & re-install VMWare Tools.
• Uninstalled and re-installed VMWare Workstation Pro 16.
• Upgraded NVIDIA drivers from 460.91.03 to 495.46.

So far, the VM works just fine for development without the SVGA driver but I would like to have 3D acceleration for the machine. Any help is appreciated.

Thank you.

[wila]

Hi,

Most likely the latest / current version of VMware Tools that is bundled with VMware Workstation doesn't work for Windows XP anymore. There's a variety of reasons why (signed certificate, c++ runtime files)

I suggest to try an older version like the one referenced here:
https://communities.vmware.com/t5/Technical-Community-Resources/latest-version-of-vmware-tools-for-w...

eg: VMware Tools 10.0.12

--
Wil

[Rattle189]

Hello,

Unfortunately, this wasn't able to solve the problem. The VM still remained stuck in 640 x 480 4-bit despite using an older version of VMware Tools. This was tested on an existing virtual machine and a new virtual machine with and without service packs installed. It's a bit strange that I had a previous Windows XP VM a few months ago work out just fine, but that was on Windows not Linux.

I assume that it's the Linux distro responsible for the problem though I yet have to test this theory out on a different distro.

[wila]

Hi,

Did you uninstall vmware tools -> reboot guest OS -> install vmware tools -> reboot guest OS?

Or did you skip the first reboot?
--
Wil

[Rattle189]

Every time VMware Tools is installed or uninstalled, it's followed by a reboot before something else happens. No reboots were skipped.

[wila]

Hi,

I was afraid you would say that.
In that case the log files (vmware.log & mkssandbox.log) might have some hints on why you're not able to get the virtual GPU to work.

Lately there also have been some issues reported with the nvidia GPU drivers (see also https://communities.vmware.com/t5/VMware-Workstation-Pro/Noticeable-typing-lag-in-Linux-VM-terminals... ) that apparently are resolved in version 496.76

Different problems, though they might be related.

--
Wil

[CarltonR]

Just a thought, what are the VM settings in Virtual Machine Settings - Hardware - Display . . . in addtion what is the version of DirectX ?

Post note: As an aside, just checked one of my Win XP Pro SP3 VMs, which had Tools version 10.0.10, and its been working since the Win XP was setup, so just updated it, as an exercise, to the 'latest' Tools 10.0.12, all still working . . . Win XP VM display setting is 1280 x 1024, with original native DirectX 9.0c.  Graphics card on the host is nvidia.

[Rattle189]

I apologize for the late reply, have been caught up with things. I re-installed Debian 11 and then installed the NVIDIA drivers version 495.29.05 and the problem still persists even on a newly created Windows XP virtual machine.

I started the VM quickly and then turned it off to generate log files and I can't really figure out what seems to be the problem that's causing the display to break.

[Rattle189]

I have Accelerate 3D graphics ticked, have 1 monitor with the maximum resolution of 1366 x 768 (my native resolution) with 64MB of graphics memory. Increasing or decreasing the graphics memory (along with the RAM) doesn't do anything.

The host doesn't support DirectX because it's Linux, but the VM runs on DirectX 9.0c

[wila]

Hi,

Like you, I don't really see in the log why this is happening.
Perhaps a VMware employee does (some are frequenting the forums so you might be lucky)

One possible workaround might be to change the virtual machine hardware of your VM to an earlier version from before the ISBRenderer backend was introduced. At least that is one thing I would try if I would bump into your issue.

--
Wil

[CarltonR]

I have a few rather dull and basic nuts and bolts question, may come to nothing as wila may well be correct:

i.   What version of VMware Tools are you currently using ?
ii.  What is the version of Win XP, and does it have the latest service packs added (Home, Pro, SP3, etc.,) ?
iii.  In the Win XP within the Device Manager - Display adapter what is shown for the 'device' and 'driver' being used ?
iv.  Within the Win XP Display Properties, what is shown ?
v.   Have you tried using ‘Use host setting for monitor’ within 'Settings' - 'Display setting' for the Win XP VM ?
vi.  Does this resolution issue occur in other versions of Microsoft Operating Systems VMs, say Win 10, or just Win XP ?
vii. Does this resolution issue occur for all of your Win XP VMs or just the one under discussion ?

It is interesting from your original 'before' and 'after' screen captures that the correct resolution was available prior to installing the VMware Tool, which although I am not familiar with the intricacies of the VMware Tools software, would imply that its Windows graphics diver related.

Note: I'm using Windows rather than Linux for the Host, so am unable to emulate what you have.

[bluefirestorm]

What I see from the vmware.log is that it tries to start with the VulkanRenderer first and failed and it seems from thereon that the 3D acceleration is no longer available even though it fell back to using GLRenderer on the GTX 1070.

What you could try to do is turn off the VkRenderer and use GLRenderer as the default.

Add these lines to the vmx configuration file.

mks.enableVKRenderer = "FALSE"
mks.enableGLRenderer = "TRUE"

I also see that the CPU is a Ryzen with Vega integrated graphics (AMD Ryzen 5 2400G with Radeon Vega Graphics). This is a desktop CPU so I assume that the monitor is connected to the GTX 1070. For Nvidia GPUs on laptops, the Nvidia GPU is a render only device and would only the integrated GPU (at least on Intel) is connected to the display.

Spoiler

2022-01-24T09:41:35.777Z In(05) mks MKS-RenderMain: PowerOn allowed MKSBasicOps GLRenderer GLBasic VKRenderer
2022-01-24T09:41:35.777Z In(05) mks MKS-RenderMain: ISB enabled by config
2022-01-24T09:41:35.777Z In(05) mks MKS-RenderMain: Collecting RenderOps caps from VKRenderer
2022-01-24T09:41:35.777Z In(05) mks MKS-RenderMain: Starting ISBRenderer (VKRenderer)
2022-01-24T09:41:35.777Z In(05) mks ISBRendererComm: ISBRendererComm DataChannel size=4294967296
2022-01-24T09:41:35.778Z In(05) mks ISBRendererComm: mksSandbox command-line: /usr/lib/vmware/bin/mksSandbox --pipeInfo 3
2022-01-24T09:41:35.780Z In(05) mks ISBRendererComm: Spawned process with pid 12550
2022-01-24T09:41:35.904Z In(05) mks ISBRendererComm: Sandbox Renderer: MKSBasicOps
2022-01-24T09:41:35.905Z In(05) mks ISBRendererComm: Unable to start a 3D renderer on mksSandbox
2022-01-24T09:41:35.909Z In(05) mks ISBRenderOps: ISBRenderer failed to start.
2022-01-24T09:41:35.909Z In(05) mks MKS-RenderMain: Failed to start the renderer ISBRenderer (VKRenderer)

[Rattle189]

I have tried downgrading the hardware compatibility as low as Workstation 12.x and also re-installed VMware Tools but unfortunately didn't change a thing.

[Rattle189]

1. I'm currently using VMware Tools 10.0.12, build-4448491
2. It's Windows XP Professional. I have tried installing VMware Tools on no service pack and SP1 to SP3.
3. VMware SVGA II is shown under Display adapters. The driver is provided by VMware, Inc., it dates at 11/17/2014 with the version 12.1.8.0 and is reported to be working properly.
4. The display is reported to be: (Default Monitor) on VMware SVGAII running on 640x480 with 4-bit as the color quality and cannot be changed.
5. Yes, it doesn't change anything.
6. Only Windows XP, Windows 10 works just fine.
7. All Windows XP virtual machines, doesn't matter which version it is nor whether it has a service pack or not.

Attachments to this message show i, iii and iv.

[Rattle189]

I have added those 2 lines above mks.enable3d = "TRUE"  and it caused an error. Upgrading the virtual machine's hardware compatibility back to the latest version caused the virtual machine to appear as a black screen after the Windows XP logo finished loading. Forcing it to shut down was the only to turn off the virtual machine as calling an ACPI shutdown causes the virtual machine to freeze and prevents you from closing it, making you resort to terminating the vmware-vmx process.

Removing the 2 lines from the .vmx file doesn't solve the issue, and re-adding then changing mks.enableVKRenderer = "FALSE" to mks.enableVKRenderer = "TRUE" doesn't solve the issue. The log with no suffix and with a -0 suffix are the logs for this one.

Restoring the virtual machine from a backup before the installation of VMware Tools and upgrade of the hardware compatibility solved the problem.

Adding the 2 lines above mks.enable3d = "TRUE" before installing VMware Tools let the virtual machine start normally. Installing VMware Tools and then restarting let the virtual machine boot normally, but the 640x480 4-bit issue is still present. Logs for this specific route are called -v12reinstall.

Upgrading the virtual machine to the latest hardware compatibility brought back the issue of it only showing a black screen. Logs for this specific route are called -v12upgrade.

Edit: Downgrading the hardware compatibility back to 12 from 16.2 fixes the black screen issue.

Additional attachments are when the virtual machine failed to start with an error, and what it looks like when it's stuck in a black screen.

[RDPetruska]

I have several Windows XP VMs here, but on Windows XP and Windows 10 hosts.  They run on multiple versions of VMware Player and Workstation.  Win 10 hosts -- Workstation 15.x, 16.x.  Win XP host - Player 7.x.

One VM has Workstation 10.x compatibility, VMware Tools 10.0.12.  The other has Workstation 5.x compatibility, VMware Tools 10.0.12.  All have working graphics stretch, etc.

So, it clearly does work.  

[Rattle189]

Maybe this is an issue exclusive to Linux only - I too also remember that I had a Windows XP VM running on Windows 10 Pro a while ago and it was working perfectly fine.

While I can uninstall VMware Tools and then re-install it, but without the SVGA driver, I still would love 3D acceleration in case I need to do anything in the virtual machine that requires 3D acceleration, such as Photoshop or something else.

[bluefirestorm]

Sorry about the mistake, I think it should be

mks.enableVulkanRenderer = "FALSE"
mks.enableGLRenderer = "TRUE"

as the log still show it still tries to start with VKRenderer with the mks.enableVKRenderer = "FALSE"

You could also try removing the GTX1070 and try with the Vega graphics. But for AMD GPUs on Linux hosts you have to add the line

mks.gl.allowUnsupportedDrivers = "TRUE"

to have 3D graphics available on guest VMs.

Try the Vega graphics also with the disabling the VulkanRenderer and have only GLRenderer.

The location of the lines is not important; just have to make sure that there are no duplicates or contradictory ones. Duplicates tend to just give a cryptic dictionary error message.

[Rattle189]

I replaced the 2 lines with the ones you've given, and even tried adding mks.gl.allowUnsupportedDrivers = "TRUE" and unfortunately it results into the virtual machine displaying a black screen only with the only way to turn it off is to either force it to shut down or terminate the vmware-vmx process. I have created a new virtual machine with the hardware compatibility set to the latest version, installed XP and then SP1 then SP3, and after installing VMware Tools the black screen problem returned.

On a positive note, the mksSandbox errors are gone in the logs. Attached to this message are logs of the newly created virtual machine. I soon will remove the GTX 1070 and try using the AMD integrated graphics instead to see if anything changes.

[CarltonR]

Hope you don't mind, but, I have a question, an observation and recommendation . . . 

Question: is there a specific reason why you need to add VMware Tools to your Win XP VM, seeing as though, from your earlier screen captures your getting the full resolution (1024 x 768) before installing the Tools.  So do you require something in particular, or is this just an exercise or expectation.

Observation: I noticed from some of your earlier screen captures that you had an 'unknown' device within Device Manager, is this relevant ?

Recommendation: Win XP Service Pack 3 is cumulative, and as such contains SP1 changes . . . so unless your going through staged diagnostics, then you only need to install SP3.

It may be the case that at the end of this investigation, that VMware Tools v11 may not play nicely when the Host is Linux based, but works on a Windows Host.  As you've outlined previously, you have had no issues with Win 10 VMs (which I suspect has VMware Tools v12) so maybe, it may simply be just a historical Linux VMware host mismatch.