Windows XP guest stuck in 640x480 4-bit

[Rattle189]

Hello,

I'm using VMWare Workstation Pro 16.2.1 build-18811642 on Debian 11 (Bullseye) 64-bit and I have a Windows XP SP3 guest for developing programs in Visual Basic 6. Installing VMware Tools' SVGA driver causes the virtual machine to reboot in 640x480 4-bit. While stuck in this mode, I'm unable to change the mode to 800x600 4-bit and trying to do so does nothing.

Before installing VMWare Tools.After installing VMWare Tools.An 800x600 4-bit mode is reported, but is unusable.Display Properties report 800x600 4-bit but is still in 640x480 4-bit.

I have tried these:

• Tried the solution in this thread but to no avail.
• Uninstalled VMWare Tools and re-installed it.
• Uninstalled the SVGA driver and repaired & re-install VMWare Tools.
• Uninstalled and re-installed VMWare Workstation Pro 16.
• Upgraded NVIDIA drivers from 460.91.03 to 495.46.

So far, the VM works just fine for development without the SVGA driver but I would like to have 3D acceleration for the machine. Any help is appreciated.

Thank you.

[Rattle189]

I install VMware Tools to mostly have 3D acceleration for the virtual machine, but also for the additional features like drag-and-drop, better mouse movement (without it, it's terrible), and allow my mouse to seamlessly enter and exit the virtual machine. I can work without the SVGA driver just fine but I would also like to have 3D acceleration. The VM is also mostly used for legacy development and testing.

The unknown device in question, I believe, is the USB controller if I'm not wrong. I don't think it's relevant to the issue, though.

Noted - I will now skip installing SP2 after SP1 as you cannot install SP3 without SP1.

I'm afraid if the cause is actually Linux itself then I will just have to wait for a future update that will hopefully fix this issue. Until then, I still have to experiment whether the cause is actually the GTX 1070 as I have yet to try running the VM while using the Vega 11 integrated graphics chip instead.

[bluefirestorm]

Over the weekend I did an upgrade of my systems from Ubuntu 18.04/Workstation 15.5.7 to Ubuntu 20.04/Workstation 16.x. I did some tests with an existing XP VM that I have.

The one scenario that worked for a Linux host was with an Intel UHD 630. There was no monitor attached to it but the integrated GPU was enabled in the motherboard BIOS. It looks like VMware Workstation just picked the first Vulkan device that it sees which was the Intel with the Nvidia GPU installed.

I don't know if Vulkan rendering works for AMD GPUs. If the AMD iGPU behaves in similar way as Intel iGPU, you may not even have to remove the GTX1070 for Vulkan rendering.

The strange thing is the black screen occurs for the XP VM (after the XP logo is suppose to switch to the logon screen; it just turns black). For a Windows 10 VM, Ubuntu 20.04 VM, the 3D acceleration works fine regardless if it is Vulkan/OpenGL on Nvidia GPU or Vulkan on Intel GPU. So it seems to be broken for XP only (I haven't tried other 16-bit/32-bit OS such as Windows 98 or Windows 2000 or some 32-bit Linux variant).

When the XP screen is black, vmware.log entries similar to this

2022-01-25T14:26:46.316Z In(05) vcpu-0 Guest: SVGA_MINI: IOCTL_VIDEO_QUERY_NUM_AVAIL_MODES: numValidModes=0
2022-01-25T14:26:46.316Z In(05) vcpu-0 Guest: vmx_fb: getAvailableModes: Unable to allocate memory for the modes
2022-01-25T14:26:46.316Z In(05) vcpu-0 Guest: vmx_fb: bInitPDEV no modes found
2022-01-25T14:26:46.317Z In(05) vcpu-0 Guest: vmx_fb: DrvEnablePDEV: bInitPDEV failed

For a successful 3D on XP VM, the numValidmodes are not zero and the bInitPDEV will show a screen resolution/parameters instead of "no modes found"

Table of XP VM 3D results

Host OS	Workstation Version	GPU	Driver Version	Renderer	XP 3D result
Windows 11	16.2.1	GTX960M	511.23	DX11	OK
Windows 11	16.2.1	GTX960M	511.23	OpenGL	OK
Ubuntu 20.04	16.2.1	Intel UHD630		Vulkan	OK
Ubuntu 18.04	15.5.7	RTX 2070 Super	470.86	OpenGL	black screen
Ubuntu 20.04	16.1.2	GTX 1070	470.86	OpenGL	black screen
Ubuntu 20.04	16.2.1	GTX 1070	470.86	OpenGL	black screen
Ubuntu 20.04	16.2.1	RTX 2070 Super	470.86	Vulkan	black screen
Ubuntu 20.04	16.2.1	RTX 2070 Super	470.86	OpenGL	black screen

Versions 15.5.7, 16.1.2 on Linux does not use the mkssandbox process and also does not support Vulkan renderer. Vulkan renderer in Workstation does not work for pre-Turing Nvidia.

[GarethF]

I can confirm the same issues (minimum screen resolution with VMware Tools, Windows XP Pro 64bit; black screen issues with host settings for monitors selected) with a RTX 3090 graphics card and the latest version of VMware Workstation Player (16.2.1 build-18811642); my system is running Ubuntu 20.04 LTS and nVidia driver 470.86.

VM is usable at a decent resolution with VMware Tools uninstalled, but without the accelerated graphics it is a bit laggy, and I do want the other functionality (shared folders and all that).

Other data points:

Same virtual machine works fine with VMware Tools on a Ubuntu 20.04 LTS machine with a GTX 960 card.

Same virtual machine works fine with VMware Tools installed on a Windows 11 machine with a RTX 3080Ti card.

Windows 10 Pro virtual machine works fine with VMware Tools on the original Linux machine I mentioned.

[Rattle189]

Unfortunately, despite the configuration of graphics card on the Linux machine, the same issue is still present whether I'm on my GTX 1070 card nor my Vega 11 integrated graphics. Running a Windows XP VM under Linux seems to be the cause of the issue, and another unrelated issue is VirtualBox has graphical glitches too under Linux with a Windows 10 guest where the visuals misbehave if transparency is turned on.

My assumption is that this is only an issue exclusive to VMware Tools and Linux, and it's maybe going to be fixed in a future update. Until then, Windows XP VMs under Linux will have no 3D acceleration.

[bluefirestorm]

The XP VM 3D acceleration appears to work when using the nouveau driver on the GTX 1070 with Ubunut 20.04. So if your host system does not have any other software that depends on the Nvidia proprietary drivers, switching to using nouveau could be an alternative.

The entry

mks.gl.allowUnsupportedDrivers = "TRUE"

needs to be present on every VM otherwise 3D acceleration will not available when nouveau driver is used.

This can be put into the /etc/vmware/config in lieu of putting in every VM vmx file.