Round corners get lost in Windows 11 with VMWare 16.2.3

[Hedgehog2]

With VMWare 16.1.2 windows in Windows 11 have round corners, with VMWare 16.2.3 they get lost.

---------------------------------------------------
VMWare 16.2.3: square corners
Tools 11.3.5

DirectX 12
VMWare Virtual SVGA 3D Graphics
VRAM: 0 MB
Treiber Version: 8.17.3.5
Direct3D-DDI: 12
Funktionsebenen: 12_1, 12_0, 11_1, 11_0, 10_1, 10_0, 9_3, 9_2, 9_1
Treibermodell: WDDM 1.3 (?)

-----------------------------------------
VMWare 16.2.3: square corners
Tools 11.2.6

DirectX 12
VMWare Virtual SVGA 3D Graphics
VRAM: 0 MB
Treiber Version: 8.17.2.14
Direct3D-DDI: 12
Funktionsebenen: 12_1, 12_0, 11_1, 11_0, 10_1, 10_0, 9_3, 9_2, 9_1
Treibermodell: WDDM 1.3

-----------------------------------------
VMWare 16.1.2: round corners
Tools 11.2.6

DirectX 12
VMWare Virtual SVGA 3D Graphics
VRAM: 4 MB
Treiber Version: 8.17.2.14
Direct3D-DDI: 11
Funktionsebenen: 11_0, 10_1, 10_0, 9_3, 9_2, 9_1
Treibermodell: WDDM 1.1

-----------------------------------------
VMWare 16.1.2: square corners
no Tools

[bluefirestorm]

It looks like the difference is whether 3D acceleration is enabled or not for the VM.

For the square corners, notice that the VRAM is 0MB and DDI version is 12 and Feature Levels up to 12_1. The means 3D acceleration in the Display Settings is disabled (or somehow got disabled during VM power up).

For rounded corners, the VRAM is 4MB DDI Version 11 Feature levels up to 11_0 only. That means 3D acceleration is enabled for the VM.

[Hedgehog2]

I didn't change the settings for this virtual machine. 'Accelerate 3D graphics' is activated.

[bluefirestorm]

You may want to check when you power up the VM if any message flashes about 3D acceleration is not available. If it flashes too quickly, you want to check either vmware.log or the mksSandbox.log or both and see whether the 3D acceleration got set up correctly or not.

What is your host OS and GPU? If your using a laptop that switches between iGPU and dGPU, the difference in capabilities between the two might also result in such scenario.

But definitely the dxdiag (feature level, DDI version, etc) output in the Windows 11 VM guest suggests that 3D acceleration difference. I can confirm to you I tried disabling 3D on a Windows 11 VM and the rounded corners got replaced by square corners.

[Hedgehog2]

My host operating system is Windows 8.1 Home 64-bit. It is a notebook with AMD Radeon HD 7500/7600 series, which has 2 GB. I can't say anything about iGPU/dGPU. The host has 8 GB RAM, 5 GB I provided for the Windows 11 VM.

After starting the Windows 11 VM I get the following messages in th message box window:

No 3D support ist available from the host.
The 3D features of the virtual machine will be turned off.

There is no relevant message in the mksSandbox.log file.

In file vmware.log I can see the message '[msg.mks.no3D] No 3D support is available from the host. The 3D features of the virtual machine will be turned off.'.

DxDiag output for the host:

The DxDiag output for the host tells that 3D acceleration ist activated. So why do I get the message 'No 3D support ist available from the host.' ?

Link for vmware.log: https://1drv.ms/u/s!AnJkahC5qjfigQWeCo0ghciKagaq 

[bluefirestorm]

During the power up, it failed to start the mksSandbox (probably why there is no relevant information in the mksSandbox.log file).

2022-04-04T10:44:58.013Z In(05) mks MKS-RenderMain: Starting ISBRenderer (DX11Renderer)
2022-04-04T10:44:58.013Z In(05) mks ISBRendererComm: ISBRendererComm DataChannel size=1073741824
2022-04-04T10:44:58.013Z In(05) mks ISBRendererComm: mksSandbox command-line: C:\Program Files (x86)\VMware\VMware Player\x64\mksSandbox.exe --pipeInfo \\.\pipe\vmware\mksSandbox\mksSandbox-52 39 6a 8b ef c1 26 d5-06 80 ed 67 a4 74 87 38
2022-04-04T10:44:58.013Z In(05) mks ISBRendererComm: Failed to create process as user (error=87)
2022-04-04T10:44:58.013Z In(05) mks ISBRendererComm: Failed to spawn the sandbox process: (87) Unknown error 87 (0x57)
2022-04-04T10:44:58.013Z In(05) mks ISBRendererComm: Failed to spawn the sandbox process
2022-04-04T10:44:58.013Z In(05) mks ISBRendererComm: Unable to spawn mksSandbox
2022-04-04T10:44:58.013Z In(05) mks ISBRenderOps: ISBRenderer failed to start.

After it failed to start the mksSandbox process, it fell back to using DX11BasicRenderer, that's why there is no 3D acceleration on the VM.

2022-04-04T10:44:58.013Z In(05) mks MKS-RenderMain: Starting DX11BasicRenderer
2022-04-04T10:44:58.013Z In(05) mks DX11BasicRenderer: Enumerating adapter 0
2022-04-04T10:44:58.013Z In(05) mks DX11BasicRenderer: `AMD Radeon HD 7500M/7600M Series` vendor=0x1002 device=0x6841 revision=0
2022-04-04T10:44:58.013Z In(05) mks DX11BasicRenderer: video=2028MB system=0MB shared=3840MB
2022-04-04T10:44:58.076Z In(05) mks DX11BasicRenderer: Using device unknown; adapter `AMD Radeon HD 7500M/7600M Series`
2022-04-04T10:44:58.076Z In(05) mks DX11BasicRenderer: Enumerating adapter 1
2022-04-04T10:44:58.076Z In(05) mks DX11BasicRenderer: `Microsoft Basic Render Driver` vendor=0x1414 device=0x008c revision=0
2022-04-04T10:44:58.076Z In(05) mks DX11BasicRenderer: video=0MB system=0MB shared=256MB

I don't know why the ISBRenderer mksSandbox process failed to start. But it might have to do with the Windows 8.1 host. Windows error 87 simply means invalid parameter. There is another thread post that has similar situation: Windows 8.1 host, Workstation 16.2.x and the installation of 16.2.x indicated 3D acceleration will be disabled despite that the GPU is Nvidia GTX1060. But this is the first time I am seeing a vmware.log related to this issue (Windows 8.1 host and 3D acceleration gets disabled).

What you could do is disable the mksSandbox by adding the following lines to the %PROGRAMDATA%\VMware\VMware Player\config.ini. These lines does not seem to work on vmx configuration level and work only on the config.ini. As it is in the config.ini it will affect all VMs. Disabling the ISBRenderer will make it like version 15.x (i.e. the mks thread is not in its own sandboxed process). The sandboxed ISBRenderer was introduced with version 16.x. To edit the config.ini you will have to open a command prompt with Administrator and navigate to the directory and edit using notepad. The config.ini also has read-only attribute checked.

mks.requireISBRenderer = "FALSE"
mks.enableISBRenderer = "FALSE"

With the mksSandbox disabled, it should not fall back to DX11BasicRenderer. And you should see DX11Renderer being used within the vmware.log.

If after disabling the sandbox the 3D acceleration is still not available, it might have to do with the DX11 feature level of the AMD GPU. I am not sure if DX11.1 is required or not with 16.2.x. If 3D acceleration is still unavailable you could try switching to GLRenderer in the vmx configuration file.

mks.enableDX11Renderer = "FALSE"
mks.enableGLRenderer = "TRUE"

I don't have a Windows PC with AMD GPU, what I have is a Windows laptop with Intel iGPU (HD530) with an Nvidia Maxwell Gen 1 (GTX960M) render only device. Most Nvidia GPUs on Windows hosts support OpenGL 4.6.

The AMD 75xx/76xxM GPUs seems to support OpenGL 4.4.
https://www.techpowerup.com/gpu-specs/amd-thames.g197

Switching to GLRenderer on Windows Nvidia GPU seems to fix a rendering issue with Chrome/Edge browsers and other applications based on Chromium or rely on ANGLE graphics (perhaps anything that uses Qt framework).
Switching to GLRenderer on Intel iGPU is not recommended as more rendering problems arise (Windows logon, menus are not rendered properly).
I don't know what happens with VMs using OpenGL rendering for Windows hosts with AMD GPU.

[Hedgehog2]

Thank you very much for your information.

I tried both of your suggestions one after the other. Unfortunately, in both cases I did not get round corners and 3D acceleration.

[bluefirestorm]

Could you try adding the lines

mks.requireISBRenderer = "FALSE"
mks.enableISBRenderer = "FALSE"

to %PROGRAMDATA%\VMware\VMware Workstation\config.ini.

I looked again at the vmware.log it looks like it also reads config.ini from that location

2022-04-04T10:44:57.716Z In(05) vmx DICT --- HOST DEFAULTS C:\ProgramData\VMware\VMware Workstation\config.ini
2022-04-04T10:44:57.716Z In(05) vmx DICT installerDefaults.simplifiedUI = "no"
2022-04-04T10:44:57.716Z In(05) vmx DICT installerDefaults.autoSoftwareUpdateEnabled = "yes"
2022-04-04T10:44:57.716Z In(05) vmx DICT installerDefaults.autoSoftwareUpdateEnabled.epoch = "6758"
2022-04-04T10:44:57.716Z In(05) vmx DICT installerDefaults.componentDownloadEnabled = "yes"
2022-04-04T10:44:57.716Z In(05) vmx DICT installerDefaults.dataCollectionEnabled = "no"
2022-04-04T10:44:57.716Z In(05) vmx DICT installerDefaults.dataCollectionEnabled.epoch = "6758"

You should then see the lines follow the installerDefaults entries in the vmware.log once those lines are added.

You can confirm that the mksSandbox is no longer used in the vmware.log

2022-04-04T10:44:58.013Z In(05) mks MKS-RenderMain: ISB enabled by config

You should see "ISB not enabled by config" instead and then see MKS-Render

<timestamp> In(05) mks MKS-RenderMain: Starting DX11Renderer
or
<timestamp> In(05) mks MKS-RenderMain: Starting GLRenderer

depending on the mks.enableDX11Renderer/GLRenderer entries.

[Hedgehog2]

I have added the two entries to the config.ini file. Now I have round window corners on Windows 11.

You wrote: 'Disabling the ISBRenderer will make it like version 15.x (i.e. the mks thread is not in its own sandboxed process). The sandboxed ISBRenderer was introduced with version 16.x.'.

Does this mean that VMWare now works as in version 15.x? Wouldn't it be better to install version 16.1.x instead of version 16.2.x and remove the two entries from the config.ini file?

[bluefirestorm]

I don't work for VMware so I am not privy to their design decisions and don't have access to any of their source code. So the next few paragraphs is just pure conjecture on my part.

I think mks stands for mouse keyboard screen (just a wild guess). With 15.x and earlier, the mks thread would be running in the same process (vmware-vmx.exe) as all other functions/capabilities/features of a VM. With version 16.x the mks thread of a VM would run in its own process (mksSandbox.exe). One possible reason with having its own sandboxed process, more resources would be available to the mks thread without running into possible limitations if it were still in vmware-vmx.exe. I believe it was with version 16.x that 8GB of VM display memory can be assigned. By having its own sandboxed process, it also prevents the mks as being a point of entry to attack/hack into a VM (its memory, guest OS, and apps running in the guest OS).

One way to visualise this: With 15.x and earlier, all the fish is in one aquarium tank (vmware-vmx.exe), and with 16.x some fish are in one tank (vmware-vmx.exe) and some fish are in another tank (mksSandbox.exe) because the original aquarium tank (vmware-vmx.exe) was getting too crowded.

As for the graphics capabilities, assuming the VMware developers organise their source code well, the bulk of the mks thread source code should be same whether it is running in its own mksSandbox.exe process or within vmware-vmx.exe. 99.99% probability that mks source code is the same. They should be really, really bad in programming practices, source code control, etc to have different branches of source for sandboxed and non-sandboxed mks feature/functionality. So if we follow that assumption, the features (such as the 3D graphics) should remain the same whether it is sandboxed or not (with the exception of the max video memory). Going back to the aquarium analogy, a clownfish is a clownfish whether it swims in tank A or tank B.

So by disabling the mksSandbox through config.ini that means the potential attack surface for a VM is bigger compared to sandboxed mks and possibly should assign <=3GB of VM display memory (3GB was the maximum in version 15.x). But with 16.x, that does not mean mksSandbox won't be vulnerable to attacks. It just mitigates that in the event of an attack via the mks thread other aspects of the VM are out of reach. On the other hand, going back to 16.1.2 would also possibly expose the VM to possible security vulnerabilities fixed in 16.2.x and future versions.

[Hedgehog2]

I understand. Thank you for your explanation.