VMware Communities
ocelik-94
Contributor
Contributor

Graphical Artifacts starting with Linux Kernel 5.10 when using Nvidia

Hello VMWare Community,

I am facing an issue which is very annoying for me.

I am using NixOS as Guest Machine I have following graphical glitches with everything webbased (webgl?):

 

248504308-7c2175ea-b7d3-423f-8081-0e6884251968.png

248504316-e7dfae12-bbce-4386-9c41-6fd09bd87403.png

248504327-a28b75db-eb5a-44e6-b83e-8da3af07bcde.png

 

The possibility is high that this issue is related to nixos itself as I dont have any problems with latest kernels on arch linux.

Facts that I already gathered:

- Kernel 5.4 is working fine and Issue starts with 5.10

- vmwglx on nixos kernel 5.4 is 2.15.0 // 5.10 has 2.18.0

- iGPU with intel/amd is working fine // nvidia GPUs are having the issues above

- VMWare Workstation Pro 17.0.2 is being used

I also opened a ticket at open-vm-tools github: https://github.com/vmware/open-vm-tools/issues/674#issuecomment-1613036175 

 

Does anyone have a clue what the problem could be?

 

 

0 Kudos
5 Replies
bluefirestorm
Champion
Champion

There has been a long standing problem with rendering on Windows VMs running on Windows hosts (dating back to at least version 14.x). But this is the first post I seen that the problem manifests in a Linux-based VM rather than Windows guest running on a Windows host. I have yet to see this problem show up when the VM runs on a Linux host or on Fusion running on a Mac.

More often than not, the rendering issue is Chrome browser. Other applications that use Chromium such as Edge, Brave browsers or Visual Studio Code can also fall victim. There are reports of apps like Viber client or Dropbox client having render issues as well. If you dig down to it, all these point to the ANGLE graphics (Viber and Dropbox are written using QT5 framework which uses ANGLE graphics).

Anyway, you could try the workaround to use OpenGL renderer by having the following lines in the vmx configuration file.

#force the use of discrete GPU
mks.forceDiscrete = "TRUE"

#disable DX11/12 renderer on the Windows host and use OpenGL instead
mks.enableDX12Renderer = "FALSE"
mks.enableDX11Renderer = "FALSE"
mks.enableGLRenderer = "TRUE"

Don't use the GLRenderer with Intel HD graphics to run a Windows VM on a Windows host. The rendering problems are even worse and shows up as soon as the Windows guest logon screen.

 

0 Kudos
ocelik-94
Contributor
Contributor

Hi @bluefirestorm,

I added the configuration you posted in the vmx File and started the vm afterwards.

Sadly it still behaves the same.

It also has to be related somehow to nvidia I think as I have no problems with intel igpu or amd igpu (i dont have a amd dedicated gpu)

 

edit:

Using  SVGA_VGPU10=0  also works but uses opengl 2.0 if I am not wrong - so it is only a workaround

ocelik94_0-1688131319567.png

 

0 Kudos
bluefirestorm
Champion
Champion

What is the model/architecture of the Nvidia GPU? One thing that rarely gets mentioned in all these posts about rendering issues is the model/architecture of the GPU.

For Nvidia, I would recommend at least Maxwell Gen 2 GPU (e.g. GTX 9xx desktop) to have Shader Model 6.x support and DX feature set 12_1 support; the newer the better of course. Even Pascal GPUs (GTX 10xx, Quadro Pxxxx) are around 7 years old already.

I have seen rendering issues with a Maxwell Gen 1 GPU (GTX 960M laptop GPU, yes it's confusing not all Nvidia 9xx GPUs are the same) and it supports only DX 11_0 feature set.

You should check the mksSandbox.log to confirm the GPU selected and renderer used (DX12 or DX11 or OpenGL 4.6).

0 Kudos
ocelik-94
Contributor
Contributor

2023-06-30T14:10:48.776Z In(05) mks  MKS-RenderMain: Collecting RenderOps caps from GLRenderer
2023-06-30T14:10:48.776Z In(05) mks  MKS-RenderMain: Starting GLRenderer
2023-06-30T14:10:48.857Z In(05) mks  GLHostWin32: : Created context with GL 2.1, core: 0, robust: 0
2023-06-30T14:10:48.891Z In(05) mks  GLHostWin32: : Created context with GL 4.0, core: 0, robust: 1
2023-06-30T14:10:48.891Z In(05) mks  GLRenderer: OpenGL Version: "4.6.0 NVIDIA 535.98" (4.6.0)
2023-06-30T14:10:48.891Z In(05) mks  GLRenderer: OpenGL Vendor: "NVIDIA Corporation"
2023-06-30T14:10:48.891Z In(05) mks  GLRenderer: OpenGL Renderer: "NVIDIA GeForce RTX 2080/PCIe/SSE2"
2023-06-30T14:10:48.891Z In(05) mks  GLRenderer: Driver Version: "4.6.0 NVIDIA 535.98" (535.98)
2023-06-30T14:10:48.891Z In(05) mks  GLRenderer: GLSL Version: "4.60 NVIDIA" (4.60.0)

I am using  Intel + Nvidia (At work nvidia t550 and private nvidia rtx 2080 as gpu)

Weird thing is that iGPU works.. so it must be related to nvidia

0 Kudos
bluefirestorm
Champion
Champion

It is hard to conclude the root cause of the rendering bugs. It has been around for many versions/years and it may not be the same root cause or it's possible it's a combination of different bugs from different components.

As for your nixos scenario, are you using KDE Plasma desktop? If so, KDE is developed with the Qt framework so it will likely bring along the ANGLE graphics rendering issues (just like Viber and Dropbox clients). Also by any chance are you using "Dark Mode" on the Workstation GUI and/or Windows host?

Just to be clear, I don't work for VMware so don't expect any fix from me. The workaround of using OpenGL came to mind years ago when I realise that the problem didn't show up on VMs running on Workstation on Linux hosts or Fusion on macOS (back then both were using OpenGL for rendering but has since moved to Vulkan on Linux (for Turing or newer cards) and Fusion has moved to using Metal.

0 Kudos