We have been working on implementing M10 vGPUs in our VMware environment and have been experiencing performance issues. We worked with NVIDIA to verify that the environment is setup correctly. Here is quick bullet point list for the environment.
- vSphere 6.7
- Host have VMware ESXi, 6.7.0
- PowerEdge R740
- Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70GHz
- 768 GB RAM
- 2x M10 GPUs
- Horizon 7.4.0
- Linked Clones
- Windows 10 1803 (also tried 1809 & 1709 builds)
- 4 core
- 8 GB RAM
- M10-1B vGPU profile
- Teradici based zero clients (PCOIP)
What we have noticed is after our first small set of user testing and no issues we began a larger test and noticed that once we hit about 15 users per M10 we began getting performance issue reports. The users are seeing lag in the interface, a right click on the desktop might take 5-10 seconds for the context menu to appear. The same could be seen with the start menu. Additionally these issues were only occurring on vGPU VMs. On the same host non vGPU VMs were not experiencing the same slowdowns. We began to notice that the pcoip_server_win32.exe was using a lot of CPU and GPU time via the task manager. We began trying different version of the vmware agent and direct connect, various revisions of the driver for esxi. We tried standalone fresh copies of windows 10 and various build numbers. Thus far no combination we have attempted has resolved the issues for vGPU machines when there are more than a few users per machine. The performance problem appears even if we use the Horizon software client and have it set to PCoIP.
We tried running it with different Horizon Agent versions (6, 7.0.2, 7.2.0, 7.4.0, 7.5.1, 7.6 & 7.7) and using direct connect bypassing Horizon Server.
We also tried running VMs using VMware Blast protocol and it didn’t have the high GPU usage issues, but unfortunately, almost all of our thin clients only support PCOIP.
Attaching screenshot below: please note the GPU utilization of PCoIP Server (32bit) process.
UPDATE 1:
After a long discussion with NVIDIA, they concluded that it's not on their side.
They pointed us at this KB: https://nvidia.custhelp.com/app/answers/detail/a_id/4156/~/nvidia-smi-shows-high-gpu-utilization-for...
Looks like it's a known issue with Teradici PCOIP protocol and it hasn't been fixed yet.
UPDATE 2:
I tried downloading and installing Teradici's PCOIP Agent (PCoIP_agent_release_installer_graphics.exe) direct from Teradici. Then I ran "NvFBCEnable.exe -disable", it disables NVFBC capture and switches back to CPU. And it works great - no GPU spike when idle and a much better performance overall.
However, when I try to do this on Horizon Agent's Teradici protocol it disables NvFBC for a brief period of time and then as soon as I reconnect via PCOIP it re-enables it back, see extract from a log below:
- Svgadevtap: NvFBC – Fixed capture by enabling NvFBC
Is there a way to permanently disable NvFBC on Horizon Agent's Teradici Protocol?
Accepted Solutions
We were able to find a workaround for our problem.
We used a combination of memory dump & Sysinternals process monitor to find the registry key.
Below is a combination of settings we use to achieve satisfactory performance:
[HKEY_LOCAL_MACHINE\SOFTWARE\VMware, Inc.\VMware SVGA DevTap]
"Win32FrameRate"=dword:0000002d
"MaxAppFrameRate"=dword:0000002d
"ForceWin32Capture"=dword:00000001
[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Teradici\PCoIP\pcoip_admin_defaults]
"pcoip.audio_bandwidth_limit"=dword:000001c2
"pcoip.enable_build_to_lossless"=dword:00000000
"pcoip.enable_console_access"=dword:00000000
"pcoip.minimum_image_quality"=dword:00000028
"pcoip.maximum_initial_image_quality"=dword:00000050
"pcoip.maximum_frame_rate"=dword:0000002d
"pcoip.use_client_img_settings"=dword:00000000
The config is for 45FPS which is the maximum we can achieve with our current NVIDIA license.
We already spoke with VMware and they confirmed that is indeed a workaround for now, until they get Teradici to fix PCOIP code.
We decided to give it another go and built everything fresh from scratch with brand new ESXi, drivers, 1803 image, etc.
The VM below isn't joined to the domain and doesn't have anything installed apart from Nvidia drivers & Horizon Agent 7.7.
See screenshot below, as you can see it's using 30% of GPU utilization in idle.
We were able to find a workaround for our problem.
We used a combination of memory dump & Sysinternals process monitor to find the registry key.
Below is a combination of settings we use to achieve satisfactory performance:
[HKEY_LOCAL_MACHINE\SOFTWARE\VMware, Inc.\VMware SVGA DevTap]
"Win32FrameRate"=dword:0000002d
"MaxAppFrameRate"=dword:0000002d
"ForceWin32Capture"=dword:00000001
[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Teradici\PCoIP\pcoip_admin_defaults]
"pcoip.audio_bandwidth_limit"=dword:000001c2
"pcoip.enable_build_to_lossless"=dword:00000000
"pcoip.enable_console_access"=dword:00000000
"pcoip.minimum_image_quality"=dword:00000028
"pcoip.maximum_initial_image_quality"=dword:00000050
"pcoip.maximum_frame_rate"=dword:0000002d
"pcoip.use_client_img_settings"=dword:00000000
The config is for 45FPS which is the maximum we can achieve with our current NVIDIA license.
We already spoke with VMware and they confirmed that is indeed a workaround for now, until they get Teradici to fix PCOIP code.
