Horizon 8.6.0
VSphere 7.0.3 lastest patches
Dell R740 hosts - latest firmware - SR-IOV enabled, Performance OS Controlled power profile
NVidia 14.2 vGPU drivers on hosts and virtual desktops
Tesla T4 - T4-2b profiles, ECC disabled
Blast protocol
Virtual Desktop OS: Windows 10 - 21H2
Primary apps - AutoDesk Civil3D
Instant Clone virtual desktop sessions will randomly lock-up. SSH to host running the desktop, run the command "nvidia-smi vgpu" and we will see the following with the highlighted being the impacted virtual desktop:
Hard reset of the impacted virtual desktop will resolve the issue. Since the desktop is reset the user will lose their work.
The occurrence is totally random with respect to host, desktop, user or applications that the user has open in the their session. We have seen this occur on sessions where the user has nothing but a web browser and Outlook open.
If other desktops with active sessions are sharing the physical GPU and we don't reset the impacted desktops quickly, their sessions will freeze as well. However, resetting the "99%" machine will unfreeze their desktops and they will be able to reconnect and not have lost work.
This issue started in early June of this year
VMWare Support shrugs shoulders and says ask Nvidia. NVidia support says the 99% utilization is a symptom of frame buffer exhaustion, which is silly. You can see from the screenshot above that nothing else is actively occurring on the card. There is no correlation to GPU intensive tasks being performed on the desktop that would cause this "exhaustion". Switching from a T4-1B to a T4-2B profile has caused the issue to occur less frequently. But this never happened before June of this year. There were no major changes to the applications being used on the desktops or in the intensity of the tasks being performed by the users. It doesn't make sense.
Any thoughts would be appreciated.
