About NVIDIA GPU and ESXi 6.5

[bellocarico]

I have 2 questions about my NVIDIA GTX 1060 6GB and ESXI 6.5

1) I have installed the above card on my server and set it to passthrough. Have reserved the same amount of memory as the RAM capacity to allow the VM to boot up.

When it gets into windows10 though, I can see the device, install the drivers but even after a reboot the device is not operational and has a yellow warning and reports:

This device cannot start. (Code 10)

Insufficient system resources exist to complete the API.

Can anybody shed some light on what windows is moaning about and especially what ESXi can do to make it happy?

2) I did a quick search on vGPU. Considering this is a standalone ESXi installation (no vSphere etc), can the physical GPU be shared amongst VMs (like the CPU does)?

Thanks

[Dave_the_Wave]

First you may want to confirm if a regular game GPU can be used as a 1:1 passthrough to a Guest OS.

That will be best case scenario, two Guests OS will not be able to share the GPU at the same time.

Would be nice to avoid those Quadro and Grid cards made for vGPU.

I too would like to see the steps to get just one Guest OS up working on this.

[bellocarico]

I actually managed to make this "seen" by wndows10 properly adding

hypervisor.cpuid.v0 = "FALSE"

to the .vmx after the NVDIA was added to the VM.

The driver installs fine as well but because the VMWARE SVGA does not understand optimus (of course) windoes is not be able to operate internal and NVIDIA GPU. e.g. opening the NVIDIA control panel it will complain that an NVIDIA card not being the primary display adapter.

I am now trying to have this card used as primary in windows but VMWARE doesn't allow me to either remove the default SVGA and if I go disabling it within windows (device manager) it still uses it!

Basically I see both cards under device manager but I'm unable to tell any program to use it.

Does anybody have any tip on how to proceed from here onwards?

Thanks!

[bellocarico]

Ok according to the latest: I need to disable the internal SVGA which is automatically added to the VM after creation.

I see no option to remove it as you would do for the hardware so I did some digging and found the "svga.present" parameter.

If I add it to the VM and set it to FALSE the video card disappears indeed.... but the VM doesn't boot up. It powers up but the VM IP is not reachable.

How can I remove the built in SVGA and manage to boot the VM?

[bellocarico]

nobody? 😞

[bellocarico · Accepted solution]

Ok DIY this time:

First of all I had to replace the GTX 1060 as it was a mining card e.g. no video output.

I got a traditional 1070 instead and plugged an old monitor via DVI. I have however ordered but not received yet a dummy HDMI plug which essentially simulate a monitor connection. Without monitor directly connected to the GPU this below is likely not to work.

0) ESXi I run

Version: 6.5.0 Update 2 (Build 10390116)

  State: Normal (not connected to any vCenter Server)

1) Once you install the GPU make sure the Server BIOS prefers internal video card to any PCIE

If you don't do this method below is not going not work; ESXi still boots but the progress bar is never to reach the end.

2) Enable pass-through on the card (via system management) and reboot

3) assign the card as PCI to the VM and reserve all the RAM you have assigned. Leave the default sVGA config untouched

4) Modify the parameter of the Windows10 VM adding:

hypervisor.cpuid.v0 = FALSE

pciPassthru0.msiEnabled = FALSE

Ref: https://nvidia.custhelp.com/app/answers/detail/a_id/4135/~/vmware-vdga-%2F-gpu-passthrough-requires-...

5)  Once you boot into Windows I first had to make the NVIDIA display act as primary. Note, vmwaretools will bypass any attempt to disable the internal sVGA or it's virtual display. So forget about device manager and head straight to the Windows Graphic Settings (right click on desktop/Display Settings):

a) highlight the display linked to the NVIDIA card

b) select "make this my main display"

c) under "multiple displays" select "show only on 2" (in my particular case)

NOTE: The only negative point since the sVGA is disable in the windows configuration, the ESXi remote console will only show up to the windows loading logo and get stuck there. That's when I guess Windows disables the sVGA and set the NVIDIA to be the primary. Hence remote console is not working but you can install your own remote admin software I suppose.

NOTE2: Reading up in Internet it seems like pass-through works better with EFI rather than BIOS. I had a pre-existing VM in BIOS so didn't want to reinstall so all this procedure refer to BIOS only.

ADDITIONAL NOTES AFTER 2 WEEKS OF USAGE:

The steps 1 to 5 above helped me to make the card operational but... it was rather unstable e.g. VM freezing while encoding video etc. After further researches I ended up applying 2 additional modifications:

6) Since my video card has 8GB of memory as an extension to point 4) above under the VM parameters also add:

pciPassthru.use64bitMMIO="TRUE"

Ref: VMware Knowledge Base

7) If an only if your VM is unstable past this point you can attempt a final step which will require the host reboot. Edit your /etc/vmware/passthru.map and modify the bridge keyword into a different value. Accepted values are: d3d0, link, bridge, default

# NVIDIA

10de  ffff  bridge   false

I had luck with link but other people with different system and dfferent GPUs are forced to use different reset values.

😎 If at this point you can operate the passwhtough video card but somehow is unstable: downgrade your VM hardware version to something lower than 10. So

a- unregister the VM

b- edit the .vmx

c- change the virtualHW.version to e.g. virtualHW.version = “8”

d- re-register the VM

My system is not 100% stable as yet but it's a huge step forward compared to what I did documented here above. I finally was able to encode 10 videos in Handbrake/NVEnc non stop without freezing.

I hope this helps other people as well.

[bellocarico]

I want to add an additional note on my setup. Hopefully I'll help other people too.

From the steps documented above I suggest a minor modification to point 5)

The modification being: do not set up your NVIDIA as primary and only display! Stick the the Intel or built in whatever you have and assign no screen to the NVIDIA. As long as Windows recognises the NVIDIA this will be used on an application basis. Obvious but likewise important: you must be on the console to do so, of course RDP will not allow you to set up display settings.

Windows 10 1909 (but even earlier version, same concept)  - Settings/System/Display/Graphic Settings

Select "Classic App" and browse the applications you want to benefit from the NVIDIA.

In my case I use Handbrake for video encoding.

Since I did the above about two months ago I did not get a single VM freeze any more.

HTH

[bellocarico]

An additional consideration on this topic.

Since the external GPU has often lots of memory (8GB in my case), you need to allocate a large amount of memory to your VM, certainly higher than the GPU RAM.

This leads into scenarios where your host RAM is used to a level probably never experienced before.

In my specific case there was a host RAM issue (minor faulty RAM module) and this became visible only with this pass-through settings.

TL;DR: make sure you fully test your host RAM before adventuring into this scenario.

[RTODD73]

Hi where did you get the vib for the 1060/1070 to install it in esxi? I have a 1060 I want to use in my esxi server but can't find any reference to a vib for it when searching the web.