VMware Cloud Community
adolfotregosa
Contributor
Contributor

6.7U1 vs 6.5U2 passthrough regression

Hello.

I'm hopping someone has some clue why this is happening.

Using esxi 6.5u2, passthrough GPU (NVIDIA 2080 TI) works perfectly. On 6.7U1 it works until I reboot the Windows VM. After rebooting I get error code 43 and the only working fix is to reboot the host. The GPU passtrhough only works on the first VM boot. If I reboot the VM it stops working with error code 43.

Downgrading to esxi 6.5U2 fixes the issue.

Any suggestions ?

20 Replies
daphnissov
Immortal
Immortal

Well, that adapter isn't certified for any release to begin with, so you'll have to take what you can get.

Reply
0 Kudos
adolfotregosa
Contributor
Contributor

I understand but since it is purely a software issue is was worth the shot asking if someone knew how to fix it.

Reply
0 Kudos
ely105
Contributor
Contributor

I've noticed issues as well with 6.7 when I had things working with 6.5+

Just a quick note of progress with 6.7...

I tried a couple different hosts with different NVIDIA cards (1030, 1080ti) and got the same results, after seeing the card initialize it would shut down/crash.  From that point forward the dreaded error code 43 on device properties.  I read many posts in the past that said 43 was an intentional disable in software when the driver detected it was running in a VM.  So it got me thinking...

I always made sure that I had "hypervisor.cpuid.v0=FALSE" in my config before ever passing through the video card.  But something changed in 6.7, or maybe Nvidia is looknig for something else?

So I tried this:

Build a Windows10 VM (v1809)

don't install vmware tools

install Chocolatey (https:\\chocolatey.org) For easier install of teamviewer and nvidia drivers

install teamviewer via chocolatey (so I can connect remotely) *also I didn't have usb hardware that I could passthrough

disable svga adapter "svga.present=FALSE"

set "hypervisor.cpuid.v0=FALSE"

Add PCI devices: 1080TI, Audio Device

boot.

Ok so the start was not pretty as windows detects video card and you need to reboot after initial install and you don't have vmware tools installed so its a cringe-worthy moment.

but alas after the reboot and the driver took, the output looked stable.

after that I installed the latest nvidia drivers and all seems ok.

So, what was it?  svga present? vmware tools?

If I had to guess it might be svga, because I saw a similar issue with Ubuntu 18.10.  I was having problems getting the nvidia driver to work in Ubuntu so on a fluke I disabled the svga driver and voila!  Ubuntu 18.10 on GPU!

I may install vmware tools and see if that makes a difference.

Anyways, I wanted to throw this out there for all the folks suffering with this issue.  If it's Nvidias doing then I suppose it'll just be a matter of time for them to find another way.

-m

Reply
0 Kudos
IvarHome
Hot Shot
Hot Shot

My AMD GPU card dont like shutdown. I can reboot Windows, but when I shutdown only windows VM, then it dont boot up without ESXi reboot.

Reply
0 Kudos
ely105
Contributor
Contributor

I saw this behavior as well after some time with a GTX 1080ti.  And later also with RTX 2080.  I could run the windows VM only once without problems, but as soon as I shutdown/rebooted then I saw error 43.  On the other hand I had an ubuntu 18.10 vm that could be cycled without issues.  But even once the Windows vm was started and shutdown, the ubuntu vm wouldn’t work after that either.  The only way to get it back working was a reboot of ESXI.  So it appeared that the card was left in some state that was not allowing it to start properly after that.  I looked into some of the PCIE bus reset methods, but only tried D3D0 other than default bridge mode.  But no change.

Then I read some articles online where others were having similar issues.  They reported one workaround was to disable the GPU device in Device Manager before a shutdown, and then re-enable it when the vm was started again.  It sounded ugly but I gave it a try and low and behold it worked.  I was able to startup/shutdown the vm with no more error code 43’s.  I found a way using devcon.exe? to automate this process as a startup/shutdown script so it winds up being painless and automatic.  So I don’t have to remember to do it manually and it just works now. 

That might be something worth trying for you.  I’m running 6.7.0 update 1 with latest patch.

-m

Reply
0 Kudos
IvarHome
Hot Shot
Hot Shot

He-hee, yes, it really sounds ugly, because when windows starts without drivers, it push all icons upside-down and also its not nice for eyes to see. Smiley Happy I have Windows 8.1. Reboot works without problems, but only when shutting windows down, then later I must also reboot host. But its not problem for me, because usually I dont need to shut down windows. When I want to shut down computer (host), I just push shutdown button and it cleanly first shuts down windows (set to do so in "VM Startup/Shutdown" automaic) and when I start computer, it boot up also windows. Boot process is only little longer, no problem. The only moment when I need to just shutdown windows, is when I want delete or make snapshots. As passthrough dont allow online snapshots and suspends. Then I use phone app ("Whatchlist" and "vmwPAD") to connect to vCenter and work with snapshots. Of course other computer can also be used for this, but phone app is very comfort. After that I make reboot to host (with the same phone app) and Windows is soon again up.

Reply
0 Kudos
HelloFelix
Contributor
Contributor

My GTX 1660Ti doesn't like reboot on 6.7u1 either.

It works like a charm on 6.5u2.

Tried all reset methods but no lucky.

See my post at Deep investigation on GPU Passthrough not working anymore after upgraded from 6.5 to 6.7, what's dif...

joetse123
Contributor
Contributor

Did you try 6.7u2? I have same problem @ 6.7u1. After initial boot the host, the gpu (GTX 1660) can passthrough to win 10 guest without any problem. After the guest reboot / shutdown, and boot again the gpu passthrough is not work and display code 43 error. I have quadro p2000 and I discovery that after the guest reboot / shutdownm, the p2000 fan will run at full speed. I think it is the pci reset work on p2000. Unfortunately, the consumer gpu can't passthrough due to incorrect pci reset. Is it a bug on 6.7? Have someone report to VMWare?

Also, I notice that the esxi is skipping to reset my gpu.

2019-05-19T15:59:54.818Z cpu0:2099331)PCI: 967: Skipping device reset on 0000:01:00.0 because PCIe link to the device is down.

2019-05-19T15:59:54.818Z cpu0:2099331)IOMMU: 2502: Device 0000:01:00.0 placed in new domain 0x430430ac2d10.

2019-05-19T15:59:54.818Z cpu0:2099331)PCI: 967: Skipping device reset on 0000:01:00.1 because PCIe link to the device is down.

2019-05-19T15:59:54.818Z cpu0:2099331)PCI: 967: Skipping device reset on 0000:01:00.2 because PCIe link to the device is down.

2019-05-19T15:59:54.818Z cpu0:2099331)PCI: 967: Skipping device reset on 0000:01:00.3 because PCIe link to the device is down.

Reply
0 Kudos
daphnissov
Immortal
Immortal

GTX series cards are consumer cards and therefore not supported anyway.

Reply
0 Kudos
joetse123
Contributor
Contributor

I know, but 6.5 it is work. So vmware change it to make consumer gpu not work?

Reply
0 Kudos
daphnissov
Immortal
Immortal

Even in 6.5 it wasn't supported. When you use unsupported hardware it can stop working at any time. Because it worked in 6.5 doesn't mean it'll work in later versions. It may. Then again, it may not.

Reply
0 Kudos
joetse123
Contributor
Contributor

Hey, I have a new idea. I have see some of people can passthrough their gtx gpu at esxi 6.7. But their cpu is xeon or the motherboard is using x99 chipset. Are you using consumer verson cpu or motherboard? Maybe the problem is the motherboard and cpu support. The pci reset method also need cpu or chipset support?? I am using i5-6500 and ASUS B150M-A/M.2 and try to passthrough GTX 1660 with no luck to keep it works after vm reboot. I want to try the x299 with i9-9900x. If it is works, I will report to you. If not, I will move on kvm.

Reply
0 Kudos
HelloFelix
Contributor
Contributor

Just go back and try latest 6.5, it works like a charm.

Reply
0 Kudos
joetse123
Contributor
Contributor

Hey guys, I have test esxi 6.7u2 on i9-9900x, ASUS WS X299 PRO/SE and GTX 1660. The GPU passthrough can works and no more code 43 after reboot problem. I think the key point is server grade motherboard or no iGPU within the cpu? Also, I have noted that the people whose susses to passthrough in 6.7 are using xeon cpu. I hope it can help you.

Reply
0 Kudos
Besterino
Contributor
Contributor

I think I figured it out: I can reboot my Win10 VM with a Nvidia Consumer GPU "passthrough'ed" without the usual workaround (disabling the GPU in device manager before reboot).

Hardware / Software used:

Board: X399D8A-2T

CPU: Threadripper 1920X

GPU: 2080Ti Founders Edition

ESXi: 6.5U2 & 6.7U3

Guest-OS: Win10 1903

1. Edit passthru.map and comment out the Nvidia wildcard setting ("10de ffff ...") and instead set for ALL individual devices EXCEPT/NOT the GPU d3d0 :

# NVIDIA original entry

#10de ffff bridge false

#Audio

10de 10f7 d3d0 false

#Serial Bus

10de 1ad7 d3d0 false

#USB

10de 1ad6 d3d0 false

2. Put all Nvidia devices via pcie-passthrough in your VM, i.e. in my case GPU, Audio, Serial Bus Controller, USB Controller.

3. Set pciPassthru.msiEnabled = FALSE for ALL Nvidia devices

4. Enjoy hassle free reboots as it should be. :smileygrin:

5. Notes:

5.1 Works for VMs with either BIOS or EFI startup option.

5.2 Audio seems to work as well.

5.3 To passthrough X399-Chipset onboard USB-Controllers add d3d0 entry in passthru.map, in my case I had to set "1022  ffff  d3d0   false" .

HelloFelix
Contributor
Contributor

Oh darling you saved my day!

I've tried all passthrough options and reset combinations, but not like the way you did.

I'll try your configurations after work. will keep you posted !

Reply
0 Kudos
Besterino
Contributor
Contributor

Good luck! If I understand your earlier posts correctly, you will use another GPU-type - so you need to change the device IDs in passthru.map accordingly. Hope it will work for you as well.

Reply
0 Kudos
starlinkgaszer
Contributor
Contributor

I just want to say thank you Besterino  I was able to get this working with my supermicro server and 1660ti using esxi 6.7. Because of you I can go to sleep now  🙂

Reply
0 Kudos
5481q8weqz
Contributor
Contributor

【Solved】

hypervisor.cpuid.v0 = FALSE

pciPassthru.msiEnabled = FALSE

 

Also in passthru.map I commented the graphics card and the default NVIDIA options, and the problem solved

 it seems to show that the display adapter works just fine without having to be marked in the passthru.map

 

Thank you very much

 

Tags (1)
Reply
0 Kudos