Passthru not working after 6.7 update

[Sinorama]

I was able to passthru my AMD gpu and a Renesas usb controller in 6.5 but after updating to 6.7 my Windows 10 vm startup just shows a black screen on the console. Normally, the VMWare logo displays and Windows boots up. Rolling back to 6.5 and passthru works again. Any ideas?

[A13x]

Are these devices listed under the compatibility matrix:

VMware Compatibility Guide - System Search

Did you check the passthrough configuration on the esxi host, did you test removing it, adding it again and then reconfigure the guest os?

[Sinorama]

Thank you for your help with my passthru problem in 6.7

My esxi is a whitebox setup with Intel Z170 chipset. When I remove the pcie devices from Windows 10 settings, the vm boots without any problem. Also tried toggling disable passthru and rebooting esxi.

[AveryFreeman]

I am having issues with passthrough in 6.7 after using passthrough since 5.5 just fine

I think it may have to do with vmkernel.boot.disableACScheck flag - want to set to = true, but all settings reset to default after rebooting the hypervisor now that I upgraded to 6.7

Seems like a weird issue, must be something I have to change to be able to save settings.

[jUDASmILExAlmig]

There's a driver switch since vSphere6.5 and completed on vSphere6.7. Then from 6.7 the legacy drivers have been removed. The device info and path might change after host upgrade with the driver switch which led to this kind of problem. Then redo the configuration will be helpful for you to re-enable the function.

[Sinorama]

I removed passthru from my esxi host and the vm settings, reboot vm and then reboot esxi.

Short of a fresh install of 6.7, what configuration needs redoing?

Where to inspect the device info and driver path that you refer to?

[jUDASmILExAlmig]

If you're gonna to passthrough usb devices on your USB PCIe card to VM then you just need to toggle passthrough of that card in pci device list then reboot your host. After reboot you can add the pci device to your VM. Once VM powered on you can find the USB devices inside gust. No other configuration needed.

For device info, you can use lspci to check your USB card data and lsusb to check your USB devices info. And there's another place you can check the most detail USB device info here: "/dev/usbdevices".

You can figure out the device path from above file. But it would take time to figure out the level/hierarchy. Another way to find USB device path that after you add the device to VM and power it on, you can check .vmx file or vmware.log to find more info about the device and it path.

[Sinorama]

Thanks for the info. My passthru setup seems to be good after 6.7 upgrade. The problem is the vm with the proper passthru setting does not boot in 6.7. It does not even start to boot.

[jUDASmILExAlmig]

I see. Then any error message you hit when you're trying to boot the VM? Generally, after enable passthrough and add the PCI device to VM, you need to reserve resources to the VM in edit settings wizard. If no resource reserved to VM the boot will fail with error saying you cannot power on the VM. You can just choose to reserve all required resources to VM and click OK to close the wizard then you should be able to boot the VM up.

[petejk]

I experienced the same problem as the OP.

Windows 10 VM, several passthru PCIe devices including Nvidia GPU, NVMe and onboard ASMedia USB controller.

Using a fresh ESXi 6.7 installation, I was able to passthrough all devices successfully with the exception of the USB controller.

The Windows 10 VM would boot to desktop, and all devices passed through as intended.

As soon as the USB controller was selected, the VM would immediately display a blank console screen after power on.

I tried two different USB controller cards, suspecting hardware - but this yielded the same result.

As soon as I blew ESXi  6.7 away, and installed 6.5U1, I was able to select all PCIe devices, including the onboard USB controller, and boot Windows 10.

I would be interested to hear if anyone has/has not been able to pass through a USB controller on 6.7 without issue.

[AveryFreeman]

I haven't tried passing through a USB controller, only my LSI 2008 controller for my FreeBSD VM. I may try it just to see if it works.

It's interesting, on the computer I was having issues with passthrough on, I removed 6.7 (which was an install from ISO to USB drive) and replaced it with a new copy of 6.5U1 (installed on same USB drive).

I then updated it to 6.7 using the following method and I no longer experience passthrough issues (although I do still see an error message while starting up the server, but it doesn't appear to affect functionality). 

I do not currently have vmkernel.boot.disableACScheck = true and it appears to be working fine.

Essentially involved using a datastore as swap by going to manage-->system-->swap-->edit (or new)

The update pretty much just involves allowing https access through the firewall and using this esxcli command:

esxcli software profile update -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml -p ESXi-6.7.0-8169922-standard

The update is detailed here:

https://tinkertry.com/easy-update-to-esxi-67http://

I do not see the error when # grep -i pass /var/log/vmkernel.log --

[code]

2018-05-09T21:42:30.877Z cpu3:2097597)vmk_MemPoolCreate passed for 64 pages

2018-05-09T21:42:31.315Z cpu0:2097642)vmk_MemPoolCreate passed for 4 pages

2018-05-09T21:42:33.428Z cpu2:2097712)vmk_MemPoolCreate passed for 256 pages

2018-05-09T21:43:09.539Z cpu0:2097526)Activating Jumpstart plugin pciPassthru.

2018-05-09T21:43:09.543Z cpu3:2098058)Loading module pciPassthru ...

2018-05-09T21:43:09.546Z cpu3:2098058)Elf: 2101: module pciPassthru has license VMware

2018-05-09T21:43:09.548Z cpu3:2098058)Device: 192: Registered driver 'PCIPassthru' from 78

2018-05-09T21:43:09.548Z cpu3:2098058)Mod: 4962: Initialization of pciPassthru succeeded with module ID 78.

2018-05-09T21:43:09.548Z cpu3:2098058)pciPassthru loaded successfully.

2018-05-09T21:43:09.553Z cpu0:2097526)Jumpstart plugin pciPassthru activated.

2018-05-09T21:43:10.472Z cpu1:2097526)Activating Jumpstart plugin autodeploy-setpassword.

2018-05-09T21:43:10.499Z cpu1:2097526)Jumpstart plugin autodeploy-setpassword activated.

2018-05-09T21:43:12.116Z cpu7:2097526)Activating Jumpstart plugin pcipassthrupool.

2018-05-09T21:43:12.140Z cpu7:2097526)Jumpstart plugin pcipassthrupool activated.

2018-05-09T21:43:23.041Z cpu5:2098605)WARNING: PCI: 178: 0000:01:00.0: Bypassing non-ACS capable device in hierarchy

2018-05-09T21:47:38.028Z cpu2:2099906)PCIPassthru: 5608: No such device: 0000:02:00.0

2018-05-09T21:48:09.951Z cpu5:2100057)PCIPassthru: 5608: No such device: 0000:02:00.0

2018-05-09T21:49:13.802Z cpu5:2098647 opID=9eb6c991)WARNING: PCI: 178: 0000:01:00.0: Bypassing non-ACS capable device in hierarchy

TSC: 898440 cpu0:1)BootConfig: 908: disablePciPassthrough = FALSE

0:00:00:04.802 cpu0:2097152)PCI: 216: enablePCIErrors: 0, enableValidPCIDevices: 1, pciSetBusMaster: 1, disableACSCheck: 0, disableACSCheckForRP: 1, enablePCIEHotplug: 1, pciBarAllocPolicy: 1, disablePciPassthrough: 0

0:00:00:04.856 cpu0:2097152)Initializing PCI passthru ...

2018-05-09T21:50:49.315Z cpu4:2097549)WARNING: PCI: 178: 0000:01:00.0: Bypassing non-ACS capable device in hierarchy

2018-05-09T21:50:50.889Z cpu7:2097597)Loading module pciPassthru ...

2018-05-09T21:50:50.893Z cpu7:2097597)Elf: 2101: module pciPassthru has license VMware

2018-05-09T21:50:50.894Z cpu7:2097597)Device: 192: Registered driver 'PCIPassthru' from 20

2018-05-09T21:50:50.894Z cpu7:2097597)Mod: 4962: Initialization of pciPassthru succeeded with module ID 20.

2018-05-09T21:50:50.894Z cpu7:2097597)pciPassthru loaded successfully.

2018-05-09T21:50:50.899Z cpu6:2097561)WARNING: PCI: 178: 0000:01:00.0: Bypassing non-ACS capable device in hierarchy

2018-05-09T21:50:50.899Z cpu6:2097561)PCIPassthru: PCIPassthruAttachDev:221: Attached to PCI device at 0000:01:00.0

2018-05-09T21:50:50.899Z cpu6:2097561)Device: 327: Found driver PCIPassthru for device 0x25894304be777d1e

2018-05-09T21:50:50.899Z cpu6:2097561)PCIPassthru: PCIPassthruStartDev:362: Received start request for PCI device at 0000:01:00.0

2018-05-09T21:50:50.899Z cpu6:2097561)PCIPassthru: PCIPassthruScanDev:264: Received scan request for PCI device at 0000:01:00.0

2018-05-09T21:51:02.675Z cpu0:2097526)Activating Jumpstart plugin pciPassthru.

2018-05-09T21:51:02.677Z cpu0:2097526)Jumpstart plugin pciPassthru activation failed: pciPassthru->start() failed: error while executing the cli

2018-05-09T21:51:03.563Z cpu4:2097526)Activating Jumpstart plugin autodeploy-setpassword.

2018-05-09T21:51:03.588Z cpu0:2097526)Jumpstart plugin autodeploy-setpassword activated.

2018-05-09T21:51:05.205Z cpu3:2097526)Activating Jumpstart plugin pcipassthrupool.

2018-05-09T21:51:05.233Z cpu3:2097526)Jumpstart plugin pcipassthrupool activated.

[/code]

[Sinorama]

I did update using vmw-depot from 6.5.0 Update 1 (Build 7967591) to 6.7 using profile as described. Unfortunately, with 6.7 setup my vm with the pci devices will not begin to boot. Can it be that it works for you because you are passing just one device (LSI 2008)?

[petejk]

As I explained in my initial reply, I was able to narrow down the problematic PCIe device to be a USB controller.

During the course of my troubleshooting, I tried passing through the onboard USB, and two different PCIE USB controller cards, one of which has a Renesas chip, the other ASMedia.

I was not able to get any result other than a blank screen, pre EFI, until I downgraded to 6.5U1, after which time, USB functioned normally.

Devices passed through successfully include Nvidia GPU, Samsung M.2. NVMe, LSI 2308, sound/video capture cards - USB is the hang-up with ESXi 6.7.

If there is a manual parameter that can be input into the VMX file, or in the host advanced system settings to enable USB to function again with ESXi 6.7, I would love to hear it!

[Sinorama]

Thanks for the info. So if trying to passthru my Renesas USB controller is the source of my problem with 6.7 then I am stuck in 6.5 until a fix or I may have to look at a USB over IP solution.

[jUDASmILExAlmig]

Hi petejk, Sinorama

As my trial in house I found this problem may relates to EFI on VM. I have several external XHCI controllers in hand and I can reproduce the black screen when using VM with EFI. But everything will be working well if use VM with BIOS. You may try on your side if this works for you. I'll contact related team to double confirm it. Thanks and have a nice day to you both.

[petejk]

Thanks for taking the time to reply - it's good to know that the issue relates in some way to the VMware virtualized EFI implementation.

For my situation, the VM that I am passing through PCIe devices, inc USB, boots in EFI mode - I have downgraded to 6.5U1g, and recreated my vSAN cluster to allow me to continue working.

I may attempt an upgrade to 6.5U2, with the potential to roll back, but will wait on a 6.7 upgrade until this problem is fixed.

[Sinorama]

I am able to passthru a Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI with EFI boot in a Windows 10 vm on 6.7 Neither this or the bios boot solution is viable for my main computer. Hoping that the Renesas usb passtrhu on EFI vm will work in the next update.

[jUDASmILExAlmig]

Hi Sinorama

The fix has been checked in 6.7U1 then you can upgrade to it once it's available in our website. And thanks for your patience.

[MaartenVM]

I'm hit by this bug as well; could you perhaps give us a rough release date for the 6.7U1 update?

[jUDASmILExAlmig]

Generally VMware follows a 6-month schedule for vSphere update releases. Then vS67U1 is expected to be released around Oct or Nov but still no exact relase date yet.