Esxi 5.1 pci passthrough broken

[HellMind]

I got a purple screen when I start up a vm with a pci device

With Esxi5 was working fine.

I tested it with 2 vm on different hosts ( but same hardware)

i7 2600 32gb

mb: dq67sw

What can I try

[madmax14304]

Has anyone tried USB Passthrough on esxi 5.5?  Not sure how long we can all stick with v5.0u2!

[derickso]

Is it downloadable anywhere? I'd give it a try

[madmax14304]

Oh, good question.  I should have looked before speaking up.  I've been watching all these youtube videos and reading blogs about it and just assumed it was available to the public.

[gerdesj]

I've been recently afflicted with this PCI passthrough stuff on one box (my smart new home ESXi) and the current state of play is:

ESXi 5.1U1 with all current patches - build 1157734

PCIe card passed through to a VM OK - it's a Sangoma telephony card 2 x FXO ports

DVBS2 USB dongle fails by trying to pass through the USB controller

That's exactly how it should be judging by previous comments.  However I did some playing with ordinary USB pass through before I bit the bullet and wiped the box to install 5.0.

Initially I accidentally ticked the option for the USB device to follow the VM through a vMotion and performance was rather bad.  Force of habit - I have many UPSs on the end of this style of connection for monitoring and it works well but a UPS doesn't exactly tax the USB redirector. I have now changed the connection to normal - ie add a host USB device to a VM without using vmdirectpath. 

Performance is excellent for this particular use case - DVBS2 USB 2.0 dongle (Pinnacle PCTV 460e) connected to a Linux VM running a MythTV backend.  From my front end I can watch full HD with no tearing or any other problems and crystal clear 5.1 audio.

I have not benchmarked anything and I won't ever be running a hard disk through this style of connection any time soon but it is certainly adequate for DVB receivers.

[pbraren]

My testing on ESXi 5.5 is going well, as far as passthrough.  Too early to rejoice yet, because I only tested my USB 3.0 card so far, with passthrough working fine.  It had worked on 4.1 and 5.0, but was broken on 5.1. The next hurtle is getting the GA level code, and public release, and sharing my experiences at TinkerTry.com.

[derickso]

Can you try your onboard USB?

[derickso]

5.5 is now GA and downloadable here: https://my.vmware.com/web/vmware/details?downloadGroup=ESXI550&productId=352&rPId=4260

Anyone try it yet to find out if motherboard USB pass through works and remains working through reboots of the server?

[gerdesj]

I've got as far as updating our vCenter and UM to 5.5. 

Just doing our sacrificial ESXi to make sure that all is well with our environment and then I will do a host that could do with the full USB bus passthrough. 

Should be doing that in an hour or two.

Cheers

Jon

[jgkurz]

I'm looking forward to your response. Thanks for taking the time to test.

[pbraren]

I'm happy to notice that things are looking quite good, as far as pass through under ESXi 5.5 (that still requires reboot to enable). More articles will be published about this soon. Meanwhile, here's a USB 3.0 passthrough discussion with screenshots.

[gerdesj]

I think I must have passed the wrong bus through, still its looking good so far:

---------------------8<-----------------------

02:00.0 USB controller: VMware USB1.1 UHCI Controller

02:02.0 USB controller: VMware USB2 EHCI Controller

03:00.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)

0b:00.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #1 (rev 05)

---------------------8<-----------------------

The last entry is an entire USB bus passed through - that is a dmesg on a Linux VM.  This is after an allocation and one reboot.  Note that it is an Intel USB controller and not a VMware one (which it also has via guest hardware - I'll remove that if the full pass through works OK)

I'll post back once I've passed the other bus through which should have the device I actually want on it and then I'll reboot the ESXi again to ensure it remains connected to the VM.

First I have to go home and sort out another DNS server for the wife to get to FB 😎

Cheers

Jon

[gerdesj]

Oh well it seemed too good to be true.  The bus appears in the VM but no devices come with it - at least for me anyway.  I tried a Linux and a Windows 2008R2 guest and both of my USB buses on a Dell T320.

Back to standard passthrough for now.

Cheers

Jon

[madmax14304]

Has anyone else tried it with onboard USB?

[derickso]

Jon:

Just to be clear, you passed through the USB controller to the VM, but within the VM the OS didn't see the controller? Or the devices plugged into the controller did not work? Did it work after the first reboot, but not subsequent ones?

Thanks,

David

[gerdesj]

>>Just to be clear, you passed through the USB controller to the VM, but within the VM the OS didn't see the controller?  Or the devices plugged into the controller did not work?  Did it work after the first reboot, but not subsequent ones?

This is exactly what I did:

Update to 5.5 (takes quite a while incidentally)

Remove all USB reassignments to VMs

Use lspci -v at the ESXi console to verify which bus is which (I have two) and that the device is attached

Pass through the USB controller's PCI device

Reboot

Add a PCI device to a VM

Start up the VM

The VM could see the controller but not devices attached to it.  In a Windows VM it came up in device manager with a yellow exclamation mark and an error saying it could not start.  Under Linux I could see kernel modules attached to the bus (lspci -k) and it listed in lsusb and dmesg (see above) but I could not see anything attached to the bus.

I then undid all of that and used ordinary single USB device passthrough to connect a DVBS2 USB device to my MythTV backend again.  Performance seems OK for that.

I have not benchmarked throughput across the USB bus - I don't have a USB HD to hand.

I wouldn't give up hope quite yet though - mine is only one data point!

Cheers

Jon

PS Another VM with a "real" PCI device (Sangoma FXO card) passed through is OK

[derickso]

Got it, thanks Jon.

As another anecdotal performance piece, I was doing pass through of some USB 2.0 enclosed HDs and found I could only get ~7MB/s of transfer rate through them when passed through via ESXi 5, when I passed through the USB controller itself I could max out the USB 2.0 link at ~35MB/s.

[mgrugel]

@derickso

This is no surprise.

Can anyone positive verify that the PCI passthrough of the USB bus is working correctly under ESXi 5.5 and that it is reboot stable?

[pbraren]

Not sure about motherboard based USB 3.0.  But as far as my PCIe card with 4 USB ports that can be individually chosen for passthrough, yes, it's stable, and reboots don't PSOD.

TinkerTry.com/vmware-vsphere-5-5-announced-with-general-availability-likely-the-week-of-september-9-...

I'll be writing more about this soon, but thankfully, it does appear that for this system that worked fine for pass through with 4.1 and 5.0, but not for 5.1, stability is back on 5.5, at least for this particular device.  Hope this helps!

[AntonZ201110141]

just checked. passthrough is working, but:

1. even after driver installation inside Guest OS - USB device still is yellow ("windows can't use it")

2. the weird thing is it was successfully detached from ESXi host, but I CANNOT attach it back! After rebooting the host USB controllers are still in passthrough mode! So be careful!

*Dell PowerEdge R620, internal USB ports

[srwsol]

If you have the yellow exclamation mark in windows try adding "pciPassthru0.msiEnabled=False" to your configuration for the VM.  You have to do this after you add the controller to the VM, and each time you remove and reinstall the controller to the VM.  Also if you have more than one PCI pass through device in that VM you might have the 0 to some other number.  Now, it could also be that there are some existing issues with the onboard controller support as I can't properly use the onboard usb 3 controller on the system which I have working with an added USB 3 controller under the version of 5.1 which requires the re-enabling of the device after every reboot.