VMware Cloud Community
HellMind
Contributor
Contributor

Esxi 5.1 pci passthrough broken

I got a purple screen when I start up a vm with a pci device

http://ft.trillian.im/940e0c6710303f7966cf57a2bcc043251745ed62/6aBIiJHevbPWvWRlZ3Q8XDKJCNbhS.jpg

With Esxi5 was working fine.

I tested it with 2 vm on different hosts ( but same hardware)

http://ft.trillian.im/940e0c6710303f7966cf57a2bcc043251745ed62/6aBIHwqu0xJF5VWcS7HooQWupa9Ly.jpg

http://ft.trillian.im/940e0c6710303f7966cf57a2bcc043251745ed62/6aBIODLiCWadA8FYb57ZlMvwVctSh.jpg

http://ft.trillian.im/940e0c6710303f7966cf57a2bcc043251745ed62/6aBIWlBZlQXgImSaCnmpX2necw5yb.jpg

i7 2600 32gb

mb: dq67sw

What can I try

275 Replies
mlai
Contributor
Contributor

I have having the same problem in trying to pass through asmedia asm1042 usb3.0 hub to vm as well.  Worked flawlessly under 5.0u1!

Please!  Fix this ASAP! 

Reply
0 Kudos
BlackBurn1983
Contributor
Contributor

Got exactly the same problem passing through a simple Firewire card (PCI).

Just downgraded back to 5.0 update 1, a fix would be nice, so I can get the console to work from Windows 8.

Reply
0 Kudos
derickso
Contributor
Contributor

Just adding my voice to this, I would like to upgrade to 5.1 but PCIe pass-through not working is a showstopper for me.

Reply
0 Kudos
THHV
Contributor
Contributor

+1

XEON E3-1230V2 / ASUS P8CWS passthrough "Panther Point AHCI Controller" = ESXi 5.1 crash!

Works in 5.0.0U1

Regards

Thomas Kristensen

Reply
0 Kudos
srwsol
Hot Shot
Hot Shot

Has anyone who has full maintenance opened an incident about this to VMWare?  Unfortunately I have just the basic essentials kit license, so it would cost me $299 to open an incident.  If somebody in this thread who is having this problem can open an incident without charge I highly encourage you to do so.

Scott W.

Reply
0 Kudos
Batmac
Contributor
Contributor

Hello all,

This to confirm that I am also having the purple screen (crash with dump) as soon as I add a PCI and/or PCIe device in passthrough mode to a VM, then launch the VM.

System stable when no passthrough selected however.

Reply
0 Kudos
NealSu
Contributor
Contributor

Confirmed, too.

i've spent more than just two hours. obviously using the wrong keywords searching for others with this issue and more frustratingly even trying to troubleshoot what exactly was the cause for my setup not working anymore. Finally i got the right keyword leding me to this thread! :smileygrin:

i had issues with installing ESXi to flashdrives (USB, SD-card) before, so i thought it was an issue with the bootbank, i believed that it was related to settings/config not saved after reboots after switching from HDD to SSD. Or maybe, because i've been using HP custom image, which had an issue of its own. Also suspected it might be, because i installed over an exisiting installation without prior formating. I've thought of so many different possibilities, that i'm really relieved to see others with this issue.

Anyway, i've successfuly used USB-passthrough with 5.0 u1 and when upgrading to 5.1 (also even after a clean install of 5.1), it is possible to mark USB for passthrough, but after reboot, it's neither available nor marked - i didn't check ESX.conf yet.

i'm using an HP ProLiant ML110 G7. The onboard USB can't be passed through, but the onboard NICs can. Those are listed as "PCI Express Root" though, so it is likely that it really might be only related to PCI and not PCIe.

Would also appreciate a patch for this and am going to switch back to 5.0 u1 for the timebeing.

Reply
0 Kudos
Bembel
Enthusiast
Enthusiast

Hi all,

having exactly the same issue with a simple hardware vendor certified PCI-X SCSI adapter which worked like charm under ESXi 5.0 for months.

If I remove the pass-through device in the VMs configuration I am having no problems at all.

The pass-through device shows up correctly in the host's configuration though.

I have already opened a call at VMware, uploaded logs and screenshots and will be waiting for a "cure".

Until now I have not rolled back to ESXi 5 since I want to be able to reproduce the crash if VMware asks me to.

System:

HP ProLiant DL 360 G6 (latest firmware)

LSI/Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra 320 SCSI with attached HP Autoloader 1/8 G2 tape library

Reply
0 Kudos
srwsol
Hot Shot
Hot Shot

Which issue are you having?  Is ESXi 5.1 crashing when you pass through the SCSI adapter, or does it just disappear from the eligible device list for the VMs after you select it as a pass through device and reboot ESXi?

Reply
0 Kudos
Bembel
Enthusiast
Enthusiast

Since the pass-through device shows up correctly when the ESXi host is running, the host only crashes when a VM is starting with this device configured.

The crash actually happens at 95% of the event "Power On virtual machine".

Reply
0 Kudos
srwsol
Hot Shot
Hot Shot

OK, so you have the crash problem.  There are two kinds of problems being talked about in this thread, the crash problem and the problem where you can't mark a USB controller as a pass through device and have it stick.  As you have an incident open would you mind testing on your setup assigning a USB controller as a pass through device and then if it doesn't stick after the reboot, which it likely won't, adding that to your incident as an issue as well?  That would help the rest of us out who have that problem by making sure that their people look into that at the same time.  I'm not sure if they are both aspects of the same problem, or two seperate software bugs concerning PCI pass through.  Unfortunately, since I just have the essentials bundle it would cost me $299 to open an incident. 

thanks,

Scott W.

Reply
0 Kudos
Bembel
Enthusiast
Enthusiast

Hi Scott,

I just got VMware on the phone - our "faulting" adapter is in the VMware HCL - so the problem will hopefully be solved by an update in the near future.

They'll take a look at our logs first thing tomorrow.

The crashing ESXi host is populated right now and I am on the run, too, I will check this tomorrow after I have evacuated the host and keep you updated.

For my information : what exactly are you trying to do with USB pass-through, using USB-Sticks or HDDs or else in a VM? What kind of device do you want me to test behind the USB-Controller?

Cheers!

Reply
0 Kudos
Batmac
Contributor
Contributor

Seems as though many different configurations are causing the crash. In my case, I use a PCIe SCSI card (LSI) for backups tied to a LT03 on a WinSBS 2008r2 VM (which was working fine with ESXi 5.0U1 with latest patches

the other PCI card is a FireWire 400 (quite old but also was working fine in ESXi 5.0U1) used to connect external drives to a VM.

I can confirm that all hardware that used to be configured with passthrough in 5.0U1 is still available for passthrough in 5.1, and I can edit and change but crashes occur when VM is launched. (regardless of hardware upgrade to vmx "9" or older "8" version

Reply
0 Kudos
Bembel
Enthusiast
Enthusiast

Hi Scott,

I tested it today and I have the same problem. After selecting the first USB Controller (at 00:1d.0) I rebooted the empty host and found the pass-though device missing after the host came up again. This is from our vmkernel.log...:

2012-09-27T08:49:04.930Z cpu2:8673)<6>uhci_hcd 0000:00:1d.0: UHCI Host Controller
2012-09-27T08:49:04.930Z cpu2:8673)<6>uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2
2012-09-27T08:49:04.930Z cpu2:8673)IRQ: 233: 0xc0 <uhci_hcd:usb2> sharable (entropy source), flags 0x10
2012-09-27T08:49:04.930Z cpu2:8673)VMK_VECTOR: 138: Added handler for shared vector 192, flags 0x10
2012-09-27T08:49:04.930Z cpu2:8673)<6>uhci_hcd 0000:00:1d.0: irq 192, io base 0x00001000
2012-09-27T08:49:04.930Z cpu2:8673)<6>usb usb2: New USB device found, idVendor=1d6b, idProduct=0001
2012-09-27T08:49:04.930Z cpu2:8673)<6>usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
2012-09-27T08:49:04.930Z cpu2:8673)<6>usb usb2: Product: UHCI Host Controller
2012-09-27T08:49:04.930Z cpu2:8673)<6>usb usb2: Manufacturer: vmklinux_9  uhci_hcd
2012-09-27T08:49:04.930Z cpu2:8673)<6>usb usb2: SerialNumber: 0000:00:1d.0
2012-09-27T08:49:04.930Z cpu2:8673)<6>hub 2-0:1.0: USB hub found
2012-09-27T08:49:04.930Z cpu2:8673)<6>hub 2-0:1.0: 2 ports detected
2012-09-27T08:49:04.930Z cpu2:8673)<6>hub 2-0:1.0: interface is claimed by hub
2012-09-27T08:49:04.930Z cpu2:8673)<6>usb usb2: device is not available for passthrough
2012-09-27T08:49:04.930Z cpu2:8673)<6>usb usb2: usbfs: registered usb0201
2012-09-27T08:49:04.930Z cpu2:8673)PCI: driver uhci_hcd claimed device 0000:00:1d.0

We de not use USB pass-through so I cannot say whether it worked under ESXi 5.0 but I will update my case at VMware to include this problem.

Cheers!

Reply
0 Kudos
Bembel
Enthusiast
Enthusiast

Hi all,

I got a call back today from VMware - it looks like non PCIe adapters have not throughly been tested, since this is a rare configuration in an enterprise environment (I will decommission the adapter in the near future, too). VMware is aware of the PCIe pass-through issue and will release a patch hopefully until the end of the year for this. While there will be an express patch coming out really soon, it will not include a fix for our PCIe pass-through problem.

When it comes to the USB pass-through - and since I did not use USB pass-through at all - I was recommended using the USB configuration within a VM and not pass-through the controller directly. Is there a configuration where pass-through for USB makes more sense than using the VM configuration for USB anyways?

To be able to use my backup again I ordered an Adaptec 29320LPE (PCIe Ultra320) which is also on VMware's HCL and hope that this solves my issue.

Cheers,

Bembel

Reply
0 Kudos
fatality88
Contributor
Contributor

Seems to be all the rage these days - releasing software that isn't up for the job...

Apple - iOS 6

VMWare - ESXi 5.1

Microsoft - don't get me started...

Reply
0 Kudos
srwsol
Hot Shot
Hot Shot

Hi Bembel:

Thanks for bringing up the USB thing with VMWare, much appreciated.   It soulds like from what they told you to do that they may be considering disallowing the passing of a USB controller to the VM, but I sure hope not.  The reason for passing the whole USB controller to the the VM is performance.  If you are using a slower device, such as a modem or dongle, it doesn't make any difference; but if you are using hard disks, especially USB 3 hard disks it makes a huge difference.

If I connect a USB 2 or USB 3 hard disk to a VM through the ESXi USB facility where ESXi is still controlling the device, at best I get USB 1 performence.  However, if I pass the whole USB controller to the VM then I can get pretty much native performance.  In my case, under ESXi 5.0u1 I'm passing a USB 3 controller with a USB 3 drive attached to a VM, which is using the drive as a backup location.  At native USB 3 speeds the backup finishes in an hour or so.  If I leave the USB 3 controller with ESXi and just connect the USB device to the VM, the same backup takes the better part of a day to run.

Bottom line is that ESXi's virtualization of a USB controller leaves a lot be desired performance wise, and if you have VMs that have significant quantities of data to backup, you are pretty much required to do it this way, or try to pass it all over the network. 

Scott W.

Reply
0 Kudos
sofakng
Contributor
Contributor

Another reason for USB controller pass-through is that certain devices do not work properly using USB arbitration (i.e. video capture cards, DVB capture cards, etc).

Hopefully VMware gets this fixed...

Reply
0 Kudos
yozh
Enthusiast
Enthusiast

Same issue PCI passthru crashes ESXi 5.1, worked fine on 5.0 U1 Can`t believe I have to roll back now...

Reply
0 Kudos
FastLaneJB
Contributor
Contributor

srwsol wrote:

The reason for passing the whole USB controller to the the VM is performance.

I have another reason for USB passthrough. If you've already passed through a PCI/PCIe device that you can no longer modify the hardware on that VM without shutting it down. Where as if you've passed a USB controller through you can plug and unplug USB devices from the VM while it's online just as if it was a physical machine. No need to go into vSphere Client and no need to turn the machine off to do it.

I thought from the fact that the USB device vanished perfectly on reboot that they'd probably done this on purpose rather than it being a bug. So if that's the case and we cannot convience them to change their minds this won't ever get fixed.

Be nice to get an offical statement from VMWare on this. I don't remember reading that they'd changed this behaviour in their release notes.

Reply
0 Kudos