VMware Cloud Community
HellMind
Contributor
Contributor

Esxi 5.1 pci passthrough broken

I got a purple screen when I start up a vm with a pci device

http://ft.trillian.im/940e0c6710303f7966cf57a2bcc043251745ed62/6aBIiJHevbPWvWRlZ3Q8XDKJCNbhS.jpg

With Esxi5 was working fine.

I tested it with 2 vm on different hosts ( but same hardware)

http://ft.trillian.im/940e0c6710303f7966cf57a2bcc043251745ed62/6aBIHwqu0xJF5VWcS7HooQWupa9Ly.jpg

http://ft.trillian.im/940e0c6710303f7966cf57a2bcc043251745ed62/6aBIODLiCWadA8FYb57ZlMvwVctSh.jpg

http://ft.trillian.im/940e0c6710303f7966cf57a2bcc043251745ed62/6aBIWlBZlQXgImSaCnmpX2necw5yb.jpg

i7 2600 32gb

mb: dq67sw

What can I try

275 Replies
derickso
Contributor
Contributor

You can't add a pass through PCI device while the VM is on, at least you cannot in 5.0.

Just so I am clear, you were passing through the entire USB controller (as a PCI pass through device) to the VM(s), correct? You were not just passing through an already listed attached USB device?

Reply
0 Kudos
vt-vmwaresjo
Contributor
Contributor

For me it sounds very promising so far, I'll update my HP Image with this patch and give it another try monday night on my last two esxi 5.0 hosts. We need it for some kind of security token which refuses to work with the normal USB Passtthrough, that was our reason why we needed to passthrough the whole usb controller.

Reply
0 Kudos
xnisupport
Contributor
Contributor

derickso,

Yes, I'm passing through the controller, not editing settings on a virtual machine, adding a USB controller then adding a virutalized USB device. The controller is added as a PCI device and then whatever is hooked up via that controller is then available to the machine directly.

Reply
0 Kudos
bpedman
Contributor
Contributor

Just confirmed that the latest patch fixes the USB controller passthrough for me as well. So excited this is finally working.

Reply
0 Kudos
derickso
Contributor
Contributor

So with these two passthrough issues fixed, are there any remaining reasons not to upgrade from 5.0?

Reply
0 Kudos
srwsol
Hot Shot
Hot Shot

It's not working for me.  After I upgraded I noticed that the USB controller that was assigned to a VM wasn't showing any longer.  I went to the config screen and added it as a PCI pass through device, but after I completed the dialog box the device didn't show.  It's basically the same behavior as before, except now the device is removed right after the dialog box rather than after the reboot.    Here are screen shots:

PassthruConfig1.jpgPassthruConfig2.jpgPassthruConfig3.jpgPassthruConfig4.jpg

Reply
0 Kudos
srwsol
Hot Shot
Hot Shot

OK.  Some updates to my previous post.  Turns out it does work, but there are some issues regarding either the vsphere client and/or vcenter.  I made the changes in my previous post from a vcenter server (the latest version).  When I logged into the host directly instead of going through vcenter it showed that the USB 3 controller was indeed marked as a PCI passthrough device even though at the same time the vsphere client connected to vcenter was still showing the above behavior.  After rebooting the system the USB 3 controller was available for for VMs, but the configuration screen, both in a vcenter connected vsphere client and a directly connected vsphere client showed that a reboot was still necessary to use the device.  However, when I brought up the VM it was able to use the USB 3 controller anyway.  Also, if I opened the dialog box for the PCI Pass through devices it showed as not being checked even though it was usable.  If I check it again and hit OK (note that it's currently in use by a running VM), it stilll shows as needing a reboot for about 20 seconds and then suddenly changed to showing a normal status.  I tried bringing up the vcenter web client and in that client the device showed a correct status (before I did the above with the vsphere vcenter connected client).

It looks like there are some presentation issues with either vcenter, the vsphere client, or both.

Now for the good news.  Every test I've run with the USB 3 controller inside the VM has worked just fine I've connected and disconnected hard drives and a card reader with a camara flash card in it, and they all work just fine in the VM.   I also tried moving the USB 3 controller to a WinXP VM and it worked there as long as I set "pciPassthru0.msiEnabled" to "False", which was the same behavior as in the previous version.  One thing you should do first before you upgrade from 5.0 to 5.1 is to remove the USB 3 controller from the VM using it because, at least in my case, the controller was identified differently than it was under 5.0, so if you leave it in the VM from 5.0 it shows up as a non-existent device and you have to remove and re-install it anyway.  I also noticed that version 5.1 seems to do a better job in correctly identifying the devices on the PCI bus.  Under 5.0 there were a couple of devices showing as unknown, which were the built-in combo USB controllers (I'm using an add-in USB 3 controller because under 5.0 I couldn't seperate out the two on board usb controllers without eliminating the ability to connect a keyboard and mouse to the host, although since they are now both identified I might be able to under 5.1, but I haven't tested that yet).

Reply
0 Kudos
NexcesThreed
Contributor
Contributor

Hi to all,

I was watching this thread for some time with the same issues. With the latest 1021289 patch I can confirm everything that srwsol wrote.

I have a USB 3.0 (Renesas Electronics - NEC) MB integrated controller now correctly passed to a VM. This was the first (USB controller) I tried, since all the other 8 show up as "USB Controller", and it would take some time to identify which ports are they conected to.

My system is also exhibiting the "must reboot system" message, but after marking the device for passthrough in the configuration and pressing ok, the controller shows as ok (green dot).

Regardles of the above status (must reboot or green dot), the VM correctly connects to the controller every time and can use it at USB 3.0 speeds.

After the update, I was also able to passthrough the (MB integrated) JMicros JMB 363 eSATA controller to a VM. This one is not exibiting any of the problems as the USB 3.0 controller above and works great. With the 914609 build this was not possible.

Unfortunately, Marvell 88SE9128 (MB integrated) and 88SE9120 (PCIe add-on) controllers are still not recognized by the VM, even though they seem to be correctly setup for passthrough.

Just my 0.02€.

BR

Reply
0 Kudos
xnisupport
Contributor
Contributor

Mine did what yours did, after reboot being there and being usable but saying it needed a reboot. When I clicked edit it showed it as not checked even though it all worked. I re-checked it, clicked OK. It was still usable and the reboot request went away and it showed all green.

As a negative I found that the machines I passed through the controller, I can't remove snapshots without shutting them down.

Reply
0 Kudos
vt-vmwaresjo
Contributor
Contributor

I think this is normal behavior with the snapshots, since its also not possible under ESXi 5.0 to take (at least quiesced) snapshots from running VMs which have pci passthrough devices attached.

Reply
0 Kudos
srwsol
Hot Shot
Hot Shot

I've found out some more details about what's apparently going on with the USB controller showing needing a reboot.  If you shutdown ESXi 5.1 with the USB controller showing that it needs a reboot, the device is removed from eligible PCI passthrough upon reboot.  If you edit the device a second time and mark the check box a second time, such that it no longer says it needs a reboot, the device will survive as a pci passthrough device for one reboot, after which it will revert to the needing a reboot again and with the checkmark removed from the dialog box.  If you reboot with it in that condition again, it will again be removed.  In order to keep the device as pass through eligible you have to remark the check box after each reboot.

It looks like what's happening here is that there must have been either two places in the code where they were removing USB controllers from PCI passthrough (one before the PCI devices were enumerated and a decision was made about their pass through status, and then again someplace later after the decision was made), or they moved the removal code from a place early in the startup to a place after the PCI devices were enumerated and the decision was made, to a place later on in the startup sequence which now makes the change happen after the next boot.

There's no way to tell which thing happened, but the net result is that now someplace after the PCI devices have been enumerated and the decision as to their pass through status for the current boot has been made, some code is marking any USB controllers that are available for pass through in the current boot to be removed at the next reboot.  By going into the dialog box and putting the check mark back in, you undo the change that code made, and the device will remain until after the next boot at which point it is again marked for removal.

I'm not sure if they deliberately attempted to fix this, but didn't realize that the code was in two places, or if they reordered something in the boot sequence that had the side effect of moving the single instance of the code that removed USB controllers marked as pass through devices to a point after the decision about pass through devices for the current boot had already been made, thus making the device available for a single boot, but marked for removal upon at the next reboot.

Can someone who has an open incident on this issue share my theory with the tech support people?  My concern is that if this was unintentional, at the next patch, if they tweak the order of what happens at boot time again, we could be back to where we were before this patch.

Reply
0 Kudos
GabrielMoga
Contributor
Contributor

Another confirmation: updated one of my host and the USB controllers still present in the passthrough list after host rebooted. The host does not show any reboot needed.

Reply
0 Kudos
derickso
Contributor
Contributor

For those reporting success, have you tried rebooting twice (without touching anything in the settings), just to be positive it still works?

Reply
0 Kudos
GabrielMoga
Contributor
Contributor

😞 ... you are right... after the second restart the controllers have disappeared ... not good!

Reply
0 Kudos
srwsol
Hot Shot
Hot Shot

I spent most of the evening last night doing testing, and at least on my system the behavior reported above is what's happening.  This morning I tried running /sbin/auto-backup.sh after replacing the check mark on the USB controller, as I had read in another thread about an issue where configuration changes were being lost due to state.tgz (where the backup goes) not being updated.  After reboot the controller was still marked for removal.  However, I did get a strange message when running the backup concerning dvstata.db, which I'm not sure what it contains or if it was backed up:

/vmfs/volumes/085fdc78-6ad9fd71-75f6-e75133552bbc # /sbin/auto-backup.sh
Files /etc/vmware/dvsdata.db and /tmp/auto-backup.31640//etc/vmware/dvsdata.db differ
Saving current state in /bootbank
Clock updated.
Time: 13:55:43   Date: 03/11/2013   UTC

Reply
0 Kudos
srwsol
Hot Shot
Hot Shot

Does anyone know if there is a way to set a PCI device as pass through enabled via the command line?  So far I haven't been able to find any reference to the ability to do that via ESXCLI.  All the examples show it being done from either the vsphere client or the web client.  If there is a command line way to do it, then this problem can be circumvented by putting the command in a startup script.

Reply
0 Kudos
stevedparry
Contributor
Contributor

Hi,

I've just uprgraded to 5.1 and like you guys, Im suffering from the inabilty to pass through a usb adapter. This thread hasnt had any activity in a few weeks and Im wondering what people have done about this bug. Is there a patch I can apply are Im I going to have to revert to 5.0u1?

Cheers

Reply
0 Kudos
MSTROWD55
Contributor
Contributor

As far as I know you will have to revert

Sent from my Verizon Wireless 4G LTE DROID

Reply
0 Kudos
srwsol
Hot Shot
Hot Shot

As I posted above, after the last patch you can make it work, but the problem is that after each ESXi reboot you have to go back into the settings and recheck the usb controller as being allowed for USB pass through, otherwise on the next reboot it will be gone again.  Go back a few pages and you can read the full explanation.

Reply
0 Kudos
Mario1976
Contributor
Contributor

Currently we're back at the latest ESXi 5.0 patch and are waiting for progress with our call we have open.

The latest patch for 5.1 reenables usb host controller passthrough, but it only survives one reboot if you don't reenable the the host Controller for passthrough after each reboot.

so to be on the safe side I'll also recommend going back to 5.0 until this is sorted out.

Reply
0 Kudos