Esxi 5.1 pci passthrough broken

[HellMind]

I got a purple screen when I start up a vm with a pci device

With Esxi5 was working fine.

I tested it with 2 vm on different hosts ( but same hardware)

i7 2600 32gb

mb: dq67sw

What can I try

[Vith]

My experience with ESXI 5.5 and the motherboard P9X79 PRO (socket 2011, 64gb ram), tested with a windows8 virtual machine, is:

-In Esxi 5.1 the machine would freeze for several minutes when booting, that didn't happen in 5.0 and it's been fixed in 5.5.

-Pass-through devices don't disappear when rebooting (I rebooted several times, tested with all the devices)

-The unofficial driver for the nic 82579V (net-e1001e-1.0.0.x86_64.vib) works fine. Vsphere client is crashing a lot (it didn't happen in 5.1), and I think it's related to the network, or maybe it's because I'm only using 1 nic. Because of that I couldn't test this in passthrough.

The following is the onboard hardware of this motherboard:

• Onboard usb 3.0 (ASMedia ASM1042 SuperSpeed USB Host Controller): passthrough works fine with both of them, the virtual machine is able to see them, no errors and it can use the devices you connect (tested with a pendrive and a keyboard)
• Onboard usb 2.0 (Intel C600/X79 series chipset usb2 Enhanced Host Controller): the device shows up in the virtual machine but with a yellow exclamation mark and can't be used. When not in passthrough, connecting devices to the host and adding them to virtual machines works fine (this can't be done with HID devices, as usual). In this mode you are able to add the onboard bluetooth to a virtual machine, and it works.
• Intel Pastburg 6 Port SATA ACHI: I can't test this in passthrough because I installed ESXI on it. It worked in 5.0 and 5.1, so it should be fine.
• ASMedia ASM 1062 Serial ATA Controller (esata): the virtual machine is able to see it, but with a yellow exclamation mark and can't be used. It doesn't work. When not in passthrough, the host doesn't see it and can't be used.
• Marvell P8P67 (sata): the virtual machine is not able to see anything. I tried with a debian virtual machine, and it gives a kernel panic when booting (the vm). Not sure what's wrong with this one, I've read it may be a bug in the hardware/firmware.
• Intel C600/X79 series chipset High Definition Audio Controller: passthrough works. I was experiencing freezes and the sound was barely playing even after installing the drivers. After removing all the other devices, it started working, but the sound was a bit choppy. Then I disabled speedstep and all the power management, attached a keyboard and mouse using the usb3 passthrough and added a videocard. The sound was perfect after that. Not sure exactly what set of changes fixed the sound, but the important thing is it can be fixed.

Also, my Raden HD 5870 works fine in passthrough. Tested with a game and the performance was great, with no errors at all. Video acceleration doesn't work in firefox/ie (but works in chrome) when playing a flash/html5 video with acceleration on. This may be related to the virtual svga.

To sum up, the marvell and esata controllers don't work. The nic, as anything with unofficial drivers, can stop working in future versions. USB3 works. USB2 doesn't work, but it can be used with the host, with the knowledge that no keyboard/mouse can be added to a virtual machine.

Edit: if the vm uses EFI (version 8, I don't know if this is important), esata and usb 2.0 work, although the usb is giving me problems. Read the following posts.

[derickso]

Thanks for the indepth response! Weird that onboard USB3 works but not 2, I wonder if there is some way to figure out what ESX 5.5 is doing differently via running lspci within the VM on each ESXi version.

[madmax14304]

Well, that was a total disappointment. I had the same experience with my usb 2.0 bus. It added to the vm but stayed yellow. pciPassthru0.msiEnabled="False" didn't fix it.  Now I'm debating rolling back.

[jgkurz]

The good news: I upgraded to 5.5 and now my USB 3.0 adapter works. I connected it to my Win7 x64 VM and so far it's functioning as expected.

My adapter is a NEC based chipset marketed as the StarTech 2 Port PCI Express SuperSpeed USB 3.0 Card Adapter Model PEXUSB3S2

The bad news: I am a home user and have uncovered an unfortunate issue with the free version of ESXi 5.5. vSphere Client no longer supports editing VM settings of a version 10 machine. You must use vSphere Web Client which requires vCenter. I downloaded a trial copy of vCenter Appliance and vSphere Enterprise Plus so that I could make the required pass-thru edit from vSphere Web Client.

I'm not sure how I'm going to administrate my host after my eval licenses expire.. : (

[Komandor89]

Hello, guys. I had the same problem on ESXi 5.5 with my onboard USB 2.0 Controller (ASRock P67 Fatal1ty Professional) - error with code 10. My solution is using EFI boot firmware instead BIOS for virtual machines that use my passthroughed internal USB controller. I hope, that this will help to other users.

[pbraren]

The good news: I upgraded to 5.5 and now my USB 3.0 adapter works. I connected it to my Win7 x64 VM and so far it's functioning as expected

My adapter is a NEC based chipset marketed as the StarTech 2 Port PCI Express SuperSpeed USB 3.0 Card Adapter Model PEXUSB3S2

The bad news: I am a home user and have uncovered an unfortunate issue with the free version of ESXi 5.5. vSphere Client no longer supports editing VM settings of a version 10 machine. You must use vSphere Web Client which requires vCenter. I downloaded a trial copy of vCenter Appliance and vSphere Enterprise Plus so that I could make the required pass-thru edit from vSphere Web Client.\

I'm not sure how I'm going to administrate my host after my eval licenses expire.. : (

jgkurz, well said, took the words right out of my mouth!

So I quoted your eloquent words in an article I just published last night:

TinkerTry.com/best-parts-of-vmwares-esxi-5-5-free-hypervisor-rely-on-vcenter-which-isnt-free-uh-oh

[jgkurz]

No problem. Glad I could help. Your write-up on Tinkertry certainly expands on the issue. I hope this is not a conscience attempt by VMware to eliminate the free license option. At this point it seems like that is the case.

My apologies to everyone for going off topic.

[derickso]

So just to confirm, when you set your VMs to use the EFI bios the onboard USB2.0 controller works correctly when passed through to them?

[Komandor89]

Yes. It solves my problem. Of course I had to convert my Windows 8 VM installation from BIOS to EFI, but after that it is working with GPU & USB passthrough. In ESXi 5.0 it works out-of-box with BIOS firmware, and with ESXi 5.1 USB controllers passthrough does not work anywhere.

[Vith]

I also confirm that usb 2.0 is working for me after reinstalling windows 8 with EFI. Not only that, but my esata (ASMedia ASM1062) is also being properly recognized. I don't have any esata device so I can't really know if it actually works. Marvell controller is the only hardware the P9X79PRO has that can't be used with esxi, and I think it's due to hardware/firmware issues. It doesn't really matter, it's too slow anyway.

However, the lack of a free vcenter version and the deprecation of vsphere client worries me. Even if it was free, the high memory requirements could be a problem for the home user, but it would be better than not having anything at all.

[dd3]

For me no success.

Adding ATI 7790 card to VM as PCI device is ok, but when i try to passthru onboard USB or onboard audio it throws a "PCI passthrough device ID (0x-73d3) is invalid." error on me (in web client). VM is win 8, vm version 10 with EFI boot.

Before when i tried it with classic client, it could be added but onboard usb in win 8 (vm version 8, bios boot) ended in code 10 error and exclamation mark. Onboard audio and GPU worked.

Host is a i5-4560 whitebox, Asrock z87 extreme 4 board.

[Vith]

My usb 2.0 didn't show any exclamation mark but nothing worked. After reconnecting everything, rebooting, adding/removing other passthrough cards and using "rescan hardware" several times, more and more things began being recognized until they all worked fine (pendrive, keyboard and mouse). I don't really know what to say other than it seems a bit unstable. I'm using vm version 8 with EFI, if it helps.

Edit:

This is how it works:

-Booting without any usb devices: it won't detect anything you connect until you "scan for new hardware" in device manager. After that, the keyboard and pendrive work, but the generic mouse doesn't. If I connect a different one (logitech mx518), it works, and the generic mouse start working after this point. You can then unplug and plug back everything as many times as you want, everything will keep working.

-Booting with at least one usb device (any device other than the generic mouse): same result as above after scanning for new hardware. It's just this time nothing has to be scanned because I suppose that step is part of the booting sequence.

So, before it works, it has to see at least one device when scanning, and it can't be my "generic mouse"

[derickso]

Has anyone tried a Win 7 or Linux VM with/without EFI? I'm curious if this is a Windows 8 specific problem.

[Vith]

I'm going to try windows7 and ubuntu now, we'll see how it goes.

Edit: usb 2.0 doesn't work in any of those systems. I used BIOS and version 8. In windows you get an exclamation mark with code 10, in ubuntu I didn't look too hard. The hardware is shown in lspci, but it doesn't work, at least not out of the box. I don't have time to test them with EFI at the moment.

[dd3]

@vith

Thanks for your tips. Now it works.

I made a new win 8 vm, vmx-8, efi, and passed thru both lynx point usb controlers, ati 7790 + hdmi sound, onboard audio, onboard i217v. For +2gb ram i needed to insert pciHole.start = 1200 and pciHole.end = 2200 lines into config.

[jangjong]

hi, long time lurker here.

I am having a slight problem with 5.5 and pci passthrough.

I have a Sil 3132 SATA controller card from monoprice (http://www.monoprice.com/Product/?p_id=2530)

In 5.1, I was able to passthrough this card to a Linux VM and able to access hdd's connected to this card.

However, after 5.5 i am not able to do so in any of my Linux VM's.

When I add the card to a linux vm and boot the vm, it gets bunch of these lines: sata_sil24: IRQ status == 0xffffffff, PCI fault or device removal?

and the esxi machine gives me a PSOD. (http://i.imgur.com/OirZghS.jpg?1)

I know PSOD is yelling something about rtl8168 (LAN Controller).. but when i remove the sata card from the vm, the vm boots up just fine.

Also, Windows 7 vm is able to load this card without any problem, so i don't think the card is defective or anything

I did some research and found out about this: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=205693...

"SiI 3132 [SATALink/SATARaid] Serial ATA Controller" is listed in the list..

So.. does anyone have Sil3132 sata controller card and is doing hw passthrough? any luck? am i just going crazy?

[madmax14304]

My motherboard has onboard USB 3.0 which I've never been able to passthrough.  With esxi 5.5, I am able to see the host detect it using dmesg.

2013-09-26T15:40:50.283Z cpu3:33332)<6> 0000:00:14.0: USB3.0 ports routing map: 0x0 (was x0f)

But this device does not show up as available passthrough devices like the onboard USB 2.0 devices do.  Does anyone have experience in forcing a device to be passed through?

Thanks

[srwsol]

I've not had good luck with onboard USB 3 controllers, even with the version of 5.1 that lets pass through work for one boot.  I tried it on an Intel DQ77KB motherboard and I couldn't get it to work right.  It seemed that even though there were two controllers on that motherboard and I only tried passing through one of them, all of the USB 3 ports became inaccessable to ESXi no matter which controller I tried (i.e. no keyboard and mouse), and also none of the ports ever became available to the VM.   I ended up buying a Startech mini pci express USB 3 add on card and that worked fine.

I haven't tried 5.5 yet, as I won't be back to where that computer is for several months, so I won't get to try it until near the end of the year.

[r0nn]

@dd3

I'm using the same z87 ex4 MB as yours, same build vm, but I can passthrough audio and usb #2 (usb3.0 ports) at the same time, otherwise windows won't boot, and usb 3.0 ports only work as usb 2.0 speed

Is there any way to wake up windows guest using mouse/keyboard connect to passthroughed usb controll ? I have to grab mouse in vsphere client vm console to wake up windows guest

[dd3]

@r0nn

For win8 vm a i have configured passthrough for usb#1, usb#2, onboard audio, gpu audio and gpu, at the same time. All working (dont know if realy usb3 speed, but will try) except front usb 3 ports, which are on asmedia usb controler (2x20 pin headers) and there is no option to pass them thru. VM has all memory reserved (4gb) and for gpu working i put pciHole.start=1200 and end=2200 into its vmx.

I also cant wake up vm with attached mouse/keyboard, so turned it off. But you can try wake on lan, it should work in win quests with vmxnet3 vnics.

Message was edited by: dd3, added memory reservation and pcihole