VMDirectPath and ATI Radeon

[taylorjonl]

I am using VMware ESXi and I am trying to setup a guest that is Windows 7 that will have an ATI Radeon video card passed through to it.  I actually had this working on a previous system but I had to reinstall.  Now when I do this the guest fails to start and I get the following:

Error message from localhost.XXXXXXXXXXX:
PCIPassthru 004:00.0: Guest tried to (null)map
32 device pages (with base address of 0xb5d20)
to a range occupied by main memory. This is
outside of the PCI Hole. Add pciHole.start =
"2909" to the configuration file and then power
on the VM.
error
12/23/2010 1:04:36 PM
media
User

When I do as it asks, the guest now starts but gets an immediate BSOD concerning memory management.  Any ideas on why this is occuring and why it worked at one point but now it fails?

[rmathis1984]

Yup the PCI Pinhole trick will get you key to more RAM on the VM.

ESXi it's self dosent require a card for output when running. At least it wont complain. You'll end up "with the VM powered off" at a point where you can no longer recover the host aka you'll see cnc-register or something like that at boot and it appear that the host isnt booting. At that point the devices has been allocated to a VM which is why passthrough works on a host with a single video card. With your setup having the 2nd video card in, you need to assign it to another VM to remove the host access. Or just dont plug a monitor in.

But I'm going to guess your talking about the console video or the VM video driver... Nope that driver has to stay but it dosent have to be enabled for output. Don't disable it in device manager. All you need to do is set your VM's primary monitor to the video card and you'll be good.

5.1 is still broken with USB passthrough. I tried that the other day but 5.0 u2 while it was unstable at first worked out on a fresh load of the host and I havent turned back since.

[The_Dave]

No, I already set the host console vga driver to not be used by setting only to display the desktop on monitor 2. What I am reffering to is the physical card. You are correct about the message. When I tried assigning the only card in the system at the time (the 5870), ESXi would boot up and I would see the last message show as "cnic_register loaded successfully". IIRC it did not finish loading though. Again IIRC it seemed that the host stopped at that point and I could not connect to it. I will try again tonight or tommorow.

If it works, I can then merge my existing server and gaming tower together until I get some more coin saved to get a new x79 mobo and core i7 with ECC ram together. I can't ditch ZFS and Windows filesystems don't do what I want nor does a raid array under Windows. So now I will have a super fast and resiliant filesystem backing my VM's and providing storage to my lan while still able to manage from a Windows machine and game with!

Is it common for the VM's not to be able to sleep with an assigned VGA card? I guess I should just let the VM run all the time? Not a big deal I guess since the server will still be on anyways.

Thanks all and thanks VMware! I did not want to try and learn/setup Xen under Fedora or something like SmartOS and make its management my 2nd job. ESXi is so easy and great to use!

[derickso]

FWIW Dave, that is exactly what I run. I have a Solaris VM with a passed through HBA and 7 disks running ZFS, exporting CIFS/NFS/iSCSI throughout my house, and to the Win 7 MCE VM that runs TV (which has a passed through Radeon 7750). I did try getting the bleeding edge Xen to work and had problems with passthrough of the HBA into Solaris so I gave up.

[jhusen]

After the cnic_register message is stuck on the console, you still need to wait a few minutes for ESXi to finish booting; there will be no indication of progress. The handoff of pcie control is somewhere in the middle of the bootup procedure. When the card is passed to a guest, the guest takes over the card when powered on and replaces the frozen console screen

Don't know if you've done this or not, but I had to be careful of which host usb controllers I selected for passthrough to VMs. It seems ESXi doesn't like it too much when you passthrough the port that its flash drive is connected to :smileysilly:. I think I had to reinstall ESXi and reconfigure the host after that one. Haha. Now that I have all the "unknown" devices mapped to the corresponding ID, that shouldn't be a problem

[The_Dave]

derickso: Mind if I send you a PM? I don't want to derail the thread but am curious about a few things. I though about including my HTPC (Win 7 - MCE for live tv with HDHomeRun tuners and XBMC for stored content) as a VM but was cautious about stability and such.

jhusen: Maybe I did not give the sytem enough time to do the switch over and was being to impatient Then again, I was doing this until 2 in the morning as I don't get too much time otherwise with the wife and baby. As for passing the wrong USB controller, I did that already. I installed ESXi on a USB stick and the first one I tried was the lucky one with the install on it.

[derickso]

Sure, happy to help.

[ohmibuffet]

Hi shlomiassaf,

I am having the exact same problem as you.  The GPU comes up once in the guest Windows 7 but only ONCE. If I shutdown/reboot the guest, the whole ESXi host hangs.  From one of your later posts, it looks like you got it working?  Can you tell me what steps you did to fit ix? I tried pcihole but it made no difference.

Thanks.

[GBowman]

I had a similar issue, it would not lock up the ESXi host but it would only pass through the card on the first power on of the machine and not on subsequient power ups.  I found an option in a white paper about RAID cards and tried it and it solved the problem for me.  Add the following to your configuration file and try it, obviously with the correct number for your pass through device.

pciPassthru0.msiEnabled = false

I now add this line as a matter of course to all GPU pass throughs and most others.  It will be deleted if you remove and re add the hardware so watch out for that.

On another note I have tested Crossfire on my x58 board and it works!  I have 2x 5770s' working in CF but with a bridge fitted as the drivers would only enable CF with the bridge in place.  I am going to borrow another 6770 as the one I have has no bridge connector so should allow software crossfire.  If this works (I see no reason why it would not) then I can have 2 VM's each with 2 cores and 2gb when needed and change to 1 VM with twice the power for gaming when needed.

[ohmibuffet]

Thanks GBowman, I put in the msiEnabled and it seems to be making a difference, although it's still not fully working.  I added msiEnabled to all the passthrough devices, which includes the onboard USB, the Radeon 5450, and the Radeon Audio device.

What's happening now is that the GPU in the guest boots, the first time as before.  When I reboot the guest, now the host doesn't crash, but the guest gets a blank screen and no screen shows up.  Both the VI client console and the direct HDMI output from the VMDP GPU.

I'm gonna try to and tinker with the different combinations of the pcienabled, putting it on different passthru devices and see if that makes a difference.

Maybe it's my particular video card, a PCIE x1 HIS Radeon HD 5450.  we'll see.

Pretty cool that you got CF working.  My motherboard only has 1 PCIE x16 slot so I can't do CF.  My goal is to get dual GPU's working, a fairly fast GPU in my x16 slot for playing games, then a slower GPU in the x1 slot for everyday stuff.

Thanks

[Deepak_Balaji]

Hi Colleague,

I am on my leave from 4th Feb to 10th Feb 2013, having no access to my mail & phone. On issues/queries please reach my colleagues at 'DL GLOBAL IT LIT SM 2nd level WIN_VM (external)'.

Thank you.

Regards,

K.Deepak Balaji.

[ohmibuffet]

It looks like it's only my secondary GPU that's causing the host to crash on reboot.  The VM using my main PCIE x16 slot Radeon seems to reboot fine.  I'm going to try a different PCIE x1 card in my secondary VM.

[GBowman]

I just tested my HD5450 in an x1 slot, it works perfectly.  This is on my x58 UD3R board but it proves that at least on some systems you can use x1 slots for GPU's.  Performance did not appear to be down either but then the 5450 is not a great performer.  I used a passive PCI x1 to x16 slot converter which just allows you to do this without having to butcher a slot, the card was working at x1 speeds but is a full x16 card.

I would suggest your problem is the card, have you tried the x1 card in the x16 slot?  Just to rule it out as a card issue.

[Zakcar1]

Je suis présentement en vacances. Pour toute urgence, contacter Taidgh Rodeck au 819 363-6133.

Date de retour: 25 fevrier 2013

[ohmibuffet]

I tried it in the x1 slot, x8 slot, and the x16 slot and get the same thing.  I have two other cards for testing, a 3450 and a 7850.  Shortly after my last post, I orderd a x8 to x16 PCI slot adapter from Orbit Micro, so I used that to try my old 3450 in one of the x8 slots and it also worked fine.  The 7850 has no issues either.

I'm thinking either it's a problem with this particular card HIS HD 5450 x1, or the card in combination with my Asus Z8NA-D6C motherboard. I'm going to try it with a different motherboard, an ASRock Z77E-ITX.  If that doesn't work, I'm going to try to return the card.

Searching through earlier posts in this thread, I get conflicting reports with the 5450 so I'm guessing it's possible it's the card.

[GBowman]

Ok just a follow up on my Crossfire success.  It appears it will have no useable merrit in this instance since it will not work without a bridge connector.  I can get 2 6770 cards that do not have the option of a bridge (no pins on PCB) to work when booted natively into Windows but not when using the same 2 cards in a VM on the same machine.  I beleive the problem is that CCC will not enable software crossfire on a system with another type of GPU present so the VMWare GPU device has thwarted me again.  I really wish there was a way to remove this from the virtual hardware.

So my plan to use 2 cards with seperate machines some of the time and then bring them together along with the CPU and RAM resources into one machine has failed.  I can only get hardware (bridged/linked) crossfire to work in a VM.  I have not tried using 2 cards with a bridge connector attached to 2 seperate VM's as I dont want to risk possibly damaging my hardware.  For most of us there will be no use in using 2 PCIe slots for 2 budget cards on one VM when we can use the same 2 slots for good cards serving 2 different VM's.

On a side note not relating to ESXi, Crossfire with no bridge is possible only when the primary card has not got the connector.  This differs to AMD's official list which says both must have the connector or both must not.  If you have one with and one without this will work aswell.

[Guizado]

Channing Benson wrote:

Hi,

I just got it working with CentOS 6.3 on a ASUS Z77 Sabertooth Mobo, ASUS HD6450 graphics card on ESX 5.0.1. The key to getting it to work was to turn of KMS (kernel mode setting) by booting with the option fglrx.modeset=0. Before that the X server would hang, sucking up 100% CPU. For those of you who were getting that message about missing BIOS, I suggest this fix.

Good luck.

I actually have something similar and im hoping someone could point me on the right direction, I think im nearly there but there is something im missing.

When I try the VGA PAssthrough (ATI HD5450) I get the Bios error and the screen does not budge and just stays there with the light flashing with no input, but if i add the "fglrx.modeset=0" on the grub option the screen comes on but stays "Gray" all the time.

Any Ideas??

Many Thanks

[gcg1]

having the exact same issue with NVIDIA quadro 2000.

[nsansari]

Excellent info here. I decided to give this a go with my HP ML110 G7. Got a HD6450 for £20.

I knew it was not going to be straight forward to set this up, but the installation of the card and configuration all went smoothly and I've now got a VM using the HD6450. However the problem I have is I've been unable to get a keyboard recognised. I've tried about 4 different ones without any joy. The mouse works and shows up as HID-compliant mouse and works but not the keyboard. Anyone have any ideas ? much appreciated.

[Guizado]

Nadeem wrote:

Excellent info here. I decided to give this a go with my HP ML110 G7. Got a HD6450 for £20.

I knew it was not going to be straight forward to set this up, but the installation of the card and configuration all went smoothly and I've now got a VM using the HD6450. However the problem I have is I've been unable to get a keyboard recognised. I've tried about 4 different ones without any joy. The mouse works and shows up as HID-compliant mouse and works but not the keyboard. Anyone have any ideas ? much appreciated.

Hi, do you have HDMI AUdio working on that??

I also have a HPML110 G7 with a HD5450, and on Win7 I get VGA PAssthough fine but my HDMI audio has a code 10 on device manager, and on Ubuntu I can only get VGA Passthrough working if I DONT passthrough the HDMI Audio.

Thanks

[nsansari]

Hi Guizado, sorry I don't have a monitor with HDMI to test. The only reason I've done this is so that I can get rid of the separate PC I have. I'm not going to be doing any kind of mulimedia on this. I only use the server / VMs for testing etc. 

What about you have you got a keyboard working passthrough?