VMDirectPath and ATI Radeon

[taylorjonl]

I am using VMware ESXi and I am trying to setup a guest that is Windows 7 that will have an ATI Radeon video card passed through to it.  I actually had this working on a previous system but I had to reinstall.  Now when I do this the guest fails to start and I get the following:

Error message from localhost.XXXXXXXXXXX:
PCIPassthru 004:00.0: Guest tried to (null)map
32 device pages (with base address of 0xb5d20)
to a range occupied by main memory. This is
outside of the PCI Hole. Add pciHole.start =
"2909" to the configuration file and then power
on the VM.
error
12/23/2010 1:04:36 PM
media
User

When I do as it asks, the guest now starts but gets an immediate BSOD concerning memory management.  Any ideas on why this is occuring and why it worked at one point but now it fails?

[BAM279]

Nice to see another report of success.

Regarding audio, I havent tried the hdmi audio passthrough on my HD3450 as I dont have an hdmi tv, however as previously mentioned, I was able to pass through a PCI Audigy2 ZS successfully.  Analogue audio output was very poor but digital out is perfect - this might be another option for you for audio from your htpc vm.

[debink]

Here are the details of my current setup:

Motherboard: SuperMicro X8SIA-F

• Intel 3420 chipset (for Lynnfield based Xeons)
• ICH10R SATA controller with 6x ports (Vendor: 8086, Device: 3B34)
  • I have this passed through to my Nexenta NAS VM via VT-d
  • There are 5x Samsung F4 2TB HDDs connected to this SATA controller

Can I ask you what the BIOS settings are for your ICH10R SATA controller?

Have a board with similar chipset (Intel S3420GPLC) and trying to pass-through the controller to Nexenta as you are however disks do not show up either on the napp-it page or when I issue "format -e".

How are your HD's mentioned under zpool? CxtxDx or as SDx?

Also my PCI id for this controller is 8086:3B22 when it is in AHCI mode which seems to be different from yours.

If I change it to either "IDE"or "Matrix RAID" the PCI id changes to 8086:3B21 or 8086:2822 but never to 3B34 as you mentioned.

Have no problem with passing through a LSI1068e controller but would like to have more than 8 disks in the VM.

Hopefully you can point me in the right direction.

[streetsofopo]

Success?  I'll add another failure to the list.  BSOD for ASUS Radeon 4350 (256MB) on Optiplex 780.  I will try the latest catalyst on Win 7, thanks for that

With regard to digital audio (SPDIF), I use this with esxi 4.1 USB Passthrough (not VT-D) and with the Windows 7 drivers it's pretty usable from inside a VM, for stereo sound at least.  http://www.ebuyer.com/106540-asonic-external-usb2-0-8-channel-sound-card-a-cm106.

I believe the bandwidth for usb passthrough is only just adequate - VT-D USB should offer much better bandwidth.

[FuNK3Y]

The matrix is a great idea!

Here is the mainboard I was using: SuperMicro X8DTH-6F (http://www.supermicro.com/products/motherboard/QPI/5500/X8DTH-6F.cfm)

With the HD5450 I was able to get the sound working fine through HDMI, but once a while the image got inverted on the TV for about 1-2 min. I am not sure that this may be related to the virtualisation as I didn't had that issue with my DVI cable (no sound though).

With the HD6450 I had the same BSOD. Too bad my guess was wrong; I was expecting this card to be very similar to the HD6850 which was reported working.

I used a W7 x64 for all my tests (but with 2gb RAM). I tried with 11.5, 11.6 and 11.7 and a much older version I cannot remember of for most of my tests.

[agent_orange]

Hello,

I have followed this thread with great interest and also like the matrix. From the replies and the matrix in this thread I have created a summary over there, too. I do not mean to distract people from this very valuable thread here, but thought that someone who stumbles over these problems in that other forum should have a good chance to find the relevant posts here. So I hope that you do not mind. In fact it may happen that more people will arrive here and report what they experiences and hopefully also how the solved their problems.

Kind rergards,

vmuser

PS:

Sorry for my stupid user name (agent orange). I would like to change it, but that does not seem to be possible.

If somebody from the admins/ mods comes along and knows how to do it, please let me know, thanks.

MoBo: Intel DQ45CB (BIOS v129, VT-d enabled)
CPU: Intel Core 2 Duo E8400 FC-LGA4 "Wolfdale" (has VT-d) @ 3.00 GHz
RAM: 4 GB
HDD: 2x 1 TB Samsung Spinpoint F3 (in AHCI mode)
Virtu.: free vSphere Hypervisor (ESXi 4.1 U1)

[agent_orange]

Hello,

I have added a Sapphire Radeon HD 5450 (1 GB DDR3 RAM) and  set it up as PCIe passthrough device (with two entries, one for the  graphics, one for the audio) on the guest OS (currently testing Windows  XP Home 32-bit).

Then I installed the latest Catalyst driver and  software package from Sapphire (are there newer versions directly from  AMD/ ATI?):

Catalyst Control Center: 2011.0728.1723.29300

AMD Catalyst Installation Manager: 3.0.838.0

AMD Screen Driver: 8.881.0.0000

AMD APP SDK Runtime: 2.5.709.2

HydraVision: 4.2.210.0

In the Windows guest display settings I now see three(!) desktops:

1: Intel GMA on-board graphics controller from my Intel DQ45CB mobo the host is running on

2+3: ATI Radeon HD 5450 Series - but both greyed out

The catalyst control center cannot be started. It tells me this:

[quote]Catalyst Control Center cannot be started because the currently active GPU is not supported[/quote]

From the above described display settings I assume that currently only  the Intel GMA is running, so I need to either switch it off and in  parallel switch the ATI on, or I switch the ATI on, too, and let the  Intel GMA still running. However, I have no clue which of these  alternatives works and how to try it.
The problem is the following:
Once  before I already had tried to adjust the display settings, but then the  screen (extended desktop) went black and it was very hard to come back  to the correct display, because I could not see the display settings  controls anymore: Everything was just black, or the mouse cursor was on  some not visible extended desktop? I do not know.
So does anybody have any recommendations how to proceed? Thank you!

PS:

I have fiddled some more with this, and after a restart I also got the well known ati2dvag bluescreen.

I had read that booting Windows in safe mode and then removing the old graphics card (the Intel GMA) form the list of the device manager (i.e. deactivating/ deinstalling that device) would help. However, when I boot in safe mode, the list of devices is competely empty. I read that this happens when the plug&play service is not running. However, in safe mode I cannot start that service, i.e. it tells me it cannot be started, because it depends on other services that are not running. When I click on the "dependencies" tab to see on which other services the non-running plug&play service depends, I again get the message that this cannot be shown, because it is not running. Stupid, isn't it?

So any more hints are very welcome. If this card does not work, I will need to give it back.

Also, it would be great of those guys for whom the HD 5450 worked  would post in more details what they did (step by step), which versions of drivers they used, if they only passed through the graphics card or also the HDMI audio part of it etc. Thank you!

[BAM279]

Hi,

how come the Intel GMA is detected by the guest OS?  Have you set this as a passthrough device in the vm hardware list?  If so, remove it so that you are only passing the ATI card, this will leave you with the ATI and the virtual vmware SVGA video adapter. 

If you installed the Intel GMA device and drivers in this vm previously, you can still remove them from the OS via safe mode.  Just go to add/remove programs to remove the drivers.  To remove the hardwrae entry, you will need to enable a system environment variable in xp, via control panel > system >  advanced tab > environment variables button, then under the system variables section at the bottom of the window add a new variable with the variable name devmgr_show_nonpresent_devices and the variable value as 1.

Then its just a matter of relaunching device manager, going to the view menu, and selecting show hidden devices and checking under your display adaptesr and removing the Intel GMA card.

[agent_orange]

Hi,

I did not set the on-board Intel GMA as a pass-through device for the VM.

There is an entry "Grafikkarte" (graphic card) in the list right from the beginning (see screenshot), but this is the VM SVGA video adapter, I guess, and I would rather keep it as you suggested (cannot remove it anyway, because the button to do so is greyed out).

So I just added the new HD 5450 with 2 PCI entries - one for the graphics and one for the HDMI audio part. I.e. there appeared 2 possible entries after I plugged the new HD 5450 to the mobo, and I added them both (first to the host, then to the VM).

I will try to get rid of the Intel GMA as you have proposed, thanks for that hint.

However, when I safe boot Windows, the device manager is completely empty, so I cannot deinstall the Intel GMA.

If I do not safe boot, it is possible to deinstall the Intel GMA (at least it then tells me that a reboot is necessary thereafter for the change to take effect). However, when I boot again (standard), I have again the VM SVGA II in the device manager.

This also shows another video controller and MS UAA bus driver for the HD audio, both with a yellow exclamation mark, but that is only, because I have not yet installed any HD 5450 drivers. When I install these, the yellow exclamation marks are gone, and the new device is correctly shown.

I did not install any drivers for the Intel GMA, i.e. it worked "out of the box",  though there are some additional Intel tools, but I did not install  them.

As long as I cannot get rid of the Intel GMA I seem to be stuck.

Would it make sense to try to completly deactivate the Intel GMA in the BIOS?

Currently, it is already setup in the BIOS as secondary video adapter (instead the "automatic"), so the HD 5450 is the primary one.

PS:

Do I maybe also have to adjust the configuration parameters under VM > options > general?

How can I get that list exported, so I could post it here if needed?

There is only a button to add new lines, but nothing to export, and copy&paste does not work either.

_______________________________________

MoBo: Intel DQ45CB (BIOS v129, VT-d enabled)
CPU: Intel Core 2 Duo E8400 FC-LGA4 "Wolfdale" (has VT-d) @ 3.00 GHz
RAM: 4 GB

Video: primary: Sapphire Radeon HD 5450 (1 GB DDR3 RAM) + secondary: Intel GMA on-board
HDD: 2x 1 TB Samsung Spinpoint F3 (in AHCI mode)
Virtu.: free vSphere Hypervisor (ESXi 4.1 U1)

[FuNK3Y]

To remove unwanted devices use this solution to display disconnected devices: http://support.microsoft.com/kb/315539/

Then remove the devices you want (and check "Also remove driver").

If you cannot achieve this from safe boot mode, remove the VMDP devices and do the same, then reenable the VMDP devices.

[agent_orange]

Yes, I had done this already, though the trick with the environment variable was not neceesary (I could see the Intel GMA in the list already without that).

But more important, now I managed to get the AMD/ ATI HD 5450 (in fact it is a Sapphire) card appearing in the device list and the Intel GMA deactivated (not completely deinstalled):

I first installed just the HD 5450 drivers (not the complete software tool package etc.) and then deactivated the Intel GMA.

After a reboot first everything looked fine at a resolution of 800x600 pixels (I left it at that for further testing).

So I then installed the complete AMD/ ATI Catalyst package v11.8 from AMD.com and was asked to reboot.

I did that, but then got a pop-up error message telling me:

The Catalyst Control Center is not supported by the driver version of your enabled graphics adapter. Please update your AMD graphics driver, or enable your AMD adapter using the Displays Manager.

Well, as I wrote above, the Catalyst driver/ package version 11.8 was the latest that I could identify for my Sapphire Radeon HD 5450 with 1 GB DDR3 RAM.

I downloaded this directly from AMD.com, i.e. not from SapphireTech.com. Is this maybe the problem?

Shall I rather take the drivers from SapphireTech, because they (slightly?) differ from the "original" AMD/ ATI drivers?

The last line of the above message ("... or enable your AMD adapter using the Displays Manager") does not make sense to me. What is the "Displays Manager"? Is it the Windows background > right-click > properties > settings? Well, there I see two displays 2 and 1 (in that order), with 2 activated and 1 deactivated, but both with "ATI Radeon HD 5400 Series" in the name.

Strange is, when I click on "identify", nothing happens, where usually a number (e.g. "1") is displayed on the display to see which number that display has. I also wonder why there are two and not only one, but the No. 1 is deactivated, only 2 is activated and sist left of 1.

Then I tried to active display No. 1 (640x480, only 8 colors) and was told in another error message window that the driver was malfunctioning. Several seconds later, I got the well-known BSOD (bluescreen of death) with ati2dvag:

**** STOP: 0x000000EA (0x8625BA78, 0x863E81A8, 0xF7A0DCBC, 0x00000001)

ati2dvag

Any other ideas how to proceed to get this darn card flying in the ESXi VM?

Thanks a lot!

[FuNK3Y]

Depending on how I was messing with my HD5450, it sometime couldn't start in the guest OS (I was recieving the same error message from the Catalyst Center, and if I went to the Device manager I could see that the HD5450 driver failed to start).

The solution was.... to reboot the whole ESXi host; which is quite annoying as I have quite a few others running VM.

If you manage to get the card working, could you test DXVA with the 11.8 driver?

Some rumors says that vsphere 5 is expected to be released tomorrow.... Finger crossed!

[agent_orange]

I will try DXVA, but first I need to understand what that is and even before I need to get the 5450 working at all.

1. Do you have a Sapphire or an original AMD/ ATI?
   • From where did you DL the drivers: Sapphire or AMD?
     (I DLed from AMD.com, not from SapphireTech.com - which may be a bad, because I have a Sapphire Radeon HD 5450 card)
2. In the VM, did you directly install the complete Catalyst 11.8 package or only the drivers and later the rest?
   (I first only installed the plain AMD/ ATI drivers, i.e. without any tools liek Catalyst Manager etc.)
3. In the VM, did you deactivate or deinstall the Intel GMA in the device manager?
   • Did you do that in safe boot mode or during the usual system use?
     (without safe boot, I could only deactivate but not deinstall it)
   • When did you do that: before or after you installed Catalyst?
     (I deactivated the Intel GMA after installation of the plain AMD/ ATI drivers)
4. In the vSphere client, did you add only the PCI entry for the graphic component of your 5450 to your VM or also the HDMI audio part?
   (I tried only the graphic component and also both, but there was no difference)
5. On the host, did you pass through both the graphic component of your  5450 in the vSphere client to your VM or also the HDMI audio part?
   (I have:
   • 00:01.0: Intel Corp 4 Series Chipset PCI Express Root Port
     • 01:00.0: ATI Tech Inc VGA compatible controller (green light)
     • 01:00.1: ATI Tech Inc Manhattan HDMI audio (green light))
6. On the host, did you pass trough any other devices, e.g. PCI bridge or on-board HDMI audio controller etc.?
   (I have non else)
7. Could you please tell me if your AMD/ ATI graphic card is displayed in the device manager > properties > resources with an IRQ?
   (I have IRQ 18)

Thanks for helping me along, I appreciate it!

[FuNK3Y]

Here are some answers.

1. The graphic card comes from an OEM PC, it is not a Sapphire. I do not really know the manufacturer (ATI?)
   
2. When I did my test I was using 11.6 or 11.7. I downloaded the Catalyst package and installed everything but the things beginning with AMD (AMD HD Coded, ....)
   
3. I never saw any GMA anywhere in my configuration; I do really wonder where yours comes from
   
4. I added the Graphic component and the HDMI passthrough device to the VM
   
5. I passed thourgh both component (graphic & audio) on the host (mandatory for 4.)
   
6. I am also passing through a RocketRaid 3520 RAID controller, which is dedicated to another VM and is not really related to that issue (it works fine though)
   
7. Sorry I can't. I removed the card from the server and it takes me quite a lot of time to get it working again (I do have to restart all the others VM)
   

If the rumors are correct and vsphere 5 is about to be released tomorrow I will do some more tests this upcoming week; I shall then be able to give you the IRQ information.

I do remember that on my previous configuration, based on a DQ45CB also I was unable to dedicate a Perc 6 raid controller with VMDP (the VM refused to start). It worked flawlessly with some others mainboards (DQ35JO for instance). Try to update the BIOS.

[agent_orange]

Hi,

thanks a lot for your feedback. I will further test, but do not know right know what I could do differently - maybe use an older Catalyst driver?

The Intel GMA adapter appears in the vSphere Client, because it is an on-board graphic adapter sitting on the Intel DQ45CB mobo, and I did not find a way to completely disable it in the BIOS. There is only an option to tell the mobo which controller (on-board or add-on/ external) is primary and which one is secondary (or automatic). I set the HD 5450 as primary, but this obviously does not help much. The BIOS version v129 I use is the second newest (before the v133 which does not change anything regarding this topic according to the changelog. But I may try to see if the update to the v133 BIOS nevertheless does help.

Since you removed the HD 5450 from your mobo, there must be some on-board graphic controller. Or am I wrong?

On the other hand, I do not think that my parallel working Intel GMA on-board graphic controller negatively affects my HD 5450. In the meantime I think that it is indeed just a matter of a faulty Windows driver from AMD/ ATI. I will try to test this on my Debian Squeeze VM now. maybe that works.

[rwatson]

I'm also trying to build a HTPC VM and have just bought the following to do VDMP with AMD IOMMU.

AMD X6 1100T

16GB RAM

Asrock 970 Extreme 4

2 x Nvidia GT520 (one for console, one for the HTPC VM...)

Misc SATA HDD in AHCI mode.

..and finally

ESX 5.0 RC in Eval mode booting from USB

...I'm kind of wishing I had read this thread first and bought a known good ATI card as I'm having no luck with DirectPath and the Nvidia GT520 card.

I've tried both Ubuntu 11.04 and Win XP guest VMs.

Under Ubuntu, I can see the card and the associated audio device under lspci but when Xorg tries to load the 'nvidia' kernel module I just get an error saying the card at PCI bus XX:XX:X isn't supported by that driver (applogies for not having the exact error messages - I'll try to post these later).

Under WinXP, I can see the VGA controller but it won't recognise the card or accept the Nvidia driver manually.

On the plus side, I'm pleasantly surprised that I didn't need any tweaks to the ESX 5.0 RC build.  It picked up the Asrock AMD SB SATA3 controller and Realtek 8111E NIC without problem.  ESX 5.0 won't support USB 3 controllers however but that is pretty widely known.

Any one have any suggestions on what I can try to get the Nvidia GT520 card working?

Failing that, any suggestions on a similar ATI card that should work (needs HDMI w/Audio)?

Thank you,

Richard

[sevet2]

So, tried the newly released ESXi 5 in eval mode and sadly no more success than before,

The worst thing is that after the Radeon 3450 didn't work I got myself a USB video card from Disply Link which by reviews sund really usable (seems great connected to my laptop)

And even that doesn't work, on ESXi 4.1u1 I didn't work, it looked like it is using the actual vmware video for something, when I disable the vmware video card, i also loose the USB card.... on ESXi 4  when I turned it on I got a black screen on the physical screen, the vmware screen kept working, on ESXi 5 both screens are blacked.... I did turn on the 3D acceleration feature of ESXi5 thought it might help....

For trying the Radeon 3450, I turned to the EFI instead of BIOSand turned on the 3D, figured staying away from ESXi 4 config as much as possible

still same problem

[agent_orange]

Same problems here...

I am going to switch to Xen now, which seems to work better with the HD 5450 I was told, and I am just frustrated to not get things done with the vSphere Hypervisor.

Besides, installing Xen e.g. on top of Debian or Ubuntu gives you a nice way to maintain the host without the need of an additional Windows installed.

Just select any of the many free (Linux) tools, e.g. described there or any listed in

• English Wikipedia > Xen Management Consoles
• German Wikipedia > Grafische Verwaltung (tools also in English, of course)

Good luck!

[rwatson]

Good news!  I've got my PowerColor HD5450 1GB running under a guest Ubuntu 11.04 VM on ESX 5.0.  The VM has 2 vCPU and 2GB vRAM.  I've got the external HDMI interface working including HDMI audio.

A couple of notes...

• I've only assigned my VM 2GB of RAM and have that marked as fully reserved on the VM settings.  I haven't tried increasing this yet...
  
• I'm using the standard "fglrx" ATI linux drivers with Ubuntu 11.04.
  
• I haven't tried getting both the VMware SVGA device and the ATI HD5450 device working in Xorg.  Right now I've just modifed the default device - including specifying the BusID - to make the H5450 the default device.  My whole xorg.conf file is about 4 lines.  This does mean the vSphere console goes black but I can see my Ubuntu desktop on the external screen.
  
• I haven't bothered setting up a dedicated keyboard or mouse for the VM as I plan just to use a USB IR remote via LIRC.  However, the VMware keyboard and mouse devices are still mapped through when the "empty" console is open and selected- though for some reason my VM mouse has the up/down direction inverted....
  
• I can run fglrx_glxgears on the desktop.  The last time I ran this I only got around 50-60 fps but I'm sure the first time I ran it I got much faster (600+).  I'm not sure now what is normal and if I've made some other change that may have effected this.  I'm not planning on using the OpenGL for 3D - just for hardware accelerated video playback so I think this will be fine.
  

I'm now trying to get MythTV Frontend (0.25) working.  I tried the MythBuntu dev build but it doesn't look like they've included VAAPI support so next step is to compile it from source...

[SomeMoose]

I Have a ATI 4890 and are using ESXi 5.0

I have been able to solve the problem with 2GB+ Ram on an vm while doing directpath

Half of the answer was the: pciHole.start

command... on a whim i also added a pciHole.end

command thinking that it might be a range....

and it worked....

I also found what i think is the solution for you who get the no hardware acceleration problem... all the graphics card probobly need to support 3d.. because for me flash worked first when i enabled 3d acceleration for my VMWare card as well...

you could see if thats your problem by starting a game (that will proboly hang)...

then wait for 5-10 minutes and go back and see if somethings happend.. if it works, then its probobly the same as i've experienced...

So, guide of what i did...

1. Enable 3d acceleration of your Vmware card...

2. add pciHole.start and pciHole.end to your configuration file... (Options/Advanced/General/Configuration Parameters)

I used the values: pciHole.start = "1200" and pciHole.end = "2200"

Thats what i needed to do extra other then passing through my graphic card and HDMI device..

As always you do it at your own risk :smileysilly:

I belive myself that the pciHole command may be for assinging a hole in the memmory map space for the passed through pci (and pcie) devices...

If what i read about drivers and memmory map space is true, a place in the 32-bit register between 1 and 4000 (MB), should work best...

Edit:      By the way, the OS i used was Win 7 x64...

[MattGagliardi]

I can confirm SomeMoose's report of success using both the pciHole.start and pciHole.end variables.  In my case I'm using an HP DL160 G6 (well-spec'd, plenty of RAM) and a single nVidia Quadro FX3800 card (which is an SLI multi-OS capable card).  4 vCPU and 8GB RAM...8GB reservation in place.  The guest VM powered on successfully when I kept the RAM at 2GB but when I moved up to 4 or 8 (which for the app I'm running is pretty much a necessity) it would fail to power on.  When I added the pciHole.start variable with the value suggested in the log the guest would start but blue screen immediately.

The addition of the pciHole.end variable seems to have cleared things up.  It's funny, Googling for that shows basically nothing...and that's too bad.