VMDirectPath and ATI Radeon

[taylorjonl]

I am using VMware ESXi and I am trying to setup a guest that is Windows 7 that will have an ATI Radeon video card passed through to it.  I actually had this working on a previous system but I had to reinstall.  Now when I do this the guest fails to start and I get the following:

Error message from localhost.XXXXXXXXXXX:
PCIPassthru 004:00.0: Guest tried to (null)map
32 device pages (with base address of 0xb5d20)
to a range occupied by main memory. This is
outside of the PCI Hole. Add pciHole.start =
"2909" to the configuration file and then power
on the VM.
error
12/23/2010 1:04:36 PM
media
User

When I do as it asks, the guest now starts but gets an immediate BSOD concerning memory management.  Any ideas on why this is occuring and why it worked at one point but now it fails?

[mindless728]

Well, I know it isn't just VMware that is having issues with passing through NVIdia cards, this also is a problem on XEN as well (alst I checked) and involved setting the vBAR=pBAR as well as giving the VM access to the original BIOS (one before booting) of the video card as the BIOS gets changed in some way after booting that it does not allow it to be used in passthrough

Keep in mind a lot of these issues stem from a year or so ago when I first started looking into the issue with GPU passthrough

[derickso]

Ya I would seriously encourage you to stop trying with an Nvidia card right now, you'll just be banging your head against the wall.  Pick a known working configuration from this thread (or one that is super close) and use it.

[MarBen]

Just giving up is against my nature!

But ... in this case, you might be right ... It's raining for straight two days now, but, unfortunately, it's not raining ATI graphic cards. Hence I'll have to buy one at some point.

Since I'm now living in Switzerland, I can no longer order one and just try it out for to weeks (as it is possible if you live within the European Union). Have to figure something out in order to make sure that the card really works with my setup ...

Cheers

[sevet]

Hi,

I already posted this but I wanted to point out what helped me in the end, It was disabling the onboard audio on the bios, it happened when I was so frasturated, I disabled everything not needed on the bios (passing the onboard audio produced horrible audio output anyways).

Try to start the system with as much disabled devices as you can on the bios - no audio, no unneeded controllers etc', maybe even remove every card but the GPU.

Also, it only worked after it with the v12 radeon drivers, did a small test and enabled the audio and it stopped working, tried default radeon drivers and it also didn't work.

It is working with default vmware display and drivers untouched (enabled), I only disable the vmware screen on the windows display settings and enable only the radeon card displays, XBMC works, usb audio works, jmicron pcie sata works so I have a whole computer, avermedia hd capture didn't work sadly.

I have Radeon 7770, it also worked with Radeon 3450 (which is still conncted on a x1 pcie slot for a second vm, after i chipped it a bit to fit)

[MrPippy]

Hey aaronvm--

I finally got around to dumping my DMAR table.

Here's my raw DMAR table from the ASUS P8B-E/4L, BIOS version 2011. Passthrough works on this board/BIOS.

DMAR @ 0xbf3ae798

  0000: 44 4d 41 52 78 00 00 00 01 e2 49 4e 54 45 4c 20  DMARx.....INTEL

  0010: 53 4e 42 20 00 00 00 00 01 00 00 00 49 4e 54 4c  SNB ........INTL

  0020: 01 00 00 00 23 01 00 00 00 00 00 00 00 00 00 00  ....#...........

  0030: 00 00 20 00 01 00 00 00 00 00 d9 fe 00 00 00 00  .. .............

  0040: 03 08 00 00 02 f0 1f 00 04 08 00 00 00 f0 0f 00  ................

  0050: 01 00 28 00 00 00 00 00 00 50 0b bf 00 00 00 00  ..(......P......

  0060: ff 0f 0d bf 00 00 00 00 01 08 00 00 00 00 1d 00  ................

  0070: 01 08 00 00 00 00 1a 00                          ........

I also made a Synalyze It grammar to interpret the DMAR table--if you have a Mac, it's a really cool and free tool. I've attached my grammar.

I used the grammar to interpret both of our DMAR tables, and they really aren't very different. The main difference is that the ALASKA/AMI DMAR declares 8 HPETs (presumably one for each logical core), whereas my INTEL/SNB one just declares one HPET. The RMRR range is also 5x bigger on the AMI table, but I don't think that makes any difference.

Take a look at the PNGs attached to see the differences.

Maybe this means that passthrough would work if all 8 cores were passed through to the VM?

I also started looking into BIOS internals: the AMI BIOS on our boards is EFI-based, so all the older AMI BIOS hacking tools don't work. Newer tools (like MMTOOL 4.50) are needed that understand the EFI-based AMI Aptio BIOS. Also, since it's EFI-based, the BIOS image itself is a bunch of executables that run to initialize the processor/chipset/platform.

I opened up the original BIOS for my board (the 1102 version which doesn't have working GFX passthrough), and easily found the DXE that generates the DMAR table (it's called VTdDxe). Figuring out how this all works is not easy though--EFI is a new world, at least it might be documented though.

(If we wanted to get into BIOS hacking, I think a separate thread is in order )

[x1nick]

A new driver for the Asus P8B-WS was released recently so thought I would give it a try

So far still getting the same result as last time

Got the catalyst drivers installed ok (with VMware tools installed,  seemed to fix that issue I had before!), then rebooted and got the  standard atikmpag.sys BSOD if a monitor is connected.

No  monitor connected results in the host crashing, and I am assuming  running at 100% CPU as the CPU fan begins spinning up 10x faster

Got 2 more pci slots I will try, might even see if I can try a different card but guessing all will have the same result!

[kthree]

Still no go with ESXi 5.1.

Both Win7 and Win8 still giving me BSOD when drivers are installed.

ASrock Z77 Extreme4-M

i7 3770

HD7850

[max888]

MSI Z77A-G45

i7 3770

HD 5450 BOSD

Geforce 8600GT could not start

Onboard Intel 4000 could not start

W2008 R2, W7 and XP

Added: Also tried Ubuntu 12.04.01 as guest. HD 5450 driver installs, but display is not detected.

VM host is version 5.0 update 1, plus patch ESXi500-201207001 (to fix VM guest auto-start/auto-shutdown for the free version).

Will try version 5.1 later.

Added 2: Got this setup working after disabling onboard GPU and Sound following kthree's setup (post 313 http://communities.vmware.com/message/2117003#2117003)

See post 315 http://communities.vmware.com/message/2117045#2117045

[derickso]

For everyone having problems, try picking up one of the known working motherboards!

[joiegils]

is any one having problem with xbmc 11.0 and mp4 files can´t seem to play them  work fine in xbmc 10,0 but after i udpate xbmc can´t seem to play them any more ???

if i disable hardware accelration then they work.

[StarAtlas]

INTEL DQ77MK

Core i3570T

Memory 8GB

RADEON HD 7550

iGPU off -> BOSD

iGPU on -> BOSD

Host: VMware ESXi 5.1

Guest(i try): Windows 7 Ultimate(32 bit), CPU 2 core, Memory 1536MB

[max888]

derickso wrote:

For everyone having problems, try picking up one of the known working motherboards!

Indeed. That would save you from pulling your hair out, or hitting your head against the wall.

I do encourage those with unsuccessful setups to post, so that others can stear clear of the non-working parts or combos.

[KRA77]

As requested I will post my "working" setup, I'm not completely done with my setup and testing yet, but I can report the following:

Since it was confirmed a success with this mobo earlier in this thread I ordered

Asus P8B 4E/L mobo

16GB ecc ram

Xeon 1230 (SB)

Asus 2008 PIKE

I flash the 2011 bios (as notet earlier in thread) did not try newer bios since 2011 bios was confirmed to change mobo from not working to working status.

I already had a 4650 pci-e ATi

Pike card is passed on to openindiana/napp-it and working automatic. (did not work automatic with freenas/freebsd based systems)

At first I tried to pass 4650 ATi card onto WHS2011, installed fine, driver install and report working order. Catalyst control panel will not start, and no way to get output to TV. Rather than try more, I tried to pass it on to xbmcbuntu, and it worked out of box, no driver install or nothing. Don't know if xbmcbuntu use free og ati driver. I do however have some problems in xbmcbuntu, sometimes when I change hz the screen gets garbled, and vaapi does not seem stable, don't know if this is driver og passthrough issue. I tried to install ATi catalyst drivers, but this failed. Please not that this card is now considered legacy, I installed Win8 preview, but ATi does not support this card for Win8. So right now I have a ATi 6450 card on the table next to me, I will report back my findings with this card. I also tried Ubuntu 12.04, but the story is same as with WHS2011, installs fine, no option to use the card as output for my tv.

Something to note about this mobo though, it lists as having 3 legacy pci slots, but slot 1 is blocked by pci-e 16x slot if you have a fan on your gfx card and slot 3 is blocked by PIKE card if you install that. Slot 2 however is free but note this port is shared with the onboard server vga card, so if you pass on this port, you will also pass on onboard vga. I've tried installing a HVR-4000 tv tuner in slot 2 and passed it on, it install fine and don't report any errors, however it fails to scan for channels.

[kthree]

After spending a lot more time playing around with BIOS settings, I've managed to get it to work!

Hardware:

Mobo: ASRock Z77 Extreme4-M (Works on all BIOS 1.00 to 1.30)

CPU: i7 3770

GPU: HD7850

Host: ESXi 5.0 U1

Guest: Win 7 64bit (Both HD7850 and it's HD audio passed through)

Mobo BIOS parameters:

Primary Graphics Adapter: PCI Express
VT-d: Enabled
IGPU Multi-Monitor: Disabled
Onboard HD Audio: Disabled

If any of the above onboard video/sound settings were enabled would result in atikmpmag BSOD.

The full catalyst 12.8 driver suite installed with no problems. Display output has been flawless, however audio over HDMI/DP frequently stutters, might test it some more otherwise I'm gonna have to buy a PCIe sound card (any working suggestions?).

Guest OS worked fine on 2GB vRAM without tweaks. More vRAM worked with the pciHole.start=1200, pciHole.end=2200 and reserved memory tweak.

I also tried ESXi 5.1, however I found that onboard USB passthrough did not work (it did in 5.0 U1), all other passthrough worked fine as in 5.01U. Can anyone else comment if they've had similar problems with 5.1?

So relieved that we finally have a Z77 chipset working, let me know there's any questions.

[derickso]

Great! Glad you could get it running. How much RAM are you giving your guest? I had a hard time interpreting your comment:

Guest OS worked fine on 2GB vRAM without tweaks. More vRAM worked with  the pciHole.start=1200, pciHole.end=2200 and reserved memory tweak.

Did you mean the guest worked fine with 2GB without tweaks, but to increase above 2G you set those pciHole settings?

I too had the same audio stuttering problem, I was able to mostly get rid of it (it still happens, but rarely), by disabling the sound device in Windows, then uninstalling all Realtek drivers, then running driver clean software to ensure all traces of it were gone, rebooting multiple times between each of those steps, then finally re-enabling and re-installing the Catalyst drivers, which include their own Realtek driver.  In my case I think this was because I did a P2V of my image and it used to have Realtek motherboard audio drivers in it, which it sounds like yours may have as well if you tried passing it in.

I haven't yet tried 5.1 to confirm the USB problem, but now I am slightly less inclined to do so.  Let us know if you get it working.

[max888]

Got VGA passthrough working, after disabling onboard IGU and Audio!

Hardware:

MSI Z77A-G45

i7 3770

HD 5450

ESX5.1 installed on USB key

W7 64bit with 2GB RAM

There is a small problem, the mouse pointer coordinates are off by quite a bit. Mouse point is at the "Computer" icon, right at the letter "C", rightclicking brings up the menu as shown in the picture.

Added: after playing with dual displays (vm console and physical monitor), and setting same resolutions on both, noticed mouse pointer moves within the VM twice as far as it is displayed on the vSphere client machine.

Also confirmed USB controller marked for passthrough is unmarked after host restarting. I need USB passthrough for kb/mouse, so this problem is holding me back from going to 5.1.

Added 2: going back to

ESX 5.0 update 1, plus patch ESXi500-201207001 (to fix VM guest auto-start/auto-shutdown for the free version)

W7 64bit w/ 2GB/4GB/8GB RAM work after disabling onboard IGU and Audio

Added 3: No go on W2008R2. Same HD5450 video card shows up in device manager on W2008R2 (2/4/8 GB or RAM). The problem is that secondary display is not detected. Was hoping to get this working on W2008R2 so that I don't need a W7 VM for VM management.

[aaronvm]

Thanks MrPippy, sorry for the delayed response.  Unfortunately your suggestion did not work (passing all 8 cores to the VM).  Still get BSOD.  I agree that our DMAR tables aren't that different.

[aaronvm]

I replaced my i5-2500 CPU in the SH67H3 with a Xeon E3-1230.  As noted earlier, this is a quad core Sandy Bridge w/ VT-d but no IGPU.  The BIOS in this machine does not have an option to disable the IGPU (the graphics options are simply "Onboard" or "Onboard/PCI-E" -- that is, no "PCI-E Only" option).  So my only hope was to try a CPU without the Intel graphics.

Sadly this did not change the outcome, still get the atikmpag.sys BSOD.

I have tried some more things, to no avail:

- Disabled Azalia HD audio in the BIOS

- Disabled Front/Rear USB 3.0 ports in the BIOS

- Disabled various other options even if they seemed unrelated (EuP control, EIST, Serial COM1)

- Passthrough both the graphics chip and the separate HDMI audio PCI devices

- Various memory configurations

- Upgraded to ESXI 5.1

- Upgraded Catalyst drivers to 12.8

My hardware again is:

Shuttle SH67H3 xPC (Motherboard V1.0, BIOS SH67H000.116)

Intel Xeon E3-1230 3.2GHz (earlier used Intel i5-2500 3.3GHz)

16GB Corsair XMS3 RAM

ATI Radeon HD5670

I've been trying this with both Windows 7 Ultimate 64-bit and Ubuntu 12.04 64-bit.  In Ubuntu I still get the drm error in dmesg about a BIOS ROM not being found.  Do folks with working setups see this too?

One major thing I haven't tried yet is a different video card.  I have the same card as the OP (taylorjonl), although he seemed to only get it working partially (no OS stability) and eventually moved to Xen.

[kthree]

@derickso: I currently have 8GB RAM on my guest. My host has 32GB.

There were some posts earlier on this thread that suggested the pcihole parameters to allow guests to have more than 2GB RAM. The exact values 1200 and 2200 is bit of a mystery, I would like to know what they were derrived from.

src: http://communities.vmware.com/message/1824763#1824763

Thanks for the audio suggestions. I'll give them a try when I get home, though I don't have the realtek mobo audio drivers installed at all.

@max888: awesome to see that the same configs work on another z77 motherboard! I wonder if the ESXi 5.1 USB problem is only with the z77 chipset. If someone else can on a different chipset can confirm onboard USB passthrough works with ESXi 5.1 then that'd be great as 5.1 has some really nice features.

[Flappje]

@Kthree, the onboard USB passthrough (and as i read from other topics), also the extention cards with usb, is not working correctly in ESXi 5.1.

i myself have a: Supermicro X8sil-f motherboard (uses the Intel® 3420 Chipset).

and for me also that i can mark it as passthrough in the list, but after a reboot it is umarked again.

here one of the few discussions i could find about it: http://communities.vmware.com/thread/417736?tstart=0