VMware Cloud Community
taylorjonl
Enthusiast
Enthusiast

VMDirectPath and ATI Radeon

I am using VMware ESXi and I am trying to setup a guest that is Windows 7 that will have an ATI Radeon video card passed through to it.  I actually had this working on a previous system but I had to reinstall.  Now when I do this the guest fails to start and I get the following:

Error message from localhost.XXXXXXXXXXX:
PCIPassthru 004:00.0: Guest tried to (null)map
32 device pages (with base address of 0xb5d20)
to a range occupied by main memory. This is
outside of the PCI Hole. Add pciHole.start =
"2909" to the configuration file and then power
on the VM.
error
12/23/2010 1:04:36 PM
media
User

When I do as it asks, the guest now starts but gets an immediate BSOD concerning memory management.  Any ideas on why this is occuring and why it worked at one point but now it fails?

814 Replies
aaronvm
Contributor
Contributor

Good info.  This is one of the cheaper builds I've seen reported to work here.  Most of the working builds for ESXi seem to use server class Intel boards (X58, X79).  I still cannot get my Shuttle SH67H3 + i5-2500 + ATI 5670 combination to work, which seems like an ESXi limitation, because GPU passthrough has been reported to work with this system using XenServer.  The same person did report success with ESXi and the Shuttle SX58J3, but I don't want to go as old as LGA1366.  There is now a Shuttle SX79R5 but it's pricy-- about $500 for the barebones system and $300 for a quad core Sandy Bridge-E.  On the other hand, the AMD FX-8120 you're using is an 8-core CPU for only $160 on Newegg.  I prefer a box with a small footprint so I will have to do some research now on AMD Micro-ATX and Mini-ITX boards and cases.

Looking back I see that BAM279 was successful with an AMD Opteron system last year.  Perhaps AMD is the way to go for an affortable desktop-class GPU passthrough solution?  For Intel, I continue to wonder what is special about the server chipsets versus desktop regarding VT-d/GPU.

Thanks!

Reply
0 Kudos
GBowman
Contributor
Contributor

My i7 870 on a Q57M-SH2 is a cheap intel build and M-ATX.  All in an HTPC case and running for more than 12 months without fault.  Up time of over 100 days now and it only got turned iff before that to fit more ram.

I would not compare VT-d on ESXi with XEN since XEN can do GPU passthrough on systems with no VT-d, they must use a different way of emmulating the gpu into the guest OS.

Reply
0 Kudos
mindless728
Contributor
Contributor

I am using the secondary PCI Express slot, the GeForce 210 is in the primary slot as I would prefer to still have a console when I need it

Also, the 2 USB 3.0 controllers (4 ports total) as also given to the Windows Gaming VM as that makes it easy to have USB input to the VM

My only pet peev right now is that I have a blu-ray player, however, ESXi seems to be limiting the speed in which the VM can read the data from the drive to a speed that makes it impossible to actually watch the movie without first using software to sip it from disk (AnyDVD HD to be exact)

Reply
0 Kudos
BrandonBBB
Contributor
Contributor

Interesting.  Thank you.

Could you maybe get a cheap SATA controller and pass it through to the VM?  For the BluRay drive, that is.


Brandon

Reply
0 Kudos
mindless728
Contributor
Contributor

I'll try that and see if it works, though for a permanent solution it will have to wait until I get a cheap card

Reply
0 Kudos
aaronvm
Contributor
Contributor

Thanks for the correction GBowman.  I now see your earlier post about that hardware.  This thread has gotten very long, I have to admit I've been a bit lazy to review it all.  Somehow I missed MrPippy's earlier post about the ACPI tables, that's very interesting.  And it jives with your earlier theory about this being purely a mainboard BIOS issue (and much less to do with chipset, specific GPU card used, etc).

So I was motivated to look into this.  Here are MrPippy's ACPI DMAR lines (before and after, respectively, of his BIOS update that got it working):

[    0.000000] ACPI: DMAR 00000000bf575fd8 000B0 (v01 ALASKA    A M I 00000001 INTL 00000001)  <- before

[    0.000000] ACPI: DMAR 00000000bf3ae798 00078 (v01 INTEL      SNB  00000001 INTL 00000001)   <- after

Now here is my ACPI DMAR line from dmesg of Ubuntu 11.10 (booted off CD on the SH67H3):

[    0.000000] ACPI: DMAR bf3b58f0 000B0 (v01 ALASKA    A M I 00000001 INTL 00000001)

My output is suspiciously similar to his before it wasn't working.  Interestingly, 000B0 and 00078 are the length fields of the structure, so it actually shrunk after his update.

Now here is some more info from my dmesg during DMAR parsing:

[    0.000000] ACPI: DMAR bf3b58f0 000B0 (v01 ALASKA    A M I 00000001 INTL 00000001)
[    2.284440] DMAR: Host address width 36
[    2.284441] DMAR: DRHD base: 0x000000fed91000 flags: 0x1

[    2.284445] IOMMU 0: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a

[    2.284447] DMAR: RMRR base: 0x000000bf5ed000 end: 0x000000bf5fbfff
[    2.284448] DMAR: No ATSR found

Hmm, "No ATSR found".  Anyone have any idea if this is critical for GPU passthrough?  It appears that ATSR stands for "Address Translation Services Remapping", which is described in the following Intel document:

http://download.intel.com/technology/computing/vptech/Intel(r)_VT_for_Direct_IO.pdf

Finally, here is the acpidump output for my DMAR entry:

DMAR @ 0xbf3b58f0
  0000: 44 4d 41 52 b0 00 00 00 01 01 41 4c 41 53 4b 41  DMAR......ALASKA
  0010: 41 20 4d 20 49 00 00 00 01 00 00 00 49 4e 54 4c  A M I.......INTL
  0020: 01 00 00 00 23 01 00 00 00 00 00 00 00 00 00 00  ....#...........
  0030: 00 00 58 00 01 00 00 00 00 10 d9 fe 00 00 00 00  ..X.............
  0040: 03 08 00 00 00 f0 1f 00 04 08 00 00 00 f0 0f 00  ................
  0050: 04 08 00 00 00 f0 0f 01 04 08 00 00 00 f0 0f 02  ................
  0060: 04 08 00 00 00 f0 0f 03 04 08 00 00 00 f0 0f 04  ................
  0070: 04 08 00 00 00 f0 0f 05 04 08 00 00 00 f0 0f 06  ................
  0080: 04 08 00 00 00 f0 0f 07 01 00 28 00 00 00 00 00  ..........(.....
  0090: 00 d0 5e bf 00 00 00 00 ff bf 5f bf 00 00 00 00  ..^......._.....
  00a0: 01 08 00 00 00 00 1d 00 01 08 00 00 00 00 1a 00  ................

I'll need to take a closer look at the aforementioned spec and match up the data here to the documented struct formats, to see if it's sensical or garbage.  I'm really not familiar enough about how these BIOS's are manufacturerd to say anything intelligent about these findings yet, other than observe the identical B0 lengths, and "ALASKA A M I" string.  If I didn't know any better I'd guess that AMI provides a default ACPI table template to the vendors that Shuttle didn't populate with proper data.

Meanwhile, if folks agree this is progress, perhaps MrPippy and others could provide full acpidump's (or at least the DMAR portion) from their working and/or non-working machines?  On Ubuntu, just use "sudo apt-get install acpidump" to install it.

Reply
0 Kudos
GBowman
Contributor
Contributor

Which player software are you using?  How have you got the drive connected?

Reply
0 Kudos
mindless728
Contributor
Contributor

I am using AnyDVD HD which is commercial software that allows me to play blu-rays in VLC and the drive is connected to the motherboard SATA connector, this works just fine when it is just windows 7 installed natively

Reply
0 Kudos
madsheep
Contributor
Contributor

I've been using vmware for a long time now, and I have a lot of vm´s too, soo switch to xen will be difficult.  Smiley Sad

But does not sound that some have successful in passthrough a gfx card on my motherboard (dq67sw).

Reply
0 Kudos
technobro1
Contributor
Contributor

Hi gentlement ,

still no luck here , its either i got bsod when i install it or when i enable it from safe mode

When i disable the svga VMware diver i can install it whiout bsod but windows  doesnt allow me to choose it in resolution settings,


I read erlier in this post the vmware svga should be WDDM 1.1, how do i install that, should i investigate more this way...

any help

Thanks you

i3770

Asrock Z77 pro3

WIn64b

Reply
0 Kudos
aaronvm
Contributor
Contributor

So a follow-up from the other day.  I used the "iasl" utility from Intel to verify that my ACPI DMAR table is in fact well-formed.  I can't confirm that all of the data in there is actually 100% correct, although PCI passthrough works for an Intel NIC (just not Radeon GPU), so I guess I should assume that DMA remapping is working.

Back to a prior theory.  Is anyone doing VGA passthrough successfully w/ ESXi on a machine with Intel graphics present?  That is, not passing through the Intel graphics per-se, but any PCI-E GPU alongside a graphics-equipped Intel CPU (i.e., my CPU is an i5-2500 w/ HD Graphics 2000).  For the next step I was thinking about picking up a Xeon E3-1230 CPU to try in my Shuttle.  It's an LGA1155 CPU that lacks graphics but does include VT-d.  This is not on the officially supported CPU list for the SH67H3, but I found reports of it working.

According to ARK, GBowman's i7-870 doesn't have integrated graphics, for example.  Nor do any of the other working configurations that I've looked up on this thread so far (I admit I lack the patience to exhaustively check every single one).  Anyway, the theory is that the Intel graphics are somehow conflicting with the PCI-E GPU.

Finally, does anyone know if there's a reliable way to disable onboard graphics entirely (like at chip level)?  In my BIOS, the only selections for graphics are "Onboard" or "PCI-E/Onboard".

Thanks all.

Reply
0 Kudos
GBowman
Contributor
Contributor

My i7 820 has no integrated graphics as said and my Q6700 has only the motherboard graphics of the intel Q45 chipset.  These are not disabled but do not appear in the list of devices that could be passed through.

I will try to run the linux diagnostic on my Q45/Q6700 machine in the next few days.  I cant shut down the i7 as it is the TV for the whole family and also now has a digium card passed through for the telephones.  Got to wait until I'm home alone!

Reply
0 Kudos
kthree
Contributor
Contributor

I went through the past 10 pages or so as well and it looks like you're onto something.

Another thing is that we could try is passing through both the intel graphics as well as the radeon. I know there has been issues with passing through the integrated graphics, but perhaps just passing it through is enough to get the radeon working.

Also believed that there was a user who got it working under OS X despite it BSODing under Win7. So might be OS dependent and worth trying on Win8.

When ESXi 5.1 comes, I'll take the opportunity to try it out as well as test the above.

Reply
0 Kudos
bwiklander
Contributor
Contributor

Hey, This is my first post in this forum so i would yust like to say hi first, and thanks to all the people who have been posting in this thread it have really helped me alot when trying to get VGA passtrough working i ESXI

1 .
Ok so first i would like to give out a tip to everyone who is trying to get pci passtrough working for a graphic adapter.

It can be a real struggle to get this working and testing diffrent setups and PCI-E ports, OS’s and so on...

I came across testing the XBMCUbuntu iso image.
http://mirrors.xbmc.org/releases/XBMCbuntu/xbmcbuntu-11.0.iso

this came in real handy as it by default disables the “onboard” in this case ESXI’s virtual graphics adapter, and puts the output directly in my case trough my radeon HD5450 HDMI port.
(ohh, and remember to only give the guest a maximum of 2GB for starters and Reserve all the mememory to the guest)

hope this can help anyone.


2.
Im also having some trouble with either my Radeon card or my Motherboard and i would like som help with solving this problem.

Simply put, Everything works great, i got the XBMCubuntu OS running for 2weeks straight, until it hanged. and the whole ESXI server freeze.

The problem is when something goes wrong, or if i just reset the virtual machine. then the whole ESXI system freezes up and i cant do anything at all, i have to do a Hard reset.

And as im planning on using a windows7 installation for my HTPC and do some casual gaming this is gona be a problem.

So have anyone else experienced something similar or do have some idea what can be wrong?’

i have seen some people having trouble with the 5450 cards, and some seem to have no problems at all, but i would like to hear if someone else can confirm it’s a problem with either my GFX or my motherboard so i can get a new one.

Posting my setup below

ESXI 5.0.0, 469512


Gigabyte GA-990FXA-UD3 (latest bios(alltough it didn’t seem to do much diffrens when i uppgraded))

AMD Phenom II X6 1055T Processor @ 2.8 ghz (stock)

PCI-E nvidia cheap graphic card running the console
MSI Radeon HD5450 1024MB
(PCI-passtrough, to Guest)

20GB ddr3 Ram

2x 30gb OCZ Vertex SSD
1x 60gb OCZ Vertex 2 SSD


Thanx in advance.
Björn

Reply
0 Kudos
derickso
Contributor
Contributor

I would suggest you try a newer card, I've had nearly no problems with my 7 series Radeon, and you have to expect that newer cards will have better hardware and software compatibility with what we are doing.

Reply
0 Kudos
mindless728
Contributor
Contributor

I don't know if it is because the card being old, before I bought my 7970 I was using passthrough just fine with my 4870.

It might have to do with the fact that it is a low end card and it works slightly different from higher end cards

Reply
0 Kudos
MarBen
Contributor
Contributor

Dear all,

since I just made my first experiences with ESXi on my new Shuttle and someone in this thread also asked about the Z77 chipset, here is my config and a question from my side:

Hardware

  • Shuttle SZ77R5
  • Intel Core i7-3770
  • 32 GB RAM (Corsair Vengeance LP)
  • 2x 3 TB Seagate Barracuda
  • nVidia 8800 GTX (from my old desktop in order to test the passthrough feature)

Everything seems to work flawlessly, though most of the hardware is listed as "unknown" within vSphere, since the Shuttle is not in the HCL.

I'm currently struggling with the passthrough of my video card: I got it through to my Windows 7 x64 system and successfully installed the certified Windows drivers for the 8800 (had to disable the VMware SVGA). When I reboot the guest OS, the output from the nVidia card still shows the host's "cnic_register loaded successfully" screen, which I guess one of you suggested correctly is the stage where the "passing through" actually happened.

But I don't get a Windows login screen.

That's the problem. Tried many things but no luck. The system boots up correctly, I can access it via the vSphere console and login to the system; but how can I get the actual output to the nVidia card?

Any help is highly appreciated ...!

Cheers

Reply
0 Kudos
derickso
Contributor
Contributor

nVidia cards are known to be problematic, try a Radeon.  Also disabling the VMware SVGA will almost certainly prevent your setup from working.

Reply
0 Kudos
mindless728
Contributor
Contributor

I can definitelty confirm that disabling the VMware adapter will cause any GPU's that are passed to the VM to not function at all

I tried this myself with both my 4870 and 7970 to see if it would work or not

Reply
0 Kudos
MarBen
Contributor
Contributor

Thanks for the quick answers!

I read in several other forums that other people got their setup to work only if they disabled the VMware SVGA. Maybe I can find these posts again ...

Anyway, if I enable it, the device manager indicates a problem with the nVidia card, which prevents me from using it. Besides that: could somebody please post a screenshot of the display settings (such as output to monitor X and stuff) and the device manager? I know, not much magic there, but I'd like to see a working config, somewhere.

Many thanks and cheers

Reply
0 Kudos