VMDirectPath and ATI Radeon

[taylorjonl]

I am using VMware ESXi and I am trying to setup a guest that is Windows 7 that will have an ATI Radeon video card passed through to it.  I actually had this working on a previous system but I had to reinstall.  Now when I do this the guest fails to start and I get the following:

Error message from localhost.XXXXXXXXXXX:
PCIPassthru 004:00.0: Guest tried to (null)map
32 device pages (with base address of 0xb5d20)
to a range occupied by main memory. This is
outside of the PCI Hole. Add pciHole.start =
"2909" to the configuration file and then power
on the VM.
error
12/23/2010 1:04:36 PM
media
User

When I do as it asks, the guest now starts but gets an immediate BSOD concerning memory management.  Any ideas on why this is occuring and why it worked at one point but now it fails?

[Artwright]

Hi,

Would you be able to confirm if the GPU passthrough is 100% stable when you tested it?  e.g  Have you tried playing games on the VM?

Would you happen to have a second GPU lying around that you can passthrough as well to see if passing through multiple GPU's work well?

Any luck with fixing the audio stuttering over HDMI?

I also notice that you have the Extreme4-M.  Do you think the passthrough will work as well for the Asrock Z77 Extreme4 (non M) version? 

Reason I ask these questions is because I am contemplating changing my motherboard.  I have the Z68 Extreme4-Gen3 motherboard but have not been able to get passthrough working.  In fact when I saw your post about disabling on-board audio and setting primary graphics to the PCI express, I tested it on my motherboard but still no luck.  So changing the motherboard to the Z77 version would be the cheapest/easiest alternative if I want this working. Although I would prefer the non M version.  Appreciate your response.

Thanks.

[kthree]

@Artwright

GPU passthrough is 100% stable, I havent come across any problems. I play Starcraft 2 quite frequently on it, videos play with no issues, I've used both Win7 and Win8.

I do have a HD4670 lying around, I'll test it when I have the time. But I don't see it being a problem, I already have other PCIe devices passed through to a different VM (a SATA controlelr passed through to an unRAID VM) and have both running simultaneously with no issues.

I have managed to minimise the audo problems by disabling sound enhancement and allowing exclusive mode as suggested by another user. However I cant gaurantee this as I soon bought a cheap PCIe x1 soundcard (Xonar DGX) and passed that through, which performs flawlessly without any tweaks.

My gut feeling is that non-M should have no issues, the 2 boards had same bios release dates and such I think that there isnt a significant difference in architechture aside from the form factor and a couple extra controllers on board.

I need to tell you though, ESXi is not detecting any devices plugged into the middle full sized pcie on my board, I'm not sure if its a faulty slot or design.

Why do you need the non-M version?

Let me know if you got any other questions.

[smanet]

Hello,

I would like to share with you my current system that is working well with VMDirectPath.

I bought this hardware and I think I made a really INTEL cheap solution (excluding the RAM):

MSI Z77A-G41 Intel Z77 Chipset

Intel Core i5-3570 (not k)

Geil Evo Corsa PC3-13600 32 GB (Quad-kit)

XFX AMD Radeon HD 6450 GDDR3 1 GB (I will buy another one, as I want to dedicate it to Hyper-v inside ESXi. I know I'm crazy )

The motherboard has the latest version of BIOS.

Disbled HD Audio and Internal Graphic of the CPU (thank you for anyone posted these two hints regarding this chipset).

I disabled the VMware SVGA card as suggested.

Tested it on VMware ESXi 5.1 (I will try USB controllers later, as I know they could have problem) and Windows 8 Enterprise 32 bit (I used 32 bit to keep it simple and use less RAM, actually I gave 1 GB).

Also HDMI worked well, tested it right now only with Youtube.

Installed latest Catalyst 12.8 with Windows 8 support.

Thank you again anyone!

Sandro

[Artwright]

@kthree, Thanks for your response.  The only reason for the non-M is that it is cheaper where I live. Also thought the extra slots might come in handy if passthrough works well. Would appreciate if when you have time whether you can confirm passing through the other GPU works well. Thanks once again,

[Artwright]

Good news.  I managed to get the directpath GPU passthrough working on my Asrock Z68 Extreme 4 Gen 3.

I initially followed the suggestions by kthree about disabling the onboard Audio, and also setting the PCI-Express card as the preferred boot option rather than onboard.  I also had set the iGPU multirender to disabled and Render Standby to disabled.  But that still resulted in a BSOD for me everytime I connected a monitor to the card (even though the drivers installs fine and Windows detects the card)

Then I read through the manual and found an interesting comment about the DVMT memory option which should be set to auto which disables the onboard GPU if a PCI-E card was present.  I had set it to 32MB.  Changing that to Auto immediately fixed my problem.

I have tested with 3D Mark 2011 and played some D3 on it and it is all stable so far and working well.

My system for those interested: -

Asrock Z68 Extreme4 Gen 3 (Bios v 2.20)

Core i7-3770

HIS ATI 7750 graphics card.

Tested with Windows 7 64Bit and Windows 8 with latest ATI 12.8 Catalyst drivers.

ESXi 5.0 with latest patch (although I found it worked without patching to the latest, I think even pre-update 1 works for me).

ESXi 5.1 works but USB passthrough doesnt as many here have indicated.

Thanks.

[Jt3rry]

Guys, thanks so much for all your efforts here. I've been following this thread on and off for a few months now. Glad to see some are on stable/working setups!

Question, has anyone been able to pass a standard PCI device through successfully, or is there something specific that will only allow pcie devices? I have a few PCI capture cards I'd love to be able to use in this type of configuration.

Also, earlier in this thread there was a matrix showing known working/non-working GPU/chipset combos... any interest in refreshing that doc? I think it'd be a huge help to everyone starting out here.

thanks!

[ciuly]

that would be a bit hard to refresh. I've made a document myself with all the stuff, mostly the working setups. I will share it once I get home and allow anyone to edit it. So it would be a good idea for people to go over it and if any info is outdated, update the document, maybe add some comments as well.

As an update to my setup (an intel 67 chipset board), I did a bios update the other days and used latest ati drivers at the time and still nothing. I'm now fishing for a board that works, but I think I'll wait a little for some of you guys to update that document I'll be sharing and see which board gets the most "working" statuses :smileygrin:

[KRA77]

JT,

This is my story with PCI devices

On my mobo all pci devices are connected, so you can only use them in one VM.

This is what I've tried:

Hauppauge HVR4000 dvb S2 tuner, not successful, card is found and installed on both win/linux, but refuse to tune. (Is listed as 4 devices in esxi)

Agrippa 3D soundcard (C-Media) confirmed working on linux.

USB/Firewire card, Both windows and linux detect and install the card, however I can only get USB devices working on it. Both loads firewire drivers, but nothing is detected when connecting.

Currently my setup.

Pike 2008 sas/sata controller pased on to Openindiana VM (1 device)

Integrated USB hub pased on to windows (for usb tv-tuner cards) (1 device)

Integrated USB hub passed on to ubuntu server (1 device)

PCIe ATi 6450 passed on to xbmcbuntu (2 devices, only pass on gfx part, not hdmi sound)

PCI Agrippa 3D soundcard passed on to xbmcbuntu (1 device)

PCI USB/FW passed on to xbmcbuntu. (4 devices, usb1+usb1+usb2+fw)

Even though all PCI devices are linked together I run into problems passing it all along, seems like it reach a limit.

[ciuly]

As promised, here's the document I was keeping in work. Hopefully people will keep it up-to-date.

https://docs.google.com/spreadsheet/ccc?key=0Aqp_xYBwP_Y7dE5EclhtaDdIV09lNWxfODd1alRUTlE

[derickso]

Thanks ciuly this is great!!  I updated my info and did some minor formatting cleanup.  I also added a BIOS version column, which I think is relevant given the BIOS seems to be a major factor in the problems folks are seeing, so if everyone can add their BIOS version when possible that would be great.

[shlomiassaf]

Hi

I am really hopeless here....

My setup is:

SuperMicro X9SCM-F

E1240 V2

HSI Radeon 5450 PCIe 1x GPU sitting on a 4x PCIe

I am using ESXi 5.1 build 799733 and the latest ATI Drivers.

I cant setup GPU passthrough, no matter what I try I get error code 43 (in windows)

The error goes away when I disable the VMWare SVGA card but the passthrough still wont work...

I got the same error trying to passthrough the onboard GPU.

Do you think it is even possiable to passthrough with this configuration?

I might need to change the motherboard?

Please help...

[derickso]

Earlier posts reported you have to fully disable the onboard video (including comments about turning off memory sharing) to get it to work.  I'd also try ESX 5, not 5.1.  Also, is your motherboard/video card listed in the wiki as working?  If not, pick a working set from that list.

[shlomiassaf]

Thanks for the help.

I disabled the onboard GPU physically...

Now, Memory sharing? is this related to VM memory sharing or memory sharing feature through the BIOS?

[derickso]

Check earlier in the thread, but I believe it referred to disabling the shared memory that the onboard GPU uses.

[shlomiassaf]

I Can confirm working on my system (Thanks derickso)

Supermicro X9SCM-F

Xeon E3-1240 V2

HIS Radeon 5450 with 1GB DDR pci-e 1x (running on pcie 4x)

16 GB Memory (6GB assigned to VM with pciHole param 1200 to 2200)

To get VMDP I had to disable the onboard GPU, this was done physically using jumpers on the MB.

I also saw a report from someone saying this can be done without physically disabling the onboard GPU, instead he used bios settings - set the offboard GPU to be the primary gpu.

This is all great but I have one serious issue - This is working only once every host boot

Whenever I reboot (or shutdown and then boot) the VM (the one with GPU VMDP) the WHOLE host goes to halt.

You can not connect to any VM and also you can not connect the host using vSphere client.

Since I vmdp`ed the gpu there is no physical console to check the response.

This means I have to do a HARD reset...

This host has an unRaid on it... very bad for business...

Any ideas?

Thanks again for this wonderful post, it was extremley helpful!!!

[GBowman]

I had this problem, to overcome it you have to add pciPassthru(n).msiEnabled and set it to 'False' in the configuration file.  I do it only for the PCI passthru device that is the GPU, all others I leave as default which is 'True'

In reply to earlier posts about PCI pass through rather than PCIe, I have the same issues with my TV tuner cards, they pass through but I cannot get any signal.  Other cards I have tried are working, a Digium telephony card is currently stable running my home phones and a soundblaster live 24 in my test PC as the onboard sound does not work.

[shlomiassaf]

I have added the value to the advaned config... didnt work...

Any other thoughts?

[Bhargava_Shriva]

Hi,

I am using Dell T3500 workstation for testing conf as follows

• 12 GB RAM
• 1 TB HDD
• 2 GB Quadro 4000

when I bypass Quadro 4000 GPU and reboot the esx 5.1. It comes up to cnic_register loaded successfull and hungs.

Kindly give suggest what is the problem how to over come this.

Thanks in advance

[shlomiassaf]

Just a little after cinic_register shows up the host actually does the vmdp

This is why the screen hangs since the the host does not control it anymore

When you start the vm which  owns the gpu  the screen will go black and then when the os loads it will show on the screen...

This is the behavior in my system

[Bhargava_Shriva]

Thanks,

Actually I am facing this not on VM this error is on ESX itself.

It is freshly installed ESX then I am going to advanced option and bypassing the GPU (Quadro 4000) after restart I am getting cnic and hung.

Kindly let me know how do I bypass graphic card. So that I can have dedicated graphics on individual VM's.