Hi Guys. I have been reading through this thread today trying to get my 3D passthrough working with ESXi. It was so frustrating knowing that I was so close to getting it working and finally I can report that I have it working though I am not sure that I could lay out the exact steps for someone to replicate 
I will be reinstalling now that I have proved that it works so I will report back with the
How I got here! I have made a new rig specifically to take advantage of VT-d and passthrough..first seeing videos and forum posts of people having success with Xen. I managed successfully follow a Howto on the Xen wiki and got 3D passthrough to Windows 7 working with this setup so I knew that the hardware could do it. I was frustrated though because my sound card drivers were buggy in Linux/Xen and would cause the whole system to hard freeze (Reset button wouldn't even work!) This led me on a voyage of discovery to KVM, then Virtualbox and eventually ESXi in search of the ultimate hypervisor for my dream dual gaming / desktop system.
First here is my hardware setup
Motherboard: ASRock Z77 Extreme 4, BIOS version 2.80
GPU: Radeon HD7950, OEM branded as Asus DirectCUII Top (Model: HD7950-DC2G-3GD5-V2)
Display: Dell U2311H connected on the DVI-I port of the graphics card
CPU: Intel i7-3770
RAM: 16 GB
PCI: Creative X-fi Fatal1ty
So I've encountered all of pci hole, 2GB limit, BSODs with atikmpag.sys, catalyst control center not detecting stuff and here is a summary of what I have found:
1. First installed ESXi 5.1 Update 1. My plan was to follow what I did in Xen, that is install windows through the console then add the PCI Passthrough devices and reboot, then install drivers. Didn't realise that I needed VMWare Tools and went ahead to provision a Windows 7 64bit VM with 6GB of RAM. I then attached the GPU and HDMI devices and booted the VM. This is when VMWare told me that I needed to put pciHole.start = "2853" into the config file. I did that then next reboot I encountered the "Not fully ACPI compliant BIOS." I then realised that I had accidentally installed 32 bit Windows so I thought what the heck I'll go with it and reduced the RAM to 2GB and installed 32 bit Catalyst drivers. The card was detected and installed but the physical display was blank. I disabled the "Standard VGA Adapter" and then catalyst control center said it couldn't find the AMD card. I thought I made a big mess of this so I started again.
2. Deleted VM #1, made a proper Win7 64 bit VM this time installing the correct version of windows. Again with 6GB (as I initially thought the ACPI thing was due to a 32 vs 64 bit thing) and followed the same steps until I encountered the ACPI thing. I tried reducing the RAM again and ended up with the same result, a successful driver install yet a blank screen. As I was desperate at this time I tried installing VMWare Tools (default installation). After that I got the atikmpag.sys bluescreen. At around this time I started reading this thread.
3. I decided to go to ESXi 5.0 Update 1. I made new VM again with 2GB of RAM (no passthrough), installed windows, this time installed VMWare Tools first but did a custom install to leave out the SVGA video driver. After this I attached the PCI cards. Surprisingly this time there was no error that suggested to use pciHole.start. In the end it was the same result, a blank screen. One interesting side-note is that in 5.1 the PCI devices list had "AMD Radeon blah blah" listed while in 5.0 it says something like "Unknown VGA compatible controller." This is when I trawled through many pages of this thread and did all of the things suggested - disabling iGPU, disabling on-board audio, booting default to PCI Express. At this time I knew I was very close as the frozen image of the console would disappear when I started the VM, yet it was still remaining blank! Finally I did a reinstall of VMWare tools with all the default options (ie installing the SVGA driver) and it worked! To prove it I ran a 3D game.
What I have learned:
- VMWare Tools is needed but only for the SVGA Driver. Without it I get successful card detection and driver install but blank screen
- I can't find any way to get it working with IGPU enabled in the BIOS. I have tried to leave IGPU on and add it to the pass-through list, but not passed to anything, figuring that this would 'hide' it from the host and guests, but no luck there. The Windows VM BSODs with atikmpag.sys whenever IGPU is enabled in the BIOS.
- X-fi PCI Fatal1ty passthrough works but sound output is just white noise
Summary of my setup
VMWare Info
ESXi 5.0.0 Update 1
VM Windows 7 64 bit
Memory 2048 MB 100% reserved
CPUs 1 socket, 4 cores
Video card 1 display, 8MB RAM, Enabled 3D Unchecked
Passed through devices: 01:00.0 (AMD GPU), 01.00.0 (AMD HDMI Audio), 07:00.0 (ASMedia USB 3.0 Controller)
No custom entries in the .vmx file
BIOS settings
IGPU Multi-monitor Disabled
Primary graphics device PCI Express
Onboard Audio Disabled
Windows
Catalyst 13.4 (Driver 12.104.0.0) custom install, all drivers and options EXCEPT Catalyst Control Center
ASMedia USB Driver (latest version)
VMWare Tools installed with the VMware SVGA 3D adapter
My next steps are to try to get more than 2GB RAM, try some of the ideas here to get the Radeon as the primary adapter, some kind of acceptable sound and if possible the IGPU working before I declare this a winner.
Windows experience index:
Processor 7.6
Memory 5.5
Graphics 7.9
Gaming graphics 7.9
Primary hard disk 7.9
Can anyone tell my why the Memory score is so low? Using roughly the same configuration with Virtualbox and Xen it was in the 7's.
After reinstall with ESX 5.0 update1 , and try passthrough again,
everything run smooth even USB mouse and the sound