I've been successful with both an AMD Radeon HD 3450 as well as a Radeon HD 6850.

The ESXi host is a SuperMicro X8SIA-F (Intel 3420 platform) with a Xeon X3440 and 16GB RAM.

The VM that's getting the graphics card is Windows 7 Ultimate x64.  Both the older (3450) and newer (6850) cards actually present two PCIe devices.  The first is the GPU and the second is an audio device (for audio over HDMI / DisplayPort).

In addition to the AMD GPU and AMD audio device, I'm passing through one of my two Intel USB host controllers (onboard device coming off of the 3420 PCH).  This lets me connect my keyboard, mouse, webcam and USB sound card for a fully functioning workstation as a VM.

I can confirm that any VM which has VMDirectPath devices must have 100% of it's configured memory reserved (this is the reason why I upgraded the host from 8GB to 16GB of RAM).

I am also running into the RAM limitation.  It took quite a few reboots of the VM, but I've determined that the RAM ceiling for my workstation VM is 2816 MB.  It's an odd number, but if I go any higher then Windows will quit at boot with a random BSOD.  It is very frustrating that I can't configure the workstation VM for a proper 4-6GB of RAM, but it's a tradeoff I'm willing to make for this setup.  Hopefully a future release of ESXi will address this limitation, but I'm not holding my breath as this setup is definitely not supported.

I initially tried using USB passthrough for the keyboard and mouse, but they didn't even show up in the list of available USB devices to pass through in vSphere.  It seems only USB storage and hardware locking keys are supported for USB passthrough.  Giving the VM direct access to the USB controller has been a great workaround.