Have you tried DXVA with something else than XBMC? My configuration is quite close to yours, and I am pretty sure that it should also be able to run DXVA.

I was unable to get it working with: Flash, FFDSHOW, W7 default WMF codecs.

Here is my configuration:

- ESXi 4.1

- SuperMicro mainboard (can't remember the ref; I'll check tonight)

- HD 5450 (VMDP)

- Highpoint RocketRaid 3520 (VMDP)

- 24gb of RAM

- Hauppauge Nova-T (VMDP, but still testing)

I shall maybe give a try tonight with a fresh VM. Having a VM as a HTPC would be awesome.