I was also unlucky with nvidia cards. I noticed the same problems with a 8400 and a GT210.

For the WDDM, I did as you did; I installed the tools provided by my ESXi (4.1U1, I haven't done any other updates).

I'd be interested in more details about the configuration working with DXVA (ESX version, driver, ...). I didn't checked if I had VDDM 1 or 1.1 though.