I've been successfully doing this for a while now, and I did try a 5450, but found it to have choppy performance, similar to what you're having.  So, IMO, it's your card, not specifically your build.  You could always keep that for your default video for your console (I'm using simple 8MB ATI Rage cards for my console graphics)

The two cards I've successfully done HTPCs in a virtual machine with are the HD6450 and HIS Radeon HD6670 PCI-e x16 Low-Profile.  The latter card I used to pass through to a VM for my son to use as his daily desktop (in fact, I've virtualized all but a single physical PC in the house) and he does gaming on it at very respectable frame rates.

You can see more of my builds and experiences at thehomeserverblog.com