Shouldn't be a host starving problem. If you're using unity or full screen mode, does it happen when in windowed mode?
One other thing - make sure you set 'prevent host from sleeping when screen blanks' in system preferences/energy, and set it to 'never' sleep. Sleeping a host while a guest is running is something that doesn't always work (sometimes it does, but others it doesn't).
