I've seen this problem in U3 and in previous releases. The first time I had this problem was due to using the EMC Control Center software. It was constantly connecting to all of our ESX systems even though most of our systems weren't set for it. We would get continual failed logins in the hostd.log. It seems that each failure would eat a little more memory and never get released until hostd crashed and restarted. The second time I saw this was because a couple of vms were stuck in a state of reconfiguring themselves over and over.

-jason