TiJa,
you are right. It is certainly not the EMC agent that originally causes the problem, since it only does read-queries to the host (via the SDK and CIM) that can be considered "legal".
The problem is hostd not correctly releasing memory again. We also noticed that hostd RAM usage climbs up with every new VM on the host, but does not drop anymore, even if we put the host into maintenance mode and move all VMs away from it. So, this *is* a memory leak in hostd that can probably be triggered by other operations as well.
One reason why VMware has implemented soft and hard memory limits for processes (it's not only hostd, but also the VirtualCenter agent process that has such limits) is probably security. DOS attacks typically lead to increased memory usage of the attacked processes, and shutting them down would prevent the attack.
It is interesting though that in ESX*i* the hostd memory check is apparently disabled by default. If you look at /etc/vmware/hostd/config.xml on an ESXi host you will find the following interesting lines there:
<!-- Frequency of memory checker -->
<!-- Disabled pending resolution of stack size issue -->
<memoryCheckerTimeInSecs> 0 </memoryCheckerTimeInSecs>
I have not tried to use these lines on an ESX host, and I don't know if the comment (stack size issue?) also applies to ESX ...
Andreas