Any special configuration for VMware Tools?
Is it working automatically?
Ballooning will not prevent a single VM from running out of memory, it is a mitigation to prevent the host from running out of memory by stealing memory from less used VMs to the ones in need.
Memory management (including ballooning) in ESXi is still based upon the ideas presented at OSDI'02:
Thus, the ballooning idea is not completed yet.