It sounds like you have set a max or min level for your mem on that VM. Make sure that mem is set to unlimited.
I have seen this on some of my servers. They where also doing swapping...
If you found this information useful, please consider awarding points for "Correct" or "Helpful". Thanks!!!
Memory ballooning is a technique where the hypervisor takes memory of a VM who doesn't need it to give it to a VM that does need it. Do you have other VM's on that ESX which are memory hungry?