Have a quick skim of the whitepaper The Role of Memory in VMware ESX Server 3. It provides some great details.
It discusses such things as how shares effect memory, the memory tax, memory sharing and how ESX dynamically expands or contracts the memory for a VM.
Considering awarding points if this is of use