I have a esxi host that has 64GB RAM and 8 CPU.
I have 3xVMs (Database) running on it each with 32GB RAM. I understand it means they are allowed to use 32GB and it does not mean they are going to use 96GB (All 3xVMs total) RAM.
I am now getting memory contention.
I'd like to know what would be the best practice in terms of RAM allocation.
Most operating systems will touch all memory pages on boot of the OS, which means that if you have 1 host with 96GB of memory allocated across 3 VMs but only 64GB of physical memory backing it that you are indeed oversubscribed. At this point you may see memory reclamation techniques kicking in even.
You should monitor the VMs and first and foremost right-size them! If they use 8 or 12GB then you assign them what they use plus some headspace.
Next you can consider enabling TPS across VMs and disabling Large Pages. This should allow TPS to "dedupe" similar memory pages and should lower the memory consumption.
If all of that fails, then you simply should buy more memory