For the free edition it is 32Gb physical RAM.
For license version, it is vRAM. This is confusing however am explaining as below.
If you have 10 VMs each allocated 4GB. Your total would be 40GB used.
E.g if you are using Essential license which is 32GB vRAM/socket. If you server has 2 socket (irregardless of cores), you would need to purchase 2 x Essential license. You will be entitled to use 64GB of vRAM.
In such, you will be allow to have 16VMs each at 4GB. However do note the vRAM is calculate base on Powered allocated on VM.
If at times due to contention e.g. balloning happens, but its not all the time which causes you to exceed the vRAM entitlement of 64GB, you will not be penalize. This is so as the allocated is base on a year of average usage.