Hi there,
the ESXi has quite complex and smart memory allocation technique. Let me break it down a little bit for you:
Once the ESXi is booted up, it consumes a certain amount of system resources for itself. When you start booting up Virtual Machines, each of them has a certain memory overhead according to its vCPU count and RAM amount which is used for VMkernel's resource scheduling purposes. The VMkernel actually needs a certain amount of memory to keep performing well (as any other OS) and therefore some thresholds are set.
Once the first ("soft") threshold is reached, the Transparent Page Sharing (see TPS | VMware vSphere Blog - VMware Blogs) mechanism is activated. If the memory commitment grows even more, VMkernel begins to reclaim some unused memory inside the OS (but never more than 35% of its allocated memory) using the Baloon driver (inflate = retrieve unused memory from the running VM and provide it to VMkernel).
When the RAM use grows further, memory compression algorithms are engaged (VMkernel analyzes the pages and these which promise at least 50% compression ratio are compressed, up to 10% of total VM's memory). When the worst scenario happens, the VMkernel will start swapping Virtual Machines' memory to their .vswp files.
Please take note that swapping should never happen, even to an SSD - if you take a look, RAM access is measured in units of nanoseconds, whereas SSD access times are measured in tenths of milliseconds, this means that swapping on SSD will be still ~10.000x slower than to RAM.
Therefore in short:
For extensive information, look here: Memory Counters