Help me understand RAM Graph

[dambart]

Hi, 

I'm trying to understand why i have such difference between ESXi VM Memory Graph value and actual guest memory usage.

My machine is a K8S node configured on debian 11. 

Spec are 8vCPU and 128GB of RAM.

I'm trying to monitor RAM usage with a zabbix server which takes the vmware: Host Memory Usage in percents. 

In ESXI Performance tab, this is the value of Usage for 'myVM'

I cannot understand why inside the VM a free -m shows : 

              total        used        free      shared  buff/cache   available
Mem:          128829       96822       17221          12       14785       31363
Swap:              0           0           0

and usage in graph shows 3% used.

I made a test on a new VM and the percentage is pretty accurate when i tried to consumed 90% of free ram and then 50%

Is there something i missed out ? 

Thanks for your help

[StephenMoll]

Does the VM have memory reservations set?

[dambart]

No reservation, see on the screenshot, memory active is 6,4GB this morning.

Htop on the second screenshot and free -m on the third one.

ESXi version : 6.7.0, 19997733

open-vm-tools installed on this machine, version 12320 (12.1.0) using bullseye backport

[StephenMoll]

I wonder if this is intra-VM TPS doing its thing?

So for example, if the VM is reserving a large proportion of that memory for caching/buffering or whatever, and preloading it with zeros, at the hypervisor it may collapse all the zero space, because ESXi can share memory pages within a VM, so with sufficient pages being all zeros ESXi would only keep one copy and map it to VM memory locations where it is used.

That is the only explanation I can think of now. 

Do you have SCA mitigations in place for Spectre/Meltdown?

[StephenMoll]

Uggh!  Don't why I mentioned SCA. 

I really meant to ask what is the TPS config?

Check out this : https://kb.vmware.com/s/article/2097593

If you have TPS enabled for the VM in question, you could try forcing TPS into an inoperative state, and see if that changes anything.

[dambart]

First, thanks for your replies Stephen it is really appreciated.

I think i'm not exposed to spectre / meltdown as the ESXi is running on two Xeon Gold 6248R (cascade lake)

For buffered & cached memory, that was also my first guess but as I can see today, the amount of buff/cache ram is 6GB inside guest.

For TPS, didn't know about it, thanks for the link. 

I have a default installation and after checking it, Mem.ShareForceSalting=2 on ESXi Host and no mention of sched.mem.pshare.salt in vmx guest file. Which as i understand means that there is no No Intra-VM TPS.

I made a screenshot of esxtop, maybe it will be helpful

[StephenMoll]

I assume you have the "2 (default)" setting in which case Inter-VM is disabled, but intra-VM is enabled. So memory pages inside the VM can be collapsed, if there are multiple memory pages with the same content.