does anybody have any idea why VM might swapping in this scenario ?
Esxi is with 2 cpu sockets.
PMEM /MB: 147445 total: 2380 vmk, 94536 other, 50528 free
VMKMEM/MB: 147060 managed: 2085 minfree, 72161 rsvd, 74899 ursvd, high state
NUMA /MB: 73727 (25330), 73716 (24813)
PSHARE/MB: 54183 shared, 2288 common: 51895 saving
SWAP /MB: 21495 curr, 19533 rclmtgt: 0.01 r/s, 0.00 w/s
ZIP /MB: 978 zipped, 535 saved
MEMCTL/MB: 149 curr, 149 target, 120152 max
So Esxi has 50528 MB free.
GID NAME MEMSZ GRANT SZTGT TCHD TCHD_W SWCUR SWTGT SWR/s SWW/s LLSWR/s LLSWW/s OVHDUW OVHD OVHDMAX
22337523 X1 65536.00 45561.91 14038.55 1310.72 1310.72 18735.35 18030.91 0.01 0.00 0.00 0.00 11.33 452.70 536.31
22326807 X2 49152.00 48950.75 47509.36 18186.24 14254.08 0.00 0.00 0.00 0.00 0.00 0.00 10.71 386.31 413.77
22330771 X3 24576.00 15906.00 10594.84 245.76 245.76 0.00 0.00 0.00 0.00 0.00 0.00 14.15 157.07 234.34
22225951 X4 24576.00 14925.84 13254.63 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 14.08 160.47 234.27
22225964 X5 24576.00 19298.36 14181.61 244.27 244.27 2757.98 1490.13 0.00 0.00 0.00 0.00 14.08 184.22 234.27
SWCUR for vm X1 and X5 is > 0 . I have no idea, why ESXi wants to swap VM memory to disk, if he still have 50 GB memory free. Host is currently in memory High state so, that would mean > 6% mem is free, if anything he should try balooning first.
I would appreciate any hints that would allow me to understand this.
If i did my math right then you have over-committed your host in terms of memory (148GB in host, 188GB configured for VMs.). To support this vmkernel might employ one or more memory reclamation techniques - TPS, balooning, compression, swap - as needed.
Additional questions to sort this out:
- do you have VMware Tools installed and active in your VMs? (if not, this could result in immediate swapping (no balooning) when memory is over-committed, looks like memctl current / target is fairly small, which makes me think that VM tools might be missing on most VMs)
- how are memory reservations configured for your VMs / resource pools? (aggressive reservations in 1 VM may result in swapping of another one. Any chance that X2, X3 and X4 might have reservations set?)
- any chance you powered all VMs on that host at once? (if memory is overcommitted and you do a boot storm, TPS has no time to kick in, baloon drivers even if installed are not running yet so vmkernel drops all the way down to compression / swap if needed)
- are there any memory limits configured for VMs and / or resource pools? (this will definitely result in balooning and / or swapping if guest tries to use more memory than specified limit)
General guidelines to avoid swapping:
Do not overcommit your memory too aggressively
Right size your VMs
Always install VMware Tools so that we could do balooning if needed, which is a lot less painful (if at all) than swapping
Enable TPS in the old-fashion way so that esxi can share pages across all VMs, if you are not concerned about the inter-vm page sharing security issue (lots of reading material out there, can start with this one: http://kb.vmware.com/kb/2080735)
Avoid boot storms, if you can.
hope this helps
Forgot one thing - once you are at the point where you have overcommitted host's / cluster's memory and VMs are under contention, in most cases I'd recommend configuring memory shares before mocking around with reservations and limits.
thanks for the reply. Today i made new screenshots ,as few vmotions were made:
And those vms running there are configured like this:
Now that you have mentioned , you are 100% correct, actually i am over committing memory. But, what i don't understand why it is shown that PMEM for host is 58 GB free ? Any idea what it says this in esxtop ? All vms have balloon driver working. all vms are swapping 😕 , 2 of them also ballooning.
Esxi looks like this from vsphere client:
Some of the memory of the host has to be reserved for the admission control i suppose.
Considering there is 48 GB reserved memory the memory scheduler has to manage 116 GB of remaining allocation in 96 GB of remaining available memory.
Also the machine X2 which is allocated 48GB seems to be using all the memory without any reservation.
The same seems to be the case with other machines, where they are using most of the allocated memory if not all.
Hence there does not seems to be room for Ballooning and obviously swapping is the next option.
If there have been recent vMotions, the math might change. But if we keep the below facts in mind we should get our answers
1) the reserved memory can never be touched
2) once the guest is using it, vmkernel will not balloon (ballooning is only possible if any machine is not using the allocated memory)
3) the next in the queue(power on sequence) will have to do with what is available.
Hope that answers the question