If i did my math right then you have over-committed your host in terms of memory (148GB in host, 188GB configured for VMs.). To support this vmkernel might employ one or more memory reclamation techniques - TPS, balooning, compression, swap - as needed.
Additional questions to sort this out:
- do you have VMware Tools installed and active in your VMs? (if not, this could result in immediate swapping (no balooning) when memory is over-committed, looks like memctl current / target is fairly small, which makes me think that VM tools might be missing on most VMs)
- how are memory reservations configured for your VMs / resource pools? (aggressive reservations in 1 VM may result in swapping of another one. Any chance that X2, X3 and X4 might have reservations set?)
- any chance you powered all VMs on that host at once? (if memory is overcommitted and you do a boot storm, TPS has no time to kick in, baloon drivers even if installed are not running yet so vmkernel drops all the way down to compression / swap if needed)
- are there any memory limits configured for VMs and / or resource pools? (this will definitely result in balooning and / or swapping if guest tries to use more memory than specified limit)
General guidelines to avoid swapping:
Do not overcommit your memory too aggressively
Right size your VMs
Always install VMware Tools so that we could do balooning if needed, which is a lot less painful (if at all) than swapping
Enable TPS in the old-fashion way so that esxi can share pages across all VMs, if you are not concerned about the inter-vm page sharing security issue (lots of reading material out there, can start with this one: http://kb.vmware.com/kb/2080735)
Avoid boot storms, if you can.
hope this helps
Forgot one thing - once you are at the point where you have overcommitted host's / cluster's memory and VMs are under contention, in most cases I'd recommend configuring memory shares before mocking around with reservations and limits.
thanks for the reply. Today i made new screenshots ,as few vmotions were made:
And those vms running there are configured like this:
Now that you have mentioned , you are 100% correct, actually i am over committing memory. But, what i don't understand why it is shown that PMEM for host is 58 GB free ? Any idea what it says this in esxtop ? All vms have balloon driver working. all vms are swapping :/ , 2 of them also ballooning.
Esxi looks like this from vsphere client:
Some of the memory of the host has to be reserved for the admission control i suppose.
Could you check if there are any memory limits configured either on VMs or Resource Pools / vApps?
VMs that are swapping are windows/linux type (VMware Tools are up to date)? Can you check sum of swapped memory?
Considering there is 48 GB reserved memory the memory scheduler has to manage 116 GB of remaining allocation in 96 GB of remaining available memory.
Also the machine X2 which is allocated 48GB seems to be using all the memory without any reservation.
The same seems to be the case with other machines, where they are using most of the allocated memory if not all.
Hence there does not seems to be room for Ballooning and obviously swapping is the next option.
If there have been recent vMotions, the math might change. But if we keep the below facts in mind we should get our answers
1) the reserved memory can never be touched
2) once the guest is using it, vmkernel will not balloon (ballooning is only possible if any machine is not using the allocated memory)
3) the next in the queue(power on sequence) will have to do with what is available.
Hope that answers the question