Highlighted
Expert
Expert

Swapping occuring on VM

Hello,

  does anybody have any idea why VM might swapping in this scenario ?

Esxi is with 2 cpu sockets.

esxtop

PMEM  /MB: 147445  total:  2380    vmk, 94536 other, 50528 free

VMKMEM/MB: 147060 managed:  2085 minfree, 72161 rsvd,  74899 ursvd,  high state

NUMA  /MB: 73727 (25330), 73716 (24813)

PSHARE/MB: 54183  shared,  2288  common: 51895 saving

SWAP  /MB: 21495    curr, 19533 rclmtgt:                0.01 r/s,  0.00 w/s

ZIP  /MB:  978  zipped,  535  saved

MEMCTL/MB:  149    curr,  149  target, 120152 max

So Esxi has 50528 MB free.

     GID NAME MEMSZ    GRANT    SZTGT     TCHD   TCHD_W    SWCUR    SWTGT   SWR/s   SWW/s  LLSWR/s  LLSWW/s   OVHDUW     OVHD  OVHDMAX

22337523 X1   65536.00 45561.91 14038.55  1310.72  1310.72 18735.35 18030.91    0.01    0.00     0.00     0.00    11.33   452.70   536.31

22326807 X2   49152.00 48950.75 47509.36 18186.24 14254.08     0.00     0.00    0.00    0.00     0.00     0.00    10.71   386.31   413.77

22330771 X3   24576.00 15906.00 10594.84   245.76   245.76     0.00     0.00    0.00    0.00     0.00     0.00    14.15   157.07   234.34

22225951 X4   24576.00 14925.84 13254.63     0.00     0.00     0.00     0.00    0.00    0.00     0.00     0.00    14.08   160.47   234.27

22225964 X5   24576.00 19298.36 14181.61   244.27   244.27  2757.98  1490.13    0.00    0.00     0.00     0.00    14.08   184.22   234.27

SWCUR for vm X1 and X5 is > 0 . I have no idea, why ESXi wants to swap VM memory to disk, if he still have 50 GB memory free. Host is currently in memory High state so, that would mean > 6% mem is free, if anything he should try balooning first.

I would appreciate any hints that would allow me to understand this.

Many thanks!

--- @blog https://grzegorzkulikowski.info
0 Kudos
6 Replies
Highlighted
Expert
Expert

If i did my math right then you have over-committed your host in terms of memory (148GB in host, 188GB configured for VMs.). To support this vmkernel might employ one or more memory reclamation techniques - TPS, balooning, compression, swap - as needed.

Additional questions to sort this out:

Primary:

- do you have VMware Tools installed and active in your VMs? (if not, this could result in immediate swapping (no balooning) when memory is over-committed, looks like memctl current / target is fairly small, which makes me think that VM tools might be missing on most VMs)

- how are memory reservations configured for your VMs / resource pools? (aggressive reservations in 1 VM may result in swapping of another one. Any chance that X2, X3 and X4 might have reservations set?)

- any chance you powered all VMs on that host at once? (if memory is overcommitted and you do a boot storm, TPS has no time to kick in, baloon drivers even if installed are not running yet so vmkernel drops all the way down to compression / swap if needed)

Secondary:

- are there any memory limits configured for VMs and / or resource pools? (this will definitely result in balooning and / or swapping if guest tries to use more memory than specified limit)

General guidelines to avoid swapping:

Do not overcommit your memory too aggressively

Right size your VMs

Always install VMware Tools so that we could do balooning if needed, which is a lot less painful (if at all) than swapping

Enable TPS in the old-fashion way so that esxi can share pages across all VMs, if you are not concerned about the inter-vm page sharing security issue (lots of reading material out there, can start with this one: http://kb.vmware.com/kb/2080735)

Avoid boot storms, if you can.

hope this helps

0 Kudos
Highlighted
Expert
Expert

Forgot one thing - once you are at the point where you have overcommitted host's / cluster's memory and VMs are under contention, in most cases I'd recommend configuring memory shares before mocking around with reservations and limits.

0 Kudos
Highlighted
Expert
Expert

Hi Kermic,

thanks for the reply. Today i made new screenshots ,as few vmotions were made:

p1.png

And those vms running there are configured like this:

vms.png

Now that you have mentioned , you are 100% correct, actually i am over committing  memory. But, what i don't understand why it is shown that PMEM for host is 58 GB free ? Any idea what it says this in esxtop ? All vms have balloon driver working. all vms are swapping 😕 , 2 of them also ballooning.

Esxi looks like this from vsphere client:

esxi.png

Some of the memory of the host has to be reserved for the admission control i suppose.

--- @blog https://grzegorzkulikowski.info
0 Kudos
Highlighted
Expert
Expert

Could you check if there are any memory limits configured either on VMs or Resource Pools / vApps?

0 Kudos
Highlighted
Contributor
Contributor

VMs that are swapping are windows/linux type (VMware Tools are up to date)? Can you check sum of swapped memory?

0 Kudos
Highlighted
Expert
Expert

Considering there is 48 GB reserved memory the memory scheduler has to manage 116 GB of remaining allocation in 96 GB of remaining available memory.

Also the machine X2 which is allocated 48GB seems to be using all the memory without any reservation.

The same seems to be the case with other machines, where they are using most of the allocated memory if not all.

Hence there does not seems to be room for Ballooning and obviously swapping is the next option.

If there have been recent vMotions, the math might change. But if we keep the below facts in mind we should get our answers

1) the reserved memory can never be touched

2) once the guest is using it, vmkernel will not balloon (ballooning is only possible if any machine is not using the allocated memory)

3) the next in the queue(power on sequence) will have to do with what is available.

Hope that answers the question

If you found my answers useful please consider marking them as Correct OR Helpful Regards, Hussain https://virtualcubes.wordpress.com/
0 Kudos