6 Replies Latest reply on Feb 23, 2016 2:56 AM by hussainbte

    Swapping occuring on VM

    Grzesiekk Expert

      Hello,

        does anybody have any idea why VM might swapping in this scenario ?

      Esxi is with 2 cpu sockets.

       

      esxtop

       

      PMEM  /MB: 147445  total:  2380    vmk, 94536 other, 50528 free

      VMKMEM/MB: 147060 managed:  2085 minfree, 72161 rsvd,  74899 ursvd,  high state

      NUMA  /MB: 73727 (25330), 73716 (24813)

      PSHARE/MB: 54183  shared,  2288  common: 51895 saving

      SWAP  /MB: 21495    curr, 19533 rclmtgt:                0.01 r/s,  0.00 w/s

      ZIP  /MB:  978  zipped,  535  saved

      MEMCTL/MB:  149    curr,  149  target, 120152 max

       

      So Esxi has 50528 MB free.

       

       

           GID NAME MEMSZ    GRANT    SZTGT     TCHD   TCHD_W    SWCUR    SWTGT   SWR/s   SWW/s  LLSWR/s  LLSWW/s   OVHDUW     OVHD  OVHDMAX

      22337523 X1   65536.00 45561.91 14038.55  1310.72  1310.72 18735.35 18030.91    0.01    0.00     0.00     0.00    11.33   452.70   536.31

      22326807 X2   49152.00 48950.75 47509.36 18186.24 14254.08     0.00     0.00    0.00    0.00     0.00     0.00    10.71   386.31   413.77

      22330771 X3   24576.00 15906.00 10594.84   245.76   245.76     0.00     0.00    0.00    0.00     0.00     0.00    14.15   157.07   234.34

      22225951 X4   24576.00 14925.84 13254.63     0.00     0.00     0.00     0.00    0.00    0.00     0.00     0.00    14.08   160.47   234.27

      22225964 X5   24576.00 19298.36 14181.61   244.27   244.27  2757.98  1490.13    0.00    0.00     0.00     0.00    14.08   184.22   234.27

       

      SWCUR for vm X1 and X5 is > 0 . I have no idea, why ESXi wants to swap VM memory to disk, if he still have 50 GB memory free. Host is currently in memory High state so, that would mean > 6% mem is free, if anything he should try balooning first.

       

      I would appreciate any hints that would allow me to understand this.

       

      Many thanks!

        • 1. Re: Swapping occuring on VM
          kermic Expert

          If i did my math right then you have over-committed your host in terms of memory (148GB in host, 188GB configured for VMs.). To support this vmkernel might employ one or more memory reclamation techniques - TPS, balooning, compression, swap - as needed.

           

          Additional questions to sort this out:

           

          Primary:

          - do you have VMware Tools installed and active in your VMs? (if not, this could result in immediate swapping (no balooning) when memory is over-committed, looks like memctl current / target is fairly small, which makes me think that VM tools might be missing on most VMs)

          - how are memory reservations configured for your VMs / resource pools? (aggressive reservations in 1 VM may result in swapping of another one. Any chance that X2, X3 and X4 might have reservations set?)

          - any chance you powered all VMs on that host at once? (if memory is overcommitted and you do a boot storm, TPS has no time to kick in, baloon drivers even if installed are not running yet so vmkernel drops all the way down to compression / swap if needed)

           

          Secondary:

          - are there any memory limits configured for VMs and / or resource pools? (this will definitely result in balooning and / or swapping if guest tries to use more memory than specified limit)

           

          General guidelines to avoid swapping:

          Do not overcommit your memory too aggressively

          Right size your VMs

          Always install VMware Tools so that we could do balooning if needed, which is a lot less painful (if at all) than swapping

          Enable TPS in the old-fashion way so that esxi can share pages across all VMs, if you are not concerned about the inter-vm page sharing security issue (lots of reading material out there, can start with this one: http://kb.vmware.com/kb/2080735)

          Avoid boot storms, if you can.

           

          hope this helps

          • 2. Re: Swapping occuring on VM
            kermic Expert

            Forgot one thing - once you are at the point where you have overcommitted host's / cluster's memory and VMs are under contention, in most cases I'd recommend configuring memory shares before mocking around with reservations and limits.

            • 3. Re: Swapping occuring on VM
              Grzesiekk Expert

              Hi Kermic,

              thanks for the reply. Today i made new screenshots ,as few vmotions were made:

              p1.png

              And those vms running there are configured like this:

              vms.png

              Now that you have mentioned , you are 100% correct, actually i am over committing  memory. But, what i don't understand why it is shown that PMEM for host is 58 GB free ? Any idea what it says this in esxtop ? All vms have balloon driver working. all vms are swapping :/ , 2 of them also ballooning.

              Esxi looks like this from vsphere client:

              esxi.png

              Some of the memory of the host has to be reserved for the admission control i suppose.

              • 4. Re: Swapping occuring on VM
                kermic Expert

                Could you check if there are any memory limits configured either on VMs or Resource Pools / vApps?

                • 5. Re: Swapping occuring on VM
                  dajbozerozum Novice

                  VMs that are swapping are windows/linux type (VMware Tools are up to date)? Can you check sum of swapped memory?

                  • 6. Re: Swapping occuring on VM
                    hussainbte Expert
                    vExpert

                    Considering there is 48 GB reserved memory the memory scheduler has to manage 116 GB of remaining allocation in 96 GB of remaining available memory.

                     

                    Also the machine X2 which is allocated 48GB seems to be using all the memory without any reservation.

                    The same seems to be the case with other machines, where they are using most of the allocated memory if not all.

                     

                    Hence there does not seems to be room for Ballooning and obviously swapping is the next option.

                     

                    If there have been recent vMotions, the math might change. But if we keep the below facts in mind we should get our answers

                    1) the reserved memory can never be touched

                    2) once the guest is using it, vmkernel will not balloon (ballooning is only possible if any machine is not using the allocated memory)

                    3) the next in the queue(power on sequence) will have to do with what is available.

                     

                    Hope that answers the question