1 2 Previous Next 21 Replies Latest reply on Jun 4, 2015 6:38 AM by KissTibor

    ESXi 5.1 incorrect guest memory usage reported.

    JCATVLee Novice

      Good morning,

       

      I've been having an issue with the guest VMs on my esxi 5.1 host for a few days now and simply cannot find any info on the web to assist me in trouble shooting and wondered if anyone here could help shed some light on my issue.

       

      Basically, when looking in vCenter or vSphere, I am seeing the active memory usage for my VMs is being reported, but when I log in to the guest OS on the VM and check the memory usage in the task manager, the figures are greatly different.

       

      It's a fairly new installation of VMWare (about 6 weeks old) and I am trying to right size my VMs, but I am unsure which figures to trust for memory usage.  Listed below is a couple of machines, operating systems and some figures that ESXi and the guest OS report for figures for memory usage.

       

      VM 1 (Win7) (2GB) vCenter reports 1.38GB used – OS reports 911MB used.

      VM 2 (Svr 2K8) (1GB) vCenter reports 307MB   - OS reports 993MB used.

      VM 3 (Win7) (2GB) vCenter reports 143MB – OS reports 850MB used.

       

      The thing that highlighted this error was when VeeamOne started emailing me errors saying that some of my guests where using too much memory, and when I investigated, I found that the figures reported in ESXi were differnet to those in the guest OS.

       

      Anyone have any suggestions?

       

      Lee

        • 1. Re: ESXi 5.1 incorrect guest memory usage reported.
          Linjo Champion
          User Moderators

          You are probably looking at different metrics, active vs allocated vs commited etc.

          vSphere will also take into account overhead for running the VM, videomemory, PCI Space memory etc.

           

          I would recommend to use a tool called vCenter Operations Manager to get better metrics and understanding on whats going on.

          That tool can also give recommendations about rightsizing.

           

          // Linjo

          • 2. Re: ESXi 5.1 incorrect guest memory usage reported.
            JCATVLee Novice

            Yeah, that I can kinda understand if vCenter was saying there was more memory being used in every instance, but as you can see from 2 of my VMs, the OS thinks that it is using way more memory than vCenter does.

             

            If vCenter was calculating video memory etc etc on top of the guest memory usage, surely vCenter would always report that more memory was being used?

            • 3. Re: ESXi 5.1 incorrect guest memory usage reported.
              JCATVLee Novice

              Does anyone out there have any ideas on this?

               

              I could undertsand if the issue was that vCenter was consistently reporting that they VM memory usage was higher than the actual OS thought it was due to the overheads etc, but some machines report less usage in vCenter than the OS and vice versa.  There seems to be no pattern to it.

               

              VeeamOne is sending me email reports out that machines are running out of memory, but when I log in to the OS, the task manager is telling me that the server is using nowhere near the amount of memory that vCenter is reporting.  And it can be a 25%-35% difference, not just a small amount.

               

              Many thanks.

              • 4. Re: ESXi 5.1 incorrect guest memory usage reported.
                elgreco81 Expert

                Hi,

                 

                When you say that vCenter is reporting x memory usage, which metric are you looking to?

                 

                Did you try to understand those numbers with "esxtop"?

                 

                I don't understand why this inconsistency buy we sure could get to the root of this. I would start by ignoring VeeamOne and limiting the first diagnose to what vCenter, esxtop and the OS have to say.

                 

                Regards,

                elgreco81

                • 5. Re: ESXi 5.1 incorrect guest memory usage reported.
                  JCATVLee Novice

                  OK.

                   

                  I am in vCentre, I have gone to my VM and am looking on the Resource Allocation Tab.  I have host memory consumed as being 2048mb and I have the guest memory showing as 163MB active and 2.00BG as private.  When I look in the performance tab at the memory consumption, I am being told that 146MB is currently active.

                   

                  However, when I look in the OS task manager, it is reporting that 671MB of memory is being used.

                   

                  That is just for one machine, if I look at another, I get the following:

                   

                  Host memory consumed 1024MB, Guest memory 409MB active1023MB private but the OS is rpeorting that 750MB is being used.

                   

                  VeeamONE seems to be reporting to me exactly what vCenter is seeing, so it would appear that VeeamONE and vCenter agree (or VeeamONE is simply reporting what vCenter is showing) and these values are different to what the guest OS thinks it is using.

                  • 6. Re: ESXi 5.1 incorrect guest memory usage reported.
                    elgreco81 Expert

                    Great! So if vCenter and Veeam are reporting the same, that's one thing less to worry about

                     

                    I always get confused with memory readings because I have bad memory and always forgot what each of the messures mean. I always have this link at hand

                     

                    http://www.vmware.com/support/developer/vc-sdk/visdk400pubs/ReferenceGuide/memory_counters.html

                     

                    The one that I always remember is that Active memory will tell you the pages that are currently being used and this is always a small number in comparison with other messures. Duncan Eping wrote about it in several posts, I found this one that it is really well explained.

                     

                    http://www.yellow-bricks.com/2010/12/20/vcenter-and-memory-metrics/

                     

                    After reading these, do you found any sense in what you are seing reported? If so, please post it !!!

                     

                    Regards,

                    elgreco81

                    • 7. Re: ESXi 5.1 incorrect guest memory usage reported.
                      JCATVLee Novice

                      Hmmmm....

                       

                      I'm still not really any wiser after reading those 2 posts.

                       

                      If vCenter was showing me that some memory was being shared, swapped or balooned or similar I woudl start to think that is why there is a difference between what the OS says it's using and what vCenter is saying.

                       

                      vCenter.jpg

                      guestos.jpg

                       

                      As you can see, there is a BIG difference between what is being reported.

                      • 8. Re: ESXi 5.1 incorrect guest memory usage reported.
                        Lurker

                        My understanding is that Active memory is memory that has been accessed recently (I don't know how recently though).

                         

                        Let's take VM3.  So if Windows tells you in Task Manager that 850MB is used, it means that processes, services, etc. are consuming 850MB of memory.  vCenter tells you that out of the 850MB used, only 143MB is active, so in essence, the rest of the memory used is kind of sleeping.

                         

                        vC Ops will focus more on the active memory because historically this is what may hurt performance.  When a Windows 2008 (Windows 7 also I think but not 100% sure) starts it marks all memory as available for buffers of some sort.  This means that it consumes all its memory upfront.  As processes are started (applications, services, etc.) it lets go of buffer memory and gives the memory to the processes.  So vCenter will report that all memory is consumed.  The reason it shows as private is because TPS is not activated until a good portion of the physical memory is consumed.

                         

                        So vCenter will tell you that your VM has consumed all 2GB, but only 143MB are currently being accessed.  Windows will report it has access to 2GB and 850MB are used.

                         

                        Veeam will report whatever vCenter is reporting.

                         

                        vC Ops will interpret the data.

                         

                        That's my understanding, but anyone feel free to correct me.

                        • 9. Re: ESXi 5.1 incorrect guest memory usage reported.
                          JCATVLee Novice

                          OK, so what you are saying is that this mismatch is normal in day to day operation?

                           

                          What about machines I have seen that have been the other way around?  vC reports that the machine is maxing out at over 95% of memory being used and the OS is only reporting that 30% is being used?  I don;t have screen shots of this happening at this point, but I witnessed this on this same environment last week.

                           

                          BTW, thanks so much for your time to reply.

                          • 10. Re: ESXi 5.1 incorrect guest memory usage reported.
                            Lurker

                            Ok, I'm stretching it a little here, but if your Windows VM has allocated 95% of its memory at some point and some of the processes have exited and memory was freed up but not zeroed out, vCenter would not know this memory can be reclaimed.

                             

                            So it could be a number of factors: freed memory not zeroed out; vmware tools not doing the reclamation; vmware tools simply not installed; etc.

                             

                            But these are just guesses.

                            • 11. Re: ESXi 5.1 incorrect guest memory usage reported.
                              JCATVLee Novice

                              Yeah, it certainly has me confused for sure!

                              • 12. Re: ESXi 5.1 incorrect guest memory usage reported.
                                TimCooke Novice

                                Might have stumbled across the answer, as I have just found something similar going on in our environment. I had the RAM increased on a couple of VMs from 8GB to 16GB. Windows reported pretty high memory usage (even though the processes running didn’t appear to be actually using that much), but in vCenter Consumed memory was much lower. I checked the Resources tab for the VMs and found the Memory Limit was set at 8GB rather than having the “unlimited” box checked, which would be normal for a new VM. On checking the box and clicking OK, memory usage in Windows did drop back down to something resembling what vCenter was reporting.

                                 

                                Did a search and found the issue is reported here: http://kb.vmware.com/kb/1003470.

                                 

                                Not happy as going to have to do some checking on 20K+ VMs to see if any of them are also showing a similar configuration.

                                 

                                Tim

                                • 13. Re: ESXi 5.1 incorrect guest memory usage reported.
                                  JCATVLee Novice

                                  Hi Tim,

                                   

                                  That sucks having to check such a huge environment.  I only have 2 hosts running about 15 VM's, so it was easy for me to check all our mahcines.

                                   

                                  I had checked this before and all of my VM's are set with no reservations or limits imposed on their memory usage.

                                   

                                  This really has me stumped as on some machines the difference between what the OS and what vSphere think the machine is using can be up to 500mb.  This is causing me some concerns when it comes to right sizing my environment as I don;t know which metric to trust when deciding how much memory to allocate to my VM's.

                                  • 14. Re: ESXi 5.1 incorrect guest memory usage reported.
                                    TimCooke Novice

                                    Yeah, it really sounds like a bug, which sucks, especially as it seems to be in 5.x too.

                                     

                                    Are you able to vmotion the VMs between the hosts, as suggested in the article, to see if that resets the usage in Windows? What is the pagefile utilization like? On the VMs I checked PF usage was really high too and also reset back to a more logical value once I had reset the memory limit.

                                     

                                    Tim

                                    1 2 Previous Next