VMware Cloud Community
navsisa
Contributor
Contributor

Guest OS high memory usage

ESX host server is an HP Proliant DL585 G1 running VMware ESX Server, 3.0.1, 32039, 2 CPUS at 2.799GHz each with 12GB of physical memory. Guest OS is running Windows Server 2003 Standard SP1, 2 vCPU, 2GB of memory.

When I power on the guest OS, the memory usage is approximately 1.2GB. I have disabled all unnecessary services and even started the vm in safe mode. For all intents and purposes, this is running the bare minimum to be an operational server. When started in safe mode, there are 13 processes running and the guest os is using approximately 900MB of memory. By looking at the processes in Task Manager, I cannot see what is using all of the memory. Adding up the memory used does not come close to what the OS says it is using.

Also, this high memory usage does not happen every time the vm is rebooted, it normally happens on every other reboot.

Thanks in advance for any help you can give.

0 Kudos
12 Replies
larstr
Champion
Champion

Are you sure about this? To find out how much available memory you have you should add up (under physical memory) Available + System Cache.

Lars

0 Kudos
navsisa
Contributor
Contributor

When I look in the task manager, I see:

Total 2096584

Available 828540

System Cache 78024

There are 31 processes running and not one of them is using more than 22,000KB.

Does this make sense to you? Again, thank you very much for your help and support!!

0 Kudos
NWhiley
Enthusiast
Enthusiast

I have found it can take 10+ hours for memory to calm down on Windows Server VMs.

Also, 2 vcpus is not normally recommended, why do you need 2?

Neil VCP
0 Kudos
navsisa
Contributor
Contributor

I can understand that there will be a higher usage when a VM starts up because of the applications starting and it will settle out after a period of time. Ours will eventually release the memory after a few hours, but in those few hours our users will be seeing high latency and this is not acceptable. There must be a reason why over half of the memory is being used, but the running applications are not using it.

We are running 2vcpus because the application administrators believe that because they had a physical server with 2 CPUs that this one should have it. I understand that VMware does not recommend using 2 and they aren't actually utilized unless the application is built for it. But, we were told by our management to appease the application folks by giving them what they want. So, here we are...

Thanks very much for your thoughts!

0 Kudos
esiebert7625
Immortal
Immortal

I started a similar thread a while back, check out the replies...

http://www.vmware.com/community/thread.jspa?threadID=71011&tstart=0

My guess is it probably has something to do with the balloon memory driver or the creation of the vswp file when the VM boots up.

Also here's a good memory white paper...

http://www.vmware.com/pdf/esx3_memory.pdf

0 Kudos
esiebert7625
Immortal
Immortal

Here's some additional info I found...

Don't rely on guest memory usage! Guest cant tell if the memory pages are from the swap file or reclaimed from another virtual machine

cat /proc/vmware/sched/mem - shows per virtual machine memory stats

vm mctl? wait shares min max size/sizetgt memctl/mctltgt swapped/swaptgt

\- Size <> sizetgt: Indicates memory pressure. Size > sizetgt : virtual machine is ballooning out. Size < sizetgt: virtual machine is reclaiming memory

\- Swappped: Currently swapped to VMkernel page file

\- Mctl: Balloon driver availability in the guest Tools installed ?

\- Wait: yes/no - VM waiting for memory !

0 Kudos
esiebert7625
Immortal
Immortal

Actually, I believe I found the answer, this white paper will tell you everything you every wanted to know about memory in ESX, here's an excerpt....

http://www.vmware.com/pdf/usenix_resource_mgmt.pdf

When the experiment is started, all five VMs boot concurrently.

Windows zeroes the contents of all pages in

physical memory while booting. This causes the system

to become overcommitted almost immediately, as

each VM accesses all of its memory. Since theWindows

balloon drivers are not started until late in the boot sequence,

ESX Server is forced to start paging to disk. Fortunately,

the share before swap optimization described

in Section 4.3 is very effective: 325 MB of zero pages

are reclaimed via page sharing, while only 35 MB of

non-zero data is actually written to disk.7 As a result of

sharing, the aggregate allocation to all VMs approaches

1200 MB, exceeding the total amount of machine memory.

Soon after booting, the VMs start executing their

application benchmarks, the amount of shared memory

drops rapidly, and ESX Server compensates by using

ballooning to reclaim memory. Page sharing continues

to exploit sharing opportunities over the run, saving approximately

200 MB.

0 Kudos
TomHowarth
Leadership
Leadership

is there any way of rebuilding the server with a single CPU, you will be getting a preformance hit on dual CPU

Tom Howarth VCP / VCAP / vExpert
VMware Communities User Moderator
Blog: http://www.planetvm.net
Contributing author on VMware vSphere and Virtual Infrastructure Security: Securing ESX and the Virtual Environment
Contributing author on VCP VMware Certified Professional on VSphere 4 Study Guide: Exam VCP-410
navsisa
Contributor
Contributor

I have created a new virtual machine from the same template that built the problem server (same specs). It is not having the abnormal memory usage and we are hitting it as hard as we can. Even when we hammer it, we still don't see memory being used like before.

I have a trouble ticket submitted with VMware and they are wondering if it not security settings that we change after the VM is built. We will be looking into that next.

I have not compared the memory usage that is reported by the OS and what is reported in VirtualCenter. I am not concerned with how much the ESX server thinks the guest OS is using, only that the OS seems to be using so much memory for no reason.

Thanks for the replies.

0 Kudos
navsisa
Contributor
Contributor

After over a month of working with VMware, we finally have found the answer to our issue. We were unknowingly limiting the amount of physical memory that the ESX host was giving the VM. Even though we were assigning 3GB of memory, we were limiting the memory within Virtual Center to 1GB. This forced the VM to use the swap file located on the ESX host when it needed more memory and in turn forced Windows to work harder. We were only seeing high memory usage on this VM, but on others, we also saw high CPU usage.

To fix the issue, Edit Settings on the VM, Resources Tab, Click on Memory, Set the limit equal to the amount of physical memory given or check the Unlimited box. You will see the memory usage in Task Manager drop very quickly.

0 Kudos
rameredit
Contributor
Contributor

THANK YOU for posting the follow up to this. I had this same problem & have been banging my head against a wall for days now over it. I am utilizing resource pools (esx 3.5, vc 2.5) and their settings were not overriding the setting on the VM?! I had several VMs w/ 2GB memory but the resource tab was limiting them to 512MB (the original setting). It caused the VM to think it has 2GB of memory - all of which was used...

Thanks again.

0 Kudos
natsu_inaho
Contributor
Contributor

HI navsisa :

  I had the same issue with you.And I follow your method looking at my source configuration like this :

pastedImage_0.png

Is this configured is right ? And I also find that some vm have a numa node ,and some vm no. I still don't know how to fixed Issue ;

0 Kudos