ESX host server is an HP Proliant DL585 G1 running VMware ESX Server, 3.0.1, 32039, 2 CPUS at 2.799GHz each with 12GB of physical memory. Guest OS is running Windows Server 2003 Standard SP1, 2 vCPU, 2GB of memory.
When I power on the guest OS, the memory usage is approximately 1.2GB. I have disabled all unnecessary services and even started the vm in safe mode. For all intents and purposes, this is running the bare minimum to be an operational server. When started in safe mode, there are 13 processes running and the guest os is using approximately 900MB of memory. By looking at the processes in Task Manager, I cannot see what is using all of the memory. Adding up the memory used does not come close to what the OS says it is using.
Also, this high memory usage does not happen every time the vm is rebooted, it normally happens on every other reboot.
Thanks in advance for any help you can give.
Are you sure about this? To find out how much available memory you have you should add up (under physical memory) Available + System Cache.
Lars
When I look in the task manager, I see:
Total 2096584
Available 828540
System Cache 78024
There are 31 processes running and not one of them is using more than 22,000KB.
Does this make sense to you? Again, thank you very much for your help and support!!
I have found it can take 10+ hours for memory to calm down on Windows Server VMs.
Also, 2 vcpus is not normally recommended, why do you need 2?
I can understand that there will be a higher usage when a VM starts up because of the applications starting and it will settle out after a period of time. Ours will eventually release the memory after a few hours, but in those few hours our users will be seeing high latency and this is not acceptable. There must be a reason why over half of the memory is being used, but the running applications are not using it.
We are running 2vcpus because the application administrators believe that because they had a physical server with 2 CPUs that this one should have it. I understand that VMware does not recommend using 2 and they aren't actually utilized unless the application is built for it. But, we were told by our management to appease the application folks by giving them what they want. So, here we are...
Thanks very much for your thoughts!
I started a similar thread a while back, check out the replies...
http://www.vmware.com/community/thread.jspa?threadID=71011&tstart=0
My guess is it probably has something to do with the balloon memory driver or the creation of the vswp file when the VM boots up.
Also here's a good memory white paper...
Here's some additional info I found...
Don't rely on guest memory usage! Guest cant tell if the memory pages are from the swap file or reclaimed from another virtual machine
cat /proc/vmware/sched/mem - shows per virtual machine memory stats
vm mctl? wait shares min max size/sizetgt memctl/mctltgt swapped/swaptgt
\- Size <> sizetgt: Indicates memory pressure. Size > sizetgt : virtual machine is ballooning out. Size < sizetgt: virtual machine is reclaiming memory
\- Swappped: Currently swapped to VMkernel page file
\- Mctl: Balloon driver availability in the guest Tools installed ?
\- Wait: yes/no - VM waiting for memory !
Actually, I believe I found the answer, this white paper will tell you everything you every wanted to know about memory in ESX, here's an excerpt....
http://www.vmware.com/pdf/usenix_resource_mgmt.pdf
When the experiment is started, all five VMs boot concurrently.
Windows zeroes the contents of all pages in
physical memory while booting. This causes the system
to become overcommitted almost immediately, as
each VM accesses all of its memory. Since theWindows
balloon drivers are not started until late in the boot sequence,
ESX Server is forced to start paging to disk. Fortunately,
the share before swap optimization described
in Section 4.3 is very effective: 325 MB of zero pages
are reclaimed via page sharing, while only 35 MB of
non-zero data is actually written to disk.7 As a result of
sharing, the aggregate allocation to all VMs approaches
1200 MB, exceeding the total amount of machine memory.
Soon after booting, the VMs start executing their
application benchmarks, the amount of shared memory
drops rapidly, and ESX Server compensates by using
ballooning to reclaim memory. Page sharing continues
to exploit sharing opportunities over the run, saving approximately
200 MB.
is there any way of rebuilding the server with a single CPU, you will be getting a preformance hit on dual CPU
I have created a new virtual machine from the same template that built the problem server (same specs). It is not having the abnormal memory usage and we are hitting it as hard as we can. Even when we hammer it, we still don't see memory being used like before.
I have a trouble ticket submitted with VMware and they are wondering if it not security settings that we change after the VM is built. We will be looking into that next.
I have not compared the memory usage that is reported by the OS and what is reported in VirtualCenter. I am not concerned with how much the ESX server thinks the guest OS is using, only that the OS seems to be using so much memory for no reason.
Thanks for the replies.
After over a month of working with VMware, we finally have found the answer to our issue. We were unknowingly limiting the amount of physical memory that the ESX host was giving the VM. Even though we were assigning 3GB of memory, we were limiting the memory within Virtual Center to 1GB. This forced the VM to use the swap file located on the ESX host when it needed more memory and in turn forced Windows to work harder. We were only seeing high memory usage on this VM, but on others, we also saw high CPU usage.
To fix the issue, Edit Settings on the VM, Resources Tab, Click on Memory, Set the limit equal to the amount of physical memory given or check the Unlimited box. You will see the memory usage in Task Manager drop very quickly.
THANK YOU for posting the follow up to this. I had this same problem & have been banging my head against a wall for days now over it. I am utilizing resource pools (esx 3.5, vc 2.5) and their settings were not overriding the setting on the VM?! I had several VMs w/ 2GB memory but the resource tab was limiting them to 512MB (the original setting). It caused the VM to think it has 2GB of memory - all of which was used...
Thanks again.
HI navsisa :
I had the same issue with you.And I follow your method looking at my source configuration like this :
Is this configured is right ? And I also find that some vm have a numa node ,and some vm no. I still don't know how to fixed Issue ;