GREnvoy
Contributor
Contributor

VMWare Tools seems to lock 75%+ of available memory

I am running ESXi 4.1.0. I have a number of 2k8 R2 machines running on this server.

What I've noticed is that some, not all VMs are using 75%-95% of the available memory. When I look at the processes running, it's very minimal with nothing really using anywhere near that much memory. I used RamMap to try to try to see who was the hidden culprit. RamMap showed that "Driver Locked" was using 4-5GB of memory. (As shown here in another site thread)

http://www.experts-exchange.com/OS/Microsoft_Operating_Systems/Server/Windows_Server_2008/Q_26259037...

Upon further research, I found some people said this was related to VMWare Tools. Sure enough, as soon as I uninstalled VMWare Tools from the VM the memory went to where I would expect, and "Driver Locked" went to almost nothing, and eventually disappeared.

I installed VMWare tools with an upgrade when I converted the VM. Of the maybe 12 VMs I think 3-4 are experienceing this.

Anyone have any thoughts?

12 Replies
Dave_Mishchenko
Immortal
Immortal

How much memory has been allocated to your VMs and how much memory does the host have. It also seems that your host is overallocated and the balloon driver has kicked in or there may be a memory limit set on the VMs that have the problem.




Dave

VMware Communities User Moderator

Now available - vSphere Quick Start Guide

Do you have a system or PCI card working with VMDirectPath? Submit your specs to the Unofficial VMDirectPath HCL.

0 Kudos
RParker
Immortal
Immortal

Have we stopped to consider that MAYBE Rammap is simply inaccurate? (for Virtual Machines)

4-5GB of RAM, seems a bit excessive, considering you can install ANY VM then install the tools, and VM Tools don't use that much RAM for ANY installation, that would be a full scale epidemic and somebody would have reported it..

I am going to discount this as being one sided, because Process Explorer shows a very DIFFERENT story for RAM.

Therefore I don't think you considered ALL the options.

0 Kudos
GREnvoy
Contributor
Contributor

The server has 40GB the VM has 8GB allocated.

According to the Summary tab on the server Memory usage is at 37GB of the 40GB.

I have since uninstalled the VMWare Tools and reinstalled. The memory seems to be normal now. This just seems like there is a problem with the tools.

0 Kudos
GREnvoy
Contributor
Contributor

Well if It's a matter of RamMap being wrong, it is wrong as well as task manager. I started digging into this because Task Manager was telling me that physical memory was at 75%-95%.

0 Kudos
GREnvoy
Contributor
Contributor

Here is a screenshot from my situation. It shows that the system is showing 3.6GB of memory being used, but RamMap is the biggest hitter at 133,632K.

The screen shot does not show it, but there are only 57 processes.

0 Kudos
GREnvoy
Contributor
Contributor

Another Update.

I am now seeing that the VMs that I uninstalled-reinstalled the tools on, are eventually going back to having the problems. I also believe that we do in fact have more memory assigned to running VMs then there is physical in the machine.

0 Kudos
Dave_Mishchenko
Immortal
Immortal

What does the memory balloon counter show for the host? If you want to verify if that is the cause, give one of the VMs a memory reservation equal to the memory assigned to the VM and you should see the problem go away (for that VM). Do you know how much you're over subscribed?




Dave

VMware Communities User Moderator

Now available - vSphere Quick Start Guide

Do you have a system or PCI card working with VMDirectPath? Submit your specs to the Unofficial VMDirectPath HCL.

GREnvoy
Contributor
Contributor

I see that the balloon value average is in the neighborhood of 18GB.

I also went into one of the VMs that was having problems, I reserved the max amount allowed under the resources, which was the same as the configured amount. After watching it for a little while I didn't see any changes to the memory. It stayed right around 82%.

If I do some quick math on allocated vs physical we Allocated 102 vs 40 physical. So 250%. I'm sure that is pretty high, but when I look at Guest Memory % we probably average around 5%. So we have allocated a lot more memory then needed, which we can probably remedy, but I'd rather allow for spikes and such assuming they don't all spike at once.

0 Kudos
Dave_Mishchenko
Immortal
Immortal

That's a pretty significant value for overcommitment. Smiley Happy Page 40 has some pointers at looking at overcommitment performance - http://www.vmware.com/pdf/Perf_Best_Practices_vSphere4.0.pdf.




Dave

VMware Communities User Moderator

Forum Upgrade Notice - the VMware Communities forums will be upgraded the weekend of December 12th. The forum will be in read-only mode from Friday, December 10th 6 PM PST until Sunday, December 12th 2 AM PST.

Now available - vSphere Quick Start Guide

Do you have a system or PCI card working with VMDirectPath? Submit your specs to the Unofficial VMDirectPath HCL.

0 Kudos
xrysmu
Contributor
Contributor

Hi,

I want to add a comment to this article.  I found it because I was experiencing the same issue; a Windows 2008 R2 server was showing a memory utilization of 85% or above. I used the Rammap utility and discovered most of it was committed to the "driver lock" category as noted above.  After doing reading this article I reviewed the resource allocation settings for the VM and found the memory limit was set to 1GB (whereas I'd assigned 4GBs to the client).  Once I reset the limit the memory usage dropped significantly to an "expected" amount of around 22%.  So basic configuration of the VM was the issue here for me and might be for others if it is not an over-allocation of the hosts (our hosts have sufficient memory).

Chris 

IQIT
Contributor
Contributor

XRYSMU - I had this exact issue and finally found that even though I had allocated 12GB of RAM to a Server 2008 R2 server it was "limited" to 4GB!  OUCH!  After clicking "Unlimited" that totally fixed the issue.  I started seeing the memory usage go from 75% to 11%!

rule1reboot
Contributor
Contributor

I can confirm this issue as well with our 2008 R2 server.

The server was allocated 20 gig of memory but inadvertently the memory was limited to 2Gig.

The server ran fine in this configuration until the free space remaining on the C drive (which contained the pagefile) dropped below 20Gig.

At that time, they transfer rate, when copying larger files, dropped to less then 10MB/sec and a phanton process thrashed the cpu.

I confirmed this was the issue with a restored VM.

As soon at the free space dropped below 20gig, the problem manifested.

Freeing space on the drive OR increasing the VM memory limit resolved the issues.

VMWare ESXi 6.5.0

VM Version 8

VMWare Tools 8.6.5 build-621624

0 Kudos