vpxd.exe uses up to 99% of available physical memory and prevents vSphere client access

[virtualG78]

We have an issue with one of our vCenter servers using

VMware vCenter Server 5.0.0.16964

This server is running on Windows Server 2008 R2 in a virtual machine with 4 vCPUs and 8 GB of memory.

It appears that the main vCenter executable, vpxd.exe can often use up physical memory to the point where Windows task manager reports 99% of memory is consumed.  Typically, this is around 6.5 GB of memory consumed by vpxd.exe.  At this point, existing vSphere client connections to vCenter are terminated and no new connections can be established.

The only way found to resolve this problem is to "End Task" on the vpxd.exe process and start the vCenter service again.  The vCenter service will not respond to a stop service request.

Also running on this vCenter server is:

VMware SRM 5 service with vSphere replication enabled

vSphere Update Manager

VirtualCenter Management Webservices

[vGuy]

Can you disable HA & DRS -> restart the vCenter server service --> re-enable HA & DRS and verify the mem utilization.

Also, how many hosts & VMs are being managed by the vCenter server. 4 vCPUs seem a bit overkill, I will suggest to configure the VM with 2 vCPUs first and upgrade as required...hth!

[virtualG78]

We are using 14 hosts and have 221 virtual machines.

Since we are currently using 25 resource pools, disabling DRS isn't feasable as all the resource pools will be removed with that action.  However, we will try disabling/re-enabling HA and reducing the vCPU count to 2 on the vCenter as you suggest.

[kjb007]

Have you checked your vCenter logs to see what is happening during this timeframe?

Does this happen frequently?  On a particular timeframe?

-KjB

[fuchsan]

Hi

We have exact the same issue here, disabling HA and DRS won't help.

Has someone a solution to it?

Thanks

[BelvedereWRX]

I just wanted to report that we're experiencing the same issue at our organization. Until a couple of reboots and vpxd.exe was force closed, the process had utilized all available memory. I'm not sure how long this has been an issue because we didn't know it was occurring until the server was taken down for a reboot and we saw the utilization at logon.

After restarting vpxd.exe and associated services, the memory utilization dropped back to normal levels, but has been slowly climbing. I'm curious to see if it's going to keep growing until all available memory is consumed.

[fuchsan]

We did a checkdisk (which found errors) and some clean up of the vcenter logs (there wehere some huge dump files), since then it seems memory usage is back to normal (running two days now).

Folders cleand:

C:\ProgramData\VMware\VMware VirtualCenter\Logs

C:\ProgramData\VMware\VMware Update Manager\Logs

regards

Andi

[iw123]

... is a good point. Do you have sufficient disk space free on your VC, given that you are running update manager from there too?

[iw123]

Also, do you have any scheduled tasks, or other plugins using resources such as vdr or other third party software? 

[helltejas]

its a vpxd memory leak issue..

can you tell me what version u r using:smileyconfused:

its common in VI3.5 and earlier versions when dealing with huge amount of VMs and Hosts.

[Carm54]

We have the same issue here. It comes and goes though. Sometimes if we kill the VDR machines it will stop... Today it started again and I restarted vCenter this morning. Its already climed to 10GB of memory within 10 minutes. If any one knows of a fix that would be great!

[clevelas]

I am running into the same issue.  Memory maxxed out (vpxd taking 10GB of RAM), CPUs (2 vCPUs) pegged.  Restarted and it's functional again.  I just upgraded to vSphere 5 about a week ago.  I'm using Update Manager, VMware Data Recovery, VMware Management Appliance and the Dell OpenManage plugin, just for reference.

It's been up for about five minutes now.  Watching vpxd.exe in Task Manager and it's already up to 4.5GB of RAM and climbing fast.  Now over 6GB.  Yikes.  Guess it's time to open a support case.

[helltejas]

Hi guyz,

     I think i figured out the root cause of this problem.. I think this problem occurs only if we deploy vCenter as VM. this problem occurs because of physical CPU and vCPU overhead

eg. if you hav 2 socket pCPU with single core with hyperthreaing enable then 4vCPU then in one time slot 4 way vSMP will be sheduled and ready time of other VM will get increased. and that causes memory leak in vpxd in vCenter because during that time vpdx is managing too many ESXi host connectivity mgmt.

and these days i am reading many problems where problems are with virtulmachine vCenter which are managing too many ESXi host not in physical

Reducing ESXi per vCenter would solve the issue

I hope this will be helpful.

Regards,

Tejas

[clevelas]

I did open a case, and the solution in my case was updating VMware Tools.  I had the latest 4.1 VMware Tools, but hadn't installed the 5.0 tools.  Since doing the update, I haven't seen any more issues.  It was explained to me that combining outdated tools with memory intensive applications like databases can cause issues like this.

Your milage may vary, but is has fixed my issues.

[Sigma201110141]

Hi,

I have the same problem with VCenter hardware V8 and tools up to date ...

After many reboot (randomly 10, 20 ....) the process vpxd uses 1,5Go RAM but after several weeks it uses again all memory....

What i did :

• Windows Update
• Isolation server on Datastore
• Upgrade VMware Tools + Hardware Virtuel V8
• Patchs Database SQL Server
• Upgrade VCenter V5 U1
• Uninstall Orchestrator.
• No limit (RAM and CPU)

If you have another idea ...

Thanks

[PeterBlatherwic]

Hi,

In hopes of restarting discussion on this topic, I started a new discussion at

   http://communities.vmware.com/message/2066825#2066825

The vpxd memory explosion issue has gotten really bad for us!

-- PeterB