vpxd consuming all memory -- vCenter slooooooow and cannot connect

[PeterBlatherwic]

As reported in http://communities.vmware.com/message/2020222 we are still experiencing the issue with vpxd consuming huge amounts of RAM and causing severe vCenter slowness.  This has gotten much worse recently.  We are hoping to renew the conversation on this, as it is becoming intollerable, and our efforts have so far failed. 

Configuratoin: VMware vCenter Server 5.0, update 1, version 5.0.0 build 623373.  Running on Windows Server 2008 R2 Standard SP1, in a virtual machine with 4 vCPUs and 8 GB of memory, SQL database on same machine.

When the issue opccurs, the main vCenter executable, vpxd.exe uses up all available memory, causing severe vCenter slowness, to the point where vSphere Client cannot connect, tasks fail due to timeouts, remote desktop to the machine are extremely slow and almost completely unusable.  After stop/start of vpxd.exe service, or restart the Windows VM, the issue initially clears, but then returns later. We are now seeing this more than once per day, sometime 3 - 4 times in a single day.

In more detail:

- Windows task manager reports 99% of  memory is consumed.  Typically, this is around 6.5 GB of memory consumed  by vpxd.exe.

- Before the issue occurs, vpxd is typically consuming around 330 MB memory (reasonable)

- Other large consumers are sqlservr.exe around 1.0 GB, java.exe around 820 MB + 380 MB (there are two processes), tomecat6.exe 690 MB.  We think these are probably normal, and they do not grow out of control like vpxd does.

- During the issue, CPU also becomes very high, near 100%, as seen through vSphere Client performance

- VM is running current VMware tools (8.6.5, build 652272)

Things we have tried:

- Upgraded to vCenter Server 5.0u1

- Rebalanced vCPUs (as suggested in other thread).  Initially was 4 CPUs on 1 socket, changed to 2 CPUs on 1 socket, now 4 CPUs split across 2 sockets.

- Moved to VM version 8

- Set service to autostart, delayed

- General cleanup -- removed a bunch of servies we are not using (Orchestrator, Update Manger...)

Any further suggestions, known issues, FIXES, whatever would be helpful! 

-- PeterB

[illvilja]

Hej!

Jag är på tjänsteresa i USA och har begränsad möjlighet att svara i telefon.

Vänligen kontakta Anders Sevelin 021-805808 så slussar han brådskande ärenden vidare.

Hi!

I'm at VMworld in San Francisco, I have limited access to answering the phone.

Please contact Anders Sevelin at +46 21805808 for urgent matters.

Mvh//Martin

[tommy269]

I have the exact smae problem too.  We were on a physcal VC 5.0 with 6 gig of ram and the DB on another SQL server too.  I p2v our VC over and gave it 16 gigs of ram and a week later it too up all the ram and cause major slowness.  Support has told me reinstall the VMware tools and if it happen again enable Verbos logging for further support??? This is what I found out and done so far that kind of help:

1. decrease VC logging to warning and errors only

2. disabled auto deployed that another admin was testing

3.create a new large page file drive for just virtual memory usage only

So far it is running better at 10 gig of ram for a week now but still would like to see this number drop lower if possible.

[tommy269]

http://pubs.vmware.com/vsphere-50/index.jsp?topic=%2Fcom.vmware.vsphere.install.doc_50%2FGUID-67C4D2...

Two services that I found is the actual cause of the high Ram but it is noted on the linked above from vmware for running inventory services and storage driven serives.  FYI

[GalNeb]

Here is what cured our problem with VPXD growing until it crashed.  We found that somewhere on our network there was a vSphere Client that had been forgotten, perhaps the user disconnected a remote session and left the client window open and the session going.  This client was trying to reconnect after vCenter had rebooted/restarted etc.  Trying and Trying and Trying.  Once we killed this client, then the memory growth issue went away.

[CELLULESYSTEME]

Thank you, that fixed the problem !!!!!

[zomx]

We are running vCenter Server 4.1 Update 3 | 30 Aug 2012 | Build 799345. Our vpxd continually would grow and eventually consume all 16GB of RAM in the VM. SQL is colocated on the same VM.  Neither vCenter vpxd service restarts or reboots helped. We saw this post and decided to both disconnect all existing client sessions and restart our management workstations which solved the problem.

Thanks so much for posting the resolution.

Here is what cured our problem with VPXD growing until it crashed.  We  found that somewhere on our network there was a vSphere Client that had  been forgotten, perhaps the user disconnected a remote session and left  the client window open and the session going.  This client was trying to  reconnect after vCenter had rebooted/restarted etc.  Trying and Trying  and Trying.  Once we killed this client, then the memory growth issue  went away.

[illvilja]

Hej!

Jag är ledig och har begränsad möjlighet att svara i telefon och mail.

Vänligen kontakta Anders Sevelin 021-805808 så slussar han brådskande ärenden vidare.

Åter i tjänst 19e nov

Hi!

I'm on vacation and have limited access to answering the phone.

Please contact Anders Sevelin at +46 21805808 for urgent matters.

Back in business the 19th of november

Regards//Martin

[ranjitcool]

Try this to fix ths issue -

http://kb.vmware.com/kb/2007600

If you found the evictions then this will certainly and hopefully fix the issue.

Thanks

RJ

[ranjitcool]

The real issue may be http://kb.vmware.com/kb/2007600

Once you killed the client, lesser cache was being consumed so you did not see the issue. VMware admits it did not increase the cache size enough.

RJ

[GalNeb]

This article is not relevant to the situations that I have seen.  First this article makes no mention of Memory growth.  It also is only an issue with very large implementations that have been upgraded from previous versions.  I have seen this problem with fresh installs on systems with as few as 2 hosts and 10 VMs running.

[ranjitcool]

Did you check if you have any evictions? Do not judge the relevance without checking for the symptoms in the article descirbed.

RJ

[rockymiho1]

spot on. killing the clients solved the problem.

Thanks heaps

[ranjitcool]

Hello

Can you tell me in detail how you killed the clients?

Thanks

RJ

[rockymiho1]

Hi,

Login to vsphere client, Administration >> administration >> Sesssions.

You will see the active sessions.  right click and terminate session. However, doing this what I noticed client was connected again automatically. So end up logging out the client from the windows.

Hope terminating the session will work for you so you dont have to force log off the user from windows.

[ranjitcool]

Thanks a lot, so once terminated did vpxd drop? Did tomcat also drop in ram?

Also how do you find who is using a session as in we have a huge dc and many admins so I want to be careful

Does killing idle sessions also resolve this?

Thanks again

[rockymiho1]

never had problem with tomcat, however while fixing the vpxd issue i did stop the vmware web service but that really dint help.

when you look at the sessions in vsphere client, it will display the username and you can check with your system administrator. I would say why not kill everyone to fix the problem?

[rockymiho1]

also you just having issue with slowness or is your vcenter server consuming all the memory? If the slow problem then this is not the correct thread, this thread is mainly for people experiencing with the mem leak issue.

[ranjitcool]

Thanks, memory leak and vcenter service dieing. VPXD maxing out. We fixed the buffer cache size issue and set it to unlimited as per my earlier posts where i have the kb article.

Thanks

RJ

[ranjitcool]

Is there any measure of how much each vcenter client session would use off of vpxd? As in 100mb per session may be?

Would anyone know?

RJ

[rockymiho1]

I would just terminate all the sessions to fix the issue.

not sure about the clients session usage off vpxd