vpxd consuming all memory -- vCenter slooooooow and cannot connect

[PeterBlatherwic]

As reported in http://communities.vmware.com/message/2020222 we are still experiencing the issue with vpxd consuming huge amounts of RAM and causing severe vCenter slowness.  This has gotten much worse recently.  We are hoping to renew the conversation on this, as it is becoming intollerable, and our efforts have so far failed. 

Configuratoin: VMware vCenter Server 5.0, update 1, version 5.0.0 build 623373.  Running on Windows Server 2008 R2 Standard SP1, in a virtual machine with 4 vCPUs and 8 GB of memory, SQL database on same machine.

When the issue opccurs, the main vCenter executable, vpxd.exe uses up all available memory, causing severe vCenter slowness, to the point where vSphere Client cannot connect, tasks fail due to timeouts, remote desktop to the machine are extremely slow and almost completely unusable.  After stop/start of vpxd.exe service, or restart the Windows VM, the issue initially clears, but then returns later. We are now seeing this more than once per day, sometime 3 - 4 times in a single day.

In more detail:

- Windows task manager reports 99% of  memory is consumed.  Typically, this is around 6.5 GB of memory consumed  by vpxd.exe.

- Before the issue occurs, vpxd is typically consuming around 330 MB memory (reasonable)

- Other large consumers are sqlservr.exe around 1.0 GB, java.exe around 820 MB + 380 MB (there are two processes), tomecat6.exe 690 MB.  We think these are probably normal, and they do not grow out of control like vpxd does.

- During the issue, CPU also becomes very high, near 100%, as seen through vSphere Client performance

- VM is running current VMware tools (8.6.5, build 652272)

Things we have tried:

- Upgraded to vCenter Server 5.0u1

- Rebalanced vCPUs (as suggested in other thread).  Initially was 4 CPUs on 1 socket, changed to 2 CPUs on 1 socket, now 4 CPUs split across 2 sockets.

- Moved to VM version 8

- Set service to autostart, delayed

- General cleanup -- removed a bunch of servies we are not using (Orchestrator, Update Manger...)

Any further suggestions, known issues, FIXES, whatever would be helpful! 

-- PeterB

[ranjitcool]

Thanks, I killed a few idle sessions - did not see any drop in vpxd. Did you see a drop right away when you killed some idle sessions?

[rockymiho1]

I had only 3 users logged in including myself, when I killed the other 2 sessions I dint see any drop as for some reason the sessions were reconnected automatically.  So I end up force log off from the windows where they were connecting from. In our environment users can only connect from 1 windows box therefore It made me easy. Not sure about your environment.

[ranjitcool]

Thanks for the reply, so you went into AD and did a force logoff? Or logged into their windows box and did a log off and once done u saw a drop right away?

[rockymiho1]

i logged into windows box from where they were connecting and did force log off. yes as soon as I kicked out of windows, memory usage dropped to 7 to 8% and its still same since yesterday

[bugblatterbeast]

I had exactly the same issue with vCenter 4.1 build 799345.  I had a VM inside a cluster that someone had left a session connected to the console.  It seems only terminating the session works.  Restarting vCenter had no effect at all.  Symptoms were ridiculous memory usage by vpxd and then subsequent crash of the vpxd service.  I also had lots of repeat events in the Cluster itself reporting this session connection attempts.

Thanks to all for pointing me in the right direction!

Danny

[GalNeb]

I have now seen the problem in vCenter 5.1 as well.  So they have not fixed it.  Try an "arp -a" from the vCenter server to see what IP addresses are talking to the vCenter server.  That will tell you where to look for bogus Client connections that are causing the memory leak.

[sbrown218]

by default mssql server will use all available ram on a server for sql query caching.  if you look at sqlservr.exe in task manager/processes it will typically say its using 128mb ram, but this is misleading. goto sql management studio and limit the amount of ram that it is allowed to use and you will see the ram use on your virtual center server immediately drop. from sql management studio right click on your sql server and goto properties - what you are looking for is the memory tab - on an 8gb server i would only allow it to use 2048mb max (not the 2tb that is the default). my virtual center is on a hard blade with 32gb ram so i configured sql server to use a maximum of 12gb.

--scott

[ThunderboltsRck]

I have pin pointed the cause of this memory issue in my 5.1 environment and stopped the issue by disabling the Single Sign on component. The steps that we performed are NOT supported by VMwrae so use at your own risk:

1. RDP to vcenter

2. Locate vpxd.cfg    C:\ProgramData\VMware\VMware VirtualCenter\

3. Edit this file with notpad++ (google it if you don't have it)

4. Modify line 54 or find code:

<sso>

          <admin>

                         <url>https://vcenter FQDN/sdk/uri>

          </admin>

          <enabled>true</enabled>

          <groupcheck>

          <lookupService>

                         <serviceId>{Large String here}:5</serviceId>

          <lookupService>

Change the Value "true" to "false"  (without quotation marks) and then re-boot your vcenter. In doing this our vcenter has gone from a woolly mammoth using 40GB of ram to using 7.5 - 8GB and it now stable and responding fast and not producing false alerts.

I hope this may be of some small help to someone but as I said at the start use at your own risk as this is not supported configuration by VMware.

[tonyhtsun]

We are using 5.0 and found out that the problem was caused by the performance monitor in vsphere client.

If there is a vsphere client keeps monitoring the graph from vsphere client, the vCenter memory will be eaten up by the vpxd,exe.

Once I shutdown the vsphere client machine and restart vcenter server everything becomes normal again.

So somehow the performance monitoring on vcenter has memory leak I guess.

Any idea?

[seycha]

We had the same issue for weeks and finally resolved it with help from VMware. We use a VMware Data Recovery appliance (v2.0.0) and have the corresponding plugin for the vSphere client installed.

It turns out this plugin was creating an awful lot of connections to the vCenter Server when it was restarted, causing the memory to go 100%. After uninstalling the plugin, upgrading the VDR appliance to the latest version (2.0.2), and re installing the plugin from its latest release, the problem went away. VMware also made us upgrade the vCenter SQL Express database to a full edition, but I'm not sure that made a difference.

So if you have the memory leak issue and are using VDR, you might want to do the same. Good luck !

[tonyhtsun]

Hi Seycha,

May I know which version of VDR plugin you are using?

Thanks

[seycha]

The latest version (2.0.2) is fine.

[3CV]

We had a similar issue, however I upped it to 10GB from 8GB, but the biggest factor by far was putting the database on a separate vdisk.  I had installed it on a seperate D partition because it wasn't going to be a full production system at the time, but as often happens, that changed.   So migrated it to a new disk and the speed increase was very dramatic.  Also, as someone has said, make sure there are no CDs or ISOs mounted.  They cause havoc if you try doing anything admin-ish!  there are scripts floating around t'interweb that will scan your system for VMs with anything mounted.  It's bad enough with 35 VMs let alone hundreds!

[DavidSilva77]

Hi my friend, check this KB , is possible that Tomcat es causing your trouble

VMware KB: VMware vCenter Management Webservices features do not function properly

[SteveEdi]

I had the same problem and its cause was that the database was full, so he couldnt write any data to the database and the cpu runs 100%

I solved the problem with this:

First check if the vcenter database is full.

If yes, try this:

VMware KB: Purging old data from the database used by VMware vCenter Server 4.x and 5.x

After that, restart the vcenter service and it should stay under 50%

Greetings EDI