Data Service Container Process Heap Memory Alarms

[usafseic]

Every so often, I'm seeing these two alarms appear in the WC.

The Data Service container process is running low on heap memory (> 90% utilized.)

followed by

Data Service is functioning properly.

They appear and go away within a minute or two every time.  On the Data Service heap memory thing, I did review this KB VMware KB: ESXi/ESX host reports VMFS heap warnings when hosting virtual machines that collectively ..., but I can't modify the suggested parameters under VCSA 6.  It says it's supposed to just do its job and grow as needed, but to continue to see these pop in and out is bothering me.  It also seems to happen more frequently when a VM CPU usage alarm is triggered.

Does anyone think this is indicative of a bigger problem that I should be concerned about, or is this normal behavior to deal with the amount of activity on the instance?  There are a lot of VMs in use and the datastore is always around 75% full, but CPU and RAM have been 50% or less.  If abnormal, where should I begin to look for any signs of trouble?  RAM capacity of the VCSA or some other config parameter(s) that might need to be tuned?

[bakerzdosen]

Just thought I'd add another data point. I've got VCSA (6.0.0.10200 Build Number 3343022) running with 24GB of RAM assigned to it. These warnings popped up.

I ssh'd in and found that free -m showed about 500mb of RAM available.

Running top, I noticed vxpd taking 13% of the RAM, vsphere-client taking 8-9%, and vmware-invsvc taking 9-12%.

I restarted vxpd first (service vmware-vpxd restart) and then vsphere-client (service vsphere-client restart) - keeping in mind that if someone is using the web client, they'll definitely notice this going down as it takes several minutes to restart - and that increased available RAM (again, free -m) to just over 2.5GB.

Restarting the inventory service (service vmware-invsvc restart) increased free RAM to just a touch under 5GB after things have stabilized.

Top now shows vpxd at 2.1%, vsphere-client at 4.7%, and invsvc at 4%.

I have no clue how long this band-aid will last (meaning, I don't know if these three are the only 3 services whose memory footprints seem to grow) but it's a possible quick fix (keep in mind there can be ramifications of restarting services during production - you're on your own for that important bit of research) to reduce the RAM utilization without a full reboot of VCSA.

Side note: I don't know if U2 (6.0.0.20000 Build Number 3634791) fixes this issue or not. We haven't been able to upgrade to U2 because of View 6 compatibility just yet, so I'm guessing there are other people in this boat for one reason or another.)

[YetAnotherVmwar]

I was able to fix this problem before following a KB of some kind, but I can't find it now for the life of me.

IIRC, increasing the memory allocated to the vcenter appliance as a whole doesn't help.  The various services that make up vcenter have memory quotas.  The fix involved ssh-ing in and upping the quota for one service, possibly vpxd. 

I'm hoping that posting here will jog someone's memory.