topic Data Service Container Process Heap Memory Alarms in vCenter™ Server Discussions

Data Service Container Process Heap Memory Alarms

usafseic — Tue, 27 Oct 2015 14:04:25 GMT

Every so often, I'm seeing these two alarms appear in the WC.

The Data Service container process is running low on heap memory (> 90% utilized.)

followed by

Data Service is functioning properly.

They appear and go away within a minute or two every time. On the Data Service heap memory thing, I did review this KB VMware KB: ESXi/ESX host reports VMFS heap warnings when hosting virtual machines that collectively use 4 TB or 20 T…, but I can't modify the suggested parameters under VCSA 6. It says it's supposed to just do its job and grow as needed, but to continue to see these pop in and out is bothering me. It also seems to happen more frequently when a VM CPU usage alarm is triggered.

Does anyone think this is indicative of a bigger problem that I should be concerned about, or is this normal behavior to deal with the amount of activity on the instance? There are a lot of VMs in use and the datastore is always around 75% full, but CPU and RAM have been 50% or less. If abnormal, where should I begin to look for any signs of trouble? RAM capacity of the VCSA or some other config parameter(s) that might need to be tuned?

Re: Data Service Container Process Heap Memory Alarms

Tinto1970 — Wed, 04 Nov 2015 10:25:05 GMT

hi,

my windows 2008 r2 vcenter, with remote MSSQL db has exactly the same behaviour.

Honestly, i'm not worried at the moment. But i'd like to investigate that one day. Maybe i will open a ticket.

Re: Data Service Container Process Heap Memory Alarms

avinchakov — Thu, 26 Nov 2015 11:09:48 GMT

I have such problem too.

Re: Data Service Container Process Heap Memory Alarms

johnny0317 — Tue, 01 Dec 2015 22:07:11 GMT

I am seeing this same issue. We are running a vCenter Server Appliance version 6.0 U1. Any updates?

Re: Data Service Container Process Heap Memory Alarms

andcro — Wed, 09 Dec 2015 03:35:34 GMT

I started seeing this as well - vcsa 6 build 3018523, external psc

As stated by OP the warnings clear shortly on their own.

I'd be curious to hear if anyone has an explanation or resolution.

Thank you,

Re: Data Service Container Process Heap Memory Alarms

johnny0317 — Wed, 09 Dec 2015 20:02:23 GMT

I put in a support case with VMware and they told me it is a known issue. They are working on correcting the bug in the next update. So for now we wait. They checked my vCenter install and there was nothing actually wrong with it so just a bug.

Re: Data Service Container Process Heap Memory Alarms

eode — Thu, 31 Dec 2015 02:35:53 GMT

FYI: I've also posted a reply here:

Re: vmware-dataservice-sca status changed from yellow to green

Gave this a look tonight. Seems like the vmware-dataservice-sca/vmware-sca is looking for a script called "dummy_path" (I know).

Found this little gem in the sca.log:

grep -Hin "dummy" /var/log/vmware/sca/sca.log|head -1

/var/log/vmware/sca/sca.log:8:2015-12-30T23:17:36.083Z [pool-5-thread-14 ERROR com.vmware.sca.servicecontrol.handlers.ScriptControlHandler] Failed to locate script; name: dummy_path, dirs: [/usr/lib/vmware-sca/scripts, /etc/vmware-sca/scripts], [java.lang.Exception: Script doesn't exist; path: /etc/vmware-sca/scripts/dummy_path]

So created a little "dummy_path"-script, which will only exit when run.

cat /etc/vmware-sca/scripts/dummy_path

#!/bin/sh

#I'm dumb, and will now exit. With a zero. Because. Awesome.

exit 0

I didn't have the time to check which process which was trying to run the script, so I gave everyone rights (I know. Not cool.).

chmod 777 /etc/vmware-sca/scripts/dummy_path

Profit (I've not seen the issue since, nor any complaints about the dummy_path-script). Oh, and I'm running the VCSA 6.0.0.10000 Build Number 3018521.

Guessing this isn't a "supported workaround", or anything like that (and may cause other problems). Just playing around with my home lab.

Regards,

Espen Ødegaard

Re: Data Service Container Process Heap Memory Alarms

DavideDeon — Thu, 07 Apr 2016 11:31:27 GMT

I had the same issue on my vcenter.

I think the problem comes out to the people who have configured 8GB of memory for the Vcenter appliance.

If you log into vcenter shell and digit "free -m" you probably see that used memory is near the total, and this cause the alarm.

I've increased the appliance VM memory to 16gb (maybe 10-12gb are enough) and the error has not come out yet.

I hope that this tweak resolve your problems.

Re: Data Service Container Process Heap Memory Alarms

johnny0317 — Wed, 13 Apr 2016 15:14:09 GMT

Good information DavideDeon. I have increased the memory on my vCenter's from 8GB to 16GB. I will keep an eye on it for about a week and report back with the results.

Re: Data Service Container Process Heap Memory Alarms

johnny0317 — Thu, 21 Apr 2016 13:15:00 GMT

I did as DavideDeon suggested and our issue has gone away. Thanks!

Re: Data Service Container Process Heap Memory Alarms

Paul_Sheard — Wed, 25 May 2016 14:11:37 GMT

I am seeing this exact bunch of errors on a VCSA 6U2 (with external PSC appliance). My VCSA already has 16GB configured (from deployment), should I go to 24GB?

Is this issue being worked on by VMware?

Re: Data Service Container Process Heap Memory Alarms

Bleeder — Wed, 25 May 2016 14:14:22 GMT

I've tried 32GB and 64GB and it still happens. It just masks the issue for a while longer, rather than actually fixing it.

Re: Data Service Container Process Heap Memory Alarms

Paul_Sheard — Wed, 25 May 2016 14:18:01 GMT

Thanks for the swift response.. much appreciated..

Does a reboot of the VCSA do exactly the same thing?

I've already got a VMware SR case open and have forwarded this community post to the VMware support engineer who I have been working with.

I'll keep you posted on how things go!

Cheers

Paul

Re: Data Service Container Process Heap Memory Alarms

Paul_Sheard — Wed, 25 May 2016 14:28:17 GMT

I also see vSphere web client health warnings appear / disappear,

along with VSAN overall health warnings.. which lead to finding zero issues with VSAN disk health

Re: Data Service Container Process Heap Memory Alarms

dfosbenner — Wed, 08 Jun 2016 17:47:30 GMT

Same issue - "The Data Service container process is running low on heap memory (>90 % utilized)"

I'm running vCenter 6.0 Build 3634793

Re: Data Service Container Process Heap Memory Alarms

jasoncain_22 — Wed, 08 Jun 2016 19:29:31 GMT

dfosbenner:

did you see the temp workaround for setting the alert from UNSET to 'warning'?

I made this change on all of the vCenters that were linked and restarted the vCenter service. (Windows server)

I am at Day 5 and still have yet to see the DSC Heap Memory Alarm. From the other posts, it seems that this is only temporary until the alarm pops again.

Re: Data Service Container Process Heap Memory Alarms

LouisReedijk — Tue, 14 Jun 2016 14:08:39 GMT

\VMware-VCSA-all-6.0.0-3634788 same issue, with no hosts connected yet!

Re: Data Service Container Process Heap Memory Alarms

LouisReedijk — Tue, 14 Jun 2016 14:42:48 GMT

I've created the mini scripts as well. Seems to be fixing the problem just fine

Re: Data Service Container Process Heap Memory Alarms

TomKalabis — Thu, 21 Jul 2016 19:10:25 GMT

hi,

i had same issue with my vcenter...

follow this post and add more RAM for VCSA :

VMware vCenter - Data Service Container Process Heap Memory Alarms

Re: Data Service Container Process Heap Memory Alarms

nlbisd — Thu, 04 Aug 2016 12:05:54 GMT

This seems to only apply on the appliance, not Windows. I've got 7GB of free RAM and mine begins to produce this error about 2-3 weeks after rebooting the vCenter server. I'm running vCenter 6.0 U2.