usafseic
Enthusiast
Enthusiast

Data Service Container Process Heap Memory Alarms

Every so often, I'm seeing these two alarms appear in the WC.


2015-10-27 10_03_43-Clipboard.png

The Data Service container process is running low on heap memory (> 90% utilized.)


followed by


Data Service is functioning properly.

They appear and go away within a minute or two every time.  On the Data Service heap memory thing, I did review this KB VMware KB: ESXi/ESX host reports VMFS heap warnings when hosting virtual machines that collectively ..., but I can't modify the suggested parameters under VCSA 6.  It says it's supposed to just do its job and grow as needed, but to continue to see these pop in and out is bothering me.  It also seems to happen more frequently when a VM CPU usage alarm is triggered.


Does anyone think this is indicative of a bigger problem that I should be concerned about, or is this normal behavior to deal with the amount of activity on the instance?  There are a lot of VMs in use and the datastore is always around 75% full, but CPU and RAM have been 50% or less.  If abnormal, where should I begin to look for any signs of trouble?  RAM capacity of the VCSA or some other config parameter(s) that might need to be tuned?

Tags (2)
21 Replies
Tinto1970
Commander
Commander

hi,

my windows 2008 r2 vcenter, with remote MSSQL db has exactly the same behaviour.

Honestly, i'm not worried at the moment. But i'd like to investigate that one day. Maybe i will open a ticket.

Alessandro aka Tinto | @tinto1970
0 Kudos
avinchakov
Enthusiast
Enthusiast

I have such problem too.

0 Kudos
johnny0317
Contributor
Contributor

I am seeing this same issue. We are running a vCenter Server Appliance version 6.0 U1. Any updates?

0 Kudos
andcro
Contributor
Contributor

I started seeing this as well - vcsa 6 build 3018523, external psc

As stated by OP the warnings clear shortly on their own.

I'd be curious to hear if anyone has an explanation or resolution.


Thank you,

0 Kudos
johnny0317
Contributor
Contributor

I put in a support case with VMware and they told me it is a known issue. They are working on correcting the bug in the next update. So for now we wait. They checked my vCenter install and there was nothing actually wrong with it so just a bug.

0 Kudos
eode
Enthusiast
Enthusiast

FYI: I've also posted a reply here:

Re: vmware-dataservice-sca status changed from yellow to green

Gave this a look tonight. Seems like the vmware-dataservice-sca/vmware-sca is looking for a script called "dummy_path" (I know).

Found this little gem in the sca.log:

grep -Hin "dummy" /var/log/vmware/sca/sca.log|head -1

/var/log/vmware/sca/sca.log:8:2015-12-30T23:17:36.083Z [pool-5-thread-14 ERROR com.vmware.sca.servicecontrol.handlers.ScriptControlHandler] Failed to locate script; name: dummy_path, dirs: [/usr/lib/vmware-sca/scripts, /etc/vmware-sca/scripts], [java.lang.Exception: Script doesn't exist; path: /etc/vmware-sca/scripts/dummy_path]

So created a little "dummy_path"-script, which will only exit when run.

cat /etc/vmware-sca/scripts/dummy_path

#!/bin/sh

#I'm dumb, and will now exit. With a zero. Because. Awesome.

exit 0

I didn't have the time to check which process which was trying to run the script, so I gave everyone rights (I know. Not cool.).

chmod 777 /etc/vmware-sca/scripts/dummy_path

Profit (I've not seen the issue since, nor any complaints about the dummy_path-script). Oh, and I'm running the VCSA 6.0.0.10000 Build Number 3018521.

Guessing this isn't a "supported workaround", or anything like that (and may cause other problems). Just playing around with my home lab.

Regards,

Espen Ødegaard

DavideDeon
Contributor
Contributor

I had the same issue on my vcenter.

I think the problem comes out to the people who have configured 8GB of memory for the Vcenter appliance.

If you log into vcenter shell and digit "free -m" you probably see that used memory is near the total, and this cause the alarm.

I've increased the appliance VM memory to 16gb (maybe 10-12gb are enough)  and the error has not come out yet.

memory.JPG

I hope that this tweak resolve your problems.

johnny0317
Contributor
Contributor

Good information DavideDeon. I have increased the memory on my vCenter's from 8GB to 16GB. I will keep an eye on it for about a week and report back with the results.

0 Kudos
johnny0317
Contributor
Contributor

I did as DavideDeon suggested and our issue has gone away. Thanks!

0 Kudos
Paul_Sheard
Enthusiast
Enthusiast

I am seeing this exact bunch of errors on a VCSA 6U2 (with external PSC appliance). My VCSA already has 16GB configured (from deployment), should I go to 24GB?

Is this issue being worked on by VMware?

Paul Sheard VMware Consultant (Contract) VCP6 DCV NV CMA DTM
0 Kudos
Bleeder
Hot Shot
Hot Shot

I've tried 32GB and 64GB and it still happens.  It just masks the issue for a while longer, rather than actually fixing it.

0 Kudos
Paul_Sheard
Enthusiast
Enthusiast

Thanks for the swift response.. much appreciated..

Does a reboot of the VCSA do exactly the same thing?

I've already got a VMware SR case open and have forwarded this community post to the VMware support engineer who I have been working with.

I'll keep you posted on how things go!

Cheers

Paul

Paul Sheard VMware Consultant (Contract) VCP6 DCV NV CMA DTM
0 Kudos
Paul_Sheard
Enthusiast
Enthusiast

I also see vSphere web client health warnings appear / disappear,

along with VSAN overall health warnings.. which lead to finding zero issues with VSAN disk health

Paul Sheard VMware Consultant (Contract) VCP6 DCV NV CMA DTM
0 Kudos
dfosbenner
Enthusiast
Enthusiast

Same issue - "The Data Service container process is running low on heap memory (>90 % utilized)"

I'm running vCenter 6.0 Build 3634793

0 Kudos
jasoncain_22
Enthusiast
Enthusiast

dfosbenner:

did you see the temp workaround for setting the alert from UNSET to 'warning'?

I made this change on all of the vCenters that were linked and restarted the vCenter service. (Windows server)

I am at Day 5 and still have yet to see the DSC Heap Memory Alarm. From the other posts, it seems that this is only temporary until the alarm pops again.

0 Kudos
LouisReedijk
Contributor
Contributor

\VMware-VCSA-all-6.0.0-3634788 same issue, with no hosts connected yet!

0 Kudos
LouisReedijk
Contributor
Contributor

I've created the mini scripts as well. Seems to be fixing the problem just fine Smiley Happy

0 Kudos
TomKalabis
Enthusiast
Enthusiast

hi,

i had same issue with my vcenter...

follow this post and add more RAM for VCSA :

VMware vCenter - Data Service Container Process Heap Memory Alarms

0 Kudos
nlbisd
Contributor
Contributor

This seems to only apply on the appliance, not Windows.  I've got 7GB of free RAM and mine begins to produce this error about 2-3 weeks after rebooting the vCenter server.  I'm running vCenter 6.0 U2.

0 Kudos