VMware Cloud Community
Sysxp
Enthusiast
Enthusiast

vCenter 6.0 - high CPU usage!

Hello!

When we was on vCenter 5.5 its CPU consumption chart was nearly flat when idle and was about ~400-500 Mhz max.

Now we have this when upgraded to vCenter 6:

vCenter6 CPU.PNG

High peaks every 5 minutes - some "java.exe" process from SYSTEM account suddenly pops out, use several CPU cores and then disappears. It shows 3 times, and use 40-100 Mb each time.

After another 5 minutes it is repeated.

The host is ESXi 5.5 and the vCenter VM is the only VM on that host. vCenter is completly idle for hours - no vMotions, etc.

How to find out which vmware service exactly uses the CPU? Maybe it is possible to disable it, or somehow restrict it.

Thank you!

Tags (3)
33 Replies
AjithDhas
VMware Employee
VMware Employee

This might be due to the IIAD daemon thread which is scheduled to be triggered every five minutes. Can you try disabling that??

Reply
0 Kudos
Sysxp
Enthusiast
Enthusiast

It is true - there was a task in Task Scheduler called "VMwareIIAD". But when I disabe it and make sure it is not starting anymore nothing changed - the "java.exe" processes continue to spawn and consume CPU.

There must be something else.

P.S. I also tried to run the task manually several times - it does not consume CPU at all. (no more than 1-2% on one core very briefly). The real CPU eater is someone else.

Reply
0 Kudos
AjithDhas
VMware Employee
VMware Employee

Can you check the program which triggers this java.exe process.

In Windows Task Manager  Go to "View" - "Select Columns..." and check "Command Line". which will show you the source of the process.

Sysxp
Enthusiast
Enthusiast

AjithDhas wrote:

In Windows Task Manager  Go to "View" - "Select Columns..." and check "Command Line". which will show you the source of the process.

Oh, that was helpful, didn't know about that useful column, shame on me. :smileyconfused:

Here's what happening:

1. "C:\Program Files\VMware\vCenter Server\jre\bin\java.exe" -Xms8M -Xmx32M -jar is.jar --get-inv

a little after:

2. "C:\Program Files\VMware\vCenter Server\jre\bin\java.exe" -Xms8M -Xmx32M -jar vc.jar --get-tickets

after that:

3. "C:\Program Files\VMware\vCenter Server\jre\bin\java.exe" -Xms8M -Xmx32M -jar wbem.jar

Can you please shed light on what is going on?

Thank you!

Reply
0 Kudos
AjithDhas
VMware Employee
VMware Employee

From the sequence I feel that this is for getting host health status and update the same... No more idea on this... Smiley Sad

Reply
0 Kudos
Sysxp
Enthusiast
Enthusiast

Tried to rename those .jar files but unsuccesful - they are in use. Not sure though that renaming .jar files is appropriate way to correct this issue.

If anyone know why this is happening and how to disable it - I would be very grateful.

This current behavior is definitely not nice at all.

Reply
0 Kudos
Sysxp
Enthusiast
Enthusiast

Ok, I found an answer to all the issues we encountered with vCenter 6. Not the one I was hoping for, but still.

We roll back to 5.5! Neat.

Everything is just normal now - plugins, CPU usage, strange "quick stats is not up to date" messages, even the WebClient is working normal again.

I guess, we'll try once again in a year. :smileygrin:

Speaking of the CPU - here's the same VM on the same host.

Just compare with the picture in the 1st post:

vCenter55.PNG

Reply
0 Kudos
wishart
Contributor
Contributor

We are having the same problem since upgrading to 6.0.  One of the many Java SE binary processes starts consuming 90%-100% CPU out of the blue for no reason.

Reply
0 Kudos
ChristophBertho
Contributor
Contributor

I too am having this issue, a java.exe process is consuming 100% CPU

Environment:

vCenter Server 6.0.0, 2559268 installed on Windows Server 2012 R2

Process Name: java.exe

Process User Name: EsxAgentManager

Command Line: "C:\Program Files\VMware\vCenter Server\jre\\bin\java" -Djava.util.logging.manager=com.springsource.tcserver.serviceability.logging.TcServerLogManager -Djava.util.logging.config.file=C:\ProgramData\VMware\vCenterServer\cfg\vmware-eam\logging.properties -Dwrap

Reply
0 Kudos
Althornin
Contributor
Contributor

So, for those of us that were having a Java process consume 100% of the CPU for extended times, I believe I have a fix:

1) Are you using replacement (non-self signed) certificates?  We are!

If so, the problem is actually this:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=211257...


From looking at the logs under 100% CPU usage, the EAM service is in a login loop, which fails due to certificate replacement.


Once we fixed that issue, the high CPU utilization went away.

Reply
0 Kudos
ATLANTIKA07
Contributor
Contributor

Im my enviroment the Java high cpu issue was the Sophos Virusscanner.

Kai

Reply
0 Kudos
Mitchcss
Contributor
Contributor

We are experiencing the same issue...  It's the busiest box I have in my environment.

cpu.png

Any info on this?!?!

Thank you,

Reply
0 Kudos
RaoulSchaffner
Contributor
Contributor

Hello!

I experience exactly the same scenario (as far as i can tell)

environment:

- windows server 2012 r2, 4 vcpu, 24 gb ram (up from 16 gb originally, didn't help)

- vcenter server 6.0.0 build 3634793

- ms sql server 2014 on same machine

i also see these three processes in rapid succession (terminated every ~20 seconds) with a short pause every 5 minutes.

"C:\Program Files\VMware\vCenter Server\jre\bin\java.exe" -Xms8M -Xmx32M -jar is.jar --get-inv

"C:\Program Files\VMware\vCenter Server\jre\bin\java.exe" -Xms8M -Xmx32M -jar vc.jar --get-tickets

"C:\Program Files\VMware\vCenter Server\jre\bin\java.exe" -Xms8M -Xmx32M -jar wbem.jar

the error file lies here:

C:\ProgramData\VMware\vCenterServer\data\core\vmware-vws\write_status-err.txt

and shows a java memory heap size problem at the end: Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

i made sure that those java processes are triggered by the process vmwarevws (vmware system and hardware health manager) by killing vmwarevws: they were gone and didn't show up anymore. that's no solution, though. 🙂

PS C:\Program Files\VMware\vCenter Server\bin> .\service-control.bat --stop vmwarevws

i tried to influence the dynamic memory allocation of vcenter 6 as pointed out by others to increase the allocation for the vmwarevws process by using these command line statements:

PS C:\Program Files\VMware\vCenter Server\bin> .\service-control.bat --stop vmwarevws

PS C:\Program Files\VMware\vCenter Server\visl-integration\usr\sbin> .\cloudvm-ram-size.bat -C 100 vmwarevws

PS C:\Program Files\VMware\vCenter Server\visl-integration\usr\sbin> .\cloudvm-ram-size.bat -S

PS C:\Program Files\VMware\vCenter Server\bin> .\service-control.bat --start vmwarevws

however, after restarting the vmwarevws process, the java processes in question did not get different command line options, i.e. the Xms/Xmx values are still the same.

it looks like the vmwarevws process does not take part in the dynamic memory allocation feature of vcenter 6.

i hope my conclustions are wrong and someone points me in the right direction! 😉

cheers,

raoul.

Reply
0 Kudos
znamor
Contributor
Contributor

Same issue on Vcenter 6U2 appliance VM that's running Windows 2008R2 Standard SP1.  We have two java.exe processes that are doing this on the command line (see attached image):

Screen Shot 2016-11-01 at 5.45.45 PM.png

I received two replies from VMware tech support on this matter:

1) There is a patch for 6U2 scheduled to be released on 10th on November 2016 to fix this. 

..a few days later this...

2) "Told CU the patch is coming by the end of this month tentetive date as 22nd nov 2016.  also made changes in conf file of the vpxd.conf set the heap size as 1024 and asked cu to observe this for 2 days and get back to us."  (aside from spelling errors they actually never called or emailed me about this and just snuck the later information in by directly posting it to the open SR online LOL.  Good think I looked at the ticket online.  I bet this was another filibuster tactic of theirs to move the date and tell the customer later -- WOW !excellent support practices Vmware engineers)

Now since I have an open SR on this I sort of have VMware support on the hook. 

VMWARE SUPPORT - when is the patch actually coming out?

Reply
0 Kudos
RaoulSchaffner
Contributor
Contributor

for what it's worth, i also have an open SR on this. let's see how well this goes. i'll keep you all posted.

does anyone of you have the "netapp virtual storage console (vsc)" installed on the troublesome vcenter as i do?

i forgot to mention that in my earlier post.

Reply
0 Kudos
abugeja
Hot Shot
Hot Shot

hi,

sorry to take over this thread but I upgraded my vCenter appliance to 6.0 Update 2a last Thursday and I have noticed an increase of 10% CPU (10% -> 20%) since I performed the upgrade.

My appliance is 4 vCPU with 16Gb of ram. (2 hosts and 120 vms)

anyone else seen this since they have upgraded?

Thanks

Reply
0 Kudos
znamor
Contributor
Contributor

We only have "NetApp OnCommand System Manager 3.1" installed but not the virtual console.  However,  although Netapp does use java.exe as well the java processes causing me high CPU problems belong to Vcenter.

Reply
0 Kudos
Matnii
Contributor
Contributor

Any known fix to this issue?

I had vCenter 6.0U1 for Windows and it runs year and half without any issue. Average CPU load was 10%. Now it was time to upgrade hosts to 6.0U2, so I upgrade vCenter to build 4637290 (U2a). After update, average CPU load increased to 40%. The picture shows that CPU load is caused by Java. The load is more like spikes, so CPU is idle couple of seconds and then it jumps to 100%.

I see that I have Xmx512M -parameter and other screen captures on this thread shows -Xmx32M -parameter.

pastedImage_0.png

Reply
0 Kudos
kgruber
Contributor
Contributor

I have exactly the same issue and the JAVA command lines that spike are exactly the same. I too have opened a case with VMware and they have taken alot of screen shots that show exactly the same thing you have shown.

This has update 2A (6.0.0 U2a) installed, an external Cert and MS SQL (full) installed locally.

We have 2 of these VCs, one for production, one for Horizon View 7.03, both act the same! !

Reply
0 Kudos