Sysxp
Enthusiast
Enthusiast

vCenter 6.0 - high CPU usage!

Hello!

When we was on vCenter 5.5 its CPU consumption chart was nearly flat when idle and was about ~400-500 Mhz max.

Now we have this when upgraded to vCenter 6:

vCenter6 CPU.PNG

High peaks every 5 minutes - some "java.exe" process from SYSTEM account suddenly pops out, use several CPU cores and then disappears. It shows 3 times, and use 40-100 Mb each time.

After another 5 minutes it is repeated.

The host is ESXi 5.5 and the vCenter VM is the only VM on that host. vCenter is completly idle for hours - no vMotions, etc.

How to find out which vmware service exactly uses the CPU? Maybe it is possible to disable it, or somehow restrict it.

Thank you!

Tags (3)
33 Replies
kgruber
Contributor
Contributor

Has anyone tried Update 3 yet? My VMware support tech says this fixes the problem. I would like to believe him but....... !

0 Kudos
sp4rt4n
Enthusiast
Enthusiast

Hi,

same here on vCenter Server Appliance 6.0 U2.

I just want to share my experience with this...

Last year, I've deployed a vCenter Server Appliance 6.0 U2 (2 Hosts, 20VMs, 8 active), as you can see, CPU usage is spiking and "building up"

60u2year.jpg60u2hour.jpg

Yesterday, I've deployed a vCenter Server Appliance 6.0U3 (no Hosts/VMs), only time will tell how CPU will perform..

60u3day.jpg

I've also deployed a vCenter Server Appliance 6.5U0 one month ago, here are the charts for comparison:

65u0month.jpg 65u0day.jpg

From my point of view: 6.5U0 is definitely better in performance comparison (Photon OS over SLES11 might be a reason for this)... however I would explicitly NOT recommend to use 6.5U0 as the vSphere Web Client is mandatory and ....a disastrous messy piece of administration software.

0 Kudos
abugeja
Hot Shot
Hot Shot

Hey... Has anyone applied update 3 yet and has it resolve the high CPU issue?

0 Kudos
chicojr
Enthusiast
Enthusiast

I too am having same issues, vcenter 5.5 on a win srv 2012 r2 box with 2cpu 16gb mem.

Vcenter hosts to datacenters with a total of 671 hosts and 1842 vms enabled.

Anyone have any idea what in the world is causing this intermittent spikes in cpu and mem?

0 Kudos
abugeja
Hot Shot
Hot Shot

671 hosts to 1842 Vms - why are there so many hosts vs Virtual machines? I assume all the VM's are very large or you have very small hosts

0 Kudos
NTShad0w
Enthusiast
Enthusiast

Hi,

as RaoulSchaffner

vmware-vwsVMware System and Hardware Health Manager

so simple temporary workaround is to stop this service by typing (from vCSA shell):

service-control --stop vmware-vws

This service is responsible for HW monitoring, but as I know it's not working for vSphere C# Client in vSphere v6.0 so for smaller environments and admins that don't want to switch to terrible slow Web Client this is a choice in my opinion.

To not change anything in vCSA appliance it's safest to stops it after each vCSA 6.0 restart.

The results after applying this workaround is seen in attachment. Generally CPU usage drops from 2200MHz to acceptable 400MHz.

pastedImage_0.png

To VMware: Mates, please don't kill C# Client, it's your worst choice we notice from the beginning of your Company and great VMware VI3 and vSphere 4/5 product.

thanks

best regards

NTShad0w

Sysxp
Enthusiast
Enthusiast

I can't believe all this still happening in 2017! This is just... WOW. No words.

This is all went wrong since the WebClient introduction and the following deprecation of fast and beautiful C# client.

By the way, have you tried the all new HTML 5 "vSphere Client"? All I know is that it still can't do much.

CollinChaffin
Enthusiast
Enthusiast

Can't agree more.  In my almost 30yrs in this business VMware towers over any other vendor I've dealt with as far as totally and unequivocally ignoring the very paying customers that built and support their existence.

I have not, in real life with the Internet trolls aside, met a living human being ranging from those supporting large worldwide VMware infrastructures, all the way down to small single-host shops that do not share my (our) belief that this MONSTROSITY they call a "web client" is IN ANY WAY superior to the actual desktop application (C# client).  NOT ONE.

82 clicks and literally an HOUR to find just which magic "MANAGE TAB" (a term I now never want to hear again thanks to VMware) nested 17 screens/tabs deep, vs the THREE CLICKS in the C# client in a SINGLE WINDOW?  Hey VMware, COUNT ME IN!  Want to burn a HALF DAY just to figure out why a plugin that has ALWAYS appeared in the C# client suddenly disappeared in the magic web client?  ABSOLUTELY!  We in I.T. absolutely were not historically busy enough and really wanted our days stretched just a bit longer.  Thanks VMware!

IMO It's got to be bad inexperienced management listening to lazy inexperienced developers who didn't have the skills to code the C# client which probably started just after they lost enough dotnet DEV talent to continue the C# client support.  Just a theory but I cannot wrap my head around why else they would have continued down this path.

Unfortunately my friend, the ONLY way VMware will listen is if customers begin bailing with their $.  Personally, I have been with VMware as their top evangelist in every room since the first version of their workstation product and in the days of the old GSX server but I have recently begun to look at Hyper-V and other competitive products to replace VMware for this exact reason I refuse to fight to manage a product where the MFG with every revision seems to make my life HARDER, all while continuing to charge rising exorbitant licensing fees.

0 Kudos
CollinChaffin
Enthusiast
Enthusiast

Oh forgot to post my screenshot to mirror what NTShad0w​ posted clearly showing my absurdly high CPU VCSA 6.0.0.30300 Build Number 7037817 before and after running:

service-control --stop vmware-vws

SNAG_12-5-2017_11-35-58.png

So I would say that NO, this has NOT been fixed in the latest VCSA 6.0u3 appliance patch available.

0 Kudos
Dave_the_Wave
Hot Shot
Hot Shot

I also got the java.exe high cpu issue when I upgraded to VCS 6.0 u3c for Windows (2008 R2).

It was a patterned spike, exactly like all the graphs in this thread.

After the typical way to flip services on and off until you hit the culprit, it was the vmwarevws "VMware System and Hardware Health Manager" service that was causing this.

Stopping and setting the Windows Service to Manual puts the box back to sleep.

I felt the Health Manager was important, so I went back to VCS 6.0 u1b, and do not get the problem.

I still use the C# Client to do any cooking in the kitchen, but the VCS is nice to monitor and observe all is kosher.

0 Kudos
goppi
Enthusiast
Enthusiast

I'm facing the same problem.

I setup a lab environment with ESXi 6.0U3 and VCSA 6.0U3f and CPU load is constantly between 20% and 50%.

This is a new Installation with one ESXi host and VCSA is the only VM running on that host.

Killing vmware-vws solves that problem but this is no solution.

Interestingly CPU load is quite OK after installation until the ESXi server is rebooted for the first time. I know this sounds strange but I verified it by completely starting from the very beginning and installed everything from scratch again. Same result.

I also verified that restarting VCSA does not cause the problem. It happened after first reboot of ESXI that causes this problem. From this point on nothing helped but deleting and installing VCSA from scratch again. But as already said this only lasts until ESXi is rebooted first time.

Update:

Just for completeness I installed latest VCSA 6.0U1 version, as from what I found in other posts problem begun when upgrading to 6.0U2 or later. My finding indeed support this assumption (see picture no 3). What is noticable is that the spikes on CPU load correlate with the messages in task pane "Acquire CIM ticket..." which you will find every 5 minutes for VCSA 6.0U1 but every minute on VCSA 6.0U1. Maybe this changed interval is the reason why CPU load is much higher on U3.

Before first ESXi reboot CPU is between 3% and 12%

1.JPG

After ESXi reboot CPU is between 20% and 50%

2.JPG

For VCSA 6.0U1 CPU is between 3% and 12% and a spike every 5 minutes

3.JPG

0 Kudos
DaneTruscott
Contributor
Contributor

Did anyone get to the bottom of this.

I have just upgraded to V6U3h and the CPU load increased by 20% as describe by others on this post.

Stopping the "VMware System and Hardware Health Manger" Services Fixes the problem but its not really a fix as such.

Just want to Add,

No.1 is when i updated from 6.0.0, 3634793 (vCenter Server 6.0 Update 2) to 6.0.0, 9313458 (vCenter Server 6.0 U3h)

No2. This is when i stopped the "VMware System and Hardware Health Manager" Service

pastedImage_54.png

Thanks

0 Kudos
Dave_the_Wave
Hot Shot
Hot Shot

I guess the bug is still there in U3h, but I didn't see anyone try U3g?

Again, the most recent version that does not have this issue is VCS 6.0 U1b, which I am using fine in a production environment.

0 Kudos
Sysxp
Enthusiast
Enthusiast

Wow, this issue is here for some time now, yeah. All the way from 23.03.2015 to Dec 2018! Just ... wow.

My solution to this back then was simple - we rolledback and lived on 5.5 and it was great, primarily because of the C# vSphere Client.

Nowadays solution - we moved to ESXi 6.7 with VCSA 6.7.

Todays VCSA is a resource monster too - it always says "Need more memory" and its daily CPU chart looks like this:

vCenterCPU.PNG

In my opinion, however, it works more predictable and surely leaves a better feeling overall. (this does not include the installation process Smiley Happy )

The only BIG dissapointment is the complete lack of the fast and bug free C# vSphere Client.

P.S. (why all of this happens)

- "Dad, what was the worst thing you know from your life?"

- "Three billion devices run Java."

0 Kudos