1 2 3 Previous Next 33 Replies Latest reply on Nov 18, 2019 9:12 AM by Sysxp

    vCenter 6.0 - high CPU usage!

    Sysxp Enthusiast

      Hello!

       

      When we was on vCenter 5.5 its CPU consumption chart was nearly flat when idle and was about ~400-500 Mhz max.

      Now we have this when upgraded to vCenter 6:

      vCenter6 CPU.PNG

       

      High peaks every 5 minutes - some "java.exe" process from SYSTEM account suddenly pops out, use several CPU cores and then disappears. It shows 3 times, and use 40-100 Mb each time.

      After another 5 minutes it is repeated.

      The host is ESXi 5.5 and the vCenter VM is the only VM on that host. vCenter is completly idle for hours - no vMotions, etc.

       

      How to find out which vmware service exactly uses the CPU? Maybe it is possible to disable it, or somehow restrict it.

       

      Thank you!

        • 1. Re: vCenter 6.0 - high CPU usage!
          AjithDhas Enthusiast
          VMware Employees

          This might be due to the IIAD daemon thread which is scheduled to be triggered every five minutes. Can you try disabling that??

          • 2. Re: vCenter 6.0 - high CPU usage!
            Sysxp Enthusiast

            It is true - there was a task in Task Scheduler called "VMwareIIAD". But when I disabe it and make sure it is not starting anymore nothing changed - the "java.exe" processes continue to spawn and consume CPU.

            There must be something else.

             

            P.S. I also tried to run the task manually several times - it does not consume CPU at all. (no more than 1-2% on one core very briefly). The real CPU eater is someone else.

            • 3. Re: vCenter 6.0 - high CPU usage!
              AjithDhas Enthusiast
              VMware Employees

              Can you check the program which triggers this java.exe process.

               

              In Windows Task Manager  Go to "View" - "Select Columns..." and check "Command Line". which will show you the source of the process.

              1 person found this helpful
              • 4. Re: vCenter 6.0 - high CPU usage!
                Sysxp Enthusiast

                AjithDhas wrote:

                In Windows Task Manager  Go to "View" - "Select Columns..." and check "Command Line". which will show you the source of the process.

                Oh, that was helpful, didn't know about that useful column, shame on me.

                Here's what happening:

                1. "C:\Program Files\VMware\vCenter Server\jre\bin\java.exe" -Xms8M -Xmx32M -jar is.jar --get-inv

                a little after:

                2. "C:\Program Files\VMware\vCenter Server\jre\bin\java.exe" -Xms8M -Xmx32M -jar vc.jar --get-tickets

                after that:

                3. "C:\Program Files\VMware\vCenter Server\jre\bin\java.exe" -Xms8M -Xmx32M -jar wbem.jar

                 

                Can you please shed light on what is going on?

                Thank you!

                • 5. Re: vCenter 6.0 - high CPU usage!
                  AjithDhas Enthusiast
                  VMware Employees

                  From the sequence I feel that this is for getting host health status and update the same... No more idea on this...

                  • 6. Re: vCenter 6.0 - high CPU usage!
                    Sysxp Enthusiast

                    Tried to rename those .jar files but unsuccesful - they are in use. Not sure though that renaming .jar files is appropriate way to correct this issue.

                     

                    If anyone know why this is happening and how to disable it - I would be very grateful.

                    This current behavior is definitely not nice at all.

                    • 7. Re: vCenter 6.0 - high CPU usage!
                      Sysxp Enthusiast

                      Ok, I found an answer to all the issues we encountered with vCenter 6. Not the one I was hoping for, but still.

                      We roll back to 5.5! Neat.

                      Everything is just normal now - plugins, CPU usage, strange "quick stats is not up to date" messages, even the WebClient is working normal again.

                      I guess, we'll try once again in a year.

                       

                      Speaking of the CPU - here's the same VM on the same host.

                      Just compare with the picture in the 1st post:

                      vCenter55.PNG

                      • 8. Re: vCenter 6.0 - high CPU usage!
                        wishart Lurker

                        We are having the same problem since upgrading to 6.0.  One of the many Java SE binary processes starts consuming 90%-100% CPU out of the blue for no reason.

                        • 9. Re: vCenter 6.0 - high CPU usage!
                          ChristophBerthoud Lurker

                          I too am having this issue, a java.exe process is consuming 100% CPU

                           

                          Environment:

                          vCenter Server 6.0.0, 2559268 installed on Windows Server 2012 R2

                           

                          Process Name: java.exe

                          Process User Name: EsxAgentManager

                          Command Line: "C:\Program Files\VMware\vCenter Server\jre\\bin\java" -Djava.util.logging.manager=com.springsource.tcserver.serviceability.logging.TcServerLogManager -Djava.util.logging.config.file=C:\ProgramData\VMware\vCenterServer\cfg\vmware-eam\logging.properties -Dwrap

                          • 10. Re: vCenter 6.0 - high CPU usage!
                            Althornin Lurker

                            So, for those of us that were having a Java process consume 100% of the CPU for extended times, I believe I have a fix:

                            1) Are you using replacement (non-self signed) certificates?  We are!

                             

                            If so, the problem is actually this:

                            http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2112577


                            From looking at the logs under 100% CPU usage, the EAM service is in a login loop, which fails due to certificate replacement.


                            Once we fixed that issue, the high CPU utilization went away.

                            • 11. Re: vCenter 6.0 - high CPU usage!
                              ATLANTIKA07 Lurker

                              Im my enviroment the Java high cpu issue was the Sophos Virusscanner.

                               

                              Kai

                              • 12. Re: vCenter 6.0 - high CPU usage!
                                Mitchcss Lurker

                                We are experiencing the same issue...  It's the busiest box I have in my environment.

                                 

                                 

                                cpu.png

                                Any info on this?!?!

                                 

                                Thank you,

                                • 13. Re: vCenter 6.0 - high CPU usage!
                                  RaoulSchaffner Novice

                                  Hello!

                                   

                                  I experience exactly the same scenario (as far as i can tell)

                                   

                                  environment:

                                  - windows server 2012 r2, 4 vcpu, 24 gb ram (up from 16 gb originally, didn't help)

                                  - vcenter server 6.0.0 build 3634793

                                  - ms sql server 2014 on same machine

                                   

                                  i also see these three processes in rapid succession (terminated every ~20 seconds) with a short pause every 5 minutes.

                                  "C:\Program Files\VMware\vCenter Server\jre\bin\java.exe" -Xms8M -Xmx32M -jar is.jar --get-inv

                                  "C:\Program Files\VMware\vCenter Server\jre\bin\java.exe" -Xms8M -Xmx32M -jar vc.jar --get-tickets

                                  "C:\Program Files\VMware\vCenter Server\jre\bin\java.exe" -Xms8M -Xmx32M -jar wbem.jar

                                   

                                  the error file lies here:

                                  C:\ProgramData\VMware\vCenterServer\data\core\vmware-vws\write_status-err.txt

                                  and shows a java memory heap size problem at the end: Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

                                   

                                  i made sure that those java processes are triggered by the process vmwarevws (vmware system and hardware health manager) by killing vmwarevws: they were gone and didn't show up anymore. that's no solution, though. :-)

                                  PS C:\Program Files\VMware\vCenter Server\bin> .\service-control.bat --stop vmwarevws

                                   

                                  i tried to influence the dynamic memory allocation of vcenter 6 as pointed out by others to increase the allocation for the vmwarevws process by using these command line statements:

                                  PS C:\Program Files\VMware\vCenter Server\bin> .\service-control.bat --stop vmwarevws

                                  PS C:\Program Files\VMware\vCenter Server\visl-integration\usr\sbin> .\cloudvm-ram-size.bat -C 100 vmwarevws

                                  PS C:\Program Files\VMware\vCenter Server\visl-integration\usr\sbin> .\cloudvm-ram-size.bat -S

                                  PS C:\Program Files\VMware\vCenter Server\bin> .\service-control.bat --start vmwarevws

                                   

                                  however, after restarting the vmwarevws process, the java processes in question did not get different command line options, i.e. the Xms/Xmx values are still the same.

                                  it looks like the vmwarevws process does not take part in the dynamic memory allocation feature of vcenter 6.

                                   

                                  i hope my conclustions are wrong and someone points me in the right direction! ;-)

                                   

                                  cheers,

                                  raoul.

                                  • 14. Re: vCenter 6.0 - high CPU usage!
                                    znamor Lurker

                                    Same issue on Vcenter 6U2 appliance VM that's running Windows 2008R2 Standard SP1.  We have two java.exe processes that are doing this on the command line (see attached image):

                                    Screen Shot 2016-11-01 at 5.45.45 PM.png

                                     

                                    I received two replies from VMware tech support on this matter:

                                     

                                    1) There is a patch for 6U2 scheduled to be released on 10th on November 2016 to fix this. 

                                    ..a few days later this...

                                    2) "Told CU the patch is coming by the end of this month tentetive date as 22nd nov 2016.  also made changes in conf file of the vpxd.conf set the heap size as 1024 and asked cu to observe this for 2 days and get back to us."  (aside from spelling errors they actually never called or emailed me about this and just snuck the later information in by directly posting it to the open SR online LOL.  Good think I looked at the ticket online.  I bet this was another filibuster tactic of theirs to move the date and tell the customer later -- WOW !excellent support practices Vmware engineers)

                                     

                                    Now since I have an open SR on this I sort of have VMware support on the hook. 

                                    VMWARE SUPPORT - when is the patch actually coming out?

                                     

                                     

                                    1 2 3 Previous Next