VMware Cloud Community
Perzlmayr
Contributor
Contributor

CPU Usage of Hyperic

Hallo Hyperic Users.
My Hyperic Server is a virtual machine with 2 CPU´s and 2048GB RAM.
On this machine we have installed Windows 2003 Server (Standard 32 Bit).

After nearly one day the CPU Usage of the Server is at 50 % (100% of 1 CPU)
and the Hyperic System is standing still.
Can Hyperic use 2 or more CPu´s?
Or do i have a problem with my virtual machine?

The hyperic Process is at 700MB of Memory Usage.
When the Server arrived this critical point, i only can restart the whole Server to shut down the hyperic Proccess.

What can i do?
Any Ideas?
Greetings
Michael
Reply
0 Kudos
17 Replies
TheChristoph
Contributor
Contributor

The Hyperic-Server can definitely use more than one CPU.
But one of it's Threads can't use more than one CPU.

What means the Hyperic System is standing still?
Aret you able to log in?

Have you take a look on your log-files?
Reply
0 Kudos
Perzlmayr
Contributor
Contributor

Hallo,
i am able to login, but the system is so slow that i get the complete dashboard after more than 15 Minutes.
Each klick in the site will take more than 5 minutes again.
After a look at the Task Manager of the Windows Server 2003 i have 50% Percent CPU Load (1 CPU full and 1 nothing)
It seems that hyperic is only using one cpu.
After a look at the Prozesses list i get nearly the same result.
The Hyperic server uses 50 % of cpu load and nearly 700 MB of used memory
(When is restart the HQ Server the Pc Resources of hyperic are 0 - 10 % CPU Load and 420 MB used memory.

What can i do besides a daily restart of the server ?
Greetings Michael
Reply
0 Kudos
excowboy
Virtuoso
Virtuoso

Hi,

which Hyperic HQ version are you running ? Are you using a version with the bundled database ?

Cheers,
Mirko
Reply
0 Kudos
Perzlmayr
Contributor
Contributor

Hi,
we are using Hyperic HQ Server 4.0.3
and the database is an outsourced (not on the same machine) Oracle 9g

Greetings
Michael
Reply
0 Kudos
excowboy
Virtuoso
Virtuoso

Hallo Michal,

could you provide a screenshot of your HQ Health screen ? Are you sure that HQ Server is running on top of a recommended SUN JRE ?

Cheers,
Mirko
Reply
0 Kudos
Perzlmayr
Contributor
Contributor

Hello Mirko,
i have running JRE 1.6.013-b03
i have attached a picture of the Windows task manager where you can see the Prozesses who are already running on top.
I have restartet the Hyperic server today, so the Memory Usage is only at 580 MB, but it is still rising.
after one day the memory usage is at nearby 750 MB and the system is standing still.

In the pictures you can see that after the resart the system is using less memory and less swap than before the restart.
Greetings
Michael
Reply
0 Kudos
excowboy
Virtuoso
Virtuoso

Hallo Michael,

sorry for the delay.

the HQ Health screen is available in the Administration tab:
http://support.hyperic.com/display/DOC/ui-Admin

Could you please make a screenshot again ?

Cheers,
Reply
0 Kudos
Perzlmayr
Contributor
Contributor

Hallo Excowboy,
thank you for your reply,
i have made the HQ Health screen and attached it here for you.

Maybe the Problem is the jvm file.
I think the server is going slower when the JVM File has arrived 700 MB.
Can is change the maximum of this file?

Thanks
Michael
Reply
0 Kudos
jvalkeal_hyperi

What I can see from the 'health' is that your 'Open FDs' value is too large. It's almost 7000 and I think it shouldn't be more than few hundreds.

This is something what you could try to check. If you can get a list of what file descriptors are opened by HQ, it could shed some light... Btw, what was the OS?

Edit: Ah, was it win2003?

Edit2: Maybe FD's are not the problem. Just checked this with my test instance on XP and FD size seems to be much higher on windows than on *nix.
Reply
0 Kudos
Perzlmayr
Contributor
Contributor

The System is Windows 2003 Standard Server.

We will install a SLES 10 VMWare soon so that i can use Hyperic on that Virtual Machine.

How can i find out the file descriptors who are opened by HQ?

Greets
Michael
Reply
0 Kudos
jvalkeal_hyperi

FD's propably are not causing the problems, but there is very good toolset from sysinternals. It's these days owned by Microsoft. Handle tool and process explorer are the ones to check what files are open.

http://technet.microsoft.com/en-us/sysinternals/bb896655.aspx
Reply
0 Kudos
Perzlmayr
Contributor
Contributor

Handle will give me the following return

(Hyperic HQ Server):
handle -s

Handle type summary:
Desktop : 58
Directory : 103
Event : 8583
File : 3509
IoCompletion : 193
Job : 2
Key : 982
KeyedEvent : 43
Mutant : 344
Port : 404
Process : 197
Section : 104
Semaphore : 1317
SymbolicLink : 5
Thread : 1905
Timer : 44
Token : 61
WaitablePort : 3
WindowStation : 86
WmiGuid : 67
Total handles: 18010

I testet the tool on an other server
and thats the issue of that server:

handle -s

Handle type summary:
Desktop : 73
Directory : 133
Event : 3045
File : 2210
IoCompletion : 249
Job : 2
Key : 1197
KeyedEvent : 55
Mutant : 419
Port : 488
Process : 248
Section : 132
Semaphore : 1116
SymbolicLink : 9
Thread : 890
Timer : 52
Token : 91
WaitablePort : 3
WindowStation : 108
WmiGuid : 81
Total handles: 10601
Reply
0 Kudos
jvalkeal_hyperi

I believe correct format of this command is: handle.exe -a -p hq-server

Btw, do you have agent installed on this server machine. If you have, you could try to check if there is some metrics which are giving higher values over time.

Sysinternals process monitor is also worth to try. There you can select process, open process properties and see the threads. There you could try to check if one single thread is taking all cpu cycles(which means jammed thread or just heavy load). Debugging java on windows is very tricky job to do.

For oracle, do you have oracle enterprise manager installed? It would be worth to check if there is any expensive queries or other sql clauses which is taking too much time.

Have you found anything weird from hq's server log?
Reply
0 Kudos
Perzlmayr
Contributor
Contributor

Hallo jvalkeal
yes there is a agent installed on this machine.

During my easter holiday my collegues say that they have to reboot the whole Server again.

I should change the system! Maybe i can resolf the problem when i change the windows 2003 server to an Linux server.

I think the problem is not the database.
Reply
0 Kudos
bjchip_hyperic
Enthusiast
Enthusiast

Windows is not unlike every other multiprocess OS.

This is almost certainly NOT a CPU issue. It is a memory issue. When the running system has to swap out memory blocks too often it is called "thrashing" and it will eat your system up.

Be concerned that something is leaking memory if the memory consumption keeps going up. The JVM needs more aggressive garbage collection?

The open-files is not good, but could be a symptom of something that is crashing out and restarting without closing files, rather than hyperic itself.



BJ
Reply
0 Kudos
Perzlmayr
Contributor
Contributor

Ah OK, than the only thing i can do is to make a daily restart of the Service?
Or is there more i can do?
Greetings
Michael
Reply
0 Kudos
bjchip_hyperic
Enthusiast
Enthusiast

The garbage collection of a JVM is pretty extensively configurable.

http://java.sun.com/docs/hotspot/gc5.0/gc_tuning_5.html

The other thing of course, is to add physical memory. This won't help a lot if there is a real leak, but it does wonders if it gives the system enough headroom to actually collect the garbage.

There are also parameters at the system level relating to process swapping and memory retention. Not sure how to manipulate this in Windows... but it has to be there.

respectfully
BJ
Reply
0 Kudos