Pulling Metrics, But Showing Server Unavailable

[ihilder]

I am in the course of setting up a server to monitor a number of agents. The Server is up and running fine, monitoring its local agent. The issue I can having is with two other agents a just added. The server seems to be pulling select metrics, and sometimes shows services, such as Apache, as being unavailable, yet the metrics are visible. The HQ server stats are:

MacOSX Server 10.4
HQ Server, Agent, Shell 3.2

I also have the following stacktrace in my log after starting the server:
2008-03-28 12:20:49,700 ERROR [http-0.0.0.0-7080-10] [org.hyperic.hq.ui.servlet.ImageServlet] Unknown error.
java.awt.HeadlessException
at sun.java2d.HeadlessGraphicsEnvironment.getDefaultScreenDevice(HeadlessGraphicsEnvironment.java:65)
at apple.awt.OSXOffScreenSurfaceData.getCopyWithBgColor(OSXOffScreenSurfaceData.java:762)
at apple.awt.OSXSurfaceData.blitImage(OSXSurfaceData.java:1388)
at apple.awt.CRenderer.blitImage(CRenderer.java:698)
at apple.awt.CRenderer.copyImage(CRenderer.java:712)
at apple.awt.CRenderer.copyImage(CRenderer.java:799)
at sun.java2d.SunGraphics2D.drawImage(SunGraphics2D.java:3160)
at org.hyperic.image.chart.AvailabilityReportChart.draw(AvailabilityReportChart.java:135)
at org.hyperic.image.chart.Chart.draw(Chart.java:460)
at org.hyperic.image.WebImage.getImage(WebImage.java:257)
at org.hyperic.image.WebImage.writeImage(WebImage.java:306)
at org.hyperic.image.WebImage.writePngImage(WebImage.java:385)
at org.hyperic.hq.ui.servlet.ChartServlet.renderPngImage(ChartServlet.java:167)
at org.hyperic.hq.ui.servlet.ImageServlet.doGet(ImageServlet.java:115)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:697)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at org.hyperic.hibernate.filter.SessionFilter$1.run(SessionFilter.java:59)
at org.hyperic.hq.hibernate.SessionManager.runInSessionInternal(SessionManager.java:79)
at org.hyperic.hq.hibernate.SessionManager.runInSession(SessionManager.java:68)
at org.hyperic.hibernate.filter.SessionFilter.doFilter(SessionFilter.java:57)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:81)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at org.hyperic.hq.product.servlet.filter.JMXFilter.doFilter(JMXFilter.java:324)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
at org.jboss.web.tomcat.security.CustomPrincipalValve.invoke(CustomPrincipalValve.java:39)
at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:159)
at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:59)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:744)
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112)
at java.lang.Thread.run(Thread.java:613)
2008-03-28 12:20:49,702 ERROR [http-0.0.0.0-7080-10] [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/].[availStoplight]] Servlet.service() for servlet availStoplight threw exception
java.awt.HeadlessException
at sun.java2d.HeadlessGraphicsEnvironment.getDefaultScreenDevice(HeadlessGraphicsEnvironment.java:65)
at apple.awt.OSXOffScreenSurfaceData.getCopyWithBgColor(OSXOffScreenSurfaceData.java:762)
at apple.awt.OSXSurfaceData.blitImage(OSXSurfaceData.java:1388)
at apple.awt.CRenderer.blitImage(CRenderer.java:698)
at apple.awt.CRenderer.copyImage(CRenderer.java:712)
at apple.awt.CRenderer.copyImage(CRenderer.java:799)
at sun.java2d.SunGraphics2D.drawImage(SunGraphics2D.java:3160)
at org.hyperic.image.chart.AvailabilityReportChart.draw(AvailabilityReportChart.java:135)
at org.hyperic.image.chart.Chart.draw(Chart.java:460)
at org.hyperic.image.WebImage.getImage(WebImage.java:257)
at org.hyperic.image.WebImage.writeImage(WebImage.java:306)
at org.hyperic.image.WebImage.writePngImage(WebImage.java:385)
at org.hyperic.hq.ui.servlet.ChartServlet.renderPngImage(ChartServlet.java:167)
at org.hyperic.hq.ui.servlet.ImageServlet.doGet(ImageServlet.java:115)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:697)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at org.hyperic.hibernate.filter.SessionFilter$1.run(SessionFilter.java:59)
at org.hyperic.hq.hibernate.SessionManager.runInSessionInternal(SessionManager.java:79)
at org.hyperic.hq.hibernate.SessionManager.runInSession(SessionManager.java:68)
at org.hyperic.hibernate.filter.SessionFilter.doFilter(SessionFilter.java:57)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:81)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at org.hyperic.hq.product.servlet.filter.JMXFilter.doFilter(JMXFilter.java:324)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
at org.jboss.web.tomcat.security.CustomPrincipalValve.invoke(CustomPrincipalValve.java:39)
at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:159)
at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:59)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:744)
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112)
at java.lang.Thread.run(Thread.java:613)

Theres a bit more to the Stacktrace, let me know if you need it or if you need any further information. Thanks in advance.

-Ian

[bclark_hyperic]

Try making sure that the time is exactly the same between all of the agents and the server. This is usually the cause of the behavior that you are seeing. I suggest setting all of them sync up their time via NTP with the same time source.

Brian

[admin]

The HeadlessExceptions can be ignored, it's a known issue when running on OSX.

See: http://jira.hyperic.com/browse/HHQ-1156

You can fix by removing the -Djava.awt.headless=true line in hq-server.sh.

-Ryan