VMware Cloud Community
heikki_hyperic
Contributor
Contributor

Buildbot monitoring with script plugin

I wrote a plugin for Buildbot (http://buildbot.net/trac) monitoring, but I am unable to get it to work like I'd want/expect.

Here's what I would like: basic process information (mem, cpu, ...), basic HTTP info (response time & code), report number of successful and failed builds. I thought I'd put the process info in a server tag, and treat the HTTP part as service. When I test the plugin from the command line it reports the server and one service were autodiscovered, but I don't get any metrics on the command line, and only the metrics for the server part in the UI. And of the server metrics, only 2 are included by default in the indicator view, even though I would have expected 3.

I am running 4.0.0 RC1. I put the buildbot-plugin.xml file into bundles/agent-4.0.0-889/pdk/plugins/buildbot-plugin.xml for the agent, and hq-engine/server/default/deploy/hq.ear/hq-plugins/buildbot-plugin.xml for the server. I restart the agent after each update of the XML file.

buildbot-plugin.xml:

<!DOCTYPE plugin [
<!ENTITY process-metrics SYSTEM "/pdk/plugins/process-metrics.xml">
]>

<plugin>

<script name="hq-buildbot-stats">
<![CDATA[
#!/usr/bin/env python

import urllib2

html = urllib2.urlopen('http://glama:8010/one_box_per_builder').read()

print 'SuccessfulBuilds=%d' % html.count('successful')
print 'FailedBuilds=%d' % html.count('failed')

]]>
</script>

<server name="Buildbot">

<property name="DEFAULT_LOG_FILE"
value="/path/to/buildmaster/testdir/twistd.log"/>

<!-- DaemonDetector requires PROC_QUERY according to msmq-plugin.xml -->
<property name="PROC_QUERY"
value="Pid.PidFile.eq=/path/to/buildmaster/testdir/twistd.pid"/>

<config>
<option name="process.query"
description="Process Query"
default="${PROC_QUERY}"/>
</config>

<plugin type="autoinventory"
class="org.hyperic.hq.product.DaemonDetector"/>

<plugin type="measurement"
class="org.hyperic.hq.product.MeasurementPlugin"/>

<plugin type="log_track"
class="org.hyperic.hq.product.LogFileTailPlugin"/>

<metric name="Availability"
template="sigar:Type=ProcState,Arg=%process.query%:State"
indicator="true"/>

<metric name="Process Cpu Usage"
template="sigar:Type=ProcCpu,Arg=%process.query%:Percent"
indicator="true"
units="percentage"/>

<metric name="Process Resident Memory Size"
template="sigar:Type=ProcMem,Arg=%process.query%:Resident"
indicator="true"
units="B"/>

<metric name="Process Threads"
template="sigar:Type=ProcState,Arg=%process.query%:Threads"
indicator="true"/>

<property name="HAS_BUILTIN_SERVICES" value="true"/>

<service name="HTTP">

<plugin type="autoinventory"/>

<metric name="Availability"
template="${http.template},port=8010:Availability"
category="AVAILABILITY"
units="percentage"
defaultOn="true"
indicator="true"/>

<metric name="HTTP Response Time"
template="${http.template},port=8010:ResponseTime"
category="THROUGHPUT"
units="ms"
defaultOn="true"
indicator="true"/>

<metric name="HTTP Response Code"
template="${http.template},port=8010:ResponseCode"
category="UTILIZATION"
defaultOn="true"
indicator="true"/>

<filter name="template"
value="exec:timeout=10,file=pdk/work/scripts/buildbot/hq-buildbot-stats,exec=%exec%:${alias}"/>

<metric name="Successful Builds"
indicator="true"/>

<metric name="Failed Builds"
indicator="true"/>

</service>
</server>
</plugin>
0 Kudos
2 Replies
excowboy
Virtuoso
Virtuoso

Hi Heikki,

please follow the instructions and deploy your plugins in a common directory. It makes the plugin deployment a lot more easier and preserves your plugins even if you upgrade your server and agents.
http://support.hyperic.com/display/hypcomm/Deploying+Plugins+-+Basic+HOWTO

>And of the server metrics, only 2 are included by default in the indicator view, even though I would >have expected 3.

I guess your Availability metric doesn't work, right ? If so, please change the metric and add the following line alias="Availability", e.g.

<metric name="Availability"
template="${http.template},port=8010:Availability"
category="AVAILABILITY"
alias="Availability"
units="percentage"
defaultOn="true"
indicator="true"/>

I'm not sure why the metrics for the plugin don't work. Could you just paste the output of the command:
java -jar pdk/lib/hq-product.jar -Dplugins.include=buildbot -Dmetric-indicator=true


Cheers,
Mirko
0 Kudos
heikki_hyperic
Contributor
Contributor

Thanks for the reply.

Using the common directory hasn't worked for me, the plugins won't get discovered at all. Any ideas what to do about this?

Which availability did you mean? The server availability did work, I just saw nothing of the service.

I changed the service availability metric to what you suggested, but I saw no changes. There is no output from the command you asked me to try.

However, now that I read the link you posted again, I then went to check if the Tools menu knows about the Buildbot HTTP service, and it did. So I added that manually, and after a while it started collecting metrics. So I guess that is better than nothing, but I'd really rather have the service automatically appear rather than users having to remember to go to the Tools menu and add services manually that way. Any ideas why it did not appear automatically in the Buildbot services section?
0 Kudos