VMware Cloud Community
GodAtum
Contributor
Contributor

Getting data from a CSV into Hyperic?

Hi all,

I have a CSV file that is produced by a script. It contain the columns: timestamp, seconds. How can i get the data into Hyperic so that it displays a graph of this? Will something like this work?

hq-plugin.xml

---------------------

<?xml version="1.0"?>

<!--  Define the type of plugin this is, and which classes implement each plugin type -->

<plugin>

     <server name="Script Reader">

       <plugin type="measurement" class="org.hyperic.hq.product.MeasurementPlugin"/>

 

     <property name="service_name" value="Monitoring Latencye"/>

         <config>       

          <option name="script" description="Data Collector script" default="/opt/monitoringDaemon/script.bat"/> 

     </config>

     <filter name="template" value="exec:file=%script%"/>

     <metric name="Script Data" category="PERFORMANCE" indicator="true" template="${template}:w/s"/>

     <metric name="Availability" template="win32:Service=${service_name}:Availability" indicator="true"/>

           

     <service name="Ping Service">

     <metric name="Availability" indicator="true"/>

     <metric name="Response Time 1" units="ms" indicator="true"/>

     <metric name="Response Time 2" units="ms" indicator="true"/>

     <metric name="Response Time 3" units="ms" indicator="true"/>

     <metric name="Response Time 4" units="ms" indicator="true"/>

     </service>

      

     </server>

</plugin>

/opt/monitoringDaemon/script.bat

---------------------------------

TYPE datafile.csv

/opt/monitoringDaemon/times.csv

---------------------------------

timestamp | 0.012

Reply
0 Kudos
14 Replies
hqpso
Enthusiast
Enthusiast

There is no method to be able to put CSV data into the HQ database as you describe.

Since your plugin looks like you're interested in Ping response, any reason you're not just using the Ping Platform Service already in HQ?

Reply
0 Kudos
GodAtum
Contributor
Contributor

The ping response is from another script which connects over IPsec to various servers and outputs the results in a csv. this cannot be changed.

I have written a shell script which echos the values and added a script service. I get a response time but no values in Hyperic under "Result Value". It is giving Execution Time values but 0% Availability. My script:

#!/bin/bash

INPUT=finalTimes.csv

OLDIFS=$IFS

IFS=,

[ ! -f $INPUT ] && { echo "$INPUT file not found"; exit 99; }

while read timestamp r1 r2 r3 r4

do

    echo "Timestamp : $timestamp"

    echo "Response 1 : $r1"

    echo "Response 2 : $r2"

    echo "Response 3 : $r3"

    echo "Response 4 : $r4"

done < $INPUT

IFS=$OLDIFS

Reply
0 Kudos
hqpso
Enthusiast
Enthusiast

Two things.

HQ wants key-value pairs returned with assignment.

Try removing the spaces, avoids premature end of variable testing.

Try changing your script to:

    echo "Response 1=$r1"

    echo "Response 2=$r2"

    echo "Response 3=$r3"

    echo "Response 4=$r4"

I'd make it more like this to get rid of the spaces.

echo Response1=$r1

etc.


I also use camelCase and the same names of the metric in both the plugin and the script to avoid confusion:

<metric name="ResponseTime1" units="ms" indicator="true"/>

... in bash:

echo ResponseTime1=$rs1

GodAtum
Contributor
Contributor

Hi, I'm just running the script through a service atm. Do i actually need a plugin as well? if so how do i get the script to use the plugin?

Reply
0 Kudos
hqpso
Enthusiast
Enthusiast

You already have a plugin in your first entry.

You just need to modify your script being called to return the metric key/value pairs correctly and make sure the string in the 'echo' matches the metric name.

GodAtum
Contributor
Contributor

Ok i have this plugin uploaded to Hyperic and set up a service to point to the script under my Hyperic server platform. However it is still getting 0% availability and only 1 "Result Values".

hq-plugin.xml

---------------------

<?xml version="1.0"?>

<!--  Define the type of plugin this is, and which classes implement each plugin type -->

<plugin>

     <server name="Script Reader">

       <plugin type="measurement" class="org.hyperic.hq.product.MeasurementPlugin"/>

     <property name="service_name" value="Monitoring Latency"/>

        <config>    

            <option name="script" description="Data Collector script" default="/local/home/hyperic/scripts/csv-reading-script.sh"/>

        </config>

     <filter name="template" value="exec:file=%script%"/>

        <service name="Ping Service">

        <metric name="ResponseTime1" units="ms" indicator="true"/>

        <metric name="ResponseTime2" units="ms" indicator="true"/>

        <metric name="ResponseTime3" units="ms" indicator="true"/>

        <metric name="ResponseTime4" units="ms" indicator="true"/>

     </service>

     </server>

</plugin>

csv-reading-script.sh

---------------------------------

#!/bin/bash

INPUT=finalTimes.csv

OLDIFS=$IFS

IFS=,

[ ! -f $INPUT ] && { echo "$INPUT file not found"; exit 99; }

while read rs1 rs2 rs3 rs4

do

    echo "ResponseTime1=$rs1"

    echo "ResponseTime2=$rs2"

    echo "ResponseTime3=$rs3"

    echo "ResponseTime4=$rs4"

done < $INPUT

IFS=$OLDIFS

Reply
0 Kudos
hqpso
Enthusiast
Enthusiast

You have 0% availability because you took out this statement:

<metric name="Availability" template="win32:Service=${service_name}:Availability" indicator="true"/>

Turn on agent debug and you will see what is being returned to the management server.

In agent.properties

agent.log.Level=DEBUG

Reply
0 Kudos
GodAtum
Contributor
Contributor

Ah OK, what is the Linux equivalent for that line of code as references Windows?

In the agent logs:

[pool-1-thread-15] [ExecutableProcess@239] [/local/home/hyperic/scripts/csv-reading-script.sh]: Timeout running [/local/home/hyperic/scripts/csv-reading-script.sh ]

19-08-2013 14:31:12,517 BST ERROR [pool-1-thread-9] [VSphereCollector@163] org.hyperic.hq.product.PluginException: ServiceInstance(https://xxx/sdk, xxx\SVC_xxx): com.vmware.vim25.InvalidLogin

Message was edited by: GodAtum

Reply
0 Kudos
hqpso
Enthusiast
Enthusiast

Depends on what is running in your system.

See the PTQL docs below.

https://support.hyperic.com/display/SIGAR/PTQL

Reply
0 Kudos
GodAtum
Contributor
Contributor

I have tried:

<metric name="Availability" template="linux:Service=${service_name}:Availability" indicator="true"/>

but still 0% avail. Looking at the logs it says:

[pool-1-thread-15] [ExecutableProcess@239] [/local/home/hyperic/scripts/csv-reading-script.sh]: Timeout running [/local/home/hyperic/scripts/csv-reading-script.sh ]

19-08-2013 14:31:12,517 BST ERROR [pool-1-thread-9] [VSphereCollector@163] org.hyperic.hq.product.PluginException: ServiceInstance(https://xxx/sdk, xxx\SVC_xxx): com.vmware.vim25.InvalidLogin

and

[system-0] [ScheduleThread@602] [3:10787:AVAILABILITY] Metric='exec:exec=,file=/local/home/hyperic/scripts/csv-reading-script.sh,args=,timeout=120:Availability' -> 0 timestamp=1376922000625

[system-0] [ScheduleThread@544] collecting data for meas=[derivedId=21066|dsnId=21066]|interval=60000]

[system-0] [ScheduleThread@602] [3:10787:THROUGHPUT] Metric='exec:exec=,file=/local/home/hyperic/scripts/csv-reading-script.sh,args=,timeout=120:ExecTime' -> 120,058 timestamp=1376922000625

[system-0] [ScheduleThread@544] collecting data for meas=[derivedId=21067|dsnId=21067]|interval=60000]

[tomcat-0] [ScheduleThread@602] [3:10315:UTILIZATION] Metric='Catalina:j2eeType=Servlet,name=jsp.resource.application.inventory.ChangeOwner_jsp,WebModule=//localhost/,J2EEApplication=none,J2EEServer=none:requestCount__RATE__=1m:jmx.url=ptql%3AState.Name.re%3Djava|jsvc%2CState.Name.Pne%3Djsvc%2CArgs.*.sw%3D-Dcatalina.base%3D/local/home/hyperic/server-5.0.0-EE/hq-engine/hq-server,jmx.username=system,jmx.password=****************' -> 0 timestamp=1376922000625

[system-0] [ScheduleThread@602] [3:10787:UTILIZATION] Metric='exec:exec=,file=/local/home/hyperic/scripts/csv-reading-script.sh,args=,timeout=120:ResultValue' -> 0.009 timestamp=1376922000626

Metric task 'exec:exec=,file=/local/home/hyperic/scripts/csv-reading-script.sh,args=,timeout=120:ResultValue' complete, duration: 67

Message was edited by: GodAtum

Reply
0 Kudos
GodAtum
Contributor
Contributor

I have edited the script so it only reads the last line in the csv:

#!/bin/bash

INPUT=finalTimes.csv

OLDIFS=$IFS

[ ! -f $INPUT ] && { echo "$INPUT file not found"; exit 99; }

lastline=`cat $INPUT | tail -1`

timestamp=`echo $lastline |cut -d, -f1`

rs1=`echo $lastline |cut -d, -f2`

rs2=`echo $lastline |cut -d, -f3`

rs3=`echo $lastline |cut -d, -f4`

rs4=`echo $lastline |cut -d, -f4`

    echo "timestamp=$timestamp"

    echo "ResponseTime1=$rs1"

    echo "ResponseTime2=$rs2"

    echo "ResponseTime3=$rs3"

    echo "ResponseTime4=$rs4"

IFS=$OLDIFS

however it does not give back any results in Hyperic. The output when runnijg the script via command line is:

Timestamp=2013-08-16 16:17:58
ResponseTime1=0.016
ResponseTime2=0.916
ResponseTime3=0.022
ResponseTime4=0.019

Reply
0 Kudos
GodAtum
Contributor
Contributor

Hi,

Thank you for your help so far. I have manged to get the script itself working but am stuck with no vlaues apperaing in hyperic. In the debug logs:

21-08-2013 11:36:37,147 BST DEBUG [pool-1-thread-7] [Execute@348] Done exit=0 /local/home/hyperic/scripts/csv-reading-script.sh

21-08-2013 11:36:37,147 BST DEBUG [pool-1-thread-7] [ExecutableProcess@247] [/local/home/hyperic/scripts/csv-reading-script.sh]: Timestamp=2013-08-16 16:17:58

ResponseTime1=0.016

ResponseTime2=0.916

ResponseTime3=0.022

ResponseTime4=0.019

21-08-2013 11:36:37,148 BST DEBUG [pool-1-thread-7] [Collector@403] name=exec, thread=pool-1-thread-7, result=Wed Aug 21 11:36:37 BST 2013 [Info] (/local/home/hyperic/scripts/csv-reading-script.sh) Timestamp=2013-08-16 16:17:58

ResponseTime1=0.016

ResponseTime2=0.916

ResponseTime3=0.022

ResponseTime4=0.019

values={Availability=1.0, ResponseTime=27.0, ResponseTime4=0.019, ResponseTime2=0.916, ResponseCode=0.0, ResponseTime3=0.022, ResponseTime1=0.016, Timestamp=2013-08-16 16:17:58}

Reply
0 Kudos
sothy97
Contributor
Contributor

Please UNSUBCRIBE ME

On Mon, Aug 19, 2013 at 6:11 PM, GodAtum <hq-users@hyperic.org> wrote:

I have edited the script so it only reads the last line in the csv:

#!/bin/bash

INPUT=finalTimes.csv

OLDIFS=$IFS

&& { echo "$INPUT file not found"; exit 99; }

lastline=`cat $INPUT | tail -1`

timestamp=`echo $lastline |cut -d, -f1`

rs1=`echo $lastline |cut -d, -f2`

rs2=`echo $lastline |cut -d, -f3`

rs3=`echo $lastline |cut -d, -f4`

rs4=`echo $lastline |cut -d, -f4`

    echo "timestamp=$timestamp"

    echo "ResponseTime1=$rs1"

    echo "ResponseTime2=$rs2"

    echo "ResponseTime3=$rs3"

    echo "ResponseTime4=$rs4"

IFS=$OLDIFS

however it does not give back any results in Hyperic.

>

Reply
0 Kudos
GodAtum
Contributor
Contributor

How do I get the script to reference the plugin?

Reply
0 Kudos