Run Telegraf as a service on Linux physical server

[toocomfortable]

Hello VMware community,

We are currently using vRealize Operations Manager 8.6. I am trying to get the Telegraf to run as a service on the physical Linux platforms. I tried using the Telegraf deployment script provided by the Cloud proxy, it runs fine when starting the Telegraf manually. But it fails to run as a service. Copied the service unit file to /etc/systemd/system.

Below is the error when trying to start it as a service. Any ideas?

user@ubuntu-20:~$ sudo systemctl status telegraf
● telegraf.service - The plugin-driven server agent for reporting metrics into InfluxDB
     Loaded: loaded (/etc/systemd/system/telegraf.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Wed 2021-12-01 20:41:50 EST; 5min ago
       Docs: https://github.com/influxdata/telegraf
     Process: 2281 ExecStart=/home/user/telegraf/usr/bin/telegraf -config /home/user/telegraf/etc/telegraf/telegraf.conf -config-directory /home/user/telegraf/etc/telegraf/telegraf.d $TELEGRAF_OPTS>
   Main PID: 2281 (code=exited, status=2)

Dec 01 20:41:50 ubuntu-20 systemd[1]: telegraf.service: Scheduled restart job, restart counter is at 5.
Dec 01 20:41:50 ubuntu-20 systemd[1]: Stopped The plugin-driven server agent for reporting metrics into InfluxDB.
Dec 01 20:41:50 ubuntu-20 systemd[1]: telegraf.service: Start request repeated too quickly.
Dec 01 20:41:50 ubuntu-20 systemd[1]: telegraf.service: Failed with result 'exit-code'.
Dec 01 20:41:50 ubuntu-20 systemd[1]: Failed to start The plugin-driven server agent for reporting metrics into InfluxDB.

Journalctl output

user@ubuntu-20:~$ sudo journalctl -xe
Dec 21 16:44:02 ubuntu-20 telegraf[4376]: 2021-12-21T21:44:02Z I! Loaded inputs: cpu disk diskio exec kernel mem processes swap system
Dec 21 16:44:02 ubuntu-20 telegraf[4376]: 2021-12-21T21:44:02Z I! Loaded aggregators:
Dec 21 16:44:02 ubuntu-20 telegraf[4376]: 2021-12-21T21:44:02Z I! Loaded processors:
Dec 21 16:44:02 ubuntu-20 telegraf[4376]: 2021-12-21T21:44:02Z I! Loaded outputs: http
Dec 21 16:44:02 ubuntu-20 telegraf[4376]: 2021-12-21T21:44:02Z I! Tags enabled: host=ubuntu-20
Dec 21 16:44:02 ubuntu-20 telegraf[4376]: 2021-12-21T21:44:02Z I! [agent] Config: Interval:5m0s, Quiet:false, Hostname:"ubuntu-20", Flush Interval:1m0s
Dec 21 16:44:02 ubuntu-20 telegraf[4376]: panic: runtime error: invalid memory address or nil pointer dereference
Dec 21 16:44:02 ubuntu-20 telegraf[4376]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x118 pc=0x1634ab4]
Dec 21 16:44:02 ubuntu-20 telegraf[4376]: goroutine 1 [running]:
Dec 21 16:44:02 ubuntu-20 telegraf[4376]: github.com/influxdata/telegraf/plugins/common/http.(*HTTPClientConfig).CreateClient(0xc0004e4898, 0x58a5d20, 0xc00010c000, 0x4, 0xc000ad1728, 0x50be90)
Dec 21 16:44:02 ubuntu-20 telegraf[4376]:         /build/mts/release/bora-18356315/cayman_telegraf/telegraf/src/github.com/influxdata/telegraf/plugins/common/http/config.go:30 +0x44
Dec 21 16:44:02 ubuntu-20 telegraf[4376]: github.com/influxdata/telegraf/plugins/outputs/http.(*HTTP).Connect(0xc0004e4840, 0x28, 0xc000ad17c0)
Dec 21 16:44:02 ubuntu-20 telegraf[4376]:         /build/mts/release/bora-18356315/cayman_telegraf/telegraf/src/github.com/influxdata/telegraf/plugins/outputs/http/http.go:105 +0x1a4
Dec 21 16:44:02 ubuntu-20 telegraf[4376]: github.com/influxdata/telegraf/agent.(*Agent).connectOutput(0xc00000e610, 0x58a5ce0, 0xc000a8b280, 0xc000106500, 0x4b27f00, 0xc0000c0a90)
Dec 21 16:44:02 ubuntu-20 telegraf[4376]:         /build/mts/release/bora-18356315/cayman_telegraf/telegraf/src/github.com/influxdata/telegraf/agent/agent.go:718 +0x125
Dec 21 16:44:02 ubuntu-20 telegraf[4376]: github.com/influxdata/telegraf/agent.(*Agent).startOutputs(0xc00000e610, 0x58a5ce0, 0xc000a8b280, 0xc00000e608, 0x1, 0x1, 0xc000f25cc0, 0x1a196c25, 0xed95441a2, 0>
Dec 21 16:44:02 ubuntu-20 telegraf[4376]:         /build/mts/release/bora-18356315/cayman_telegraf/telegraf/src/github.com/influxdata/telegraf/agent/agent.go:701 +0x115
Dec 21 16:44:02 ubuntu-20 telegraf[4376]: github.com/influxdata/telegraf/agent.(*Agent).Run(0xc00000e610, 0x58a5ce0, 0xc000a8b280, 0x1, 0x1)
Dec 21 16:44:02 ubuntu-20 telegraf[4376]:         /build/mts/release/bora-18356315/cayman_telegraf/telegraf/src/github.com/influxdata/telegraf/agent/agent.go:116 +0x286
Dec 21 16:44:02 ubuntu-20 telegraf[4376]: main.runAgent(0x58a5ce0, 0xc000a8b280, 0x82f65e8, 0x0, 0x0, 0x82f65e8, 0x0, 0x0, 0x0, 0x0)
Dec 21 16:44:02 ubuntu-20 telegraf[4376]:         /build/mts/release/bora-18356315/cayman_telegraf/telegraf/src/github.com/influxdata/telegraf/cmd/telegraf/telegraf.go:240 +0x859
Dec 21 16:44:02 ubuntu-20 telegraf[4376]: main.reloadLoop(0x82f65e8, 0x0, 0x0, 0x82f65e8, 0x0, 0x0)
Dec 21 16:44:02 ubuntu-20 telegraf[4376]:         /build/mts/release/bora-18356315/cayman_telegraf/telegraf/src/github.com/influxdata/telegraf/cmd/telegraf/telegraf.go:132 +0x263
Dec 21 16:44:02 ubuntu-20 telegraf[4376]: main.run(...)
Dec 21 16:44:02 ubuntu-20 telegraf[4376]:         /build/mts/release/bora-18356315/cayman_telegraf/telegraf/src/github.com/influxdata/telegraf/cmd/telegraf/telegraf_posix.go:7
Dec 21 16:44:02 ubuntu-20 telegraf[4376]: main.main()
Dec 21 16:44:02 ubuntu-20 telegraf[4376]:         /build/mts/release/bora-18356315/cayman_telegraf/telegraf/src/github.com/influxdata/telegraf/cmd/telegraf/telegraf.go:397 +0x790
Dec 21 16:44:02 ubuntu-20 systemd[1]: telegraf.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- An ExecStart= process belonging to unit telegraf.service has exited.
--
-- The process' exit code is 'exited' and its exit status is 2.
Dec 21 16:44:02 ubuntu-20 systemd[1]: telegraf.service: Failed with result 'exit-code'.

[RickVerstegen]

Did you follow the steps in https://docs.vmware.com/en/vRealize-Operations/8.6/com.vmware.vcom.core.doc/GUID-FA44D453-76E9-4BBF-...

[toocomfortable]

Hi Rick,

Yes, followed those exact steps in the guide. Manually starting Telegraf works but running it as a service fails. I am using the service unit file that came with the sample script.

This is the service unit file:

[Unit]
Description=The plugin-driven server agent for reporting metrics into InfluxDB
Documentation=https://github.com/influxdata/telegraf
After=network.target

[Service]
EnvironmentFile=-/etc/default/telegraf
User=user
ExecStart=/home/user/telegraf/usr/bin/telegraf  -config /etc/telegraf/telegraf.conf -config-directory /etc/telegraf/telegraf.d $TELEGRAF_OPTS
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartForceExitStatus=SIGPIPE
KillMode=control-group

[Install]
WantedBy=multi-user.target

[toocomfortable]

Actually I was wrong, it doesn't start correctly even when manually starting it. The error is in the journalctl output

Dec 21 16:44:02 ubuntu-20 telegraf[4376]: 2021-12-21T21:44:02Z I! [agent] Config: Interval:5m0s, Quiet:false, Hostname:"ubuntu-20", Flush Interval:1m0s
Dec 21 16:44:02 ubuntu-20 telegraf[4376]: panic: runtime error: invalid memory address or nil pointer dereference
Dec 21 16:44:02 ubuntu-20 telegraf[4376]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x118 pc=0x1634ab4]
Dec 21 16:44:02 ubuntu-20 telegraf[4376]: goroutine 1 [running]:

[RickVerstegen]

Did you have a look onto the manage agents tab in vRops UI to see if the server is listed?
You can perform several actions from the UI and have a look at the status.

[toocomfortable]

No, I can only see VMs from the Manage Agents tab in vRops UI.

The server I want to install Telegraf on and monitor in vRops is a physical server. Is there a way to get the physical server to be listed under the Manage Agents tab?