Re: vRLI and webhook alerts

sxnxr · ‎11-21-2017

All I am new this so help is appreciated

We have vRLI setup to send alerts to a webhook shim server. The shim server has and endpoint configured and forwarding on modified payloads to our upstream service

This is working well for our vrops alerting but there are some event based alerts that we want to setup for auditing in our upstream service. some of which vrops cant alert on so we set up vrli to alert through to vrops which in turn sent the rest payload to the SHIM but vrops does not seem to pass on any of the description or recommendations for Notification alerts so i get an audit event on the object in the up stream service but no data as to what it is.

I then setup a new endpoint for vrli and sent the alert directly from vrli to the SHIM server. This sends the description but i cant seem to find the source in the payload that the SHIM receives

Below is the log of the SHIM ( this is a simple event for SSH being enabled on a host for testing) as you can see there is no host name listed. I change the host name to colour in the py script just to get it to forward out of the SHIM and not error out

Nov 21 04:19:50 xxxxxx iptables[197]: 2017-11-21 04:19:50,364 INFO Parsed={'hookName': 'Log Insight', 'color': 'red', 'AlertName': 'SSH Enabled (W

ebhook)', 'info': 'Test webhook Discription', 'Messages': [], 'url': 'https://vrli-xxxxxxxxxxx', 'editurl': 'https://vrli-xxxxxxxxx/s/nnv2dq', 'HasMoreResults': 'False', 'NumHits': '0', 'icon': 'http://blogs.vmware.com/management/files/2015/04/li-logo.png', 'more

info': 'Hello from the webhook shim! This is a test webhook alert.\n\nAlert Name: SSH Enabled (Webhook)\nAlert Info: Test webhook Discription', 'fi

elds': [{'name': 'HasMoreResults', 'content': 'False'}, {'name': 'NumHits', 'content': '0'}]}

Nov 21 04:19:50 xxxxx iptables[197]: 2017-11-21 04:19:50,364 INFO URL=http://xxxxxx:50070

Nov 21 04:19:50 xxxxx iptables[197]: 2017-11-21 04:19:50,364 INFO Headers={'Content-type': 'application/json', 'Accept': 'application/json'}

Nov 21 04:19:50 xxxxx iptables[197]: 2017-11-21 04:19:50,365 INFO Body={"result": {"AlertMessage": "Test webhook Discription", "Criticality": "re

d", "EventInstance": "red", "EventName": "SSH Enabled (Webhook)", "HostName": "red", "TransformerName": "VMWare", "type": "link", "url": "https://v

rli-xxxxxxxxxx/s/barcgx"}}

Nov 21 04:19:50 xxxxx iptables[197]: 2017-11-21 04:19:50,365 INFO Check=True

Nov 21 04:19:50 xxxxx iptables[197]: 2017-11-21 04:19:50,367 DEBUG Starting new HTTP connection (1): xxxxxxxxx

Nov 21 04:19:50 xxxxx iptables[197]: 2017-11-21 04:19:50,373 DEBUG http://xxxxxxx:50070 "POST / HTTP/1.1" 200 None

Nov 21 04:19:50 xxxxx iptables[197]: 2017-11-21 04:19:50,374 INFO 10.195.239.11 - - [21/Nov/2017 04:19:50] "POST /endpoint/vrli HTTP/1.1" 200 -

Am i doing something wrong?

This is the event i am alerting on.

jasnyder · ‎11-21-2017

Could you provide a little more information about the webhook server you have set up? Admittedly, I have never configured this aspect of vRLI before, but I just threw together an example. I wrote an HTTP server in golang which accepts the POST method to a specific path. I have it dumping the body of the request to the log so I can see natively what vRLI is sending. I created a simple alert in vRLI that sends to my simple web server on any event that contains the word "logon". I am seeing that it's sending the hostname information. Here is an example POST payload from vRLI to my golang server:

{

"AlertType":1,

"AlertName":"Login Received",

"SearchPeriod":300000,

"HitCount":0.0,

"HitOperator":2,

"messages":[

{

"text":"2017-11-21 13:44:52.783 vcs01.lab.local vcenter-server: User LAB\\svc_vcloginsight@192.168.12.41 logged out (login time: Tuesday, 21 November, 2017 13:44:52, number of API invocations: 2, user agent: JAX-WS RI 2.2.9-b130926.1035 svn-revision#5f6196f2b90e9460065a4c2f4e30e065b245e51e)",

"timestamp":1511271892783,

"fields":[

{

"name":"vmw_vc_api_invocations",

"content":"2"

},{

"name":"vmw_vc_auth_user",

"content":"LAB\\svc_vcloginsight"

},{

"name":"vmw_host",

"content":"esxi07.lab.local"

},{

"name":"vmw_vc_auth_source",

"content":"192.168.12.41"

},{

"name":"vmw_vcenter",

"content":"vcs01"

},{

"name":"vmw_vcenter_id",

"content":"E0DBBBD5-9136-4305-9EAF-9E2EF2FBF0D3"

},{

"name":"vmw_vr_ops_id",

"content":"a7f00838-3c9c-430a-87e3-4e520e3c1557"

},{

"name":"hostname",

"content":"vcs01.lab.local"

},{

"name":"vmw_cluster",

"content":"cl-02"

},{

"name":"vmw_datacenter",

"content":"Lab"

},{

"name":"vc_event_type",

"content":"com.vmware.vim25.UserLogoutSessionEvent"

},{

"name":"vmw_object_id",

"content":"vm-11322"

},{

"name":"appname",

"content":"vcenter-server"

},{

"name":"vc_username",

"content":"LAB\\svc_vcloginsight"

},{

"name":"__li_source_path",

"content":"vcs01.lab.local"

}

...additional messages removed...

]

},

"HasMoreResults":false,

"Url":"https://logs01.lab.local/s/yvgobg",

"EditUrl":"https://logs01.lab.local/s/kn8y06",

"Info":null,

"Recommendation":null,

"NumHits":9

}

This alert tracked a logout of the service account vRLI was using to access vCenter. The Hostname shows up in the messages[0].fields[7].content value. My guess is whatever you're using to process the web hook isn't parsing it correctly, but I'd need to know what that is to look into it.

If I had to guess, I would say you might be using this - webhook-shims/loginsightwebhookdemo at master · vmw-loginsight/webhook-shims · GitHub

If so, I took a real quick look through it, and it appears that for every handler type, it is parsing the body of the request and placing it in a payload. For example, this is from the template.py:

a = parse(request)

payload = {

"body": a['info'],

"title": a['AlertName'],

"type": "link",

"url": a['url'],

}

As you can see, it doesn't access the messages field or the fields field within each message object. So if you write a handler based on this template, the resultant JSON payload would have basically not good info in it.

Another example from the ServiceNow handler:

a = parse(request)

payload = { "comments": a['moreinfo'] }

This one only gets the "moreinfo" field. I don't see the "moreinfo" field present in my example. So this one doesn't look like it would have a lot of useful info to post to Service Now.

If this is indeed the solution you're using, you'll need to modify the section that builds the payload to have it access the correct portion of the request body JSON. If you provide some more details about the mechanism you're using to handle the POST from vRLI, I will try to help you piece together the right code/changes.

sxnxr · ‎11-21-2017

yea i am using - webhook-shims/loginsightwebhookdemo at master · vmw-loginsight/webhook-shims · GitHub

the problem is i cant get the host name added this is what is configured

payload = {

"result": {

"TransformerName": "VMWare",

"AlertMessage": a['Info'],

"EventInstance": a['AlertName'],

"Criticality": a['color'],

"EventName": a['AlertName'],

"HostName": a['color'],

"type": "link",

"url": a['url'],

But if i look at what the SHIM shows me i dont have a hostname field ( i have change it in the above to color to get it to work)

Nov 21 04:19:50 xxxxxx iptables[197]: 2017-11-21 04:19:50,364 INFO Parsed={'hookName': 'Log Insight', 'color': 'red', 'AlertName': 'SSH Enabled (W

ebhook)', 'info': 'Test webhook Discription', 'Messages': [], 'url': 'https://vrli-xxxxxxxxxxx', 'editurl': 'https://vrli-xxxxxxxxx/s/nnv2dq', 'HasMoreResults': 'False', 'NumHits': '0', 'icon': 'http://blogs.vmware.com/management/files/2015/04/li-logo.png', 'more

info': 'Hello from the webhook shim! This is a test webhook alert.\n\nAlert Name: SSH Enabled (Webhook)\nAlert Info: Test webhook Discription', 'fi

elds': [{'name': 'HasMoreResults', 'content': 'False'}, {'name': 'NumHits', 'content': '0'}]}

Nov 21 04:19:50 xxxxx iptables[197]: 2017-11-21 04:19:50,364 INFO URL=http://xxxxxx:50070

Nov 21 04:19:50 xxxxx iptables[197]: 2017-11-21 04:19:50,364 INFO Headers={'Content-type': 'application/json', 'Accept': 'application/json'}

Nov 21 04:19:50 xxxxx iptables[197]: 2017-11-21 04:19:50,365 INFO Body={"result": {"AlertMessage": "Test webhook Discription", "Criticality": "re

d", "EventInstance": "red", "EventName": "SSH Enabled (Webhook)", "HostName": "red", "TransformerName": "VMWare", "type": "link", "url": "https://v

rli-xxxxxxxxxx/s/barcgx"}}

Nov 21 04:19:50 xxxxx iptables[197]: 2017-11-21 04:19:50,365 INFO Check=True

Nov 21 04:19:50 xxxxx iptables[197]: 2017-11-21 04:19:50,367 DEBUG Starting new HTTP connection (1): xxxxxxxxx

Nov 21 04:19:50 xxxxx iptables[197]: 2017-11-21 04:19:50,373 DEBUG http://xxxxxxx:50070 "POST / HTTP/1.1" 200 None

Nov 21 04:19:50 xxxxx iptables[197]: 2017-11-21 04:19:50,374 INFO 10.195.239.11 - - [21/Nov/2017 04:19:50] "POST /endpoint/vrli HTTP/1.1" 200 -

I have bolded the vrli headings that i see in the SHIM and also the resulted output using the payload.

What i am not sure is if what i am seeing when i use the journalctl -n 30 when the alert is triggered is all the data passed to the SHIM from vrli or is it a pharsed version that is cutting out the host name.

Again i am new to this so i dont know if i am using the correct command to see the original payload from vrli or if i have a config somewhere blocking some of the payload.

jasnyder · ‎11-21-2017

I narrowed the problem down to this: it is receiving the 'messages' field in the body from the POST method from Log Insight. It's not doing anything with it, except putting it into the ['messages'] field. You can access this field as a['Messages']. But the way the data is posted, you need to do a little bit of processing on it. First of all, Log Insight might send you several events of the same type. They might be from the same host or not. This method will only pull the host from the first message and put it in a field called "Hostname" on the alert object. So if any other hosts are affected and sent in the same payload, you will miss them. If you want each of them, you might have to get creative because you'll want to send to your upstream server once for each message object. You would handle that by iterating over the a['Messages'] object.

For my fix you need to update the /loginsightwebhookdemo/__init__.py file and update the parseLI function to this:

def parseLI(payload, alert):

"""

Parse LI JSON from alert webhook.

Returns a dict.

"""

if (not 'AlertName' in payload):

return alert

alert.update({

"hookName": "Log Insight",

"color": "red",

"AlertName": payload['AlertName'] if ('AlertName' in payload) else "<None>",

"info": payload['Info'] if ('Info' in payload and payload['Info'] is not None) else "",

"Messages": payload['messages'] if ('messages' in payload) else "",

"url": payload['Url'] if ('Url' in payload and payload['Url'] is not None) else "",

"editurl": payload['EditUrl'] if ('EditUrl' in payload and payload['EditUrl'] is not None) else "",

"HasMoreResults": str(payload['HasMoreResults']) if 'HasMoreResults' in payload else False,

# may be less than length of messages, if there's more events

"NumHits": str(payload['NumHits']) if 'NumHits' in payload else False,

"icon": "http://blogs.vmware.com/management/files/2015/04/li-logo.png",

})

if ('Info' in payload and payload['Info'] is not None):

alert.update({"info": payload['Info']})

elif ('messages' in payload and payload['messages'] is not None and len(payload['messages']) > 0):

alert.update({"info": payload['messages'][0]['text']})

else:

alert.update({"info": ""})

if ('Messages' in alert and not alert['Messages']): # If a test alert

alert.update({

"moreinfo": ("Hello from the webhook shim! This is a test webhook alert.\n\n") + \

("Alert Name: ") + alert['AlertName'] + \

("\nAlert Info: ") + alert['info'],

})

else:

alert.update({

"moreinfo": alert['AlertName'] + ("\n\n") + alert['info'] + \

("\n\nYou can view this alert by clicking: %s" % alert['url'] if alert['url'] else "") + \

("\nYou can edit this alert by clicking: %s" % alert['editurl'] if alert['editurl'] else ""),

})

if alert['HasMoreResults']:

alert.update({

"fields": [

{ "name": 'HasMoreResults', "content": alert['HasMoreResults'], },

{ "name": 'NumHits', "content": alert['NumHits'], }

]

})

else:

alert.update({"fields": []})

if alert['Messages']:

for v in alert['Messages'][0]['fields']:

if v['name'] == 'hostname':

alert.update({"Hostname":(v['content'])})

return alert

The section highlighted in red above is what I added. This will iterate over the fields in the first message, find the one named "hostname" and add it in the alert object as "Hostname". Then you can access it in your module as a['Hostname']. I updated the test endpoint in my implementation to output the value of the "Hostname" field and then fired an event. This is the output of that:

2017-11-21 12:50:12,520 INFO vcs01.lab.local

2017-11-21 12:50:12,521 INFO 192.168.12.41 - - [21/Nov/2017 12:50:12] "POST /endpoint/test HTTP/1.1" 200 -

In your custom handler, you may want to do wrap the entire logic in a for loop of the "Messages" object and use my method to find the hostname for each message. Then for each message, fire off an upstream HTTP/S method for each message. Whether you need to do that is up to you, however.