JSON from nprobe into Log Insight API?

[pboguszewski]

Hi all,

     I would like to ingest data from our netflow data collected via nprobe directly into the Log Insight API.  I am currently writing the output to disk and using the liagent to read the files.  My biggest issue with this is the 10 second burst of data as it writes to disk not lining up with application and syslog events.  One can currently directly write to Splunk via the API and would like the flexibility to do that direct to Log Insight.  I wrote to the nprobe / ntop list asking if it is possible to export the flow json differently but not getting any responses.  Thanks for any input.

Pete

[sflanders]

Writing to Log Insight ingestion API is fully supported today. For more information see: About vRealize Log Insight Developer's Guide.

[pboguszewski]

What I am looking for is a free form non-POST to web API option.  I have non-conforming JSON coming from nprobe that I want to directly stream into an open port, ingest it, and write custom dashboards with the data. I have access via liagent to the locally dumped text data but it does not support adding labels, therefore field extraction gets complex and slow because I have to build fairly complex regex statements to ensure the proper fields are associated with the columns.

[sflanders]

Still not sure I am entirely following. In regards to labels are you referring to the agent? It supports tags, which should be the same thing. If you are interested in better parsing options you might want to look on loginsight.vmware.com and the tech preview program. As for the ingestion API, it does not care the format of your event you just need to send it via the REST API.

[pboguszewski]

I apologize, I am trying to be vague to be helpful in the 'bigger picture' but not getting my specific example down.  Broadly, I am asking for acceptance of constant flowing stream(s) of JSON or other structured data to a TCP port.  Here is an example of part of a stream nprobe sends to a tcp port (Splunk can read):

{"IPV4_SRC_ADDR":"xxx.xxx.xxx.xxx","L4_SRC_PORT":7000,"IPV4_DST_ADDR":"xxx.xxx.xxx.xxx","L4_DST_PORT":37523,"PROTOCOL":6,"IN_BYTES":542100,"OUT_BYTES":1557981,"FIRST_SWITCHED":1431101095,"LAST_SWITCHED":1431101236,"IN_PKTS":10425,"OUT_PKTS":13453,"IP_PROTOCOL_VERSION":4,"APPLICATION_ID":"0","L7_PROTO_NAME":"Unknown","ICMP_TYPE":0,"FLOW_START_MILLISECONDS":1431101095000,"FLOW_END_MILLISECONDS":1431101236000,"APPL_LATENCY_MS":0.000,"SRC_IP_COUNTRY":"","SRC_IP_CITY":"","DST_IP_COUNTRY":"","DST_IP_CITY":"","TOTAL_FLOWS_EXP":64890}

I do not have the option to post this data to a web page, just stream to an IP:Port.

Here is the nprobe help line:

      --tcp <server:port>                 | Deliver flows in JSON format to the specified server via TCP.

I can imagine other instances this would be helpful - ingesting data from a tcp port outside of syslog or Restful API posts.  Please let me know if you would like more information - I know I am so immersed in the issue I miss the bigger picture and have a difficult time explaining the issue at hand.

Thanks!

Pete

[pboguszewski]

I have another API issue - I am attempting to stream data via ntopng to the API and getting further.  This is what my packet capture tells me (just small sample):

POST /api/v1/messages/ingest/42030d8f-a682-71da-8206-cadaf18b370e HTTP/1.1

Host: 10.128.234.14:9000

Connection: keep-alive

Accept: application/json

User-Agent: li-agent-2.5.0.2347850 Beta

Content-Type: application/json; charset=utf-8

Content-Encoding: gzip

Content-Length: 338

^_<8b>^CU<91 <85>ÿK<9f>  R<95><89>¼±<82>Y3 ­^Z²<98><86>        N^R^D#]ܲòßW`^Z÷öÝsNNso^?À©hÛ

ì£h<81>ûö^CÊ^\¸3ÇY:3Û^@<87>²¨ò 82>fò^H^L°ojYÔR<8b><93><<93>ÙäP5×v2µ mÙ<96>#à^S´!4^G^UtƽáØ´rÀÇ<86>Z6g³*ß/Ùå

Û¼<96>íÞ,òOÐí^L <8b>¯><80>éf.X<82><84>çû<89>

G¦$ájp|ÆïNÏ<83>C^S     "¡Â±xNyÀ^TY ÇØ^Vsô^Rø*ô^^'<9d>§¯|<8c>^O<80>©¸<95><89>M<90>0LbåQ^ZbäqÍ^Bë<86>Å^X^Q±^W^E

£<88>

<9e>Ò@­B²^U<8c>{        ^W^Q^NCÌ^BDb<9f><8d>z^PûÿÕ¾T<84>^^^Ob<94><8a><88>©~Ký8"ë<98>'é}Ä<Uý<9a>^OÖmÔ<96>¾¬,õ§Êìt^F.<9

c>Ï <84>ËÅ^TN<97>V·ë~^Ar<8d>^S&ð^A^@^@HTTP/1.1 200 OK

ACCESS-CONTROL-EXPOSE-HEADERS: X-LI-Build

Content-Type: application/json; charset=utf-8

Date: Fri, 08 May 2015 21:15:21 GMT

X-LI-Build: 2347850

Content-Length: 58

{"status":"ok","message":"messages ingested","ingested":1}POST /api/v1/messages/ingest/42030d8f-a682-71da-8206-cadaf1

8b370e HTTP/1.1

Host: 10.128.234.14:9000

Connection: keep-alive

Accept: application/json

User-Agent: li-agent-2.5.0.2347850 Beta

Content-Type: application/json; charset=utf-8

Content-Encoding: gzip

Content-Length: 3875

Expect: 100-continue

HTTP/1.1 100 Continue

Which appears to be ingesting data but I do not see it in interactive analytics.  Not sure what is going on but I am pretty sure it is me.  I need to walk away for a bit and look at this tomorrow or Monday.  As always, any insight or ideas are appreciated!

[sflanders]

Ah, now I understand the stream request -- can you please open this as a feature request on loginsight.vmware.com?