VMware Cloud Community
HywelBurris
Enthusiast
Enthusiast

LI Agent Log Formatting

Currently looking at a design whereby the LI agent is being used to forward logs to both LI and a SIEM. The issue the SIEM integrator is having, is the format of the logs compared to some other syslog agents which enable easier parsing.

Example from LI Agent

{"@timestamp":"2016-03-23T13:25:30.959Z",

"message":"<38>1 2016-03-23T13:24:21.095731Z aaa-xxx-ss-test.XXX.XXX Microsoft-Windows-Security-Auditing - 4688 [liagent@6876 eventrecordid=\"50747\" keywords=\"Audit Success\" opcode=\"Info\" tas

k=\"Process Creation\"] A new process has been created.\r\n\r\nSubject:\r\n\tSecurity ID:\t\tS-1-5-18\r\n\tAccount Name:\t\tXXX-XXX-SS-TEST$\r\n\tAccount Domain:\t\tXXXX\r\n\tLogon ID:\t\t0x3E7\r\n\r\nProcess Information:\r\n\tNew Proc

ess ID:\t\t0x11bc\r\n\tNew Process Name:\tC:\\Program Files (x86)\\McAfee\\VirusScan Enterprise\\x64\\Scan64.Exe\r\n\tToken Elevation Type:\tTokenElevationTypeDefault (1)\r\n\tCreator Process ID:\t0xb24\r\n\tProcess Command Line:\t\r\n\r

\nToken Elevation Type indicates the type of token that was assigned to the new process in accordance with User Account Control policy.\r\n\r\nType 1 is a full token with no privileges removed or groups disabled.  A full token is only us

ed if User Account Control is disabled or if the user is the built-in Administrator account or a service account.\r\n\r\nType 2 is an elevated token with no privileges removed or groups disabled.  An elevated token is used when User Acco

unt Control is enabled and the user chooses to start the program using Run as administrator.  An elevated token is also used when an application is configured to always require administrative privilege or to always require maximum privil

ege, and the user is a member of the Administrators group.\r\n\r\nType 3 is a limited token with administrative privileges removed and administrative groups disabled.  The limited token is used when User Account Control is enabled, the a

pplication does not require administrative privilege, and the user does not choose to start the program using Run as administrator.",

"@version":"1",

"tags":["multiline"],

"host":"xx.xx.xx.xx",

"port":13681,

"type":"loginsight",

"logstash_checksum":"aad474d7b793eb5fa3ff0b30e0c72950142d206cffbb4d16bf6972a3dc444767"}

Example format from NXLog:

{

  "_index": "XXX,

  "_type": "winevt",

  "_id": "AVPLvPVAhiObdUc_LyEB",

  "_score": null,

  "_source": {

    "EventTime": "2016-03-31 09:14:26",

    "Hostname": "XXXX",

    "Keywords": -9214364837600035000,

    "EventType": "AUDIT_SUCCESS",

    "SeverityValue": 2,

    "Severity": "INFO",

    "EventID": 4688,

    "SourceName": "Microsoft-Windows-Security-Auditing",

    "ProviderGuid": "{54849625-5478-4994-A5BA-3E3B0328C30D}",

    "Version": 1,

    "Task": 13312,

    "OpcodeValue": 0,

    "RecordNumber": 3913469,

    "ProcessID": 4,

    "ThreadID": 24120,

    "Channel": "Security",

    "Message": "A new process has been created.\r\n\r\nSubject:\r\n\tSecurity ID:\t\tS-1-5-21-286448784-2901391547-1905694321-1188\r\n\tAccount Name:\t\ttest\r\n\tAccount Domain:\t\ttest\r\n\tLogon ID:\t\t0x288C8A3\r\n\r\nProcess Information:\r\n\tNew Process ID:\t\t0x4424\r\n\tNew Process Name:\tC:\\Program Files (x86)\\Java\\jre1.8.0_51\\bin\\jp2launcher.exe\r\n\tToken Elevation Type:\tTokenElevationTypeLimited (3)\r\n\tCreator Process ID:\t0x1944\r\n\tProcess Command Line:\t\"C:\\Program Files (x86)\\Java\\jre1.8.0_51\\bin\\jp2launcher.exe\" -secure -javaws -jre \"C:\\Program Files (x86)\\Java\\jre1.8.0_51\" -vma LWNsYXNzcGF0aABDOlxQcm9ncmFtIEZpbGVzICh4ODYpXEphdmFcanJlMS44LjBfNTFcbGliXGRlcGxveS5qYXIALURqYXZhLnNlY3VyaXR5LnBvbGljeT1maWxlOkM6XFByb2dyYW0gRmlsZXMgKHg4NilcSmF2YVxqcmUxLjguMF81MVxsaWJcc2VjdXJpdHlcamF2YXdzLnBvbGljeQAtRHRydXN0UHJveHk9dHJ1ZQAtWHZlcmlmeTpyZW1vdGUALURqbmxweC5ob21lPUM6XFByb2dyYW0gRmlsZXMgKHg4NilcSmF2YVxqcmUxLjguMF81MVxiaW4ALURqYXZhLnNlY3VyaXR5Lm1hbmFnZXIALURzdW4uYXd0Lndhcm11cD10cnVlAC1YYm9vdGNsYXNzcGF0aC9hOkM6XFByb2dyYW0gRmlsZXMgKHg4NilcSmF2YVxqcmUxLjguMF81MVxsaWJcamF2YXdzLmphcjtDOlxQcm9ncmFtIEZpbGVzICh4ODYpXEphdmFcanJlMS44LjBfNTFcbGliXGRlcGxveS5qYXI7QzpcUHJvZ3JhbSBGaWxlcyAoeDg2KVxKYXZhXGpyZTEuOC4wXzUxXGxpYlxwbHVnaW4uamFyAC1Eam5scHguanZtPUM6XFByb2dyYW0gRmlsZXMgKHg4NilcSmF2YVxqcmUxLjguMF81MVxiaW5camF2YXcuZXhl -ma LVNTVkJhc2VsaW5lVXBkYXRlAC1ub3RXZWJKYXZh\r\n\r\nToken Elevation Type indicates the type of token that was assigned to the new process in accordance with User Account Control policy.\r\n\r\nType 1 is a full token with no privileges removed or groups disabled.  A full token is only used if User Account Control is disabled or if the user is the built-in Administrator account or a service account.\r\n\r\nType 2 is an elevated token with no privileges removed or groups disabled.  An elevated token is used when User Account Control is enabled and the user chooses to start the program using Run as administrator.  An elevated token is also used when an application is configured to always require administrative privilege or to always require maximum privilege, and the user is a member of the Administrators group.\r\n\r\nType 3 is a limited token with administrative privileges removed and administrative groups disabled.  The limited token is used when User Account Control is enabled, the application does not require administrative privilege, and the user does not choose to start the program using Run as administrator.",

    "Category": "Process Creation",

    "Opcode": "Info",

    "SubjectUserSid": "S-1-5-21-286448784-2901391547-1905694321-1188",

    "SubjectUserName": "test",

    "SubjectDomainName": "test",

    "SubjectLogonId": "0x288c8a3",

    "NewProcessId": "0x4424",

    "NewProcessName": "C:\\Program Files (x86)\\Java\\jre1.8.0_51\\bin\\jp2launcher.exe",

    "TokenElevationType": "%%1938",

    "CommandLine": "\"C:\\Program Files (x86)\\Java\\jre1.8.0_51\\bin\\jp2launcher.exe\" -secure -javaws -jre \"C:\\Program Files (x86)\\Java\\jre1.8.0_51\" -vma LWNsYXNzcGF0aABDOlxQcm9ncmFtIEZpbGVzICh4ODYpXEphdmFcanJlMS44LjBfNTFcbGliXGRlcGxveS5qYXIALURqYXZhLnNlY3VyaXR5LnBvbGljeT1maWxlOkM6XFByb2dyYW0gRmlsZXMgKHg4NilcSmF2YVxqcmUxLjguMF81MVxsaWJcc2VjdXJpdHlcamF2YXdzLnBvbGljeQAtRHRydXN0UHJveHk9dHJ1ZQAtWHZlcmlmeTpyZW1vdGUALURqbmxweC5ob21lPUM6XFByb2dyYW0gRmlsZXMgKHg4NilcSmF2YVxqcmUxLjguMF81MVxiaW4ALURqYXZhLnNlY3VyaXR5Lm1hbmFnZXIALURzdW4uYXd0Lndhcm11cD10cnVlAC1YYm9vdGNsYXNzcGF0aC9hOkM6XFByb2dyYW0gRmlsZXMgKHg4NilcSmF2YVxqcmUxLjguMF81MVxsaWJcamF2YXdzLmphcjtDOlxQcm9ncmFtIEZpbGVzICh4ODYpXEphdmFcanJlMS44LjBfNTFcbGliXGRlcGxveS5qYXI7QzpcUHJvZ3JhbSBGaWxlcyAoeDg2KVxKYXZhXGpyZTEuOC4wXzUxXGxpYlxwbHVnaW4uamFyAC1Eam5scHguanZtPUM6XFByb2dyYW0gRmlsZXMgKHg4NilcSmF2YVxqcmUxLjguMF81MVxiaW5camF2YXcuZXhl -ma LVNTVkJhc2VsaW5lVXBkYXRlAC1ub3RXZWJKYXZh",

    "EventReceivedTime": "2016-03-31 09:14:26",

    "SourceModuleName": "eventlog",

    "SourceModuleType": "im_msvistalog",

    "@version": "1",

    "@timestamp": "2016-03-31T08:14:31.377Z",

    "host": "10.X.X.X:1063",

    "type": "winevt"

  },

As you can see from the two examples the LI and (in this case) nxlog have very different outputs for windows events.

Can the LI agent pre-parse and add the fields ready for use?

Thanks

Labels (1)
Reply
0 Kudos
4 Replies
MichaelRyom
Hot Shot
Hot Shot

It does parsing of logfiles, but im not sure about windows events.

This seems to surgest it can:

"agentClasses":[ {

    "name":"Windows Firewall Advanced",

    "info":"",

    "agentConfig":"[winlog|WindowsFirewall]\nchannel=Microsoft-Windows-Windows Firewall With Advanced Security/Firewall\n\n[filelog|win2012_Firewall]\ndirectory=C:\\Windows\\system32\\logfiles\\firewall\\\ninclude=*.log\nevent_marker=^\\d{4}-\\d{2}-\\d{2}\ntags={\"ms_product\":\"firewall\"}\n"

  }

Blogging at https://MichaelRyom.dk
Reply
0 Kudos
admin
Immortal
Immortal

Yes it will parse the events. I assume you took the agent config from the Windows content pack? Are you seeing any issues? If yes, what does the agent log say? I am not sure I fully understand what the issue is.

Thanks.

Reply
0 Kudos
HywelBurris
Enthusiast
Enthusiast

Sorry this may be a misunderstanding from my point of view. We've tested this without a Log Insight Server and associated windows content pack.

So when you install the windows content pack this sends the information to the agent for what to parse and and format accordingly? So without the content pack it would be sent un-parsed?

Thanks

Reply
0 Kudos
admin
Immortal
Immortal

All events will be ingested unparsed (or rather parsed with defaults) if there it no parser config specified. When you install a content pack you have to explicitly apply the agent config in the content pack to your agent to get the agent to parse messages as per the agent config. Hope this helps.

Reply
0 Kudos