new feature: JSON output schema

[profnandaa]

Preamble

The aim is to come up with a standard JSON output schema as the default output for the LogMonitor tool v2 (due to the breaking changes). This will make it easier for log analysis tools. In Azure Monitor for instance, you can use functions like parse_json the LogEntry column and extract specific data points that you can analyze on.

In this document, we would like to propose a top-level schema for the log entries. This is top-level because, in the inner JSON entries, especially for ETW logs, the (custom) fields will vary between different ETW providers.

Will be also good to note that we will be using NDJSON (Newline Delimited JSON) for the log entry since the logs are streams that should be processed one record at a time.

Schema

Here is our proposed schema for JSON outputs for the various log sources:

{
	"Source": "File|Process|ETW|EventLog",
	"LogEntry": { 
			// diffent per source
	},
        "SchemaVersion": ""
}

// LogEntry - File
{
	"Logline": "...",
	"FileName": "..."
}

// LogEntry - Process
{
	"Logline": "...",
        "ProcessName": "...",
        "ProcessId": xxx
}

// LogEntry - EventLog
{
	"Time": "",
	"Channel": "",
	"Level": "Error|Information",
	"EventId": "",
	"Message": ""
}

// LogEntry - ETW
{
	"Time": "",
	"ProviderId": "",
	"ProviderName": "",
	"Execution": {
		"ProcessId": <int>,
		"ThreadId": <int>
	},
	"Level": "Error|Information",
	"Keyword": "",
	"EventId":  <int>,
	"EventData": { 
			// varies depending on provider
	}
}

Examples

EventLog

{
    "Source": "EventLog",
    "LogEntry": {
        "Time": "2022-11-30T16:39:18.000Z",
        "Level": "Error",
        "EventId": 701,
        "Message": "Task Scheduler service failed to start Task Compatibility module. Tasks may not be able to register on previous Window versions. Additional Data: Error Value: 2147942526."
    },
    "SchemaVersion": "1.0.0"
}

ETW

Example 1:

{
    "Source": "ETW",
    "LogEntry": {
        "Time": "2022-11-30T09:40:22.000Z",
        "ProviderId": "{7E8AD27F-B271-4EA2-A783-A47BDE29143B}",
        "ProviderName": "Microsoft-Windows-IIS-Logging",
        "DecodingSource": "DecodingSourceXMLFile",
        "Execution": {
            "ProcessId": 8300,
            "ThreadId": 4440
        },
        "Level": "Information",
        "Keyword": "0x8000000000000000",
        "EventId": 6200,
        "EventData": {
            "EnabledFieldsFlags": 2478079,
            "date": "2022-11-30",
            "time": "09:40:27",
            "c-ip": "10.224.0.4",
            "cs-username": "-",
            "s-sitename": "W3SVC1",
            "cs-computername": "iislogmonitor-847fbf7766-rqr4b",
            "s-ip": "10.224.0.171",
            "cs-method": "GET",
            "cs-uri-stem": "/.env",
            "cs-uri-query": "-",
            "sc-status": 404,
            "sc-win32-status": 2,
            "sc-bytes": 1383,
            "sc-bytes": 232,
            "time-taken": 89,
            "s-port": 80,
            "csUser-Agent": "Mozilla/5.0+(X11 +Linux+x86_64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/81.0.4044.129+Safari/537.36",
            "csCookie": "-",
            "csReferer": "-",
            "cs-version": "-",
            "cs-host": "-",
            "sc-substatus": 0,
            "CustomFields": "null"
        }
    },
    "SchemaVersion": "1.0.0"
}

Example 2:

{
    "Source": "ETW",
    "LogEntry": {
        "Time": "2022-12-06T13:40:28.000Z",
        "ProviderId": "{DAA6A96B-F3E7-4D4D-A0D6-31A350E6A445}",
        "ProviderName": "Microsoft-Windows-WLAN-Driver",
        "DecodingSource": "DecodingSourceXMLFile",
        "Execution": {
            "ProcessId": 13804,
            "ThreadId": 13900
        },
        "Level": "Information",
        "Keyword": "0x8000000000000001",
        "EventId": 0,
        "EventData": {
            "FrameUniqueID": 5434658,
            "PortNumber": 0,
            "TID": 0,
            "PeerID": 0,
            "PayloadLength": 1286,
            "QueueLength": 0,
            "QueueState": "false",
            "CustomData1": 24,
            "CustomData2": 0,
            "CustomData3": 0
        }
    },
    "SchemaVersion": "1.0.0"
}

NOTE: EventData will differ in naming convention depending on the provider. It could be a little difficult to try standardize the convention e.g. sc-status to scStatus or ScStatus as it's easy to miss some other cases that we haven't met yet. Perhaps it will be easy to just leave the naming as is, provided that they are valid key names for JSON (since they are already valid XML tags).
Also, for the items that are numbers, we will present them as numbers to make analysis better. e.g. | where d.port < 450

File Log

{
    "Source": "File",
    "LogEntry": {
        "Logline": "2022-12-01 23:14:16 10.224.0.163 HEAD /memberadmin.cfm - 80 - 10.224.0.149 Mozilla/5.0+(compatible;+Nmap+Scripting+Engine;+https://nmap.org/book/nse.html) - 404 0 2 95",
    },
    "SchemaVersion": "1.0.0"
}

Process Log

{
    "Source": "Process",
    "LogEntry": {
        "Logline": "Name: C:\\ProgramData\\ssh",
        "ProcessExe": "C:\\LogMonitor\\log_writer.exe"
    },
    "SchemaVersion": "1.0.0"
}

[profnandaa]

/cc. @mloskot

[iankingori]

For File and Process can we include FileName or ProcessName in addition to the Logline. This is relevant for File since we currently support monitoring multiple files at the same time, I imagine may also be relevant for process should we ever go with multiple child process support

[profnandaa]

Sure, sounds good. Will combine that with @mloskot suggestion, please check the update in a few.

[profnandaa]

BTW for FileName, I'm wondering if it will be prudent to provide the full path e.g. C:\inetpub\logs\LogFiles\W3SVC1\u_ex221201_x.log for the case of a sample IIS log file...

[iankingori]

I don't have any strong opinion here, we can go Log Directory and FileName or just FilePath. I'm guessing the first one allows for easier/richer filtering but would love to hear suggestions

[profnandaa]

Went with displaying the whole path relative to what was set in the config file, e.g. C:\inetpub\logs -> LogFiles\W3SVC1\u_ex221201_x.log;

[mloskot]

Overall the idea looks good to me.
I agree that standardising the EventData may be impractical as it is a bag of source/event/entry-specific details.

For "Source": "File", wouldn't it be sensible to include location of actual log file?
For "Source": "Process", similary, a process ID and/or executable/image location?

[profnandaa]

For "Source": "File", wouldn't it be sensible to include location of actual log file?

Good catch, I'd forgot that one. I think it doesn't hurt to provide that by default. Let me edit.

For "Source": "Process", similary, a process ID and/or executable/image location?

This sounds reasonable too. Executable location is fair enough.

[cdenneen]

Any thoughts on making it ECS compatible? https://www.elastic.co/guide/en/ecs/current/index.html

[profnandaa]

Good suggestion! I think we can handle this as separate request; start off with this as the base format and have transformers/adapters for the other major formats like ECS. How does that sound?

[mloskot]

That sounds like a flexible and extensible implementation, much desired indeed.

[profnandaa]

Just posting here for reference. Here is a sample ECS logline from Elastic Winlogbeat:

{
    "log.level": "warn",
    "@timestamp": "2022-12-07T14:49:35.198Z",
    "log.origin": {
        "file.name": "eventlog/wineventlog.go",
        "file.line": 335
    },
    "message": "WinEventLog[Application] error salvaging message (event id=0 qualifier=0 provider=\"WindowsAzureGuestAgent\" created at 2022-12-07 13:12:33.9856488 +0000 UTC will be included without a message): failed in EvtFormatMessage: The message resource is present but the message was not found in the message table.",
    "service.name": "winlogbeat",
    "ecs.version": "1.6.0"
}

[profnandaa]

The WIP implementation is here, pending some e2e testing, reviews and a few TODOs - #110

[profnandaa]

UPDATE: all the TODOs done now, except one minor one to include the process name and pid on the Process logs. Free to try it out + review and give feedback -> #110

[profnandaa]

UPDATE: PR now ready for final review for the v2.0.0-alpha release (TBD).

[CharityKathure]

Closing this discussion. feature is now available here