Lambdas: extension to Rich Attributes that allow transformation of objects and to render arbitrary widgets

[psrok1]

Your checklist for this pull request

• I've read the contributing guideline.
• I've tested my changes by building and running the project, and testing changed functionality (if applicable)
  - [ ] I've added automated tests for my change (if applicable, optional)
• I've updated documentation to reflect my change (if applicable)

This is next rework of great idea from #951 and its extension from #955. It introduces a concept of transformers (called using pipeline syntax) that modify objects and renderers that allow to put any object we want within a render.

This time I was able to render a very complex view with collapsible lists, sections, custom icons and so on.

Technical details

Marked&Mustache rendering looks as follows.

{object} => Mustache => [Markdown string] => Markdown => <rendered React object>

Transformers are simple, it's just a function that gets an object as an argument and returns a modified object. To make it possible, I introduced pipeline syntax

{{value|sort|entries}}

which is equivalent to entries(sort(view["value"])).

More complicated thing is renderer. It is usually called using a section syntax e.g.

{{#indicator}}This is important message{{/indicator}}

If lambda returns an object that is not a simple string (e.g. rendered component), it is stored in MustacheContext with an unique id e.g. lambda_result417. Then proper reference is put in Markdown string [](lambda#lambda_result417). Finally, Markdown renderer retrieves object from MustacheContext and puts it in rendered React object tree.

Rendering process can be nested and lambda may subrender section part of template into React. It may also run only the Mustache renderer and only the context-related Markdown renderer if needed.

Finally, lambdas can be parametrized using previously rendered values in special sections.

{{#indicator.type}}danger{{/indicator.type}}
{{#indicator}}This is important message{{/indicator}}

indicator.type is a lambda that renders subtemplate (danger) using only the Mustache renderer and preserve the resulting string in indicatorType key in MustacheContext.lambdaContext. Then, this value is retrieved by indicator lambda to use proper icon. I used this pattern to implement if-clause, groupBy and other parametrized lambdas. This means we are not only able to call functions but also to parametrize them with values from the object.

Example template code and attribute value

{{#value}}
{{#section.header}}PE indicators{{/section.header}}
{{#section}}
{{#signed}}
{{#indicator}}PE file is **signed**.{{/indicator}}
{{/signed}}
{{#signature_verification}}
{{#indicator.type}}success{{/indicator.type}}
{{#indicator}}Signature looks valid!{{/indicator}}

{{/signature_verification}}
{{/section}}
{{#section.header}}Basic PE information{{/section.header}}
{{#section}}

|Property|Value|
|--------|-----|
{{#pdb}}
|**PDB name:**|{{.}}|
{{/pdb}}
{{#dll_name}}
|**DLL name:**|{{.}}|
{{/dll_name}}
{{#compilation_timestamp}}
|**Compilation timestamp:**|{{.}}|
{{/compilation_timestamp}}
{{/section}}

{{#if}}{{sections}}{{/if}}
{{#then}}
{{#section.header}}PE sections{{/section.header}}
{{#section}}

|Name|Offset|VA|Virtual Size|Raw Size|Entropy|MD5|Characteristics|
|-|-|-|-|-|-|-|-|
{{#sections}}
|{{name}}|{{offset}}|{{virtual_address}}|{{virtual_size}}|{{sizeof_raw_data}}|{{entropy}}|{{md5}}|{{charactersitics_list}}|
{{/sections}}

{{/section}}
{{/then}}

{{#if}}{{imports}}{{/if}}
{{#then}}
{{#section.header}}PE imports{{/section.header}}
{{#section}}
{{#group.by}}dll{{/group.by}}
{{#imports|group|entries}}
{{#collapse.header}}**{{key}}** ({{value|count}} methods){{/collapse.header}}
{{#collapse}}
{{#value}}
- {{name}}
{{/value}}
{{/collapse}}
{{/imports|group|entries}}
{{/section}}
{{/then}}

{{/value}}

Attribute value:

{
    "imports": [
        {"dll": "kernel32.dll", "name": "GetModuleHandleA"},
        {"dll": "kernel32.dll", "name": "GetModuleHandleW"},
        {"dll": "kernel32.dll", "name": "GetProcAddress"},
        {"dll": "kernel32.dll", "name": "GetFileAttributesW"},
        {"dll": "kernel32.dll", "name": "GetVersion"},
        {"dll": "kernel32.dll", "name": "LoadLibraryW"},
        {"dll": "kernel32.dll", "name": "GetLastError"},
        {"dll": "kernel32.dll", "name": "SetLastError"},
        {"dll": "kernel32.dll", "name": "GetModuleFileNameW"},
        {"dll": "kernel32.dll", "name": "OutputDebugStringA"},
        {"dll": "kernel32.dll", "name": "LoadLibraryExW"},
        {"dll": "kernel32.dll", "name": "FreeLibrary"},
        {"dll": "kernel32.dll", "name": "GetSystemDirectoryW"},
        {"dll": "kernel32.dll", "name": "IsDebuggerPresent"},
        {"dll": "kernel32.dll", "name": "SetUnhandledExceptionFilter"},
        {"dll": "kernel32.dll", "name": "UnhandledExceptionFilter"},
        {"dll": "kernel32.dll", "name": "GetCurrentProcess"},
        {"dll": "kernel32.dll", "name": "TerminateProcess"},
        {"dll": "kernel32.dll", "name": "GetStartupInfoA"},
        {"dll": "kernel32.dll", "name": "InterlockedCompareExchange"},
        {"dll": "kernel32.dll", "name": "Sleep"},
        {"dll": "kernel32.dll", "name": "InterlockedExchange"},
        {"dll": "kernel32.dll", "name": "GetProcessHeap"},
        {"dll": "kernel32.dll", "name": "GetSystemTimeAsFileTime"},
        {"dll": "kernel32.dll", "name": "GetCurrentProcessId"},
        {"dll": "kernel32.dll", "name": "GetCurrentThreadId"},
        {"dll": "kernel32.dll", "name": "GetTickCount"},
        {"dll": "kernel32.dll", "name": "QueryPerformanceCounter"},
        {"dll": "kernel32.dll", "name": "VirtualProtect"},
        {"dll": "msvcr90.dll", "name": "_crt_debugger_hook"},
        {"dll": "msvcr90.dll", "name": "?terminate@@YAXXZ"},
        {"dll": "msvcr90.dll", "name": "_unlock"},
        {"dll": "msvcr90.dll", "name": "__dllonexit"},
        {"dll": "msvcr90.dll", "name": "_lock"},
        {"dll": "msvcr90.dll", "name": "_onexit"},
        {"dll": "msvcr90.dll", "name": "_decode_pointer"},
        {"dll": "msvcr90.dll", "name": "_invoke_watson"},
        {"dll": "msvcr90.dll", "name": "_controlfp_s"},
        {"dll": "msvcr90.dll", "name": "_except_handler4_common"},
        {"dll": "msvcr90.dll", "name": "_encode_pointer"},
        {"dll": "msvcr90.dll", "name": "__p__fmode"},
        {"dll": "msvcr90.dll", "name": "__p__commode"},
        {"dll": "msvcr90.dll", "name": "_adjust_fdiv"},
        {"dll": "msvcr90.dll", "name": "__setusermatherr"},
        {"dll": "msvcr90.dll", "name": "_configthreadlocale"},
        {"dll": "msvcr90.dll", "name": "_initterm_e"},
        {"dll": "msvcr90.dll", "name": "_initterm"},
        {"dll": "msvcr90.dll", "name": "_acmdln"},
        {"dll": "msvcr90.dll", "name": "exit"},
        {"dll": "msvcr90.dll", "name": "_ismbblead"},
        {"dll": "msvcr90.dll", "name": "_XcptFilter"},
        {"dll": "msvcr90.dll", "name": "_exit"},
        {"dll": "msvcr90.dll", "name": "_cexit"},
        {"dll": "msvcr90.dll", "name": "__getmainargs"},
        {"dll": "msvcr90.dll", "name": "_amsg_exit"},
        {"dll": "msvcr90.dll", "name": "wcsncat_s"},
        {"dll": "msvcr90.dll", "name": "__set_app_type"}
    ],
    "imports_count": 57,
    "dll_name": "winword.exe",
    "signed": true,
    "signature_verification": true,
    "pdb": "t:\\word\\x86\\ship\\0\\winword.pdb",
    "compilation_timestamp": "2010-03-27T15:35:19"
}

[yankovs]

Looks great.

Something that might be useful to test is sort of "lazy load" for large tables/lists.

What I mean by that is that sometimes in rich attributes we display for example a table, but the payload is unknown to us and dynamic (for example, PE imports like in your example). And sometimes, the payload itself is really big, so it makes the object view huge and takes a lot of space in the page.

Would be cool if builtInLambdas had an option to paginate the data like VT does for resources as an example:

This probably means keeping a pageNum state in the collapse built in lambda but I'm not sure how the syntax will look in the template. Maybe something like {{#collapse|paginated}}?

[PBia]

Suggestions for minor language changes and some formatting in the docs.

[psrok1]

Something that might be useful to test is sort of "lazy load" for large tables/lists.

What I mean by that is that sometimes in rich attributes we display for example a table, but the payload is unknown to us and dynamic (for example, PE imports like in your example). And sometimes, the payload itself is really big, so it makes the object view huge and takes a lot of space in the page.

This probably means keeping a pageNum state in the collapse built in lambda but I'm not sure how the syntax will look in the template. Maybe something like {{#collapse|paginated}}?

@yankovs: Yeah, it would be something that is both a transformer and renderer 🤔

{{#huge_array|paginated}}
<template that shows how to render single item>
{{/huge_array|paginated}}

There is no lazy loading in typical case because all attributes are eagerly loaded by MWDB. But items itself may contain widgets that are implemented by plugin and lazy-load some additional data.

[psrok1]

{{#value}}
{{#paginated.header}}
|SHA256|Type|Language|File type|Entropy|
|------|----|--------|---------|-------|
{{/paginated.header}}
{{#resources|paginated}}
|{{sha256}}|{{type}}|{{language}}|{{file_type}}|{{entropy}}|
{{/resources|paginated}}
{{/value}}

Well.. I have done some pagination. I see these tables are not the most beautiful thing 😆

I have also not tested whether components within rows are preserving their state after clicking "More", but it should work.

Solution is a bit hacky because lambda emits React objects, so we need to build Markdown from parts and call Markdown renderer ourselves.

[psrok1]

Ok, I'm going to merge it and give it a try.