Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Plugins Support with WebAssembly #2056

Closed
wants to merge 128 commits into from
Closed

Add Plugins Support with WebAssembly #2056

wants to merge 128 commits into from

Conversation

AmmarAbouZor
Copy link
Member

@AmmarAbouZor AmmarAbouZor commented Jul 18, 2024 •
edited
Loading

This PR still works in progress...

It will provide a plugin system for Chipmunk using WASM with wasmtime runtime and the component model definitions.

The Communication API between Chipmunk and the plugins is defined via WIT files, and the plugins could be written basically in any language which supports compiling to web assembly and the component model proposal.

For Plugins written in Rust we will provide a crate which will take care of generating the bindings form the wit files and provides all the needed functionalities for development (like logging).

The plugin system will offer writing plugins in the following areas:

  • Parser: The parser will be given a buffer of bytes that needed to be parsed to the data types defined in chipmunk to make this data useful for analyzing.
  • Byte Source: This part is responsible of providing data to chipmunk as array of bytes, then the app can parse and analyze this data.
  • Provider: This part is responsible of both getting the data from the given configurations, parse them and then provide Chipmunk with parsing results then it can analyze them. (This is postponed for now)

Current State:

Host:

  • A single Wasmtime engine will manage all plugins.
  • The host loads plugins, validates them, retrieves the API version from plugin metadata, and selects the matching plugin host version.
  • The PluginsParser struct on the host implements the Parser trait, abstracting plugin communication complexity.
  • The PluginsByteSource struct implements the standard library's Read trait and is passed to BinaryByteSource to provide data as bytes.
  • The ParserType enum now includes plugin types with configuration settings. These configurations specify the plugin path, general settings for all parser plugins, and the path to the plugin's custom configuration file.
  • Plugins are not yet integrated into the UI and are temporarily enabled via environment variables.

Plugins:

  • The general plugin API is defined via .wat files located in the plugins-api stand-alone crate.
  • Rust plugin authors don't need to generate plugins from .wat; instead, they can use the plugins-api library, implement required traits, and export them with the provided macro.
  • There are currently two plugin types: Parser and ByteSource, with the primary focus on the parser plugin.
  • The parser plugin must implement two methods: one for initialization and another for parsing bytes. Additional methods are needed for delimiter handling, header management, and rendering custom options, as the current configuration via a file isn't user-friendly.
  • The ByteSource plugin requires two methods: initialization and providing byte chunks of a given length. Its configuration is also limited to a file and needs a more user-friendly approach.
  • Examples of parser and ByteSource plugins are available in the Chipmunk repository.

Next steps:

  • Define types and methods in the parser plugins API to allow each plugin to provide its delimiters and headers, if applicable.
  • Define types and methods for plugins to handle their required configurations, enabling rendering in the UI and passing them back to the plugin instead of relying on custom configuration files.
  • Extend the Parser trait in Core to support custom delimiters, headers, and configurations, and deliver these values to the UI for rendering. (Consider making this change universal for all parsers in a separate PR.)
  • Create templates for plugins using the Build CLI tool.
  • Set up an additional repository for approved plugins and implement functionality in Chipmunk to fetch and download these plugins directly.

@AmmarAbouZor AmmarAbouZor force-pushed the plugins_support branch 2 times, most recently from 6e1387b to 1edaec3 Compare August 1, 2024 10:00
@AmmarAbouZor AmmarAbouZor force-pushed the plugins_support branch 5 times, most recently from 8fcc42b to 9941076 Compare December 5, 2024 11:23
@AmmarAbouZor
Plugins System: Multiple squashed commits
ce31fe2 
*** These commits has been squashed for rebase ***

*** Changes while rebase included too ***

********************************************************

Change parse function return type

Parse function returns how many bytes have been consumed instead the
rust of the slice to eliminate the reference and the lifetime

Parse returns vector of results (in progress)

- Change parse signature to return a vector of values instead of one
  item at a time.
- Basic adjustments for producer logic to accept vector of parse
results.
- Adjust the current parsers temporally by wrapping their single return
  value inside a vector
- Adjust parsers unit tests assuming that each parser will still return
  a vector of one value on each call.

Plugin: Add simple temporary benchmarking code

Assert parsing ends on first error temporarlly

- Producer logic assumes that parsing will stop after encountering the
  first error. This check should be done in unit tests but for now it's
  done in code for now

Plugin Host and API (in progress)

- Create library for plugins host
- Create wit file for parser on directory where plugins API crate will
  be initialized

Parse return `impl IntoIterator` instead of `vec`

- This can give more flexibility for other implementations of pares to
  return other types than vector
- Parsers with a single return value can use `iter::onc` or an array to
  enable more compiler optimization on the code

Move return statement in producer loop

- Returning the results should happen after the parse iterator is
  finished. It was positioned at the end of the first iteration of parse
  call.

Plugin: Wit files definitions & Prototype Clients

- Wit files:
  - Use a directory for each version.
  - Use a separate interface for shared types.
  - Create wit file for bytesource
  - Small changes on parse wit file
- Initialize rust lib and created place holder plugins

Plugin: Initialize WASM host lib

Plugins: Initialize WASM Engine

Plugins: Provide general configuration for plugins

- Parser and Bytesource plugins should get general chipmunk
configuration
  besides their user-define configurations as string (or path to a file)
- Currently the configurations structs are just palce-holders for
  chipmunk actual configs

Plugin: Implementing Parser Host (in progress)

- Create Parse Message type for parser plugins supporting only strings
- Create Plugin Parser State, but its implementation is still missing
- Bindgen Macro

Plugin: Implementing Parser Host (in progress)

- Implement Initialization of Parser Plugin
- Error definitions for plugins
- Move shared types and functionality to shared module
- Move bindings module to their own file
- WasmPlugin trait with basic implementation
- Create Bytesoruce Plugin place holder with implementation for input
  types only

Plugin: Parser Host & Versioning (in progress)

- Move version specific parts to their own module to have one module for
  each version for the parts that can be changed
- Define shared types and functions among different versions in their
  separate module
- Provide Mapping between bindings types and shared types.
- Create semantic version structure with parsing
- Implement reading component meta data by loading and perform simple
  validation and version parsing to initiate the corresponding host
  version
- Use std::OnceLock instead of tokio::OnceCell for creating wasm host
  since we don't need the function to be async.
- Change error type `unsupported` in wit file to provide a custom
  message too.

Plugin: Clippy Fixes

Plugin Parser Host: Implement Parser Trait

- Implement both parse functions using add method or returning a
  collection of results
- Provide the needed mapping between guest and host results

Plugin Parser: Add Unrecoverable Error type

- Add Unrecoverable error to parse results to cover panics inside the
  plugins
- Currently we log the error and print it to stderr and stop the parsing

Plugins: Parser Integrations (in progress)

- Introduce new parser type for the plugins with it's settings data type
- Add Error Kind to the native errors for the plugins and map plugin
  initialization error to it.
- Implement plugins parser Integrations in sessions export and observing
- Make configurations path optional for parser and source plugins
- Set `duplication_if_necessary` to true on parser host bindgen

Integrate Plugin parser in CLI interactive tool

- Option `plugin` is added to CLI interactive sessions
- Plugin path will be defined via environment variables temporally.

Plugins: Use interfaces in wit files

Interfaces in wit files contain the current version meta data as well
which is used to detect the plugin version to select the matching
host version.

Plugins: Implement missing mapping for parser host

Plugin: Fix producer loop logic

- Return was called from within the inner loop

Producer Stream: Change return type to vector of values

- Producer stream will yield all the available items at once instead of
  caching them

Force using plugin parsers via env vars temporally

Plugins: Parser Client Defs & Macro (in progress)

- Define `Parser` trait which must be implemented by plugins authors
- Implementing `export_parser` macro to be used by users
  - global paths for crate types still missing
  - Unit tests still missing
@AmmarAbouZor
Plugins: Parser Client Defs & Macro (in progress)
65be020 
- Define `Parser` trait which must be implemented by plugins authors
- Implementing `export_parser` macro to be used by users
  - global paths for crate types still missing
  - Unit tests still missing
@AmmarAbouZor
Plugins: Parser Client Exports & Macro (in progress)
94c0ffe 
- Define which types from bindings will be exported to the plugins
  implementers
- Export types needed in export macros as hidden
- Modify generate! Macro attributes to make its export macro public and
  set the bindings types for its needed types
- Set all types withing export macros with their full path.
@AmmarAbouZor
Plugin: Fixes & Adjustments to Parser Plugin Client
ab0f282 
Apply fixes and changes for the parser client after trying to use it
with real external plugin. Changes include:
- Separate parser and bytesource with features to prevent generate!
  Macro from being called multiple times.
- Move add function on the host to it's own interface that will be
  imported separately to avoid compilation errors because of duplicate
  definitions of types that occur when using the crate with separate
  projects as plugins only + Make the needed adjustments on the host
  side.
- Make generated bindings public but hide them in documentations to
  because they are needed with export Macro
- Move Prototyping code to test module temporally.
@AmmarAbouZor
Parser Plugin Macro: Insure Parser trait in scope
ec5c19a 
- Insure Parser trait from Chipmunk is in scope inside export macro so
  we can call parse method successfully.
- Remove prototyping suppressing warning + Corresponding fixes
@AmmarAbouZor
Integration tests for parser export macro
1390d65 
- Use the crate `TryBuild` to insure that the wrong patterns won't
  compile
- Add successful test case when Parser trait is implemented.
- Add failing tests for not implementing Parser Macro or for completely
  wrong input with trait or expression.
- Ignore the comparison of compilation error messages for the tests
  since we still not sure if we gonna include this comparison on the
  future, because the comparison is text-based and can be broken on
  different Rust editions or different environments
- Write script to run tests since they are behind inactive features
@AmmarAbouZor
Plugins API: Add build check for enabled features
4d365c4 
- Insure that one feature form Plugin-API crate at most is enabled at a
  time to provide the users with meaningful error message instead of the
  encrypted linking error if they enable many features at once since
  doing so will lead to linking errors on release builds only with very
  weird error messages
@AmmarAbouZor
Integration tests for parser export macro
80c1da8 
- Add test for the case when parser implementation and export macros are
  in different modules
@AmmarAbouZor
Plugins API: Add warning for internal module + renaming.
39ee2d3 
- Internal modules now start with underscore to make it more clear for the
  users that they shouldn't use them
- A warning have been added for the users if they explored the source
  code
- This idea is taken from the crate 'Log'
@AmmarAbouZor
Plugins API: Adding logging functionality
092a2fb 
- Add logging definitions and functions to wit file
- Creating module on the client side to provide clean logging public API
  while handling log level complexity internally to direct avoid log calls
  to the host when the log level isn't allowed
- Integrating the log level in the macro is still missing
@AmmarAbouZor
Plugins API: Adding logging functionality
c19a15e 
- Provide custom logger that implements `log::Log` trait and register
  it using boxed logger.
- This enables the users from using the general log macros from `log`
  crate and the message will be validated on the client side then sent
  to the host.
- Added log level to the custom configurations for the parser in the wit
  file and implement it on the client side only for now.
@AmmarAbouZor
Plugins API: Adding logging functionality
080435c 
- Use static logger instead of the boxed one because since the current
  max log level can be retrieved from log crate itself.
@AmmarAbouZor
Plugins API: Adding logging functionality
a8346f5 
- Add unit tests for plugins parser
- Change attributes and module name for prototyping code to make sure
  this wouldn't be compiled on any use case beside activating it
  manually with commenting out the attribute
@AmmarAbouZor
Plugins API: Adding logging functionality
266aa2a 
- Implement logging on parser host.
- Log Level will be always the current leg level in chipmunk since we
  use the same log functionality to log plugins' messages
@AmmarAbouZor
Plugins API: Adding logging functionality
1d3a905 
Reexport log crate from chipmunk client crate because it's used in
export macro, and the users can use it directly without having import
their own version of the same library.
@AmmarAbouZor
Parser Plugin: Send collection of results as default
3b30d1a 
- Use sending parse results to the host instead of using the host add
  function by default since it's more readable and until know there is
  no noticeable performance difference between the two approaches
@AmmarAbouZor
Plugin API Docs: Configurations & Improvements
3963c2d 
- Add configurations, attributes to show all the features in crate
  documentation, showing a hint for items available on features only.
- Provide a script file to show the documentation with all needed
  attributes and environment variable to replicate what will be shown on
  docs.rs once the crate is published
- Provide documentations and comments the implemented part of the crate
@AmmarAbouZor
Plugin API: Docs & Example for export parser macro
8a4715e
@AmmarAbouZor
Plugins: Add render options to string parser.
1546966 
Extend `string parser` showing how to return render options changing
its return type to two columns one for the message and the other
for its length.
@AmmarAbouZor
Plugins Parser: UI Integration with files (in progress)
4939419 
* Extend file types with `Plugins` type to avoid changing the current
  logic.
* Extend the code everywhere where the file types are used with suitable
  code for the plugins.
* Add menu command to open folder with plugins.
@AmmarAbouZor
Plugins Parser: UI Integration with files (done)
02b5c9c 
* Rename file type `plugins` to `parserPlugin` since it's intended to
  work with the parser plugins only and not the bytesource or other
  plugin kinds.
* Add missing dialogs to electron file and folder services.
* Extend types and implementation in rust core and their corresponding
  proptests
@AmmarAbouZor
Plugins: Parser on all stream formats.
1b85cdb 
Add plugins parser to all kinds of stream inputs.
@AmmarAbouZor
Plugins: Search results & Todos & Cleanup
0f268de 
* Fixes exporting the results of plugins into new tap.
* Resolve various todos in code.
* Remove deprecated communication types.
@AmmarAbouZor
Plugins: Fix warnings.
67f5058
@AmmarAbouZor
Plugins: Split WIT into three packages & Fixes
f7cdab4 
* Split WIT interface into three packages: (plugins, bytesource, shared)
  to generate the shared items once only and enable removing the
  features on the plugins-api crate since the bindings don't clash
  anymore.
* This separation changed the whole structure of the WIT files and
  made it similar to the structure of wasi crate.
* Generate unified bindings for the shared types and move all their code
  into the shared module in plugins-api crate.
* Adjust the unit tests for the macros accordingly, and remove all the
  wild card imports.
* Adjust example plugins and make then compile.
* Plugin host doesn't compile currently.
* Documentations in WIT and plagins-api still missing.
@AmmarAbouZor
Plugins: API Documentations & Cleanup
2eac9b0 
* Document all modules and types in the public API of the plugins_api
  crate
* Small improvements for the wit files.
* Fix warnings in bytesource tests.
@AmmarAbouZor
Plugins: Apply changes for shared types in host.
f2ac717 
* Make the need adjustments after separating the plugins to different
  packages unifying the implementation for the shared types.
* Generating the shared types doesn't generate not referenced types with
  `bindgen!()` macro. The current workaround is to generate the types
  from parser package inside the shared module.
* State `generate_used_types` explicitly to true in `generat!()` macro
  for plugins_api because this macro supports this option
@AmmarAbouZor
Plugins: Functions for retrieving configuration values
955723d 
* Provides functions to validate the retrieve the values of the
  configurations to reduce the boilerplates for the plugins developers.
* A macro has been used to generate the functions with a reminder to get
  compiler errors on type changes.
* Change the code in the example plugins to use the new functionality.
@AmmarAbouZor
Plugins: Owning variant for some config getter functions.
2434293 
Introduce another macro to return owned value for configuration values
where it doesn't make sense to return a reference to a boolean or for an
integer.
@AmmarAbouZor
Plugins: Move example to root directory.
43e56c2 
Example shouldn't be tracked while building Chipmunk and they should be
visible on the top level of the repo.
@AmmarAbouZor
Plugins: Export only needed logging functions.
026b72f 
Not all logging functions can be run in the plugins environment and
changes within the logging configurations can cause problem.
Therefore we re export logging macros only and the other functions
needed withing the export macros will be undocumented and start with
underscore.
@AmmarAbouZor
Plugins: Adding DLT parser plugin.
e399436 
* Create an example of DLT parser plugin that replicates the dlt
  parser built in Chipmunk to check how powerful the current
  implementation is and for benchmarking.
* Current solution is working without filter support and without
  timestamps and formatting options.
* Solution still have TODOs in the code for improvements.
@AmmarAbouZor
Merge branch 'master' into plugins_support
7ee8572 
Conflicts:
      application/apps/indexer/stypes/Cargo.toml
      application/client/src/app/ui/tabs/module.ts
      application/client/src/app/ui/tabs/observe/parsers/general/module.ts
      application/holder/src/service/actions/index.ts
      application/platform/ipc/request/actions/index.ts
@AmmarAbouZor
Plugins: Fixes after merging master.
1c947d0 
* Fixes included adapting the UI components to current angular version.
@AmmarAbouZor
Plugins: Configuration default values on plugins side
799ddc1 
* Define default value for plugins in wit file and plugins api
* Matching adjustments on the example plugins.
* Implementation in host is still missing (Host won't compile currently)
@AmmarAbouZor
Plugins: Configs default values on host.
a463d19 
* Implement support for configurations default values in backend and
  front end of the host, changing the types and fixing there proptests
  as well.
* All control on front-end provide a default value on initialization to
  avoid having some configurations not existed if users haven't changed
  them.
* Move shared configurations functions to state struct to be reachable
  in multiple places in code.
@AmmarAbouZor
Plugins: README for plugin development.
3dc3202 
Provide documentation for developing plugins, building the examples and
integrating them within Chipmunk.
@AmmarAbouZor
Plugins: Resolve TODOs & Refactoring & Documentation.
833cf90 
Rust:
* Plugin api: Remove checks for building with all features since they
  are not conflicting anymore.
* Remove all code for forcing parser plugin via environment variables
  since it's not needed anymore for parser plugins.
* Resolve smaller todos in plugins host and api.
* Documentation and comments about plugins in Rust code.

Type Script:
* Remove not needed methods for byte-source plugins.
* Remove not needed angular imports.
* Change some TODOs asking Dmitry to help with them since I don't have
  enough overview on the front-end aspect.
* Resolve other TODOs in front-end
@AmmarAbouZor AmmarAbouZor mentioned this pull request Feb 10, 2025
Merged
@AmmarAbouZor
Merge branch 'master' into plugins_support
7991de9 
Conflicts:
      application/client/src/app/service/actions.ts
      application/client/src/app/service/actions/index.ts
      application/holder/src/service/actions/index.ts
      application/platform/ipc/request/actions/index.ts
@AmmarAbouZor
Copy link
Member Author

Will be continued in branch plugins_support in mainstream repo
#2196

@AmmarAbouZor AmmarAbouZor deleted the plugins_support branch February 10, 2025 11:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@AmmarAbouZor