You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The goal is to make Sigma rules a real workload in LynxDB, not a bolt-on. Two modes I want to support from day one.
Serverless: pipe events through, get matches, no persistence - cat events.ndjson | lynxdb sigma run --rules ./rules --fieldmap ./ecs-to-lynx.yaml.
And server mode, where Sigma becomes a new MV kind (ViewTypeDetection) sitting next to projection and aggregation:
Matched events land in a compact detection index, enriched with sigma.rule_id, sigma.title, sigma.level, sigma.tags, sigma.rule_hash, and friends. Then it's just another queryable source:
from detections_windows
| where sigma.level in ("high", "critical")
| stats count by sigma.rule_title
| sort -count
The reason to make this an MV and not a "run rules at query time" feature: re-evaluating the whole ruleset against the entire corpus on every query is the wrong shape. The MV lifecycle already does what detection workloads need - backfill, continuous dispatch, retention, pause/resume, versioned rebuild. Sigma slots in naturally as a third MV kind, with extra metadata for ruleset_id, fieldmap_hash, per-rule hashes, list of unsupported rules, and the detection output schema.
The actually hard part isn't running rules it's the compilation interface. Whichever runtime we pick, the matcher has to expose a structured IR, not just a query string. LynxDB needs rule id and hash, required fields (for column pruning), prefilter terms (for inverted index seek), the field-mapped logical predicate, the metadata payload, and per-rule diagnostics for unsupported features. With that, the planner does cheap source/time/term filtering first, loads only the columns it needs, then runs the exact predicate same shape as the rest of the engine. Without it, we're forced into "compile to opaque string and pray", which kills any planner optimization.
For the runtime itself there are roughly four ways to do it, each with different trade-offs on coverage, hot-path performance, release coupling, and how much of the Sigma spec we end up owning ourselves:
Native Go Sigma compiler in LynxDB. Full control over the IR, zero FFI, deepest planner integration. Highest cost, and we're on the hook for tracking SigmaHQ compatibility forever.
Embed RSigma directly. Reuse Mostafa's work. Needs RSigma to expose the structured IR above (collaboration with @mostafa). FFI shape depends on language - if it stays Rust, that's CGO or Wasm.
RSigma via the extension runtime (depends on Experiment: evaluate plugin/extension architectures #15). Only makes sense if the extension spike picks Wasm and per-batch FFI cost turns out acceptable. Decouples release cycles, but constrains the IR to whatever crosses the Wasm boundary cheaply.
External preprocessor. RSigma compiles rules offline into a LynxDB-native predicate format; LynxDB only consumes the IR. Cheapest integration, loses hot-reload, adds a build step for users.
A few things still open: field mapping format (ECS-to-LynxDB YAML, inline in the rule, or auto-derived from Sigma logsource?); behavior when a rule is updated mid-flight (versioned rebuild, which is the existing MV pattern, vs in-place reload); aggregation/correlation rules with condition: count() > 10 (should fit naturally into streamstats/eventstats but worth confirming); and whether one view holds multiple rulesets or it's one-view-per-ruleset.
Out of scope for this issue: building a detection content library (LynxDB consumes Sigma, doesn't author it), alert routing (existing alerts subsystem handles it once detections are queryable), and any UI for rule management.
Done when there's a recorded decision on which execution path we're taking and why, a spec for the structured IR (independent of which compiler produces it), a design doc for ViewTypeDetection, the CLI surface for both sigma run and sigma view create, and a test corpus subset of SigmaHQ rules plus synthetic events with expected matches.
The goal is to make Sigma rules a real workload in LynxDB, not a bolt-on. Two modes I want to support from day one.
Serverless: pipe events through, get matches, no persistence -
cat events.ndjson | lynxdb sigma run --rules ./rules --fieldmap ./ecs-to-lynx.yaml.And server mode, where Sigma becomes a new MV kind (
ViewTypeDetection) sitting next to projection and aggregation:Smth like
Matched events land in a compact detection index, enriched with
sigma.rule_id,sigma.title,sigma.level,sigma.tags,sigma.rule_hash, and friends. Then it's just another queryable source:The reason to make this an MV and not a "run rules at query time" feature: re-evaluating the whole ruleset against the entire corpus on every query is the wrong shape. The MV lifecycle already does what detection workloads need - backfill, continuous dispatch, retention, pause/resume, versioned rebuild. Sigma slots in naturally as a third MV kind, with extra metadata for
ruleset_id,fieldmap_hash, per-rule hashes, list of unsupported rules, and the detection output schema.The actually hard part isn't running rules it's the compilation interface. Whichever runtime we pick, the matcher has to expose a structured IR, not just a query string. LynxDB needs rule id and hash, required fields (for column pruning), prefilter terms (for inverted index seek), the field-mapped logical predicate, the metadata payload, and per-rule diagnostics for unsupported features. With that, the planner does cheap source/time/term filtering first, loads only the columns it needs, then runs the exact predicate same shape as the rest of the engine. Without it, we're forced into "compile to opaque string and pray", which kills any planner optimization.
For the runtime itself there are roughly four ways to do it, each with different trade-offs on coverage, hot-path performance, release coupling, and how much of the Sigma spec we end up owning ourselves:
A few things still open: field mapping format (ECS-to-LynxDB YAML, inline in the rule, or auto-derived from Sigma
logsource?); behavior when a rule is updated mid-flight (versioned rebuild, which is the existing MV pattern, vs in-place reload); aggregation/correlation rules withcondition: count() > 10(should fit naturally into streamstats/eventstats but worth confirming); and whether one view holds multiple rulesets or it's one-view-per-ruleset.Out of scope for this issue: building a detection content library (LynxDB consumes Sigma, doesn't author it), alert routing (existing alerts subsystem handles it once detections are queryable), and any UI for rule management.
Done when there's a recorded decision on which execution path we're taking and why, a spec for the structured IR (independent of which compiler produces it), a design doc for
ViewTypeDetection, the CLI surface for bothsigma runandsigma view create, and a test corpus subset of SigmaHQ rules plus synthetic events with expected matches.Refs: Sigma, RSigma, LynxDB MV docs.