Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,8 @@

Python orchestration, contracts, schemas, and CLI code live in
`python/nfi_backtest_engine/`. The Rust workspace is under `rust/`; its main crates
are `nfi-sim-core`, `nfi-vector-io`, `nfi-sim-cli`, and the PyO3 bridge `nfi-py`.
are `nfi-sim-core`, `nfi-vector-core`, `nfi-vector-io`, `nfi-sim-cli`, and the PyO3
bridge `nfi-py`.
Python tests are in `tests/`, with parity-focused cases in `tests/parity/`. Captured
fixtures and published evidence belong in `benchmarks/fixtures/` and
`benchmarks/evidence/`. Keep user documentation in `docs/` and roadmap state in
Expand Down
3 changes: 3 additions & 0 deletions docs/indicator-inventory.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,3 +51,6 @@ lookback. Function and node IDs are structural data; strategy and Signal names n
become opcodes. Negative shifts, centered rolling windows, backward fill, dynamic
window sizes, and prefilled informative merges fail closed. An unsupported source
construct must be generalized deliberately before it can enter the Native lane.

The safe Rust execution boundary and its memory/accuracy claim are documented in
[Native Vector Core](native-vector-core.md).
54 changes: 54 additions & 0 deletions docs/native-vector-core.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
# Native Vector Core

`nfi-vector-core` is the safe Rust execution substrate for
`indicator-program-v1`. It is intentionally independent of strategy names,
Signal numbers, pairs, timeranges, upstream commits, and expected results.

## Responsibility boundaries

- `nfi-vector-core` owns program validation, output reachability, typed Arrow
views, deterministic scalar behavior, bounded state, and batch execution.
- `nfi-vector-io` continues to own Feather paths, hashes, IPC decoding, and
transport projection.
- `nfi-sim-core` continues to own trades, orders, wallets, callbacks, funding,
liquidation, protections, and the event loop.
- The Python compiler remains the source-to-IR boundary.

The core uses Arrow2 0.18, matching the existing transport. It has no filesystem,
PyO3, simulator, Polars, ndarray, Rayon, or SIMD dependency. Workspace policy
forbids unsafe Rust and no fast-math option is enabled.

## Deterministic execution

Input columns are borrowed from Arrow buffers after exact physical-type checks.
Only columns reachable from requested outputs enter the plan. Unused source
columns and unrelated program branches are neither decoded nor allocated by the
core.

All NaN payloads are normalized to one quiet NaN at kernel boundaries. Null and
NaN remain distinct; signed zero and infinities are preserved. Arithmetic order
is explicit and comparisons use exact IEEE behavior. Approximate equality is not
an execution or promotion rule.

Shift and rolling-window state uses bounded ring buffers that survive record-batch
boundaries. Long-stream tests assert that retained state never exceeds the
compiled lag or window. A discard sink demonstrates this live-value bound:

```text
batch_rows × (projected + intermediate + final columns) + bounded state
```

## Current claim boundary

M20-03 provides the engine, typed columns, generic scalar operations, causal
shift, rolling storage, and source-located fail-closed errors. TA-Lib indicators,
rolling reducers, EWM algorithms, and the complete latest-NFI kernel set belong
to M20-04 and require official Python/TA-Lib column-exact fixtures before Native
promotion. An unimplemented reachable opcode stops; it is never guessed.

The Rust boundary can validate a compiled contract directly:

```bash
cargo run --manifest-path rust/Cargo.toml -p nfi-vector-core \
--example validate_indicator_program -- .nfi/indicator-program.json
```
63 changes: 56 additions & 7 deletions planning/roadmap-state.json
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
{
"schema_version": "1.0.0",
"roadmap_id": "nfi-backtest-engine-post-v1.1.0",
"revision": 127,
"updated_at": "2026-08-10T21:17:45+09:00",
"revision": 129,
"updated_at": "2026-08-10T23:08:02+09:00",
"acceptance_commands": "planning/acceptance-commands.json",
"active_task_id": null,
"execution_policy": {
Expand Down Expand Up @@ -4693,7 +4693,7 @@
"order": 2003,
"milestone": "M20",
"title": "Implement the safe Rust nfi-vector-core engine",
"status": "pending",
"status": "completed",
"depends_on": [
"M20-02"
],
Expand All @@ -4713,10 +4713,16 @@
"Rust vector-kernel tests",
"memory-bounded column execution"
],
"started_at": null,
"completed_at": null,
"commit_sha": null,
"evidence": [],
"started_at": "2026-08-10T22:16:20+09:00",
"completed_at": "2026-08-10T23:08:02+09:00",
"commit_sha": "248d0a9a335a127df5da3d9b15da510e5fbbd386",
"evidence": [
".nfi/roadmap-acceptance/M20-03/248d0a9a335a127df5da3d9b15da510e5fbbd386/acceptance-report.json",
".nfi/roadmap-acceptance/M20-03/248d0a9a335a127df5da3d9b15da510e5fbbd386/memory-bound-report.json",
".nfi/roadmap-acceptance/M20-03/248d0a9a335a127df5da3d9b15da510e5fbbd386/SHA256SUMS.txt",
".nfi/roadmap-acceptance/M20-03/248d0a9a335a127df5da3d9b15da510e5fbbd386/x7-spot/run.json",
".nfi/roadmap-acceptance/M20-03/248d0a9a335a127df5da3d9b15da510e5fbbd386/x7-futures/run.json"
],
"blocker": null
},
{
Expand Down Expand Up @@ -8243,6 +8249,49 @@
"next_eligible_task": "M20-03",
"next_task_manual_gate": false
}
},
{
"sequence": 175,
"timestamp": "2026-08-10T22:16:20+09:00",
"task_id": "M20-03",
"event": "task_started",
"details": {
"objective": "implement a safe memory-bounded Rust execution core for indicator-program-v1",
"new_crate": "nfi-vector-core",
"arrow_version_must_match_transport": true,
"whole_nfi_indicator_kernel_claim": false,
"unused_column_materialization_allowed": false,
"unsafe_rust_allowed": false,
"fast_math_allowed": false,
"runtime_hardcoding_allowed": false
}
},
{
"sequence": 176,
"timestamp": "2026-08-10T23:08:02+09:00",
"task_id": "M20-03",
"event": "task_completed",
"details": {
"implementation_commit": "248d0a9a335a127df5da3d9b15da510e5fbbd386",
"crate": "nfi-vector-core",
"python_contract_fingerprint": "cb278c7294995c0a8c018d44485990612e8370ba4fff7e5d71df0c3c862fc8ca",
"python_contract_validated_by_rust": true,
"future_compatibility_test_count": 134,
"python_test_count": 729,
"rust_workspace_test_count": 169,
"memory_bound_input_rows": 32000,
"memory_bound_peak_live_value_upper_bound": 98,
"unused_input_columns_materialized": 0,
"spot_trade_surface_and_full_state_exact": true,
"futures_trade_surface_and_full_state_exact": true,
"unsafe_rust_added": false,
"fast_math_added": false,
"runtime_hardcoding_added": false,
"all_nfi_indicator_kernels_claim": false,
"full_native_strategy_claim": false,
"next_eligible_task": "M20-04",
"next_task_manual_gate": false
}
}
]
}
11 changes: 11 additions & 0 deletions rust/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions rust/Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
[workspace]
members = [
"crates/nfi-sim-core",
"crates/nfi-vector-core",
"crates/nfi-vector-io",
"crates/nfi-sim-cli",
"crates/nfi-py",
Expand Down
22 changes: 22 additions & 0 deletions rust/crates/nfi-vector-core/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
[package]
name = "nfi-vector-core"
version.workspace = true
edition.workspace = true
license.workspace = true
rust-version.workspace = true
repository.workspace = true
homepage.workspace = true
description = "Safe, deterministic vector execution core for NFI indicator programs"

[lib]
path = "src/lib.rs"

[dependencies]
arrow2 = "0.18"
serde = { version = "1", features = ["derive"] }
serde_json = "1"
sha2 = "0.10"
thiserror = "1"

[lints]
workspace = true
26 changes: 26 additions & 0 deletions rust/crates/nfi-vector-core/examples/validate_indicator_program.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
//! Validate a Python-compiled indicator program at the Rust boundary.

use std::error::Error;
use std::fs;
use std::io::{Error as IoError, ErrorKind};

use nfi_vector_core::program::IndicatorProgram;

fn main() -> Result<(), Box<dyn Error>> {
let path = std::env::args_os().nth(1).ok_or_else(|| {
IoError::new(
ErrorKind::InvalidInput,
"usage: validate_indicator_program <indicator-program.json>",
)
})?;
let encoded = fs::read_to_string(path)?;
let program = IndicatorProgram::from_json(&encoded)?;
println!(
"validated {}: {} functions, {} nodes, fingerprint {}",
program.schema_version,
program.functions.len(),
program.nodes.len(),
program.fingerprint
);
Ok(())
}
Loading