diff --git a/CLAUDE.md b/CLAUDE.md index 50bb416..abda180 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -5,8 +5,9 @@ transaction logic as sandboxed WebAssembly, ~5.7M tx/s. Rust, edition 2024, MSRV > **Status nuance (read this):** The public `README.md` describes a *single-node* engine and marks > cluster/raft as "planned." The code is ahead — the `cluster`, `raft`, `client`, and `control` -> crates implement Raft consensus + replication and are under **active development**. Recovery is -> mid-redesign on branch `feature/redesign-recover-process` (ADR-019/020/021 + ActiveSnapshot). +> crates implement Raft consensus + replication and are under **active development**. The +> `ActiveSnapshot`-based recovery redesign (ADR-019/020/021) is **merged** and lives in +> `crates/ledger/src/recover.rs`. > Treat README's "single node" claims as marketing-lag, not ground truth — check the code. ## Mental model @@ -30,10 +31,11 @@ transaction logic as sandboxed WebAssembly, ~5.7M tx/s. Rust, edition 2024, MSRV `Deposit`, `Withdrawal`, `Transfer`, and `Function` (invoke a registered WASM module as a first-class atomic op — the only extension point). Accounts are `u64`, balances `i64`, account `0` is the system -source/sink. WASM host API is deliberately narrow (`credit`, `debit`, `get_balance`) and fully -deterministic so followers can replay entries without re-running code. See `docs/wasm-runtime.md`. +source/sink. WASM host API is a small set of deterministic verbs — balance ops (`credit`/`debit`/`get_balance`), +account-layout/flag ops, and typed KV/constant ops (ADR-022/023/026) — so followers can replay entries +without re-running code. See `docs/wasm-runtime.md`. -## Repository map (8-crate Cargo workspace) +## Repository map (9-crate Cargo workspace) | Crate | Path | Role | |---|---|---| @@ -45,6 +47,7 @@ deterministic so followers can replay entries without re-running code. See `docs | `client` | `crates/client` | gRPC client library. | | `control` | `crates/control` | Control plane (web/gRPC) for multi-node scenarios; drives the `ui/`. Owns the scenario primitives + catalogue (`scenario`/`scenarios` modules). | | `ctl` | `crates/ctl` | Offline CLI (`roda-ctl`): pack/unpack/validate WAL segments. | +| `roda-wasm-abi` | `crates/roda-wasm-abi` | Guest-side WASM SDK/ABI: `execute!`/`register!` macros, typed `Params`, `key!` builder, safe host-verb wrappers — how WASM modules are written (ADR-026). | `cluster` is the default workspace member, so `cargo run` starts `roda-server`. @@ -111,10 +114,11 @@ Docker quick start: `docker run -p 50051:50051 -v $(pwd)/data:/app/data tislib/r Read in order: `README.md` → `docs/01-concepts.md` → `docs/02-api.md` → `docs/03-architecture.md` → `docs/internal.md` → `docs/wasm-runtime.md`. Perf numbers: `docs/load.md`. -(Note: README links `docs/04-internals.md`, but the file on disk is `docs/internal.md`.) -ADRs live in `docs/adr/` (`docs/adr/README.md` indexes them, though the index table lags behind disk — -ADRs 021/022 exist but aren't listed). Status ≠ implementation state; the most load-bearing +ADRs live in `docs/adr/` (`docs/adr/README.md` indexes them, though that index lists two **phantom** +rows — ADR-024 `0024-temporal-index.md` / ADR-025 `0025-point-in-time.md` — that don't exist on disk +(numbering jumps 0023→0026→0027); `docs/adr.md` is a stale duplicate stopping at ADR-010). Status ≠ +implementation state; the most load-bearing **implemented** ones: - **001** entries-based execution · **002** Vec balance storage · **006** WAL/snapshot/seal durability diff --git a/Cargo.lock b/Cargo.lock index 2f23d59..29e3dfb 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -424,7 +424,6 @@ dependencies = [ "proto", "raft", "rand 0.8.5", - "roda-latency-tracker", "roda-wasm-abi", "serde", "signal-hook", @@ -472,6 +471,7 @@ dependencies = [ "proto", "rand 0.8.5", "spdlog-rs", + "testing", "thiserror 1.0.69", "tokio", "tokio-stream", @@ -2313,6 +2313,13 @@ dependencies = [ "winapi-util", ] +[[package]] +name = "testing" +version = "0.1.0" +dependencies = [ + "anyhow", +] + [[package]] name = "thiserror" version = "1.0.69" diff --git a/crates/control/src/main.rs b/crates/control/src/main.rs index 04592f6..fdf602d 100644 --- a/crates/control/src/main.rs +++ b/crates/control/src/main.rs @@ -69,14 +69,12 @@ async fn main() -> anyhow::Result<()> { let events = Arc::new(EventStore::new()); let shutdown = CancellationToken::new(); - // Wire SIGINT to the cancellation token. + // Wire SIGINT/SIGTERM to the cancellation token so `kill` (containers + // send SIGTERM) tears the cluster down via RAII instead of orphaning it. let signal_shutdown = shutdown.clone(); tokio::spawn(async move { - if let Err(e) = tokio::signal::ctrl_c().await { - tracing::warn!("failed to install SIGINT handler: {e}"); - return; - } - info!("received SIGINT; initiating shutdown"); + wait_for_shutdown_signal().await; + info!("received shutdown signal; initiating shutdown"); signal_shutdown.cancel(); }); @@ -97,6 +95,38 @@ async fn main() -> anyhow::Result<()> { } } +/// Resolve on the first SIGINT or SIGTERM so shutdown is cooperative +/// whether the process is Ctrl+C'd or `kill`ed. +#[cfg(unix)] +async fn wait_for_shutdown_signal() { + use tokio::signal::unix::{SignalKind, signal}; + let mut term = match signal(SignalKind::terminate()) { + Ok(s) => s, + Err(e) => { + tracing::warn!("failed to install SIGTERM handler: {e}"); + return wait_for_ctrl_c().await; + } + }; + tokio::select! { + _ = wait_for_ctrl_c() => {} + _ = term.recv() => {} + } +} + +#[cfg(not(unix))] +async fn wait_for_shutdown_signal() { + wait_for_ctrl_c().await; +} + +/// Never resolve if the handler can't be installed, so a failure can't +/// masquerade as a shutdown request. +async fn wait_for_ctrl_c() { + if let Err(e) = tokio::signal::ctrl_c().await { + tracing::warn!("failed to install SIGINT handler: {e}"); + std::future::pending::<()>().await; + } +} + /// Sane defaults so a blank cluster has something coherent to start /// from. Mirrors the cluster's TOML rendering defaults at /// `provisioner/process.rs::render_config_toml`. diff --git a/crates/ledger/src/index.rs b/crates/ledger/src/index.rs index fb60ab5..938e956 100644 --- a/crates/ledger/src/index.rs +++ b/crates/ledger/src/index.rs @@ -1,14 +1,13 @@ -//! In-memory transaction and account index — ADR-008 / ADR-022. +//! In-memory transaction index — ADR-008 / ADR-022. //! -//! Three pre-allocated, fixed-size structures; zero heap allocation after +//! Two pre-allocated, fixed-size buffers; zero heap allocation after //! construction, no cold start on segment rotation: //! -//! `circle1` — maps `tx_id → (TxMetadata, location in circle2)` (direct-mapped) -//! `circle2` — `IndexedTxEntry` storage: each follower's raw `WalEntry` -//! stored as-is, plus the per-account `prev_link` chain -//! `account_heads` — maps `account_id → latest circle2 index` (direct-mapped) +//! `circle1` — maps `tx_id → (TxMetadata, location in circle2)` (direct-mapped) +//! `circle2` — `IndexedTxEntry` storage: each follower's raw `WalEntry` as-is //! -//! Both circle sizes must be powers of two so modulo reduces to a bitmask. +//! Both sizes must be powers of two so modulo reduces to a bitmask. Account +//! history is served by a backward WAL scan, not this index (see `ledger.rs`). use bytemuck::Zeroable; use storage::EntryBuf; @@ -42,11 +41,8 @@ impl Default for TxSlot { // ── IndexedTxEntry (circle2) ────────────────────────────────────────────────── /// One slot in `circle2`: a follower's raw `WalEntry`, stored as-is, wrapped -/// with the bookkeeping the index needs. -/// -/// - `tx_id` groups the slot with its transaction and detects eviction. -/// - `prev_link` is the 1-based per-account history chain pointer — set only -/// for `WalEntry::Entry` followers; `0` otherwise and at chain ends. +/// with the bookkeeping the index needs. `tx_id` groups the slot with its +/// transaction and detects eviction. #[derive(Copy, Clone, Debug, PartialEq, Eq)] pub struct IndexedTxEntry { pub record: WalEntry, @@ -108,9 +104,8 @@ impl TransactionIndexer { } } - /// Index a committed transaction: its `TxMetadata` and all followers - /// (stored as raw `WalEntry`). `WalEntry::Entry` followers are chained into - /// the per-account `prev_link` history; all other followers are stored too. + /// Index a committed transaction: its `TxMetadata` and all followers, + /// each stored as a raw `WalEntry` in `circle2`. pub fn insert_transaction(&mut self, meta: &TxMetadata, followers: &[WalEntry]) { self.insert_with(meta, followers.len(), followers.iter().copied()); } @@ -189,7 +184,7 @@ mod tests { use storage::entities::{EntryKind, FailReason, WalEntryKind}; fn small() -> TransactionIndexer { - // circle1=16, circle2=64, account_heads=16 — small enough for fast tests + // circle1=16, circle2=64 — small enough for fast tests TransactionIndexer::new(16, 64) } diff --git a/crates/ledger/src/transactor/wasm_runtime.rs b/crates/ledger/src/transactor/wasm_runtime.rs index 3b628a7..731b725 100644 --- a/crates/ledger/src/transactor/wasm_runtime.rs +++ b/crates/ledger/src/transactor/wasm_runtime.rs @@ -451,10 +451,9 @@ impl WasmRuntime { } } -/// Build the shared [`Linker`] with the three host imports -/// (`ledger.credit` / `ledger.debit` / `ledger.get_balance`) routed -/// directly to the transactor state. Called exactly once per -/// `WasmRuntime`. +/// Build the shared [`Linker`] with the `ledger` host verbs — balances, +/// account flags, and KV/constants (ADR-022/023) — routed to the +/// transactor state. Called once per `WasmRuntime`. fn build_host_linker(engine: &Engine) -> Linker { let mut linker: Linker = Linker::new(engine); diff --git a/crates/ledger/src/tx_ring/README.md b/crates/ledger/src/tx_ring/README.md index 659f86a..74f9136 100644 --- a/crates/ledger/src/tx_ring/README.md +++ b/crates/ledger/src/tx_ring/README.md @@ -40,12 +40,12 @@ let (writer, reader) = TxRing::new(capacity); | Handle | Count | Role | |---|---|---| | `TxRingWriter` | exactly one | the **only** producer; drives `reserve` / `push` / `commit` / `rollback` | -| `TxRingReader` | exactly one | the **only** consumer; reads (`walk` / `get`) **and** moves the release index (`release_to`) | +| `TxRingReader` | exactly one | the **only** consumer; reads (`walk`) **and** moves the release index (`release_to`) | -- Both handles hold an `Arc`; the ring drops when both do. +- Both handles hold an `Arc`; the ring is never exposed and drops when both do. - `TxRingWriter` / `TxRingReader` are **not `Clone`** and are yielded once from `new()`. -- The reader holds its own local release cursor and reads any index inside the readable - window and read any element or sub-range. +- The reader holds its own local release cursor and walks the readable window + `[released, write_index)`. --- @@ -54,7 +54,7 @@ let (writer, reader) = TxRing::new(capacity); The ring **never blocks or sleeps internally**. Capacity pressure is expressed as a *clamped grant*: -1. `writer.reserve()` grants *all* currently-free slots — everything the releaser has +1. `writer.reserve()` grants *all* currently-free slots — everything the reader has freed: `granted = capacity − (write − release)` — and returns the new `capacity()`. 2. `writer.capacity()` reports how many slots you may still `push` — possibly `0` when the ring is full. @@ -75,17 +75,17 @@ unsynchronized slot write, and a cursor bump. - **Capacity is a power of two.** Enforced by an assert in `new()` (slot indexing is `cursor & (capacity − 1)`). -- **One writer, one releaser.** Guaranteed by construction — don't smuggle extra producers - in by other means. -- **Readers stay in the window.** A reader's cursor must remain within - `[release_index(), write_index())`. Reading below `release_index` races a writer - overwrite; reading at/above `write_index` reads unpublished data. **`get()` enforces this - at runtime** (see below) — a violation panics instead of returning stale or torn data. -- **The releaser only moves forward, and never past the writer.** `release` / `advance_to` - are monotonic and must not exceed `write_index()`. Both are checked with `debug_assert!`. -- **The releaser must coordinate all readers.** Because the ring has one mover but many - readers, the application is responsible for advancing the release index only once *every* - reader is done with those slots (typically the slowest/last stage drives it). +- **One writer, one reader.** Guaranteed by construction — `new()` yields exactly one of each + and neither is `Clone`; don't smuggle extra producers or consumers in by other means. +- **The reader stays in the window.** The reader's cursor must remain within + `[release_index(), write_index())`. Reading below `release_index` races a writer overwrite; + reading at/above `write_index` reads unpublished data. `walk` stops at the `write_index` + snapshot it takes on entry and `debug_assert!`s `from >= released`. +- **The reader only releases forward, and never past the writer.** `release_to` is monotonic + and must not exceed `write_index()`; both are checked with `debug_assert!`. +- **The reader gates the writer.** Because the reader is the sole release mover, it advances + the release index only once it is done with those slots — that is what frees them back to + the writer. **Element type:** the ring stores `WalEntry`. It is `Copy`, so `get()` returns a value and no reference into a slot ever escapes (hence no `Sync` requirement on the element). Slots @@ -97,15 +97,11 @@ window. ## API -### `TxRing` (via `Arc`, used by readers) +### `TxRing` (constructor only; the ring itself is never exposed) | Method | Ordering | Description | |---|---|---| -| `new(capacity) -> (Arc, TxRingWriter, TxRingReleaser)` | — | allocate; assert power-of-two | -| `write_index() -> usize` | `Acquire` | exclusive upper bound of the readable window | -| `release_index() -> usize` | `Acquire` | lower bound; slots below may be overwritten | -| `capacity() -> usize` | — | ring size (slot count) | -| `get(idx) -> WalEntry` | read → `Acquire` fence → load | copy of slot `idx`; **panics if `idx` was outside the live window** | +| `TxRing::new(capacity) -> (TxRingWriter, TxRingReader)` | — | allocate (assert power-of-two) and hand out the writer + reader; the `Arc` lives behind them | ### `TxRingWriter` (the single producer) @@ -117,7 +113,7 @@ window. | `writer.cursor() -> usize` | — | the next `ring_index` this writer will write to (the head) | | `writer.push(entry: WalEntry) -> usize` | — | write at the head, return the `ring_index` written; **panics if `capacity() == 0`** | | `writer.walk(start, end, handler)` | — | visit `ring_index` range `[start, end)` (wrapping) lending each `&WalEntry` — no copy | -| `writer.commit()` | `Release` | publish the head so readers/releaser observe it | +| `writer.commit()` | `Release` | publish the head so the reader observes it | | `writer.rollback_to(ring_index)` | — | move the head back to `ring_index`, discarding the uncommitted tail | | `drop(writer)` | `Release` | auto-`commit()` of whatever was pushed | @@ -128,12 +124,19 @@ physical slots only on access. > its target `[commit, cursor)` is pushed-but-unpublished, so no reader's window (`[release, > write)`) overlaps it and the borrow cannot race. The reference never escapes the call. -### `TxRingReleaser` (the single consume-index mover) +### `TxRingReader` (the single consumer — reads *and* releases) | Method | Ordering | Description | |---|---|---| -| `advance_to(index)` | `Release` | advance the release index to an absolute logical index (monotonic; debug-asserted) | -| `released() -> usize` | — | current release index (local) | +| `reader.walk(from, handler)` | `Acquire` | visit published entries from `from` up to the `write_index` snapshot taken on entry, handing each a **copy** to `handler`; stop early when it returns `false`. `from` must be `>= released` | +| `reader.release_to(index)` | `Release` | advance the release index to an absolute logical index, freeing those slots back to the writer (monotonic, `≤ write_index()`; debug-asserted) | +| `reader.released() -> usize` | — | current release index (local) | +| `reader.write_index() -> usize` | `Acquire` | exclusive upper bound of the readable window | +| `reader.capacity() -> usize` | — | ring size (slot count) | + +> `walk` hands `handler` a `WalEntry` **by value** — the reader copies out of the slot and never +> lends a reference into a live one, so the borrow can't race the writer. (`get(idx)`, a random-access +> copy, exists only behind `#[cfg(test)]`.) --- @@ -144,7 +147,7 @@ physical slots only on access. ```rust use storage::entities::WalEntry; -let (ring, mut writer, mut releaser) = TxRing::new(1024); +let (mut writer, mut reader) = TxRing::new(1024); // Push a slice of entries, honoring backpressure with the long-lived writer. let batch: &[WalEntry] = &entries; @@ -152,9 +155,9 @@ writer.reserve(); let mut i = 0; while i < batch.len() { if writer.capacity() == 0 { - // Re-grant space the releaser has freed (also commits prior pushes). + // Re-grant space the reader has freed (also commits prior pushes). if writer.reserve() == 0 { - std::hint::spin_loop(); // ring full — let a reader/releaser catch up + std::hint::spin_loop(); // ring full — let the reader catch up continue; } } @@ -164,30 +167,24 @@ while i < batch.len() { // drop(writer) commits the tail ``` -### Reader (any number, each with its own cursor) +### Reader (the single consumer — reads, then releases) ```rust -let ring = Arc::clone(&ring); -let mut cursor = 0usize; +let mut consumed = 0usize; loop { - let w = ring.write_index(); // Acquire: see all slots written before this point - while cursor < w { - let entry = ring.get(cursor); // copy out; panics if cursor left the window + // walk copies out every published entry from `consumed` to the current + // write_index; the reader never lends a slot reference. + reader.walk(consumed, |entry| { handle(entry); - cursor += 1; - } + consumed += 1; + true + }); + // Done with those slots — free them back to the writer. + reader.release_to(consumed); std::hint::spin_loop(); } ``` -### Releaser (single mover — e.g. the last/slowest stage) - -```rust -// Once every reader is done up to an absolute consumed position, free those -// slots back to the writer. -releaser.advance_to(global_consumed); -``` - --- ## Memory ordering & safety @@ -195,46 +192,32 @@ releaser.advance_to(global_consumed); Slots live in `Box<[UnsafeCell]>`; correctness rests on a release/acquire edge on each cursor: -- **Publish:** the writer fully writes a slot, then `write.store(Release)` in `flush`. A +- **Publish:** the writer fully writes a slot, then `write.store(Release)` in `commit`. A reader's `write_index()` (`Acquire`) that observes the new value is guaranteed to see the slot write. -- **Free:** the releaser publishes freed space with `release.store(Release)`; the writer - observes it via an `Acquire` load in the `reserve` gate before reusing those slots. +- **Free:** the reader publishes freed space with `release.store(Release)` in `release_to`; the + writer observes it via an `Acquire` load in the `reserve` gate before reusing those slots. - **`unsafe impl Send/Sync for TxRing`** is sound because the writer gates on `release` - (never overwrites a readable slot) and readers only ever take a *copy*. - -### Why `get()` reads before it checks - -`get()` reads the slot value **first**, then — after an `Acquire` fence — loads `release` -and `write` and asserts `release ≤ idx < write`: - -```rust -let value = unsafe { *slot(idx) }; // read first -fence(Acquire); -let (release, write) = (load(release), load(write)); -assert!(release <= idx && idx < write); // validate after -value -``` + (never overwrites a readable slot) and the reader only ever takes a *copy*. -Checking *before* reading would be a TOCTOU race: the writer could advance `release` and -overwrite the slot between the check and the read. Reading first makes the check meaningful. -`release` is monotonic, so observing `release ≤ idx` *after* the read proves it was `≤ idx` -*throughout* the read — i.e. the writer had not been cleared to overwrite this slot, so the -value is not torn. If `idx` left the window, `get()` panics instead of returning stale data. +### Why reads need no per-slot window check -> A genuine concurrent overwrite (a caller that ignores the window) is still a data race; -> the in-`get()` check is a guard that turns most window violations into a panic rather than -> a silent stale/torn read, not a license to read out of range. +`walk` takes the `write_index` (`Acquire`) on entry and only reads `[from, write_index)`. Because +the single reader is also the sole release mover, the writer is gated on a release index that this +reader has already advanced past those slots — so an in-window slot is never overwritten mid-read, +and the read copies out by value with no per-entry fence-and-assert. The caller upholds the window +(`from >= released`), checked by a `debug_assert!`. Reading out of the window is a data race, not a +guarded panic. -> The whole module currently carries `#![allow(dead_code)]`: it is finished, tested -> infrastructure that is **not yet wired into the pipeline**. Remove the allow once a stage -> consumes it. +> The module still carries `#![allow(dead_code)]` because not every handle method has a caller yet +> (e.g. the test-only `get`); the ring itself **is** wired into the pipeline — the WAL stage holds +> the `TxRingReader` and the transactor holds the `TxRingWriter` (ADR-021). --- ## Tests `cargo test -p ledger --lib tx_ring` covers grant clamping, full-ring → zero grant, -release-then-reserve, monotonic-release panics, wrap-around, the read-window guards -(`get` at/above `write_index` and below `release_index` both panic), and a multi-threaded -multi-reader broadcast. See [tests.rs](tests.rs). +release-then-reserve, monotonic-release debug-asserts (release backward / past the write +cursor both panic), the power-of-two assert, in-window reads, wrap-around, `push`'s returned +ring index, half-open `walk` ranges, and `rollback_to`. See [tests.rs](tests.rs). diff --git a/docs/01-concepts.md b/docs/01-concepts.md index 5d3e1ae..dea508b 100644 --- a/docs/01-concepts.md +++ b/docs/01-concepts.md @@ -14,12 +14,14 @@ This has a concrete implication: if you lose your balances but keep your transac ## Accounts and Balances -An **account** is an identifier — a `u64` that represents a participant in the ledger. Accounts are not created explicitly; they come into existence the first time a transaction references them. +An **account** is an identifier — a `u64` that represents a participant in the ledger. Accounts must be opened explicitly via the `OpenAccount` operation before they can be used; a balance operation that touches an unopened account fails with `ACCOUNT_NOT_FOUND` ([ADR-022](./adr/0022-account-layouts-and-program-defined-accounts.md)). -A **balance** is the current state of an account, represented as an `i64`. Balances can be positive or negative. By default, roda-ledger protects accounts from going below zero — a transaction that would produce a negative balance is rejected. This protection can be intentionally bypassed via WASM-defined functions that explicitly check `get_balance` themselves (for example, to model overdraft accounts or internal system accounts). +A **balance** is the current state of an account, represented as an `i64`. Balances can be positive or negative. The built-in operations that *reduce* a user balance — `Withdrawal` and the sender side of a `Transfer` — guard against going below zero: if the account balance is less than the requested amount, the transaction is rejected with `INSUFFICIENT_FUNDS` before any entry is produced. This is an operation-level check, not a property of the balance store itself: the underlying `credit` / `debit` host calls use saturating arithmetic and enforce only account existence and the zero-sum invariant. A WASM-defined `Function` therefore does not inherit the funds check automatically — a module is free to drive a balance negative (e.g. to model overdraft or internal system accounts), or to implement its own `get_balance` check. **Account 0** is the system account. It serves as the source and sink for all money entering or leaving the ledger. A `Deposit` produces a debit on the user account (balance goes up) and a credit on account 0; a `Withdrawal` produces a credit on the user account (balance goes down) and a debit on account 0. The convention used throughout the engine and the WASM host API is **`debit` adds, `credit` subtracts** — see the [WASM Runtime guide](./wasm-runtime.md#host-api) for the host-call contract. This is not a stylistic choice — it is required by the zero-sum invariant described below. +**Account layouts.** An account is more than a bare balance ([ADR-022](./adr/0022-account-layouts-and-program-defined-accounts.md)). Each account cell carries the `i64` balance plus an eight-lane `flags` word. Lane 0 is the **status** lane and doubles as the existence marker — a zero status means the account does not exist, which is what lets `OpenAccount` and `ACCOUNT_NOT_FOUND` work. Beyond user accounts opened explicitly, a `Function` can create **program-defined sub-accounts**: it asks for the account linked to a parent under a type id, and the engine lazily allocates and links a new `PROGRAMMED` account on first reference. The remaining flag lanes are free for module-defined metadata, readable and writable from WASM. These mechanics live entirely behind the host API; the [WASM Runtime guide](./wasm-runtime.md) documents the verbs. + --- ## Operations and Transactions @@ -135,7 +137,9 @@ The window is defined by transaction count, not wall-clock time. This makes idem roda-ledger is designed around two dimensions of flexibility that set it apart from opinionated ledger systems. -**Programmable ledger via WebAssembly.** Beyond the built-in `Deposit`, `Withdrawal`, and `Transfer` operation types, the `Function` operation type is the single extension point for arbitrary multi-account logic. The caller uploads a compiled WebAssembly module, registers it under a name (a durable operation in its own right — see below), and from then on invokes it by name with up to eight `i64` parameters. The runtime exposes only three host calls — `credit`, `debit`, `get_balance` — so the module can move value and inspect balances but cannot break out of the ledger's invariants. This covers any financial logic that does not fit the named types: split payments, fee deductions, multi-leg settlements, or domain-specific balance rules. New transaction types can be added at runtime without recompiling or redeploying the engine, while keeping the same correctness, durability, and audit guarantees as the built-in operations. +**Programmable ledger via WebAssembly.** Beyond the built-in `Deposit`, `Withdrawal`, and `Transfer` operation types, the `Function` operation type is the single extension point for arbitrary multi-account logic. The caller uploads a compiled WebAssembly module, registers it under a name (a durable operation in its own right — see below), and from then on invokes it by name with up to eight `i64` parameters. The runtime exposes a small, deliberately narrow set of deterministic host calls — moving value (`credit`, `debit`, `get_balance`), managing account layouts (`linked_account`, `get_flag` / `has_flag` / `set_flag`), and reading and writing programmable state (`kv_get` / `kv_set`, plus the constant verbs described below) — so a module can express rich logic but cannot break out of the ledger's invariants or its determinism. This covers any financial logic that does not fit the named types: split payments, fee deductions, multi-leg settlements, or domain-specific balance rules. New transaction types can be added at runtime without recompiling or redeploying the engine, while keeping the same correctness, durability, and audit guarantees as the built-in operations. + +**Programmable state.** A registered function is not limited to account balances. The runtime exposes a single typed **key→value map** ([ADR-023](./adr/0023-programmable-state.md)) that a `Function` reads and mutates through the `kv_get` / `kv_set` host calls — a place to keep counters, rates, configuration, or any other state a transaction type needs. Functions stay stateless across calls (no module globals): named values a module depends on are declared once at registration time as **constants** and resolved by name at execution time. Every mutation is logged as an ordinary WAL record and replayed on recovery exactly like a balance entry, so programmable state is as durable and as deterministic as the balances themselves, and is never re-executed by a follower replaying the log. **Choose your guarantee level.** The staged pipeline exposes a dial between performance and consistency. The caller chooses how much to wait per submission: @@ -162,7 +166,7 @@ A `Function` operation is, from the ledger's perspective, indistinguishable from **Durable registration.** `RegisterFunction` is itself transactional. The call only returns after the binary is on disk, a `FunctionRegistered` record is committed to the WAL, and the handler is loaded into the live runtime. A subsequent `Operation::Function` is therefore guaranteed to see the new version. After a crash, recovery rebuilds the registry from a paired function snapshot plus the `FunctionRegistered` records that follow it — the live runtime always matches what the WAL says it should be. -**Determinism by construction.** The host API does not expose clocks, randomness, file or network I/O, threads, or atomics. Every legal function is a pure mapping from `(params, observed balances) → (status, credits, debits)`. This is what makes the runtime safe for future Raft replication: the leader executes the function, and followers apply the WAL entries it produced without ever re-running the WASM code. +**Determinism by construction.** The host API does not expose clocks, randomness, file or network I/O, threads, or atomics. Every legal function is a pure mapping from `(params, observed balances, programmable state) → (status, entries, state mutations)`. This is what makes the runtime safe for Raft replication: the leader executes the function, and followers apply the WAL entries it produced without ever re-running the WASM code. In short, the `Function` operation gives the caller a way to express *new* transaction shapes — fee splits, multi-leg settlements, conditional transfers, accounting templates — that behave as if they had been built into the engine from day one. @@ -176,8 +180,8 @@ Understanding the boundaries is as important as understanding the capabilities. **Not an authorization layer.** roda-ledger does not authenticate callers or enforce access control. This is the responsibility of the layer above it. -**Not a distributed system today.** roda-ledger is a single-node engine. There is no replication, no leader election, no multi-node coordination. This is a current limitation, not a design principle — Raft-based multi-node replication is planned. +**Runs single-node, with clustering under active development.** roda-ledger runs as a single node, and a single node is a complete, correct deployment. Multi-node operation is not deferred to the future: Raft consensus and streaming WAL replication are implemented across the `raft`, `cluster`, `client`, and `control` crates and are under active development. The pipeline already carries the machinery for it — a `WAIT_LEVEL_CLUSTER_COMMIT` wait level for quorum-durable submits, and a cluster-commit gate that holds back sealing until a transaction is quorum-committed. Treat clustering as an evolving capability rather than a finished, GA feature. **What it does have today.** A gRPC interface and a Docker image. roda-ledger is not only an embedded library — it can run as a standalone service. -**Planned additions.** Raft-based multi-node replication, mTLS authentication, and per-function CPU / memory metering for the WASM runtime. \ No newline at end of file +**Planned hardening.** Beyond the consensus and replication already in progress, the near-term roadmap includes mTLS authentication and per-function CPU / memory metering for the WASM runtime. \ No newline at end of file diff --git a/docs/02-api.md b/docs/02-api.md index c0d6f26..2b28128 100644 --- a/docs/02-api.md +++ b/docs/02-api.md @@ -35,8 +35,9 @@ max_connections = 1000 max_message_size_bytes = 4194304 # 4MB [ledger] -max_accounts = 1000000 -wait_strategy = "balanced" # low_latency | balanced | low_cpu +initial_account_size = 1000000 # starting account-array capacity; grows on demand +resize_factor = 0.75 # geometric growth factor when capacity is exceeded +wait_strategy = "balanced" # low_latency | balanced | low_cpu [ledger.storage] data_dir = "/data" @@ -68,7 +69,7 @@ use roda_ledger::ledger::{Ledger, LedgerConfig}; use roda_ledger::storage::StorageConfig; let config = LedgerConfig { - max_accounts: 1_000_000, + initial_account_size: 1_000_000, wait_strategy: WaitStrategy::Balanced, storage: StorageConfig { data_dir: "./data".to_string(), @@ -110,12 +111,22 @@ grpcurl -plaintext -d '{ grpcurl -plaintext -d '{ "transfer": {"from": 1, "to": 2, "amount": "300", "user_ref": "44"} }' localhost:50051 roda.ledger.v1.Ledger/SubmitOperation + +# OpenAccount — open 3 sequential accounts (read the id range back via GetTransaction) +grpcurl -plaintext -d '{ + "open_account": {"count": 3, "user_ref": "45"} +}' localhost:50051 roda.ledger.v1.Ledger/SubmitOperation + +# Function — invoke a registered WASM function (params padded/truncated to 8 i64) +grpcurl -plaintext -d '{ + "function": {"name": "my_fn", "params": ["1", "2"], "user_ref": "46"} +}' localhost:50051 roda.ledger.v1.Ledger/SubmitOperation ``` -Response: +Response (`term` is the leader term at reply time — `0` in single-node mode): ```json -{"transactionId": "42"} +{"transactionId": "42", "term": "0"} ``` **Rust library:** @@ -141,6 +152,15 @@ let tx_id = ledger.submit(Operation::Transfer { amount: 300, user_ref: 44, }); + +let tx_id = ledger.submit(Operation::OpenAccount { count: 3, user_ref: 45 }); + +// `params` is a fixed [i64; 8]; pass 0 for unused slots. +let tx_id = ledger.submit(Operation::Function { + name: "my_fn".to_string(), + params: [1, 2, 0, 0, 0, 0, 0, 0], + user_ref: 46, +}); ``` ### Operation Types @@ -151,7 +171,18 @@ let tx_id = ledger.submit(Operation::Transfer { **Transfer** — moves funds between two accounts atomically. Fails with `INSUFFICIENT_FUNDS` if the source account balance is insufficient. If `from == to`, the operation succeeds immediately as a no-op. -**Function** — invokes a registered WASM function by name with up to 8 `i64` parameters. See [WASM Runtime](./wasm-runtime.md) for the full guide on writing, registering, versioning, and invoking functions. +**OpenAccount** — opens `count` sequential accounts (account ids are allocated sequentially from 1; `count == 0` is treated as 1). The committed transaction carries an `AccountOpened` record with the first allocated id and the count — read it back via `GetTransaction`, or use the library's `open_accounts`, which returns the id range directly. + +**Function** — invokes a registered WASM function by name. The ABI is a fixed `8 × i64`: over gRPC, `params` is a `repeated int64` that the server zero-pads (short lists) or truncates (lists longer than 8) to exactly 8; in the Rust library, `params` is a `[i64; 8]` (pass `0` for unused slots). The guest entry point is `execute(i64 × 8) -> i32`. See [WASM Runtime](./wasm-runtime.md) for the full guide on writing, registering, versioning, and invoking functions. + +#### Account layouts / linked buckets + +Beyond the flat `u64 → i64` balances, programs can carve out **linked buckets** — child accounts attached to a parent under a program-defined `type_id` (e.g. a `HOLD` or `BONUS` lane). Every account also carries an 8-lane `flags` word (lane 0 is the status lane). `GetBalance` exposes this: + +- `include_linked` (request) — when set, the response's `linked` list carries one `LinkedBalance { type_id, balance }` per bucket linked under the queried account. +- `flags` (response) — the account's raw 8-lane flags word. + +Accounts whose status lane marks them `PROGRAMMED` hold program-internal money and are **not** directly queryable — `GetBalance` rejects them with a `failed_precondition` status. Account `0` (the system source/sink) stays observable. See [WASM Runtime](./wasm-runtime.md) for how programs open and manipulate these layouts. ### The `user_ref` Field @@ -160,6 +191,8 @@ Every operation carries a `user_ref` — a `uint64` supplied by the caller. It s - **Idempotency key** — when `user_ref > 0`, if the same `user_ref` appears within the active window — the deduplication cache uses a flip-flop (active + previous) keyed off `transaction_count_per_segment`, giving an effective window of N to 2N transactions — the second submission is detected as a duplicate. It is sequenced and recorded, but linked to the original via a `TxLinkRecord { kind: DUPLICATE }` rather than re-executed. This prevents double-processing on client retries. Deduplication is always on and cannot be disabled. Pass `user_ref = 0` to opt out of the idempotency check for individual transactions. - **Correlation reference** — stored in the WAL alongside the transaction. Use it to link to your own database record — an order ID, payment ID, or any external reference. +Separately, every transaction's WAL metadata also carries a `tag` — an 8-byte free-form field (rendered as trailing-null-trimmed UTF-8 when valid, otherwise 16-char lowercase hex). Unlike `user_ref`, `tag` plays no part in deduplication; it is a pure annotation surfaced when reading a transaction back. + --- ## Custom Operations (WASM Function) @@ -172,7 +205,7 @@ See the **[WASM Runtime guide](./wasm-runtime.md)** for the full story: writing ## Submit and Wait -Block until the operation reaches a specific pipeline stage. Returns the full result including whether the transaction was rejected and why. +Block until the operation reaches a specific pipeline stage, then return its transaction id (and, in cluster mode, the leader term). This does **not** return the entry-level outcome — to learn whether the transaction was rejected and why, either poll `GetTransactionStatus` afterwards or use [Submit and Wait (Result)](#submit-and-wait-result) below. **gRPC:** @@ -183,48 +216,107 @@ grpcurl -plaintext -d '{ }' localhost:50051 roda.ledger.v1.Ledger/SubmitAndWait ``` -Response: +Response (`term` is `0` in single-node mode): ```json -{"transactionId": "43", "failReason": 0} +{"transactionId": "43", "term": "0"} ``` **Rust library:** +`submit_and_wait` returns a `TransactionStatus` — the pipeline stage the transaction settled at. This is purely a stage indicator and carries no accept/reject signal: a rejected transaction still gets a metadata record written to the WAL for audit, so it advances through commit and snapshot like any other and reports `OnSnapshot`. To learn whether it was accepted, read the transaction back (`submit_and_wait_result`, below) and inspect its `fail_reason`. + ```rust use roda_ledger::transaction::WaitLevel; -let result = ledger.submit_and_wait( +let status = ledger.submit_and_wait( Operation::Deposit { account: 1, amount: 1000, user_ref: 46 }, WaitLevel::OnSnapshot, ); -if result.fail_reason.is_none() { - println!("committed: tx {}", result.tx_id); +if status == TransactionStatus::OnSnapshot { + println!("reached snapshot (durable + visible)"); } ``` **Wait levels:** -| Level | gRPC | Library | -|--------------|------------------------|-------------------------| -| Computed | `WAIT_LEVEL_COMPUTED` | `WaitLevel::Computed` | -| Committed | `WAIT_LEVEL_COMMITTED` | `WaitLevel::Committed` | -| On Snapshot | `WAIT_LEVEL_SNAPSHOT` | `WaitLevel::OnSnapshot` | +| Level | gRPC | Library | +|----------------|-----------------------------|-------------------------| +| Computed | `WAIT_LEVEL_COMPUTED` | `WaitLevel::Computed` | +| Committed | `WAIT_LEVEL_COMMITTED` | `WaitLevel::Committed` | +| On Snapshot | `WAIT_LEVEL_SNAPSHOT` | `WaitLevel::OnSnapshot` | +| Cluster Commit | `WAIT_LEVEL_CLUSTER_COMMIT` | — | + +`WAIT_LEVEL_COMPUTED` is the proto zero value — it is the level used by fire-and-forget `SubmitOperation` when `wait_level` is omitted. -**`WAIT_LEVEL_COMPUTED`** — The Transactor has executed the operation. Validation has run. If the operation violated any constraint (insufficient funds, zero-sum violation, etc.), `fail_reason` is set and the transaction is permanently rejected. If `fail_reason = 0`, the transaction is accepted and balance changes are live in memory — but not yet durable. A crash at this point would lose the transaction. +**`WAIT_LEVEL_COMPUTED`** — The Transactor has executed the operation. Validation has run. If the operation violated any constraint (insufficient funds, zero-sum violation, etc.), the transaction is permanently rejected: its balance changes are rolled back and a metadata record carrying the `fail_reason` is written for audit (read it via `GetTransaction`). The rejection is final, but the record still advances through commit and snapshot — the pipeline does not halt. If the operation succeeded, balance changes are live in memory but not yet durable; a crash at this point would lose the transaction. -**`WAIT_LEVEL_COMMITTED`** — The WAL Storer has flushed the transaction to disk. Durability is guaranteed. The transaction will survive a crash and be replayed on restart. Balance changes are not yet visible via `get_balance`. +**`WAIT_LEVEL_COMMITTED`** — The WAL Storer has flushed the transaction to disk. Local durability is guaranteed. The transaction will survive a crash and be replayed on restart. Balance changes are not yet visible via `get_balance`. **`WAIT_LEVEL_SNAPSHOT`** — The Snapshotter has applied the transaction to the balance cache. `get_balance` now reflects this transaction and all transactions before it. This is the linearizable read guarantee — any `get_balance` call after this point will see a consistent, fully settled balance. +**`WAIT_LEVEL_CLUSTER_COMMIT`** — The transaction is quorum-durable: replicated to a majority of cluster nodes. This is the strongest level and is meaningful only in cluster mode; in single-node mode it coincides with local commit. It is gRPC-only — the embedded `WaitLevel` enum has just the three local stages — and is reached via the `cluster_wait` flag on the `*_AndWaitResult` RPCs below. + To get a transaction ID without waiting for anything, use `SubmitOperation` / `submit` instead. +### Submit and Wait (Result) + +`SubmitAndWait` returns only an id. When you want the **committed transaction read back** — its entries, computed balances, and any links — use `SubmitAndWaitResult` (or `SubmitBatchAndWaitResult` for batches). These always wait at least for the snapshot stage and return the full record. + +A boolean `cluster_wait` flag selects the durability gate: `false` (default) waits for the local snapshot stage; `true` waits for `CLUSTER_COMMIT` (quorum-durable) before reading the result back. Because `CLUSTER_COMMIT` implies snapshot, the committed transaction is always present once the call returns. + +**gRPC:** + +```bash +# Wait for the local snapshot stage (cluster_wait omitted / false) +grpcurl -plaintext -d '{ + "deposit": {"account": 1, "amount": "1000", "user_ref": "47"} +}' localhost:50051 roda.ledger.v1.Ledger/SubmitAndWaitResult + +# Wait for quorum durability before returning the result +grpcurl -plaintext -d '{ + "deposit": {"account": 1, "amount": "1000", "user_ref": "48"}, + "cluster_wait": true +}' localhost:50051 roda.ledger.v1.Ledger/SubmitAndWaitResult +``` + +Response (a `CommitedTransaction` — see [Transaction Details](#transaction-details) for its shape): + +```json +{ + "transaction": { + "meta": {"txId": "44", "failReason": 0, "userRef": "47"}, + "items": [ + {"accountId": "1", "amount": "1000", "kind": "DEBIT", "computedBalance": "1000"}, + {"accountId": "0", "amount": "1000", "kind": "CREDIT", "computedBalance": "-1000"} + ] + }, + "term": "0" +} +``` + +**Rust library:** + +`submit_and_wait_result` waits for the snapshot stage and returns the `CommittedTransaction`; `submit_batch_and_wait_result` does the same for a batch. + +```rust +let tx = ledger.submit_and_wait_result( + Operation::Deposit { account: 1, amount: 1000, user_ref: 47 }, +); + +if tx.is_err() { + println!("rejected: {:?}", tx.get_fail_reason()); +} else { + println!("committed: tx {}", tx.tx_id()); +} +``` + --- ## Batch Operations -Submit multiple operations in a single round trip. Each operation is independent — failure of one does not affect others. There is no atomicity guarantee across operations in a batch. +Submit multiple operations in a single round trip. The whole batch is sequenced and executed as **one contiguous, in-order block** — no other client's transactions interleave between them. The batch is **not** atomic as a unit: a crash mid-batch keeps the operations already committed (it is not rolled back wholesale), and a rejection of one operation does not roll back the others. Each operation is individually atomic — it commits all-or-nothing on its own. **gRPC:** @@ -237,12 +329,14 @@ grpcurl -plaintext -d '{ ] }' localhost:50051 roda.ledger.v1.Ledger/SubmitBatch -# With wait: +# With wait — wait_level is set once at the request level and applies to the +# whole batch (the server waits for the last operation to reach it): grpcurl -plaintext -d '{ "operations": [ - {"deposit": {"account": 1, "amount": "1000", "user_ref": "53"}, - "wait_level": "WAIT_LEVEL_COMMITTED"} - ] + {"deposit": {"account": 1, "amount": "1000", "user_ref": "53"}}, + {"deposit": {"account": 2, "amount": "2000", "user_ref": "54"}} + ], + "wait_level": "WAIT_LEVEL_COMMITTED" }' localhost:50051 roda.ledger.v1.Ledger/SubmitBatchAndWait ``` @@ -255,14 +349,17 @@ let ops = vec![ Operation::Transfer { from: 1, to: 2, amount: 500, user_ref: 52 }, ]; -let results = ledger.submit_batch_and_wait(ops, WaitLevel::Committed); +// One TransactionStatus per operation, in submit order. +let statuses = ledger.submit_batch_and_wait(ops, WaitLevel::Committed); -for result in results { - println!("tx {}: {:?}", result.tx_id, result.fail_reason); +for status in statuses { + println!("{:?}", status); } ``` -Operations within a batch are sequenced in the order provided. However, other clients' transactions may be interleaved between them — there is no reservation of a contiguous block in the global order. +`submit_batch_and_wait` returns one `TransactionStatus` per operation, in submit order. To get the committed transactions read back (entries, computed balances, links), use `submit_batch_and_wait_result` / `SubmitBatchAndWaitResult` instead — see [Submit and Wait (Result)](#submit-and-wait-result). + +Operations within a batch occupy a contiguous run of transaction ids in the global order; the block is sequenced and executed in the order provided, with no other client's transactions interleaved. Prefer batch submission over repeated single calls for high-throughput ingestion — it reduces round trips significantly. @@ -284,6 +381,8 @@ grpcurl -plaintext -d '{"transaction_ids": ["42", "43", "44"]}' \ localhost:50051 roda.ledger.v1.Ledger/GetTransactionStatuses ``` +The gRPC `GetStatusRequest` also accepts an optional fencing `term` (pass `0` to skip the check). When set and the transaction's actual term differs, the response carries `term_mismatch = true` plus the superseding `term` and its `term_start_tx_id` — letting a cluster client detect that its transaction was lost to a leader change and redirect. + **Rust library:** ```rust @@ -297,8 +396,11 @@ let status = ledger.get_transaction_status(tx_id); | `PENDING` | Sequenced, not yet executed. Balance unchanged. Not durable. | | `COMPUTED` | Executed by Transactor. Balance updated in memory. Not yet durable. | | `COMMITTED` | Flushed to WAL. Durable. Survives a crash. `get_balance` does not yet reflect it. | -| `ON_SNAPSHOT` | Applied to balance cache. `get_balance` reflects it. Final successful state. | -| `ERROR` | Rejected. No balance changes. Check `fail_reason` for the cause. | +| `ON_SNAPSHOT` | Applied to balance cache. `get_balance` reflects it. Terminal stage. | +| `ERROR` | Defined in the proto enum but **not returned by this RPC** — see the note below. | +| `TX_NOT_FOUND` | The id was never sequenced on this node (too old, never existed, or — under a term fence — not covered by the supplied term). | + +> **Status is a stage, not a verdict.** `GetTransactionStatus` reports only the pipeline stage, and a rejected transaction still advances through it (its metadata record is written for audit). The status RPC therefore never returns `ERROR` — a rejected transaction reports `COMPUTED` / `COMMITTED` / `ON_SNAPSHOT` like any other. The `ERROR` enum value exists in the proto but is not emitted here; to tell a rejection from a success, read the transaction's `fail_reason` (via `GetTransaction` / `get_transaction_block`, or use `SubmitAndWaitResult`). The **embedded** `TransactionStatus` enum has no `Error` variant at all — its values are `NotFound` / `Pending` / `Computed` / `Committed` / `OnSnapshot`. **Rejection codes (`fail_reason`):** @@ -310,11 +412,16 @@ let status = ledger.get_transaction_status(tx_id); | `3` | `ZERO_SUM_VIOLATION` | Credits and debits in the transaction do not net to zero | | `4` | `ENTRY_LIMIT_EXCEEDED` | Transaction emitted more than 255 entries (per-transaction limit) | | `5` | `INVALID_OPERATION` | Operation is malformed or contains invalid parameters | -| `6` | `ACCOUNT_LIMIT_EXCEEDED` | Account ID exceeds `max_accounts` configuration | -| `7` | `DUPLICATE` | `user_ref` was seen within the dedup window — linked to the original transaction | -| `8–127` | — | Reserved for future standard reasons | +| `6` | — | Retired (formerly `ACCOUNT_LIMIT_EXCEEDED`) | +| `7` | `DUPLICATE` | (Not a rejection — see below.) `user_ref` seen within the dedup window; the duplicate is linked, not failed | +| `8` | `PROHIBITED_HOST_CALL` | A WASM module called a host verb prohibited in the current phase (e.g. a constant-registration verb from `execute`) | +| `9` | `CONSTANT_NOT_FOUND` | A WASM module looked up a constant name that was never registered | +| `10` | `CONSTANT_NAME_TOO_LONG` | A constant name exceeds the maximum byte length and cannot be stored | +| `11–127` | — | Reserved for future standard reasons | | `128–255` | — | User-defined custom reasons | +> **`DUPLICATE` is not an `ERROR`.** A transaction whose `user_ref` was seen within the dedup window is *not* rejected — it is sequenced, recorded, and linked to the original via a `TxLinkRecord { kind: DUPLICATE }` rather than re-executed. It does not surface as an `ERROR` status. Code `7` is reserved as the link/dedup discriminant, not a failure reason. + --- ## Reading Balances @@ -326,20 +433,34 @@ let status = ledger.get_transaction_status(tx_id); ```bash grpcurl -plaintext -d '{"account_id": 1}' \ localhost:50051 roda.ledger.v1.Ledger/GetBalance + +# Also return the account's flags word and per-type linked-bucket balances: +grpcurl -plaintext -d '{"account_id": 1, "include_linked": true}' \ + localhost:50051 roda.ledger.v1.Ledger/GetBalance ``` -Response: +Response (`linked` is present only when `include_linked` was set): ```json -{"balance": "1500", "lastSnapshotTxId": "42"} +{ + "balance": "1500", + "lastSnapshotTxId": "42", + "flags": "1", + "linked": [{"typeId": 1, "balance": "200"}] +} ``` +`flags` is the account's 8-lane flags word (lane 0 = status). See [Account layouts / linked buckets](#account-layouts--linked-buckets). Querying a `PROGRAMMED` bucket returns a `failed_precondition` error. + **Rust library:** ```rust let balance = ledger.get_balance(1); // balance.balance: i64 // balance.last_snapshot_tx_id: u64 + +let flags = ledger.get_flags(1); // u64, the 8-lane flags word +let linked = ledger.linked_balances(1); // Vec<(u16 type_id, Balance)> ``` ### Multiple Accounts @@ -393,6 +514,39 @@ break; --- +## Key-Value Store + +Beyond balances, the engine holds a programmable key-value state — keys and values written by WASM functions during execution and committed to the WAL alongside the transaction's entries. The read side is exposed for forward lookups. + +A KV **key** is an ordered path of typed components (a `WalKvKeyPath`), rendered as a `"/"`-joined string such as `products/123`. A **value** is either an interned constant string or an `i64`. Programs may also register **constants** — a `u32` key bound to a fixed string — which both keys and values can reference. Writing KV state and registering constants happen only from inside WASM functions; see [WASM Runtime](./wasm-runtime.md) for the host API. + +**gRPC — `GetKv`:** resolves the value for a key. Provide the key **either** as `key.str` (a `"/"`-joined path, parsed server-side) **or** as structured `key.items` — passing both is `INVALID_ARGUMENT`. + +```bash +grpcurl -plaintext -d '{"key": {"str": "products/123"}}' \ + localhost:50051 roda.ledger.v1.Ledger/GetKv +``` + +Response: + +```json +{"found": true, "value": {"integer": "42", "str": "42"}} +``` + +`found` is `false` (and `value` absent) when the key is unset. + +**Rust library:** + +```rust +// Forward KV lookup; returns None if the key is unset. +let value: Option = ledger.get_kv(key); + +// Resolve an interned constant name to its id; None if unregistered. +let id: Option = ledger.get_constant("usd"); +``` + +--- + ## Pipeline Index Observe the current progress of each pipeline stage. Useful for health checks, lag monitoring, and determining when a bulk load is complete. @@ -404,29 +558,36 @@ grpcurl -plaintext -d '{}' \ localhost:50051 roda.ledger.v1.Ledger/GetPipelineIndex ``` -Response: +Response (`term`, `clusterCommitIndex`, and `isLeader` are cluster fields — in single-node mode `term`/`clusterCommitIndex` are `0` and `isLeader` is always `true`): ```json { - "computedIndex": "10500", - "committedIndex": "10498", - "snapshotIndex": "10496" + "computeIndex": "10500", + "commitIndex": "10498", + "snapshotIndex": "10496", + "term": "0", + "clusterCommitIndex": "0", + "isLeader": true } ``` **Rust library:** ```rust -let computed = ledger.last_compute_id(); // processed by Transactor -let committed = ledger.last_commit_id(); // flushed to WAL +let computed = ledger.last_compute_id(); // executed by Transactor (in memory) +let written = ledger.last_write_id(); // written to the WAL page cache (pre-fsync) +let committed = ledger.last_commit_id(); // flushed (fsync'd) to WAL let snapshotted = ledger.last_snapshot_id(); // applied to balance cache ``` +`last_write_id` (the buffered, pre-fsync WAL position) has no field in the gRPC response — it is an embedded-only accessor. + **Common patterns:** -- **Health check** — all three indexes should be advancing. A stalled index indicates a stuck pipeline stage. -- **Commit lag** — `computed_index - committed_index`. High lag means WAL writes are falling behind. -- **Snapshot lag** — `committed_index - snapshot_index`. Should be near zero under normal load. +- **Health check** — the indexes should be advancing. A stalled index indicates a stuck pipeline stage. +- **Commit lag** — `compute_index - commit_index`. High lag means WAL writes are falling behind. +- **Snapshot lag** — `commit_index - snapshot_index`. Should be near zero under normal load. +- **Cluster lag** — `commit_index - cluster_commit_index`. How far local durability is ahead of quorum durability. - **Bulk load completion** — submit all operations, then poll until `snapshot_index >= last_tx_id`. --- @@ -442,76 +603,105 @@ grpcurl -plaintext -d '{"tx_id": "42"}' \ localhost:50051 roda.ledger.v1.Ledger/GetTransaction ``` -Response: +Response — a `CommitedTransaction { meta, items }`. `meta` is the transaction's closing metadata (`txId`, `failReason`, `userRef`, `tag`, `timestamp`); `items` is the list of follower `WalEntry` records that make up the transaction — balance entries, plus any links, account-layout records (`accountOpened` / `accountLinked` / `accountFlagsUpdated`), or KV records: ```json { - "txId": "42", - "entries": [ - {"accountId": "1", "amount": "1000", "kind": "DEBIT", "computedBalance": "1000"}, - {"accountId": "0", "amount": "1000", "kind": "CREDIT", "computedBalance": "-1000"} - ], - "links": [] + "transaction": { + "meta": {"txId": "42", "failReason": 0, "userRef": "42", "tag": ""}, + "items": [ + {"txEntry": {"accountId": "1", "amount": "1000", "kind": "DEBIT", "computedBalance": "1000"}}, + {"txEntry": {"accountId": "0", "amount": "1000", "kind": "CREDIT", "computedBalance": "-1000"}} + ] + } } ``` -Each entry shows the account, amount, direction (`CREDIT` or `DEBIT`), and the balance of that account immediately after the entry was applied. +Each balance entry shows the account, amount, direction (`CREDIT` or `DEBIT`), and the balance of that account immediately after the entry was applied. There is no separate top-level `txId` / `entries` / `links` — everything lives under `transaction.meta` and `transaction.items`. -**Links** connect related transactions: +**`link` items** connect related transactions (a `WalTxLink` carries `toTxId` and a `kind`): | Link kind | Meaning | |---|---| | `DUPLICATE` | This transaction is a duplicate of the linked one (same `user_ref` within dedup window) | | `REVERSAL` | This transaction reverses the linked one | +**Rust library:** `get_transaction_block(tx_id)` returns `Option` (`meta` + an `entries: Vec`); `get_transactions_block(&[tx_id])` fetches a batch. + --- ## Account History -Retrieve the transaction history for an account, newest first, with pagination. +Retrieve the transaction history for an account, newest first. The request is a transaction-id **range**, not a count: the server scans backward from `from_tx_id` (`0` = latest) and stops below `to_tx_id` (the oldest id to include; `0` = scan to the start of the WAL). It returns the matching transactions plus `scan_last_tx_id` — the oldest id the scan reached. To page further into the past, re-query with `from_tx_id = scan_last_tx_id`. **gRPC:** ```bash -# First page -grpcurl -plaintext -d '{"account_id": 1, "from_tx_id": 0, "limit": 20}' \ +# Most recent transactions touching account 1, back to the WAL start +grpcurl -plaintext -d '{"account_id": 1, "from_tx_id": 0, "to_tx_id": 0}' \ localhost:50051 roda.ledger.v1.Ledger/GetAccountHistory -# Next page — pass next_tx_id from previous response -grpcurl -plaintext -d '{"account_id": 1, "from_tx_id": "38", "limit": 20}' \ +# Next page — resume from the previous response's scan_last_tx_id +grpcurl -plaintext -d '{"account_id": 1, "from_tx_id": "38", "to_tx_id": 0}' \ localhost:50051 roda.ledger.v1.Ledger/GetAccountHistory ``` -Response: +Response — `transactions` is a list of `CommitedTransaction` (same shape as [Transaction Details](#transaction-details)), newest first: ```json { - "entries": [ - {"accountId": "1", "amount": "500", "kind": "CREDIT", "computedBalance": "500"}, - ... + "transactions": [ + { + "meta": {"txId": "42", "failReason": 0}, + "items": [ + {"txEntry": {"accountId": "1", "amount": "500", "kind": "CREDIT", "computedBalance": "500"}} + ] + } ], - "nextTxId": "38" + "scanLastTxId": "38" } ``` -`next_tx_id = 0` means there are no more entries. Pass `from_tx_id = 0` to start from the latest transaction. Default limit is `20`, maximum is `1000`. +**Rust library:** `get_account_history(account_id, from_tx_id, to_tx_id)` returns an `AccountHistory { transactions, scan_last_tx_id }`. Use cases: account statements, balance verification, dispute resolution, reconciliation. --- +## Other RPCs + +The `Ledger` service exposes several more RPCs not detailed above. Most have a (sometimes thinner) embedded equivalent on `Ledger`, noted per entry; the cluster-only RPCs do not. + +**`WaitForTransaction`** — blocks until the transaction reaches a given `wait_level`, or until its outcome is otherwise known. The response is a `WaitOutcome`: `REACHED`, `NOT_FOUND` (unknown id), or `TERM_MISMATCH` (the term changed before commit and the transaction was lost — the response carries the superseding `term` / `term_start_tx_id`). An optional request `term` fences the wait (`0` = no check). Embedded: `wait_for_transaction_level`. + +**WASM function registry:** + +- **`RegisterFunction`** — registers a WASM binary under a name (`snake_case`, max 32 bytes; the binary must export `execute(i64 × 8) -> i32`). Returns the new per-name `version` (monotonic from 1) and the binary's `crc32c`. With `override_existing = false`, re-registering a name returns `ALREADY_EXISTS`. Blocks until the runtime reflects the new handler, so a subsequent `Function` op sees it. Embedded: `register_function(name, binary, override_existing)`. +- **`UnregisterFunction`** — removes a handler; later `Function` ops on the name fail with `INVALID_OPERATION`. Returns the version stamped on the unregister record. Embedded: `unregister_function(name)`. +- **`ListFunctions`** — lists registered functions as `FunctionInfo { name, version, crc32c }`. Embedded: `list_functions()`. + +See [WASM Runtime](./wasm-runtime.md) for the full registration / versioning story. + +**`GetLog`** — reads raw `WalEntry` records over the range `[from_tx_id, to_tx_id]` from this node's **local** WAL (never returns uncommitted bytes). This is the count-paginated reader: `limit` defaults to `1000` with a server hard-cap of `10000`, and the response's `next_tx_id` is the cursor for the next page (`0` = no more records in range on this node). The response also carries `last_commit_tx_id`, the node's commit watermark at read time. + +**`GetTerms`** (cluster only) — returns this node's Raft term boundaries (`term.log`) and vote decisions (`vote.log`) as `TermInfo` rows, paginated by `from_term` / `limit` with a `next_term` cursor. A term may appear in only one log — check `has_term_record` / `has_vote_record`. + +Operational and diagnostic RPCs (e.g. latency probing) live on the cluster side and are feature-gated — they are not part of this client-facing `Ledger` API surface. + +--- + ## Library Extras These methods are available in the Rust library only and have no gRPC equivalent. -**`wait_for_transaction(tx_id)`** — blocks until the given transaction reaches `ON_SNAPSHOT`. Times out after 10 seconds. +**`wait_for_transaction(tx_id)`** — blocks until the given transaction reaches `ON_SNAPSHOT`. Times out after 100 seconds. **`wait_for_transaction_until(tx_id, duration)`** — same as above with a custom timeout. -**`wait_for_transaction_level(tx_id, level)`** — blocks until the given transaction reaches the specified `WaitLevel`. Returns immediately on rejection regardless of wait level. +**`wait_for_transaction_level(tx_id, level)`** — blocks until the given transaction reaches the specified `WaitLevel` (times out after 10 seconds). Returns once a rejected transaction settles at its terminal stage. **`wait_for_pass()`** — blocks until the last submitted transaction reaches `ON_SNAPSHOT`. Useful after a bulk load to confirm everything is visible. -**`get_rejected_count()`** — returns the total number of rejected transactions since startup. Useful for metrics and health monitoring. +**`open_accounts(count)`** — opens `count` accounts and returns an `OpenAccountsResult { tx_id, fail_reason, begin_account_id, count }` with the allocated id range (the gRPC path uses the `OpenAccount` operation plus `GetTransaction` to read the range back). **`query(request)` / `query_block(request)`** — low-level access to the Snapshot stage query queue. `query` is non-blocking; `query_block` blocks until the Snapshot stage processes the request. Used for advanced read patterns not covered by `get_balance`. \ No newline at end of file diff --git a/docs/03-architecture.md b/docs/03-architecture.md index de11ab1..53cb91f 100644 --- a/docs/03-architecture.md +++ b/docs/03-architecture.md @@ -134,7 +134,7 @@ Stages 5–7 are the atomicity boundary. Nothing is published to the transaction - **Single-writer correctness.** A function executes on the Transactor thread, in strict sequence, with the same in-memory balance cache native operations use. Two functions cannot race; a function cannot race with a native operation. - **No new persistence path.** Function-produced entries are normal `TxEntry` records. WAL segmentation, sealing, snapshotting, and replay treat them identically to entries from a `Transfer`. - **No new failure mode.** A WASM trap, an unbalanced credit/debit set, or a domain-specific reject all use the existing transaction-rollback machinery. -- **Determinism for replication.** The host API is intentionally narrow: no clocks, no randomness, no I/O, no atomics, no threads. The leader executes a function; any future Raft follower can apply the resulting WAL entries directly without re-running the WASM code. +- **Determinism for replication.** The host API is intentionally narrow: no clocks, no randomness, no I/O, no atomics, no threads. The leader executes a function; a Raft follower applies the resulting WAL entries directly without re-running the WASM code. ### Registration as a first-class WAL event @@ -170,7 +170,7 @@ The WAL runs as **two concurrent threads** communicating through shared atomics: - **WAL Writer** — reads published records from the transaction ring, writes them to the active segment file, advances `last_written_tx_id`, and rotates segments when full. - **WAL Committer** — runs independently, calls `fdatasync` whenever `last_written_tx_id > last_committed_tx_id`, then advances `last_committed_tx_id` (which the pipeline exposes as `commit_index`). -The Writer never blocks waiting for `fdatasync` — it continues writing while the Committer syncs. The WAL forwards nothing: the Snapshotter reads the same ring independently and gates itself on `commit_index`, so a transaction becomes visible to readers only after it is durable. This decoupling is why the WAL sustains high write throughput despite the inherent latency of `fdatasync` (~100s µs, disk-bound). +The Writer never blocks waiting for `fdatasync` — it continues writing while the Committer syncs. The Snapshotter does not read the ring: it tails the durable WAL and gates itself on `commit_index`, so a transaction becomes visible to readers only after it is durable ([ADR-021](./adr/0021-wal-sole-releaser-snapshot-tails-wal.md)). This decoupling is why the WAL sustains high write throughput despite the inherent latency of `fdatasync` (~100s µs, disk-bound). The WAL is **segmented** — divided into files based on transaction count (`transaction_count_per_segment`). When the transaction count in the active segment reaches the configured limit, the Writer rotates to a new one. Segment files are dynamically sized on disk — a segment with many complex (multi-entry) transactions will be larger than one with simple deposits — but the transaction count per segment is always fixed and predictable. Sealed segments are complete, consistent units used for recovery. @@ -238,7 +238,7 @@ The Snapshotter draws from two independent sources: -- **The transaction ring** — it reads published records, buffering a transaction's entries until the trailing metadata arrives and its `tx_id` is at or below `commit_index` (the durability gate). Only then does it apply the entries to the indexes and balance cache together, advance `snapshot_index`, and — as the ring's sole releaser — reclaim the consumed slots. Readers therefore never see a partially-applied or not-yet-durable transaction. +- **The durable WAL** — it tails the WAL rather than the transaction ring ([ADR-021](./adr/0021-wal-sole-releaser-snapshot-tails-wal.md)), buffering a transaction's entries until the trailing metadata arrives and its `tx_id` is at or below `commit_index` (the durability gate). Only then does it apply the entries to the indexes and balance cache together and advance `snapshot_index`. Because it reads from already-durable storage, it never sees a partially-applied or not-yet-durable transaction — and it no longer touches the ring or its reclamation point. - **A query-only queue** — `GetTransaction` / `GetAccountHistory` requests execute inline against current state and call the response callback. The Snapshotter does no computation — balances are pre-computed by the Transactor and stored in entries. The gap between `COMMITTED` and `ON_SNAPSHOT` is typically nanoseconds. @@ -247,11 +247,11 @@ The Snapshotter does no computation — balances are pre-computed by the Transac ## The transaction ring -The Transactor, WAL, and Snapshotter share a single lock-free **transaction ring** — the one transport for the record stream, replacing the former transactor→WAL and WAL→snapshot queues ([ADR-019](./adr/0019-transaction-ring.md)). +The Transactor and the WAL share a single lock-free **transaction ring** — the transport that carries the record stream from execution to durability, replacing the former transactor→WAL queue ([ADR-019](./adr/0019-transaction-ring.md)). The Snapshotter no longer sits on the ring: it tails the durable WAL instead ([ADR-021](./adr/0021-wal-sole-releaser-snapshot-tails-wal.md)). - **One producer.** The Transactor appends records into ring slots and publishes them. It also builds each transaction *in place* in the uncommitted region, so the ring doubles as its per-transaction scratch space — no separate staging buffer. -- **Independent copy-out readers.** The WAL and Snapshotter each track their own absolute cursor and copy records out at their own pace. A slot may be reused once released, so readers copy rather than borrow. -- **One releaser, durability-gated.** The Snapshotter is the only party that advances the reclamation point, and only up to the durability watermark. A slot is reused only after the record it held is durable on disk *and* consumed by every reader — making "never overwrite an undurable record" a structural property of the transport. +- **One reader.** The WAL is the only consumer of the ring. It tracks an absolute cursor and copies records out into its active segment as fast as it can write them. +- **One releaser, write-gated.** The WAL is also the sole releaser: it advances the reclamation point right after it has written a batch into the active segment's page cache, *not* after `fdatasync` ([ADR-021](./adr/0021-wal-sole-releaser-snapshot-tails-wal.md)). Decoupling release from the commit fence keeps the writer from blocking on the disk while still back-pressuring the Transactor whenever writes stall. Durability is enforced downstream by the independent WAL Committer, and the Snapshotter — which reads the durable WAL — never gets ahead of `commit_index`. Positions are absolute, monotonically increasing indices (mapped to physical slots on access), so progress and ordering stay unambiguous across wraps. Because a transaction is published atomically, the ring capacity (`ring_size`) must be at least the largest possible single transaction. @@ -261,7 +261,7 @@ Positions are absolute, monotonically increasing indices (mapped to physical slo The record stream rides the transaction ring described above; the **submit and query paths** use lock-free **SPSC queues** (`sequencer → transactor`, and the Snapshotter's query queue) — one producer, one consumer, no locks. Each stage owns its data completely. -**Backpressure** propagates naturally. The Snapshotter releases ring slots only up to the durability watermark, so if durability or indexing stalls, the ring fills and the Transactor stops finding free slots; that stalls the Transactor, which fills the `sequencer → transactor` queue, which eventually stalls `submit()`. No explicit flow control is needed, and the producer never blocks inside the ring — it retries under the shared wait strategy. +**Backpressure** propagates naturally. The WAL releases ring slots only after writing them to the active segment, so if disk writes stall, the ring fills and the Transactor stops finding free slots; that stalls the Transactor, which fills the `sequencer → transactor` queue, which eventually stalls `submit()`. No explicit flow control is needed, and the producer never blocks inside the ring — it retries under the shared wait strategy. The wait strategy under backpressure is controlled by the `wait_strategy` config field (see [API → Setup](./02-api.md#grpc-server)): `low_latency` (spin forever), `balanced` (spin → yield → park), `low_cpu` (park quickly). @@ -282,9 +282,9 @@ Because every committed transaction is in the WAL, and the WAL is always replaye ## Design boundaries -**Single node.** All pipeline stages run on one machine. Raft-based multi-node replication is planned — the segmented, append-only WAL is a natural fit for log replication. +**Single node is the zero-peer case.** All pipeline stages run on one machine, and a single node is a complete deployment. Cluster mode — Raft consensus plus streaming WAL replication — is implemented and under active development across the `raft`, `cluster`, `client`, and `control` crates; the segmented, append-only WAL is a natural fit for log replication. See [ADR-0015](./adr/0015-cluster-mode.md), [ADR-0016](./adr/0016-leader-election.md), [ADR-0017](./adr/0017-roda-raft.md), and [ADR-0027](./adr/0027-reactive-index-propagation.md). -**Pre-allocated account space.** Balances are stored in a structure sized to `max_accounts`. O(1) reads and writes always, but memory is committed at startup — a capacity planning decision. +**Growable account array.** Balances live in a flat `Vec` of account cells indexed directly by account id, so reads and writes are O(1). There is no fixed `max_accounts`: the array starts at `initial_account_size` and grows geometrically (by `resize_factor`) on demand as `OpenAccount` and program-defined sub-accounts allocate new ids — no startup capacity planning ([ADR-022](./adr/0022-account-layouts-and-program-defined-accounts.md)). Each cell holds the `i64` balance plus an 8-lane `flags` word (lane 0 = status/existence). **Disk-bound durability throughput.** `fdatasync` latency determines `COMMITTED` throughput. On NVMe this is hundreds of µs per batch. `COMPUTED` throughput (in-memory only) reaches millions of transactions per second. diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index b7d5521..8d1bd6a 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -25,12 +25,18 @@ - [ADR-0008 Transaction account index](adr/0008-transaction-account-index.md) - [ADR-0009 Tx link dedup reversal](adr/0009-tx-link-dedup-reversal.md) - [ADR-0010 Sync submit](adr/0010-sync-submit.md) + - [ADR-0011 WAL write/commit separation](adr/0011-wal-write-commit-separation.md) + - [ADR-0012 E2E testing strategy](adr/0012-e2e-testing.md) + - [ADR-0013 Transaction-count-based segments](adr/0013-transaction-count-based-segments.md) - [ADR-0014 WASM function registry](adr/0014-wasm-function-registry.md) + - [ADR-0015 Cluster mode](adr/0015-cluster-mode.md) + - [ADR-0016 Leader election](adr/0016-leader-election.md) + - [ADR-0017 roda-raft](adr/0017-roda-raft.md) - [ADR-0018 Fault injection framework](adr/0018-fault-injection.md) - [ADR-0019 Transaction ring](adr/0019-transaction-ring.md) - [ADR-0020 Trailer metadata — commit-record WAL layout](adr/0020-wal-trailer-metadata.md) - [ADR-0021 WAL sole ring releaser; snapshot tails WAL](adr/0021-wal-sole-releaser-snapshot-tails-wal.md) - [ADR-0022 Account layouts and program-defined accounts](adr/0022-account-layouts-and-program-defined-accounts.md) - - [ADR-0024 Temporal index](adr/0024-temporal-index.md) - - [ADR-0025 Point-in-time](adr/0025-point-in-time.md) - - [ADR-0026 roda-wasm-abi](adr/0026-roda-wasm-abi.md) \ No newline at end of file + - [ADR-0023 Programmable state](adr/0023-programmable-state.md) + - [ADR-0026 roda-wasm-abi](adr/0026-roda-wasm-abi.md) + - [ADR-0027 Reactive index propagation](adr/0027-reactive-index-propagation.md) \ No newline at end of file diff --git a/docs/adr.md b/docs/adr.md index bca7f08..2e1aab6 100644 --- a/docs/adr.md +++ b/docs/adr.md @@ -1,16 +1,3 @@ # Architecture Decision Records -This section contains Architecture Decision Records (ADRs) documenting significant design choices made during the development of Roda-Ledger. - -| ADR | Title | Status | -|------------------------------------------------------|------|--------| -| [ADR-001](adr/0001-entries-based-execution-model.md) | Entries-Based Execution Model | Accepted | -| [ADR-002](adr/0002-vec-based-balance-storage.md) | Vec-Based Balance Storage | Accepted | -| [ADR-003](adr/0003-ledger-api-redesign.md) | Ledger API Redesign | Accepted | -| [ADR-004](adr/0004-grpc-interface.md) | gRPC External Interface | Accepted | -| [ADR-005](adr/0005-pressure-manager.md) | Adaptive Pipeline Execution Mode | Postponed | -| [ADR-006](adr/0006-wal-snapshot-durability.md) | WAL, Snapshot, and Seal Durability | Accepted | -| [ADR-007](adr/0007-cli-tools.md) | CLI Operational Tools | Proposed | -| [ADR-008](adr/0008-transaction-account-index.md) | Transaction Index and Query Serving | Accepted | -| [ADR-009](adr/0009-tx-link-dedup-reversal.md) | Transaction Links, Deduplication, and Reversal | Accepted | -| [ADR-010](adr/0010-sync-submit.md) | Sync Submit | Accepted | +The canonical ADR index lives at [`adr/README.md`](adr/README.md). diff --git a/docs/adr/0008-transaction-account-index.md b/docs/adr/0008-transaction-account-index.md index 8580b0a..7f5fff7 100644 --- a/docs/adr/0008-transaction-account-index.md +++ b/docs/adr/0008-transaction-account-index.md @@ -6,6 +6,14 @@ --- +> **⚠️ Superseded in part (2026-06).** The in-memory account-history design described +> below — the `account_heads` table and the `circle2` `prev_link` chain — was later +> removed. `GetTransaction` is still served from the `circle1`/`circle2` indexer, but +> **account history is now a backward WAL scan** (`Ledger::get_account_history` → +> `wal_scanner().scan`; no `account_heads`, no `prev_link`). This ADR is retained as the +> original decision record; for the current design see `docs/internal.md` §9–§10.3 and +> `crates/ledger/src/index.rs`. A follow-up ADR should formally amend this one. + ## Context roda-ledger currently has no way to retrieve a specific transaction by ID or retrieve diff --git a/docs/adr/0015-cluster-mode.md b/docs/adr/0015-cluster-mode.md index f395a55..7293249 100644 --- a/docs/adr/0015-cluster-mode.md +++ b/docs/adr/0015-cluster-mode.md @@ -3,6 +3,7 @@ **Status:** Proposed **Date:** 2026-04-20 **Last-Updated:** 2026-05-26 +**Author:** Taleh Ibrahimli ## 2026-05-26 Decisions diff --git a/docs/adr/0017-roda-raft.md b/docs/adr/0017-roda-raft.md index 1f6f0b4..f7d2d52 100644 --- a/docs/adr/0017-roda-raft.md +++ b/docs/adr/0017-roda-raft.md @@ -1,6 +1,9 @@ -# roda-raft Design +# ADR-017: roda-raft Design +**Status:** — +**Date:** 2026-05-26 **Last-Updated:** 2026-05-26 +**Author:** Taleh Ibrahimli ## 2026-05-26 Decisions diff --git a/docs/adr/0022-account-layouts-and-program-defined-accounts.md b/docs/adr/0022-account-layouts-and-program-defined-accounts.md index 8f25e8f..bae8385 100644 --- a/docs/adr/0022-account-layouts-and-program-defined-accounts.md +++ b/docs/adr/0022-account-layouts-and-program-defined-accounts.md @@ -2,7 +2,7 @@ **Status:** Proposed **Date:** 2026-06-08 -**Author:** Taleh +**Author:** Taleh Ibrahimli **Amends:** - ADR-001 — Entries-Based Execution Model: adds three account-layout WAL record kinds; transactor rollback extends to account-layout mutations. diff --git a/docs/adr/README.md b/docs/adr/README.md index edc6fb9..762cbf1 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -26,8 +26,8 @@ This section contains Architecture Decision Records (ADRs) documenting significa | [ADR-020](0020-wal-trailer-metadata.md) | Trailer Metadata — Commit-Record WAL Layout | Accepted | | [ADR-021](0021-wal-sole-releaser-snapshot-tails-wal.md) | WAL Sole Ring Releaser; Snapshot Tails the WAL | Accepted | | [ADR-022](0022-account-layouts-and-program-defined-accounts.md) | Account Layouts and Program-Defined Accounts | Proposed | -| [ADR-023](0023-programmable-state.md) | Programmable State — Typed KV Stores and the WASM State ABI | Proposed | -| [ADR-024](0024-temporal-index.md) | Temporal Index | Proposed | -| [ADR-025](0025-point-in-time.md) | Point-in-Time | Proposed | +| [ADR-023](0023-programmable-state.md) | Programmable State — Typed KV Store and the WASM State ABI | Proposed | | [ADR-026](0026-roda-wasm-abi.md) | roda-wasm-abi — Guest-Side ABI Crate | Proposed | -| [ADR-027](0027-reactive-index-propagation.md) | Reactive Index Propagation — One Ledger Index Hook Drives Replication and Waiters | Proposed | \ No newline at end of file +| [ADR-027](0027-reactive-index-propagation.md) | Reactive Index Propagation — One Ledger Index Hook Drives Replication and Waiters | Proposed | + +> ADR numbers 024–025 are unused (skipped); the sequence jumps from 023 to 026. \ No newline at end of file diff --git a/docs/internal.md b/docs/internal.md index d02376a..fdffcf8 100644 --- a/docs/internal.md +++ b/docs/internal.md @@ -121,10 +121,11 @@ wrapping is purely for type-level encapsulation; there is no runtime cost. ### §2.4 Inter-stage transport The record stream rides a single lock-free **transaction ring** (§13.4): the -Transactor is the sole producer; the WAL and Snapshotter are independent -copy-out readers; the Snapshotter is the sole releaser. The former -`transactor → wal` and `wal → snapshot` queues are gone — both stages now read -the same ring ([ADR-0019]). +Transactor is the sole producer and the **WAL is the sole consumer** — it both +walks the ring and releases the slots it has consumed ([ADR-0021]). The +Snapshotter no longer touches the ring at all; it tails the *durable WAL* via a +`WalTailer` (§8.4). The former `transactor → wal` and `wal → snapshot` queues +are gone ([ADR-0019]). Two lock-free, fixed-capacity, single-producer / single-consumer queues remain, for the paths that are not the record stream: @@ -132,20 +133,31 @@ remain, for the paths that are not the record stream: - `sequencer → transactor` — submitted operations awaiting execution. - `snapshot query` — `GetTransaction` / `GetAccountHistory` requests (§8.4). -Queue capacity is fixed at construction from `queue_size` (§15.2); ring -capacity from `ring_size` (§15.10). A full queue or a full ring is the only -backpressure signal; producers spin/yield until space exists. There is no +Queue capacity is a fixed internal constant (`1024`, §15.2); ring +capacity comes from `ring_size` (§15.10). A full queue or a full ring is the +only backpressure signal; producers spin/yield until space exists. There is no other flow-control mechanism. ### §2.5 Global progress indexes -The pipeline holds five monotonic atomic counters and one shutdown flag: +The pipeline holds six monotonic progress counters plus a seal-progress +counter, a cluster seal gate, and a shutdown flag: - `sequencer_index` — next `tx_id` to be handed out. - `compute_index` — last `tx_id` executed by the Transactor. -- `commit_index` — last `tx_id` durably written by the WAL. +- `write_index` — last `tx_id` the WAL Writer wrote to the active segment's + *page cache* — buffered only, **not yet fsynced** (that gate is + `commit_index`). It is the high-water mark the ring's release is keyed to + (§13.5) and lets callers observe pipeline transit minus the fsync. +- `commit_index` — last `tx_id` *durably* written by the WAL (post-`fdatasync`). - `snapshot_index` — last `tx_id` reflected in the Snapshotter. -- `seal_index` — last segment id sealed. +- `seal_index` — last segment id sealed (a `u32` segment id, not a `tx_id`). +- `seal_step_id` — monotonic counter the seal stage bumps each pass, so + `wait_for_seal` can observe progress without holding the `Seal`. +- `seal_watermark` — cluster-commit seal gate (§34, [ADR-0016 §10]): a segment + may be sealed only once every transaction it contains is `≤ seal_watermark`. + Defaults to `u64::MAX` ("no gate") for standalone ledgers; the cluster + supervisor drives it from the raft cluster-commit index. - `running` — global shutdown flag. The indexes are padded to separate cache lines so that a publication on one @@ -158,10 +170,10 @@ threshold check on a single counter resolves any wait. ### §2.6 Backpressure and shutdown Backpressure is implicit and propagates through ring and queue saturation: a -slow Snapshotter stops releasing ring slots (it releases only up to the -durability watermark, §13.5), so the ring fills and the Transactor stops -finding free slots; the stalled Transactor fills the sequencer→transactor -queue, which stalls `submit()`. There is no rate limiter, leaky bucket, or +slow WAL Writer (the ring's sole consumer) stops releasing ring slots (it +releases right after the page-cache `write`, §13.5), so the ring fills and the +Transactor stops finding free slots; the stalled Transactor fills the +sequencer→transactor queue, which stalls `submit()`. There is no rate limiter, leaky bucket, or admission control. Shutdown is the inverse signal: `Pipeline::shutdown()` clears `running`; every stage's idle loop checks the flag and exits. The shutdown signal does not need to be observed @@ -231,19 +243,25 @@ mutate the same state through interior mutability with no contention. ### §4.3 Balance cache layout The Transactor's balance cache is a flat array indexed directly by -`account_id`. Lookups and updates are O(1) array accesses with no hashing, -no probing, no resizing. The array is pre-allocated to `max_accounts` at -construction and never grows. [ADR-0002] - -### §4.4 The `max_accounts` ceiling - -Any operation referencing an `account_id ≥ max_accounts` is rejected with -`ACCOUNT_LIMIT_EXCEEDED` before the operation can mutate any state. -This is the price of the direct-indexed array layout (§4.3) — the -account-ID space must be sized at startup. Resizing would require pausing -the Transactor and reallocating every read-side cache (§8.3), which is not -worth the complexity given how rarely a deployed ledger needs to grow its -ceiling. [ADR-0002] +`account_id`. Lookups and updates are O(1) array accesses with no hashing +and no probing. The array is seeded to `initial_account_size` at construction +and **grows on demand** as higher account ids are referenced — it is not a +fixed ceiling. [ADR-0002, ADR-0022] + +### §4.4 The growable account array + +There is no `max_accounts` ceiling. When an operation references an +`account_id` beyond the current array length, the Transactor grows the array +before mutating any state: `grow_capacity` computes the next capacity +geometrically — `new_cap = ceil(cap × (1 + resize_factor))`, clamped up to +cover the requested id — and the array is resized in place, defaulting the new +slots. Growth is amortised O(1); the geometric factor keeps reallocations rare +even under a sparse id space. The `u64` id space is effectively inexhaustible +(`u32::MAX` accounts already exceeds 50 GB of state), so exhaustion is treated +as a bug and the allocator panics rather than returning a status. The retired +`ACCOUNT_LIMIT_EXCEEDED` reason (former status `6`) no longer exists. The same +geometric growth is applied symmetrically to the read-side cache (§8.3) and the +seal-side balance vector. [ADR-0002, ADR-0022] ### §4.5 Linearizability of the write path @@ -647,26 +665,31 @@ There is no other coordination — no condvars, no channels. The Writer copies records out of the ring into a bounded pending-write buffer, batched so a whole iteration's worth of records reaches disk in one `write` syscall. The buffer is sized in advance — never resized — so the WAL stage's -memory footprint is fixed. It forwards records nowhere: the Snapshotter reads -the ring independently (§8.4). +memory footprint is fixed. As the **sole ring consumer**, the Writer releases +the ring slots it has copied out (§13.5) immediately after that buffered +`write`. It forwards records nowhere: the Snapshotter tails the durable WAL +independently (§8.4). ### §7.4 Writer per-iteration loop Each iteration the Writer copies the newly published ring records into the active segment's pending-write buffer, performs a single buffered `write` -syscall (one per iteration, not per record), advances *last-written*, and -rotates the segment if the transaction-count threshold is met (§7.6). It does -not gate or forward anything to the Snapshotter — durability visibility is the -Snapshotter's own concern (§7.5). As records are written and the Committer -syncs them, `commit_index` (§2.5) advances. +syscall (one per iteration, not per record), advances *last-written* +(`write_index`, §2.5), **releases the consumed ring slots** (§13.5), and rotates +the segment if the transaction-count threshold is met (§7.6). The release +happens right after the page-cache `write`, gated on `write_index` — not on +`fdatasync` — so a stalled Writer back-pressures the Transactor while the fsync +stays decoupled. It does not gate or forward anything to the Snapshotter — +durability visibility is the Snapshotter's own concern (§7.5). As records are +synced by the Committer, `commit_index` (§2.5) advances. ### §7.5 Durability rule, enforced reader-side The durability rule still holds: the Snapshotter — and therefore `get_balance`, queries, and the snapshot wait level — never sees a transaction that is not yet on disk. But the WAL no longer enforces it by forwarding only durable records. -Instead the Snapshotter reads the ring and gates *itself*: it applies a -transaction only once the trailing `TxMetadata`'s `tx_id ≤ commit_index` +Instead the Snapshotter **tails the durable WAL** and gates *itself*: it applies +a transaction only once the trailing `TxMetadata`'s `tx_id ≤ commit_index` (§8.4). The Writer therefore keeps no per-transaction "expected record count" bookkeeping; it writes whatever the ring publishes, and a transaction is recognised as complete at its trailing metadata. A transaction may span a @@ -696,13 +719,16 @@ Writer's buffer, and never inspects any queue — its single responsibility is moving *last-committed* forward as fast as storage allows. -### §7.8 Backpressure from a slow reader +### §7.8 Backpressure from a slow WAL -There is no outbound queue to fill. A persistently slow Snapshotter stops -releasing ring slots (it releases only up to the durability watermark, §13.5), -so the ring eventually fills and the Transactor — the sole producer — stalls -for want of free slots (§2.6). The WAL Writer, an independent reader, is never -itself blocked by the Snapshotter; the two consume the ring at their own pace. +There is no outbound queue to fill. Because the WAL Writer is the ring's **sole +consumer and sole releaser** (§13.5), a persistently slow Writer stops releasing +ring slots, so the ring eventually fills and the Transactor — the sole producer +— stalls for want of free slots (§2.6). The Snapshotter is not on this path at +all: it tails the durable WAL (§8.4) rather than the ring, so a slow Snapshotter +can fall behind on read-side visibility but never throttles the ring or the +Transactor. Snapshotter slowness only bounds how stale `get_balance` is, not +ingest throughput. ### §7.9 Failure handling @@ -736,18 +762,25 @@ threads. ### §8.3 Read-side balance cache -The read-side cache is a flat array of atomic balances, sized to -`max_accounts`. The Snapshotter writes; many readers can load -concurrently without locks. The array is pre-allocated at startup and -never resized. [ADR-0002] +The read-side cache is a flat array of atomic balances, seeded to +`initial_account_size`. The Snapshotter writes; many readers can load +concurrently without locks. Like the write-side array (§4.4) it **grows on +demand** rather than sitting at a fixed ceiling: when an `AccountOpened` +follower covers an id beyond the current length, the Snapshotter grows the +array geometrically (`resize_factor`) via `ensure_capacity`, swapping in a +larger generation through the `ArcSwap` so concurrent readers never see a +torn vector. [ADR-0002, ADR-0022] -### §8.4 Ring entries and the query queue +### §8.4 The durable-WAL tailer and the query queue The Snapshotter draws from two independent sources, in a fixed order each iteration: -- **The transaction ring** — the published record stream (§13.4), buffering a - transaction's followers until the trailing `TxMetadata` (§8.5). +- **A `WalTailer` over the durable WAL** — *not* the transaction ring. The + tailer streams the on-disk segments forward from a cursor and, via + `tail_transactions`, groups each transaction's followers with its closing + `TxMetadata` for the apply step (zero-copy: a borrowed view, no owned `Vec`). + The ring is the WAL's private input (§2.4); the Snapshotter never reads it. - **A query-only SPSC queue** — `GetTransaction` / `GetAccountHistory` requests enqueued by callers (§10). @@ -761,15 +794,17 @@ issued without that wait races the apply loop — the same trade-off a ### §8.5 Apply algorithm -The Snapshotter walks the ring from its cursor, buffering a transaction's -`TxEntry` / `TxLink` followers. When the trailing `TxMetadata` arrives it checks -the durability gate: if `tx_id > commit_index` the transaction is not yet -durable, so the walk stops and resumes next iteration. Otherwise it applies the +The Snapshotter tails the durable WAL from its cursor; `tail_transactions` +hands it each transaction as its `TxEntry` / `TxLink` followers grouped with the +trailing `TxMetadata`. It first checks the durability gate: if +`meta.tx_id > commit_index` the transaction is not yet fsynced, so the tailer +retains it and the walk resumes once it commits. Otherwise it applies the buffered followers together — updating the hot index (§9.4) and publishing each `TxEntry`'s `computed_balance` to the read-side balance cache — then advances -`snapshot_index` (§2.5) to this `tx_id` and, as the ring's sole releaser, -reclaims the consumed slots (§13.5). Applying at the trailer is the *only* point -the read-side index advances, so readers never observe a partially-applied or +`snapshot_index` (§2.5) to this `tx_id`. The Snapshotter does **not** release +any ring slots: that is the WAL's job, as the ring's sole consumer (§13.5). +Applying at the trailer is the *only* point the read-side index advances, so +readers never observe a partially-applied or not-yet-durable transaction. `FunctionRegistered` takes the registry path of §8.7. @@ -809,24 +844,22 @@ thread. ## §9 Hot indexes — `TransactionIndexer` -### §9.1 Three buffers, all pre-allocated +### §9.1 Two buffers, all pre-allocated -The hot transaction index is a single in-memory structure with three +The hot transaction index is a single in-memory structure with two pre-allocated buffers, sized at construction and never resized: - **circle1** — for each `tx_id`, a pointer to that transaction's - entries in circle2. -- **circle2** — entry storage with per-account chain links. -- **account_heads** — for each account, a pointer to that account's - latest entry in circle2. + followers in circle2. +- **circle2** — follower storage: each transaction's followers (entries + and links alike) stored as raw `WalEntry` records, as-is. -A separate map keyed by `tx_id` holds link records (sparse — most -transactions have none — so a hash map is acceptable here). All three -array sizes must be powers of two. [ADR-0008] +Both array sizes must be powers of two. The index is a `tx_id`-keyed +cache only; account history is *not* served from here (§10.3). [ADR-0008] ### §9.2 Why power-of-two -All three lookups reduce to "id AND mask" to find a slot, where mask is +Both lookups reduce to "id AND mask" to find a slot, where mask is `size − 1`. The power-of-two requirement turns the lookup into a single bitwise operation. The cost is at most ~2× memory in the worst case (desired size just above a power of two); the alternative (modulo) @@ -845,49 +878,21 @@ eviction queue, no cleanup pass. Lookups detect eviction by comparing the slot's `tx_id` to the queried one and falling back to disk on mismatch (§10.2). -### §9.4 circle2 — entry storage and per-account chains - -Each circle2 slot stores one `TxEntry`'s payload (`tx_id`, `account_id`, -`amount`, `kind`, `computed_balance`) plus a *previous-link* — the -circle2 index of the same account's previous entry, or zero if there -is no previous. circle2 is written sequentially with a write head that -advances on every insert and wraps modulo circle2's size. On insert, -the indexer reads the account's head from `account_heads`; if the head -is for the same account, the new entry's previous-link is set to it, -otherwise the link is empty. Then the entry is written, `account_heads` -is updated to point at the new slot, and the write head advances. -Account history walks (§10.3) follow the previous-link chain backwards -in time until the chain ends, an evicted slot is detected (account-id -mismatch), the transaction predates the requested lower bound, or the -caller's limit is reached. - -### §9.5 account_heads — direct-mapped account head table - -Each `account_heads` slot stores `(account_id, circle2_idx)`. The slot -index is the account id masked by the account_heads size. The slot -stores the account id it belongs to so a colliding lookup can detect -eviction the same way circle1 does: a mismatch means the head was -overwritten and the chain head is gone, and the caller must fall back -to disk for any history older than the current chain. The size is -`next_power_of_two(max_accounts)`, which means in practice every -account has its own slot; collisions only occur if the account-id space -is sparser than the table. - -### §9.6 Sizing rule - -The three sizes are derived from `transaction_count_per_segment` and -`max_accounts`: - -- circle1 — one slot per transaction in the active window, rounded up - to a power of two. -- circle2 — enough entry slots for the active window assuming roughly - two entries per transaction (transfers), rounded up. -- account_heads — one slot per account, rounded up. +### §9.4 circle2 — follower storage -Changing the active window changes the hot indexes and the dedup window -in lockstep; there are no separate knobs. [ADR-0013] +Each circle2 slot stores one follower record as a raw `WalEntry` (a +`TxEntry`'s `tx_id`, `account_id`, `amount`, `kind`, `computed_balance`, +or a link record — stored as-is), tagged with the owning `tx_id` so a +colliding overwrite is detectable. circle2 is written sequentially with +a write head that advances on every insert and wraps modulo circle2's +size. On insert, the indexer writes the transaction's followers into +consecutive slots from the current write head and records the start +offset and follower count in that transaction's circle1 slot. There is +no per-account chaining and no per-account head table — circle2 is a +flat, `tx_id`-keyed follower store. Per-account history is reconstructed +on demand by scanning the WAL (§10.3), independent of this index. -### §9.7 Recovery seeding +### §9.5 Recovery seeding The recovery sequence (§12) seeds the indexer directly: as it walks WAL records from the last snapshot through the active segment's tail, @@ -896,6 +901,18 @@ through the Snapshotter's recovery hooks. By the time the Snapshotter thread starts, the indexer is already populated for everything in the active window. +### §9.6 Sizing rule + +Both sizes are derived from `transaction_count_per_segment` (§15.1): + +- circle1 — one slot per transaction in the active window, rounded up + to a power of two. +- circle2 — `4 ×` the circle1 size, giving headroom for several + followers per transaction (entries plus any links). + +Changing the active window changes the hot index and the dedup window +in lockstep; there are no separate knobs. [ADR-0013] + --- ## §10 Query path — hot/cold routing @@ -931,27 +948,30 @@ read from disk via the on-disk transaction index (Stage 2). ### §10.3 `GetAccountHistory` resolution -`GetAccountHistory` reads `account_heads[account_id & mask]`; if the -stored account_id does not match, returns an empty vec (account head -evicted). Otherwise walks `prev_link` backwards through `circle2`, -stopping on `account_id` mismatch (chain broken by eviction), -`tx_id < from_tx_id` (past the requested window), `limit` reached, or -`prev_link == 0` (chain head). Results are returned newest-first. An -empty vec from a query that should have had results is the -caller's signal to fall back to the on-disk account index (Stage 2). +`GetAccountHistory` does not consult the in-memory transaction index at +all — there is no per-account head table or link chain. +`Ledger::get_account_history` runs a backward WAL scan +(`wal_scanner().scan(from_tx_id, …)`), sweeping transactions +newest→oldest from `from_tx_id` and keeping those whose entries +reference the account (`entry_references_account`: a matching +`TxEntry.account_id`, an account-open range, a link, or a flag update). +Results are returned newest-first, bounded by the requested window and +`limit`. Because it reads the durable WAL directly, it is uniform across +recent and sealed history — there is no hot-miss/cold-fallback step. ### §10.4 Hot tier vs cold tier -The hot tier is everything currently in the indexer; the cold tier is -everything sealed to disk. The boundary is not a function of `tx_id` -absolute value but of how recently each `tx_id`'s slot has been -overwritten — a low-traffic account's chain can survive deep into -history, while a hot account's chain may be evicted within one segment. +Hot/cold tiering applies to `get_transaction` lookups, not to account +history. The hot tier is whatever `tx_id`s are currently resident in the +indexer; the cold tier is everything sealed to disk. The boundary is not +the absolute `tx_id` value but how recently each slot has been +overwritten — a `tx_id` from a quiet period can survive deep into +history, while a busy stretch may be evicted within one segment. Cold-tier reads are sealed-segment lookups (the on-disk transaction -index and the on-disk account index built by Seal — §11.4). The -mechanism by which the caller routes a hot miss to the cold tier is -the same for both query types: the hot result returns `None` / empty, -and the caller reissues against the cold-tier API. +index built by Seal — §11.4); on a hot miss the indexer returns `None` +and the caller reissues against the cold-tier API. `GetAccountHistory` +is tier-agnostic: it always scans the durable WAL (§10.3), so it has no +hot-miss/cold-fallback path. ### §10.5 Synchronous response via callback @@ -1019,7 +1039,8 @@ segment is left unsealed and retried on the next poll. Seal holds two pieces of state separate from the rest of the pipeline: -- A balance array sized to `max_accounts`, updated from each entry's +- A balance array seeded from `initial_account_size` (and grown on demand the + same way as the other account arrays, §4.4), updated from each entry's `computed_balance` as Seal walks the segment. This is the source for the balance snapshot (§11.5); it is *not* the read-side cache (§8.3). @@ -1090,9 +1111,14 @@ sealed segment. Production must always have it `false`. `Ledger::start` calls into `Recover` before spawning Sequencer, Transactor, WAL, Snapshotter, or Seal threads. Recovery is therefore single-threaded and observes a quiescent disk; nothing else is reading -or writing the data directory. Once recovery returns, the pipeline is -in a state that is observationally identical to the state that existed -at `last_committed_tx_id` immediately before the previous shutdown. +or writing the data directory. `Recover` builds a single `ActiveSnapshot` +struct — the reconstructed `last_tx_id`, account/flag map, function set, links, +KV/constants, dedup `user_ref → tx_id` maps, and the active segment's +transactions — and threads it into each stage's `recover_from`/seed step, so +every stage starts from one consistent reconstructed baseline. Once recovery +returns, the pipeline is in a state that is observationally identical to the +state that existed at `last_committed_tx_id` immediately before the previous +shutdown. ### §12.2 Pre-seal of unsealed segments @@ -1162,8 +1188,8 @@ a known-good state. The record stream uses a single lock-free **transaction ring** (§13.4); the submit and query paths use lock-free, fixed-capacity, single-producer / single-consumer queues (§2.4). Each stage owns its data completely; no shared -mutable state crosses stage boundaries. Queue capacity is sized at construction -from `queue_size` (§15.2), the ring from `ring_size` (§15.10). +mutable state crosses stage boundaries. Queue capacity is a fixed internal +constant (`1024`, §15.2); the ring is sized from `ring_size` (§15.10). ### §13.2 Wait strategies @@ -1185,27 +1211,31 @@ is no per-stage override. ### §13.3 Backpressure as ring and queue saturation There is no rate limiter, leaky bucket, or admission control. Slowness anywhere -propagates upstream: a stalled Snapshotter stops releasing ring slots (§13.5), -filling the ring and stalling the Transactor, which fills the -sequencer→transactor queue and eventually stalls `submit()` (§2.6). This is -intentional — the ring and queues are the only place stages observe each other's -progress, and using them as the backpressure mechanism keeps the design free of -explicit coordination. +propagates upstream: a stalled WAL Writer (the ring's sole consumer) stops +releasing ring slots (§13.5), filling the ring and stalling the Transactor, +which fills the sequencer→transactor queue and eventually stalls `submit()` +(§2.6). This is intentional — the ring and queues are the only place stages +observe each other's progress, and using them as the backpressure mechanism +keeps the design free of explicit coordination. ### §13.4 Transaction ring transport The transaction ring is a fixed-capacity, lock-free buffer that is the sole -transport for the record stream between the Transactor and the read stages -([ADR-0019]). It is single-producer, multi-reader: +transport for the record stream between the Transactor and the WAL +([ADR-0019], [ADR-0021]). It is single-producer, **single-consumer**: - The **Transactor** is the only writer. It writes records into the ring's uncommitted region (slots above the *write frontier*), builds each transaction - in place there (§4.6), and publishes by advancing the write frontier — so - readers only ever observe whole, committed transactions. -- The **WAL** and **Snapshotter** are independent copy-out readers, each - tracking its own absolute cursor. They copy records out (a slot may be reused - once released, so they cannot hold borrows across reclamation). -- The **Snapshotter** is the sole releaser (§13.5). + in place there (§4.6), and publishes by advancing the write frontier — so the + consumer only ever observes whole, committed transactions. +- The **WAL** is the only reader, *and* the only releaser. It walks the ring, + copies each record out (a slot may be reused once released, so it never holds + a borrow across reclamation), and advances the release frontier itself + (§13.5). The read window is `[released, write)`; because the writer is gated + on the release index that only the WAL advances, in-window slots are never + overwritten, so reads copy out directly with no per-entry window check. +- The **Snapshotter is not a ring reader.** It tails the durable WAL (§8.4), + decoupled from the ring entirely. Positions are absolute, monotonically increasing indices mapped to physical slots on access; the capacity (`ring_size`, §15.10) is a power of two and must @@ -1215,18 +1245,22 @@ blocks inside the ring: when no free slots exist it backs off under the shared wait strategy (§13.2) and retries — backpressure is caller-driven, not hidden in the transport. -### §13.5 Durability-gated reclamation - -A ring slot may be reused only after the record it held is durable on disk *and* -consumed by every reader. The Snapshotter, as sole releaser, advances the -reclamation point (the *release frontier*) only up to what it has applied, and -it applies a transaction only once its trailing `TxMetadata.tx_id ≤ -commit_index` (§8.5). The frontiers therefore stay ordered -`write ≥ persisted ≥ durable ≥ applied = released`. This makes "never overwrite -a record that is not yet durable" a single structural property of the transport, -rather than something each stage enforces separately. If durability stalls, -reclamation stalls and the producer eventually blocks once the ring fills — the -explicit, durability-coupled form of bounded-transport backpressure. +### §13.5 Write-gated reclamation + +A ring slot may be reused once the record it held has been **written to the +active segment's page cache** — *not* once it is fsynced ([ADR-0021]). The WAL, +as the ring's sole consumer and sole releaser, advances the reclamation point +(the *release frontier*) right after each buffered `write` syscall, in lockstep +with `write_index`: it publishes `write_index` and then calls `release_to` on +the same batch. The frontiers stay ordered `write_index ≥ released`, and +reclamation is therefore **write-gated, not durability-gated** — the +`fdatasync`/`commit_index` step is fully decoupled and runs on the Committer's +own schedule (§7.7). Durability is instead enforced *reader-side*: the +Snapshotter applies a transaction only once `TxMetadata.tx_id ≤ commit_index` +(§8.5), and it reads the durable WAL, not the ring, so the read side never +depends on a slot still being live. If the WAL Writer stalls (slow disk +`write`s), reclamation stalls with it and the producer eventually blocks once +the ring fills — the bounded-transport form of backpressure. --- @@ -1246,12 +1280,22 @@ The caller picks one of four wait levels per submission: [ADR-0010] -### §14.2 Implementation - -A wait is a poll of the relevant pipeline index against the requested -`tx_id`, driven by the shared wait strategy (§13.2). When the index -passes the threshold, the wait completes and synchronises with every -store that produced the advance. +### §14.2 Implementation — reactive index hook + +A wait is **edge-triggered, not a poll** ([ADR-0027]). The pipeline exposes a +single `IndexHook` — `Fn(PipelineIndexKind, u64)` — fired inline on the +publishing thread each time `compute_index`, `commit_index`, or `snapshot_index` +advances (the sequencer index is deliberately *not* hooked: waiters never block +on it, so firing per-submit would be pure hot-path cost). A consumer registers +one hook via `set_index_hook` (first registration wins) and routes each +`(kind, value)` into a per-index reactive watch. The network wait path (§36.4, +the cluster `Waiter`) does exactly this: each watch is a `u64` plus a `Notify`, +and an async waiter calls `wait_reach(target)` — it registers interest, checks +the value, and parks on the `Notify` edge with no spinning. When the index +crosses the threshold the parked future is woken and synchronises with the store +that produced the advance. The legacy `CommitHandler` / `on_commit` callback +(a `Fn(u64)` fired only on commit) is **superseded and dead** — it has no +production caller; the unified `IndexHook` replaced it. ### §14.3 Per-call dial @@ -1274,20 +1318,25 @@ reads. ## §15 Configuration — ledger-side knobs -### §15.1 `max_accounts` +### §15.1 `initial_account_size` and `resize_factor` -Pre-allocated capacity of every balance vector and read-side atomic -vector. Fixed at startup; cannot grow at runtime (§4.4). Accounts -referencing IDs ≥ this value are rejected with -`ACCOUNT_LIMIT_EXCEEDED`. Sized so the balance vector fits comfortably -in L2/L3 even with read-side and seal-side duplicates. +The account arrays are **growable**, not fixed (§4.4). `initial_account_size` +is the seed capacity of every balance vector and read-side atomic vector at +construction; `resize_factor` is the geometric growth increment applied when a +higher id is referenced — `new_cap = ceil(cap × (1 + resize_factor))`. Both are +exposed via `config.toml`. There is no `max_accounts` ceiling and no +`ACCOUNT_LIMIT_EXCEEDED` rejection. Size the seed so the common-case balance +vector fits comfortably in L2/L3 even with read-side and seal-side duplicates; +growth past it is amortised and rare. [ADR-0022] -### §15.2 `queue_size` +### §15.2 Inter-stage queue capacity -Capacity of the inter-stage queues. Not exposed via `config.toml`; an internal -tuning knob. The two remaining queues — sequencer→transactor and the -Snapshotter's query queue (§2.4) — are both sized from this single value. The -record-stream ring is sized separately (`ring_size`, §15.10). +The two inter-stage queues — sequencer→transactor and the Snapshotter's query +queue (§2.4) — are **not** sized from a config field. Their capacity is a fixed +internal constant (`1024`) set in `Pipeline::new`. `queue_size` is only a +test/bench parameter (`Pipeline::with_sizes`), not a `LedgerConfig` knob. The +record-stream ring is sized separately (`ring_size`, §15.10), which *is* a real +field. ### §15.3 `wait_strategy` @@ -1332,14 +1381,14 @@ the cost of more snapshot I/O during operation. ### §15.9 Hot-index sizes are derived Hot-index sizes (§9.6) are derived from `transaction_count_per_segment` -(§15.5) and `max_accounts` (§15.1). There are no separate knobs; -changing the active window changes both the hot indexes and the dedup +(§15.5) and the current account-array length (§15.1). There are no separate +knobs; changing the active window changes both the hot indexes and the dedup window in lockstep. ### §15.10 `ring_size` Capacity of the transaction ring (§13.4). A power of two; not exposed via -`config.toml`, an internal tuning knob like `queue_size`. Because a transaction +`config.toml`, an internal tuning knob. Because a transaction is published atomically and assembled in place, the ring must be at least as large as the biggest possible single transaction (`sub_item_count ≤ u16::MAX` followers plus the trailing metadata); otherwise the Transactor cannot make @@ -1910,15 +1959,16 @@ surface: that bypasses the Transactor and feeds pre-validated bytes directly to the WAL stage. - `wal_tailer()` — produces a stateful raw-WAL byte cursor (§23.1) - used by the leader to ship bytes to followers. -- An `on_commit` hook fired when the local commit index advances, - used by the leader to feed its own slot of the quorum tracker - (§30.4). + used by the leader to ship bytes to followers and by the Snapshotter + to tail the durable WAL (§8.4). +- `set_index_hook()` — registers the single `IndexHook` (§14.2) the cluster + uses to drive its reactive index watches (compute/commit/snapshot), which in + turn feed the cluster-commit driver (§30.3) and the client wait paths. - `start_with_recovery_until(watermark)` — an alternative to `start()` invoked only by the runtime divergence path (§33). Nothing else changes inside the Ledger. A standalone Ledger ignores -all four. [ADR-0015, ADR-0016 §10] +all four. [ADR-0017, ADR-0027, ADR-0016 §10] ### §26.3 Two gRPC surfaces @@ -1950,10 +2000,11 @@ served by another transport without renaming the role types. A node is in exactly one of four roles at any time: `Initializing` (post-boot or post-teardown, awaiting a signal), `Candidate` (running an election round), `Leader` (writable), or `Follower` (read-only, -receiving `AppendEntries`). The role is encoded as a `u8` inside a -single atomic; reads use `Acquire`, writes use `Release`. The atomic -is shared by every gRPC handler so the writability check on every -submit RPC is one cache-line read. +receiving the replication stream, §35.2). The role is published through a +`tokio::sync::watch` channel that tasks subscribe to (`role_subscribe`); reads +of the underlying atomic use `Acquire`, writes use `Release`. The role is shared +by every gRPC handler so the writability check on every submit RPC is one +cache-line read. ### §27.2 The supervisor is the sole writer @@ -1964,36 +2015,36 @@ role cannot leave handlers observing a half-applied state: the supervisor publishes the new role only after the old role's tasks have been torn down and the new role's tasks have been brought up. -### §27.3 Drop-and-rebuild on every transition +### §27.3 Cancel-and-respawn on every transition -Role-specific state — peer replication tasks, the Quorum tracker, -the gRPC handler's posture — lives inside a `LeaderHandles` or -`FollowerHandles` struct. A transition drops the outgoing struct -(which joins or aborts every owned task) before constructing the new -one. There is no "modify role in place" path; a node that goes -Leader → Follower → Leader has fully re-instantiated its peer tasks, -fresh quorum slots, and a freshly-bound writable handler each time. -The discipline is what eliminates stale-state bugs during failover. +There is no `LeaderHandles`/`FollowerHandles` struct. Role-specific work — the +leader's per-peer pusher tasks — is governed by `CancellationToken`s, not an +RAII bundle. The long-lived `replication_push_loop` (§31.1) watches the role +channel: on `→ Leader` it spawns the peer pushers under a child token; on +`Leader →` anything it cancels that token and joins the tasks. There is no +"modify role in place" path; a node that goes Leader → Follower → Leader has +fully re-spawned its peer tasks, re-seeded each peer's match index from a fresh +handshake (§30.4), and flipped the handler's writable posture each time. The +discipline is what eliminates stale-state bugs during failover. ### §27.4 What survives a transition -Three pieces of state are deliberately carried across a role -boundary: the `Arc` (so the node's data does not have to be -recovered on every transition), the `Arc` (so the durable term -log keeps a single owner), and the `Arc` (same reason for vote -durability). Every other piece of state is owned by a `Handles` -struct and dies with it. The one exception to even the Ledger -surviving a transition is divergence (§33), in which case the Arc is -dropped and rebuilt via `start_with_recovery_until`. +The durable, node-lifetime state is deliberately carried across role +boundaries: the `Arc` slot (so the node's data does not have to be +recovered on every transition), the durable `term.log` and `vote.log` (so each +keeps a single owner), the shared `Waiter` (§14.2), and the `RaftNode` itself +(its quorum/cluster-commit state, §30). Only the cancellable per-role tasks die +on a transition. The one exception to even the Ledger surviving is divergence +(§33), in which case the slot's Arc is reseeded via `start_with_recovery_until`. ### §27.5 `Initializing` is the post-boot and post-teardown state `Initializing` is the role a node is in immediately after the supervisor starts (before the first election timer fires) and -immediately after any `Handles` is dropped (before the next role's -bring-up). The Node gRPC server is already running so the node can -participate in elections and receive `AppendEntries`, but neither the -writable client handler nor any peer-replication task is up. The +immediately after a teardown (before the next role's bring-up). The Node gRPC +server is already running so the node can participate in elections and accept a +replication stream (§35.2), but neither the writable client handler nor any +peer-pusher task is up. The client-facing handler in this state behaves like a follower's read-only handler: queries succeed; submits return `FAILED_PRECONDITION`. [ADR-0016 §2, §3] @@ -2054,8 +2105,8 @@ to the cold path more often but still see correct answers. Every node not in `Leader` role runs an election timer with a randomised deadline in `[election_timer_min_ms, election_timer_max_ms]`. The deadline is re-randomised on every -reset. Reset triggers: any valid -`AppendEntries` from a current leader (empty heartbeat included), and +reset. Reset triggers: any valid leader activity on the replication stream +(`WalUpdate` or `Heartbeat`, §35.2) noted via `note_leader_activity`, and the granting of a `RequestVote`. Without a reset, the timer fires and the node transitions to `Candidate`. The randomisation defeats split votes: two nodes that timed out at exactly the same instant would @@ -2079,7 +2130,7 @@ deadline. Majority counts the candidate itself: a 5-node cluster needs 3 votes (of which 1 is the self-vote, so 2 peer grants suffice). A single-node cluster trivially wins by self-vote with no RPCs sent. The arithmetic -matches `Quorum`'s majority calculation (§30.1) but is computed +matches the RaftNode's cluster-commit majority (§30.1) but is computed independently — the election's vote count is not the running quorum. ### §29.4 `RequestVote` grant rule @@ -2105,133 +2156,138 @@ term high enough to depose the current leader. ### §29.6 Step-down on higher term -Any RPC in any direction (request or response, `RequestVote` or -`AppendEntries` or even a `Ping`) that carries a term higher than the -node's current term triggers an immediate step-down: `Term::observe` -durably records the new term, `voted_for` is cleared, and the node -transitions to `Initializing`. A node in `Leader` role drops its -`LeaderHandles` (which drains all peer tasks); a node in `Follower` -or `Candidate` drops the equivalent. The step-down is the universal -correctness anchor — no Raft node ever continues operating in an old +Any term observed in any direction (a `RequestVote`, a replication +handshake/frame, or even a `Ping`) that is higher than the node's current term +triggers an immediate step-down: `Term::observe` durably records the new term, +`voted_for` is cleared, and the node transitions to `Initializing`. The +role-change watch then cancels whatever per-role tasks were running — a leader's +peer pushers (§31.1, §31.6), a follower's session. The step-down is the +universal correctness anchor — no Raft node ever continues operating in an old term once it has seen a higher one. [ADR-0016 §5, §6, §13] --- -## §30 Quorum tracker - -### §30.1 One slot per node - -`Quorum` is an array of `AtomicU64`s, one slot per node in the -cluster. By convention slot 0 is the leader itself; slots `1..` are -the peers in configuration order. The majority size — -`(node_count / 2) + 1` — is computed at construction and never -changes. A separate atomic caches the current majority-committed -index for lock-free reads. - -### §30.2 `advance` publishes via `fetch_max` - -`advance(slot, index)` writes the slot's atomic with `Release`, -snapshots all slots with `Relaxed`, sorts the snapshot descending, -takes the `(majority - 1)`-th element (the highest index acknowledged -by a majority), and publishes it via `fetch_max(Release)` on the -cached majority atomic. The `fetch_max` ensures the published majority -never regresses, even when concurrent `advance` calls from different -slots interleave. This is the lock-free property that lets every peer -task call `advance` without coordination. - -### §30.3 Lock-free reads via `get` - -`Quorum::get()` is one `Acquire` load of the cached majority atomic. -There is no allocation, no fast-path lock, no per-call computation. -Client wait paths (§34.1) call `get()` repeatedly while polling for a -target index; the cost per poll is one cache-line load. - -### §30.4 The leader's slot is fed by the on-commit hook - -On `Leader` bring-up, the leader registers an `on_commit` callback -with the Ledger. The callback fires every time the local commit index -advances and calls `advance(self_slot, ledger.last_commit_id())`. -Without this callback the leader's own progress would be invisible to -the quorum calculation and the cached majority would be stuck at the -slowest peer. The hook is the reason the leader counts toward its own -quorum. - -### §30.5 `reset_peers` zeros peer slots on bring-up - -When a leader is freshly elected, the prior leader's match-index -state may still be reflected in the slots; carrying it forward could -let a stale peer slot inflate the new leader's perceived majority. -`reset_peers(leader_slot)` is called as part of `Leader` bring-up and -zeros every slot except the leader's own. The next `advance` call -from each peer task republishes from a clean baseline. -[ADR-0015 decisions 3, 5; ADR-0016 §3] +## §30 Cluster-commit in the pure RaftNode + +### §30.1 Quorum lives in the consensus state machine, not the cluster crate + +There is no standalone `Quorum` slot-array in the cluster crate. Quorum and +cluster-commit are owned by the **pure `RaftNode`** in the `raft` crate +([ADR-0017]) — the consensus state machine has no async, no I/O, and no upward +dependency on `ledger`. The cluster layer (`Consensus`, `consensus/state.rs`) +holds the `RaftNode` behind a `Mutex` and only feeds it observations. The +majority size — `(node_count / 2) + 1` — is a property of the RaftNode's peer +set. + +### §30.2 Two indexes: local and cluster + +The RaftNode tracks a *local* index (how far this node's own WAL has +progressed) and a *cluster-commit* index (the highest `tx_id` a majority has +acknowledged). `advance_local_index(new_local)` raises the local index (and the +leader's own quorum contribution); `advance_cluster_index(new_cluster)` raises +the cluster-commit index, clamped to the local index so a node never claims to +have cluster-committed past what it holds. `cluster_commit_index()` is the +lock-free read the wait paths (§34) consult. The leader's per-peer match indexes +live in the RaftNode's `Replication` view, advanced by +`replication().peer(id).append_result(...)` as acknowledgements arrive. + +### §30.3 The leader's self-progress is fed reactively, not by a callback + +The leader's own contribution to quorum is fed **reactively off the +snapshot-index watch**, not by an `on_commit` hook. `run_cluster_commit_driver` +parks on the ledger's snapshot `IndexWatch` (§14.2) and, on every advance, calls +`self_advance` → `advance_local_index(ledger.last_snapshot_id())`, then +publishes the resulting `cluster_commit_index()` into the waiter's +`cluster_commit` watch. This is what advances cluster-commit for a singleton +(no peers) and keeps the leader's self-progress fresh without polling. It runs +for every role; on a follower `self_advance` is a no-op on quorum. The legacy +`on_commit` callback path is gone (§14.2). + +### §30.4 Peer acknowledgements advance the leader's view + +Each peer's match index advances when an `IndexUpdate` arrives from that +follower over the replication stream (§31): the leader applies +`append_result(Success { term, last_commit_id })` to the peer's `Replication` +slot and then calls `self_advance`, which recomputes `cluster_commit_index()` +and republishes it. A freshly-elected leader seeds each peer's match index from +the handshake response (`last_term_curr_tx_id`) rather than carrying stale +state forward, so quorum is always recomputed from a clean per-session baseline. +[ADR-0016 §3, ADR-0017] --- -## §31 Leader replication tasks - -### §31.1 One task per peer - -A `Leader` spawns one `tokio` task per peer (`PeerReplication`). The -leader owns these tasks directly — there is no per-peer-supervisor -indirection — and they all share the leader's `Arc`, -`Arc running` shutdown flag, and a transition channel for -out-of-band signals. Spawning is part of `Leader::run_role_tasks`; -joining is part of `LeaderHandles::drop`. - -### §31.2 Each task's private state - -Each peer task owns: a `WalTailer` cursor seeded from the leader's -ledger (§23.1); a tonic Node-service client connected to the peer; -the peer's slot index in `Quorum`; and two watermarks — `from_tx_id` -(Raft's `nextIndex`, the next `tx_id` to ship) and `peer_last_tx` -(Raft's `matchIndex` snapshot, the highest `tx_id` the peer -acknowledged). Nothing else is shared between peer tasks. - -### §31.3 The replication loop - -Each iteration: call `tailer.tail(from_tx_id, &mut buf)`; if it -returns zero bytes, send an empty `AppendEntries` heartbeat carrying -only `leader_commit_tx_id`, then park until the commit index advances -(reactive) or the raft heartbeat interval elapses, whichever comes -first; if it returns bytes, call the ship-until-accepted path which retries on -transport failures and observes higher-term replies. On a successful -non-empty reply the task advances `from_tx_id` past the shipped -batch's last `tx_id`, updates `peer_last_tx`, and calls -`quorum.advance(slot, peer_last_tx)`. - -### §31.4 Lagged single-phase replication - -A successful `AppendEntries` returns the follower's *current* -fsynced `last_commit_id` — *not* a fresh fsync of the shipped batch. -The follower queues the bytes and replies immediately; its WAL stage -fsyncs on its own schedule (§7.7) and the next RPC's reply reflects -the result. Replies are therefore one batch stale relative to the -shipped data, which is harmless for quorum tracking because the gap -closes on the next reply. The model avoids blocking the network RPC -on disk latency. [ADR-0015] - -### §31.5 Idle heartbeats close the staleness gap - -Without idle heartbeats, the leader's last knowledge of a peer's -commit progress would be the reply to the previous shipment — itself -one batch stale. After the writer goes idle, the peer's true commit -watermark would never be observed. Sending an empty `AppendEntries` -every raft heartbeat interval fixes this: the reply carries the peer's -now-fsynced `last_commit_id`, which advances the peer's quorum slot. -Transport errors on heartbeats are intentionally swallowed — the -next interval retries; a heartbeat is purely observational, not a -durability event. - -### §31.6 Step-down on higher-term reply - -A peer reply whose `term` is higher than the leader's current term -triggers an immediate step-down: the peer task posts a step-down -transition to the supervisor and returns. The supervisor drains all -peer tasks (drops `LeaderHandles`), observes the higher term, clears -`voted_for`, and re-enters `Initializing`. The write side is also -drained: any in-flight client submit returns the error generated by -the dropping handler. [ADR-0015, ADR-0016 §3] +## §31 Replication driver and per-peer streams + +### §31.1 One driver, two long-lived loops + +Replication is not "one RPC per heartbeat." `run_replication_driver` spawns two +long-lived loops that live for the node's whole lifetime, regardless of role: + +- `replication_stream_loop` — the **follower** side: it accepts inbound + `Replication` streams handed in by the gRPC handler (§35.2) and runs a + follower session per stream. +- `replication_push_loop` — the **leader** side: it subscribes to role changes + and, on becoming `Leader`, spawns one `run_peer_push` task per peer + (`spawn_peer_pushers`); on losing leadership it cancels them. There is no + `PeerReplication` struct and no `LeaderHandles` — task lifetime is governed + by `CancellationToken`s, not an RAII handles bundle. + +### §31.2 Each peer pusher's session + +A peer pusher (`run_peer_push`) opens **one bidirectional `Replication` stream** +to the peer, sends a single `Handshake` frame, and waits for the +`HandshakeResponse`. On accept it seeds the peer's match index from the +response's `last_term_curr_tx_id` (§30.4) and runs a `run_peer_session`, which +splits into two parallel tasks over the *same* stream: + +- a **sender** (`run_peer_sender`) owning a `WalTailer` pre-positioned just + after the handshake anchor; and +- a **receiver** (`run_peer_receiver`) consuming the follower's `IndexUpdate` + frames. + +On a rejected handshake the pusher records the follower's reported +`last_term_curr_tx_id` as an anchor override and reconnects; transient +connect/stream failures back off and retry. + +### §31.3 The sender loop (streaming, not request/reply) + +The sender does not wait for a per-batch ack. Each pass it calls `tailer.tail` +into a fixed buffer: if bytes come back it sends a `WalUpdate` frame (the raw +WAL bytes plus the leader's current `cluster_commit_id`) and, while the buffer +keeps filling, stays tight (`yield_now`, no park) so leader→follower throughput +is not artificially capped; if the WAL is drained (partial buffer) or empty it +sends a `Heartbeat` frame carrying only `cluster_commit_id`, then parks. The +park is reactive: it waits on the `commit` `IndexWatch` to reach +`last_sent_tx_id + 1` (new durable bytes to ship) or on the raft heartbeat +interval, whichever fires first — never a fixed sleep while WAL is pending. + +### §31.4 Acks are a separate, reactive stream + +Acknowledgement is decoupled from sending. On the follower side the session +splits into a *receiver* (applies leader frames to the ledger, no inline ack) +and an *acker* (`run_follower_acker`) that parks on the follower's snapshot +`IndexWatch` and emits an `IndexUpdate(local_commit_id)` each time it advances — +reporting the follower's *true* post-fsync applied index, not a stale read taken +at enqueue. The leader's receiver feeds each `IndexUpdate` into the peer's match +index and calls `self_advance` (§30.4). Because applying never blocks on the ack +and the ack reflects real durability, quorum tracking is both live and accurate. + +### §31.5 Idle heartbeats keep cluster-commit fresh + +When the WAL is idle the sender still emits a `Heartbeat` every raft heartbeat +interval, carrying the leader's `cluster_commit_id`; a data `WalUpdate` resets +the timer (a data frame counts as the heartbeat, per Raft). Followers mirror the +carried `cluster_commit_id` via `advance_cluster_index` (clamped to their own +local commit). Transport errors tear down the stream and the pusher reconnects; +a heartbeat is observational, not a durability event. + +### §31.6 Step-down on higher term + +A higher term observed anywhere in the consensus state machine drives the node +out of `Leader`; the role-change watch then fires and `replication_push_loop` +cancels every peer pusher (§31.1). The write side is drained the same way it is +on any role transition (§27.3): in-flight client submits return the error +generated by the rebuilt handler. [ADR-0016 §3, ADR-0017] --- @@ -2239,60 +2295,62 @@ the dropping handler. [ADR-0015, ADR-0016 §3] ### §32.1 Raft naming map -The Roda cluster module uses Raft's terminology where possible: -`from_tx_id` is `nextIndex`; `peer_last_tx` and the `Quorum` peer -slot together correspond to `matchIndex`. The `tx_id` stream is the -Raft log; the term boundary records in `term.log` (§28) supply the -term for any historical `tx_id`. This mapping is documented so a -reader who knows Raft can navigate the implementation without -re-deriving the correspondence. - -### §32.2 `prev_tx_id` and `prev_term` consistency check - -Every `AppendEntries` request carries `prev_tx_id` (the `tx_id` -immediately before the batch) and `prev_term` (the term of that -`tx_id`). The follower validates: it has `prev_tx_id` durably on -disk, and the term covering `prev_tx_id` matches `prev_term`. If the -follower's log is shorter, the reply carries `REJECT_PREV_MISMATCH` -with the follower's own `last_commit_id`. If the follower's log has -the `tx_id` but with a different term, the reply also carries -`REJECT_PREV_MISMATCH` and divergence handling kicks in (§33). - -### §32.3 The first RPC - -For the first `AppendEntries` against an empty follower (or a -fully-replicated follower starting a fresh batch from `tx_id = 0`), -both `prev_tx_id` and `prev_term` are zero. The follower treats -`(0, 0)` as "no precondition" and accepts. Subsequent RPCs use the -actual preceding `tx_id` and its covering term. - -### §32.4 Reject reasons are explicit - -`AppendEntriesResponse` carries an enum reject reason rather than a -bare bool. The four currently emitted reasons are: `TERM_STALE` -(request's term is below the follower's current term), -`PREV_MISMATCH` (consistency check failed — see §33), `CRC_FAILED` -(the bytes did not parse), `WAL_APPEND_FAILED` (the follower's WAL -stage rejected the queued bytes). The leader uses the reject reason -to decide whether to step down (`TERM_STALE`), trigger a divergence -path on the follower side (`PREV_MISMATCH`), or log and continue. +The Roda cluster module uses Raft's terminology where possible: the sender's +`WalTailer` cursor is `nextIndex`; the peer's match index in the RaftNode's +`Replication` view is `matchIndex` (§30.4). The `tx_id` stream is the Raft log; +the term boundary records in `term.log` (§28) supply the term for any historical +`tx_id`. This mapping is documented so a reader who knows Raft can navigate the +implementation without re-deriving the correspondence. + +### §32.2 The consistency anchor lives in the handshake, once per stream + +Roda does **not** carry a `prev_tx_id`/`prev_term` precondition on every +appended batch. The Log-Matching anchor is sent **once, in the `Handshake` +frame** that opens a replication stream (§35.2): the leader sets +`prev_log_tx_id = local_commit_index` and `prev_log_term = term_at_tx(that)`, +plus `leader_term`, `leader_term_first_tx_id`, and `leader_commit`. The follower +validates the anchor in `validate_handshake`; once accepted, every subsequent +`WalUpdate` on that stream is appended in order with no per-frame precondition. +If the leader's term changes, a fresh stream (and fresh handshake) is +established. This is the central difference from textbook AppendEntries: +consistency is a per-*session* check, not a per-*message* one. + +### §32.3 The handshake decision + +`validate_handshake` returns `Accept` or `Reject { reason, truncate_after }`. +Accept seeds the leader's match index from the response and lets the WAL stream +flow. Reject maps to a `RejectReason` on the `HandshakeResponse`: `TermBehind` +→ `REJECT_TERM_STALE` (leader term below the follower's), `LogMismatch` → +`REJECT_PREV_MISMATCH` (the anchor is not in the follower's log, or sits at a +different term — divergence, §33). On a `LogMismatch` carrying a +`truncate_after`, the follower reseeds its ledger to that watermark *before* +replying, so the leader's first `WalUpdate` lands on the truncated state. + +### §32.4 The follower reports its anchor back + +A rejected handshake is not a dead end: the `HandshakeResponse` carries the +follower's own `last_term`, `last_term_first_tx_id`, and `last_term_curr_tx_id`. +The leader's pusher records `last_term_curr_tx_id` as an anchor override and +reopens the stream from there (§31.2), walking back until the handshake is +accepted. The remaining `RejectReason` variants in the proto +(`REJECT_CRC_FAILED`, `REJECT_SEQUENCE_INVALID`, `REJECT_WAL_APPEND_FAILED`, +`REJECT_NOT_FOLLOWER`) are reserved for append-side and role failures. ### §32.5 Batch sizing is exact -WAL records are exactly 40 bytes (§18.1), so the leader's per-RPC -byte cap is always a clean multiple of 40 — there is no padding, no -partial record at the end of a batch. The configured -`append_entries_max_bytes` is rounded down to the nearest multiple of -40 at startup. +WAL records are exactly 40 bytes (§18.1), so the leader's per-frame byte cap is +always a clean multiple of 40 — there is no padding, no partial record at the +end of a `WalUpdate`. The configured `append_entries_max_bytes` (still the knob +name) is rounded down to the nearest multiple of 40 at startup and sizes the +sender's tail buffer. ### §32.6 gRPC message-size override -Tonic's default decoding/encoding limit of 4 MiB would otherwise -reject batches sized at exactly `append_entries_max_bytes`. The Node -server and client both raise `max_decoding_message_size` and -`max_encoding_message_size` to `append_entries_max_bytes × 2 + 4 KiB` -to cover both the WAL byte payload and protobuf framing overhead with -margin. [ADR-0015, ADR-0016 §8] +Tonic's default decoding/encoding limit of 4 MiB would otherwise reject +`WalUpdate` frames sized at exactly `append_entries_max_bytes`. The Node server +and client both raise `max_decoding_message_size` and `max_encoding_message_size` +to `append_entries_max_bytes × 2 + 4 KiB` to cover both the WAL byte payload and +protobuf framing overhead with margin. [ADR-0016 §8, ADR-0017] --- @@ -2309,26 +2367,26 @@ divergence path (§33.3) achieves truncation by *restarting the Ledger*, which lets the Recover path do the truncation safely against a quiescent disk. -### §33.2 Divergence detection on the follower +### §33.2 Divergence detection at the handshake -When `AppendEntries` arrives with a `prev_tx_id` that the follower -has on disk but with a `prev_term` that disagrees with the follower's -own term covering that `tx_id`, the follower has diverged from the -new leader's history. The follower stashes `leader_commit_tx_id` from -the request as the *recovery watermark* and replies -`REJECT_PREV_MISMATCH`. The supervisor's divergence watcher reads the -stashed watermark and triggers the reseed. +Divergence is detected when a replication **handshake** arrives whose anchor — +`prev_log_tx_id`/`prev_log_term` (§32.2) — the follower has on disk but at a +disagreeing term. `validate_handshake` returns `Reject { reason: LogMismatch, +truncate_after }`, where `truncate_after` is the highest `tx_id` the follower +may keep. This is a per-session decision, not a per-message one: there is no +`AppendEntries` carrying a precondition on each batch. -### §33.3 Reseed sequence +### §33.3 Reseed sequence (inline in the handshake handler) -The supervisor: drops the current `FollowerHandles` (so all tasks -holding the `Arc` are gone); calls a `LedgerSlot::replace` -with a freshly-built `Ledger::start_with_recovery_until(watermark)`, -which gives the slot a new `Arc` and returns the old one for -asynchronous drop on a background task; brings up a fresh -`FollowerHandles` over the new Ledger. The next `AppendEntries` from -the leader either resumes cleanly (`prev_tx_id ≤ watermark`) or -triggers a normal catch-up backfill from the leader. +The reseed happens **inline in `replication_follower_handshake`, before the +handshake response is sent** (so the leader's first `WalUpdate` lands on the +truncated state). On a `LogMismatch` carrying a `truncate_after`, the handler +calls `LedgerSlot::reseed(after)`, which builds a fresh +`Ledger::start_with_recovery_until(after)` and atomically swaps it into the slot +via `ArcSwap` (the old `Arc` drops once outstanding handlers release it, +§33.6). The follower then reports its post-reseed anchor in the handshake +response; the leader resumes streaming from there (§32.4). There is no separate +`FollowerHandles` to drop — the session simply runs against the reseeded slot. ### §33.4 `start_with_recovery_until` semantics @@ -2353,14 +2411,14 @@ knows how to reconstruct balances from a WAL prefix. ### §33.6 `LedgerSlot` is the indirection that lets handlers survive A reseed swaps the Ledger out from under the gRPC handlers without -restarting the gRPC servers. The mechanism is `LedgerSlot`: an atomic -swap of `Arc`. Every handler reads the current Arc on every -call (`slot.ledger()`) and never retains it across operations. The -swap is lock-free; the old Arc's strong count drops to zero on a -background task that joins the old Ledger's pipeline threads -synchronously, which never blocks a gRPC request because gRPC handlers -have already released their reference. The supervisor is the sole -writer of the slot. [ADR-0016 §9, §10] +restarting the gRPC servers. The mechanism is `LedgerSlot` — +`Arc>`. Every handler reads the current Arc on every call +(`slot.current()`) and never retains it across operations. The swap is +lock-free; the old Arc's strong count drops to zero once outstanding handlers +release it, joining the old Ledger's pipeline threads on drop, which never +blocks a gRPC request because handlers have already released their reference. +The single ledger `IndexHook` is registered through the slot so it survives the +swap (§26.2). [ADR-0016 §9, §10] --- @@ -2368,39 +2426,43 @@ writer of the slot. [ADR-0016 §9, §10] ### §34.1 A fifth wait level -Stage 1 introduces four wait levels (§14.1). The cluster adds a -fifth: `cluster_commit` blocks the submitter until -`quorum.get() ≥ tx_id`. The handler driving the wait is the same -shared wait strategy (§13.2) used for the in-process levels; the only -difference is which atomic the predicate reads. - -### §34.2 The leader feeds its own slot - -The leader's slot in `Quorum` is advanced by the on-commit hook -(§30.4); each peer task advances its peer's slot on every successful -`AppendEntries`. Both contributions are required for the majority to -move forward — without the leader's own contribution the cached -majority would be stuck at the slowest peer's progress. +Stage 1 introduces four wait levels (§14.1). The cluster adds a fifth: +`cluster_commit`. It is reactive like the rest (§14.2): the waiter awaits the +**`commit`, `snapshot`, and `cluster_commit` `IndexWatch`es in turn** — +requiring the tx to be locally durable *and* locally queryable *and* +quorum-replicated, completing when the last of the three crosses `tx_id`. There +is no spin/poll on a quorum atomic; the `cluster_commit` watch is fed from the +RaftNode's `cluster_commit_index()` at every advance site +(`publish_cluster_commit`). + +### §34.2 The leader feeds its own progress reactively + +The leader's contribution to quorum is not a callback-driven slot write. The +snapshot-index driver (§30.3) calls `self_advance` on each snapshot advance, +raising the RaftNode's local index and republishing `cluster_commit_index()`; +each peer's `IndexUpdate` advances that peer's match index and triggers another +`self_advance` (§30.4). Both contributions are required for the cluster-commit +index to move — without the leader's own progress it would be stuck at the +slowest peer. ### §34.3 Local commit is independent of quorum -The leader's *own* `commit_index` (§2.5) advances from its local WAL -stage as soon as the local `fdatasync` returns, independent of any -peer's progress. Only the client's wait choice determines the -guarantee: a client that picks `wal` sees the local commit; a client -that picks `cluster_commit` waits for quorum. The two indexes coexist -in the same pipeline; nothing about cluster mode slows down the local -commit path. [ADR-0015 decision 5] +The leader's *own* `commit_index` (§2.5) advances from its local WAL stage as +soon as the local `fdatasync` returns, independent of any peer's progress. Only +the client's wait choice determines the guarantee: a client that picks `wal` +sees the local commit; a client that picks `cluster_commit` waits for quorum. +The two indexes coexist in the same pipeline; nothing about cluster mode slows +down the local commit path. [ADR-0017] -### §34.4 Followers do not durably track cluster commit +### §34.4 Followers mirror, but do not own, cluster commit -`leader_commit_tx_id` arrives on every `AppendEntries` (including -heartbeats) but the follower does not persist it. Its only consumer -on the follower side is divergence handling (§33.2), where it is -captured as the recovery watermark. A follower's view of "cluster -commit" is not authoritative — only the leader's `Quorum::get()` is. -Reads from a follower that need cluster-commit semantics are out of -scope for this implementation. +`cluster_commit_id` arrives on every `WalUpdate` and `Heartbeat` (§31.5). The +follower mirrors it via `advance_cluster_index`, clamped to its own local commit +index, so a follower can answer `cluster_commit`-level waits for transactions it +has both applied and seen acknowledged — but its view is derived from the +leader's stream, not authoritative on its own. The leader's +`cluster_commit_index()` remains the source of truth. A divergent handshake also +carries a `truncate_after` watermark used to reseed the follower (§32.3, §33.2). --- @@ -2416,12 +2478,18 @@ default; without the mutex, two concurrent `RequestVote` handlers could race on `vote.log` and grant the same term to two candidates. [ADR-0016 §7] -### §35.2 `AppendEntries` +### §35.2 `Replication` (bidirectional stream) -An empty `wal_bytes` is a valid heartbeat: the follower runs the -consistency check and returns its commit id without queueing any -records. The reply's `last_tx_id` carries the follower's *current* -fsynced commit id, which lags the shipped batch by one round (§31.4). +There is no unary `AppendEntries` RPC. Peer replication is a single +**bidirectional streaming** RPC, `Replication(stream +ReplicationLeaderMessage) returns (stream ReplicationFollowerMessage)`. The +leader→follower stream carries one `Handshake` (the consistency anchor, §32.2) +followed by an open-ended sequence of `WalUpdate` (raw WAL bytes + +`cluster_commit_id`) and `Heartbeat` (`cluster_commit_id` only) frames. The +follower→leader stream carries one `HandshakeResponse` followed by `IndexUpdate` +frames reporting the follower's applied commit id (§31.4). The handler hands the +inbound stream to the replication driver (§31.1), which runs the per-session +loops; a term change opens a fresh stream rather than mutating an existing one. ### §35.3 `RequestVote` @@ -2549,12 +2617,13 @@ indirection (§33.6). ### §38.2 Supervisor boot -The role supervisor builds the long-lived shared resources: `Quorum` -(sized to `peers + 1`), the cooperative `running: Arc`, -and the transition channel. It then spawns the client-facing gRPC -server, the peer-facing gRPC server, the divergence watcher, and the -role driver loop. The two gRPC servers are bound for the entire -lifetime of the process; they never restart on a role transition. +The supervisor builds the long-lived shared resources: the `Consensus` wrapping +the pure `RaftNode` (which owns quorum/cluster-commit state, §30), the shared +`Waiter` (§14.2), and a node-wide `CancellationToken`. It then spawns the +client-facing gRPC server, the peer-facing gRPC server, the replication driver +(§31.1), the cluster-commit driver (§30.3), and the role-driver loop. The two +gRPC servers are bound for the entire lifetime of the process; they never +restart on a role transition. ### §38.3 Initial role @@ -2568,21 +2637,20 @@ election timer to expire before becoming a candidate. The driver awaits the appropriate signal per role: an election-timer expiry while in `Initializing` or `Follower`; the candidate round's -outcome while in `Candidate`; a transition-channel message while in -`Leader`. Each transition drops the previous role's `Handles` (which -joins or aborts every owned task) before constructing the next role's -`Handles`. The driver itself owns no role-specific state; it is purely +outcome while in `Candidate`; loss of leader contact while in `Leader`. Each +transition publishes the new role on the watch channel (§27.1); the long-lived +replication driver reacts by cancelling and respawning the per-role tasks +(§27.3, §31.1). The driver itself owns no role-specific task state; it is purely the coordinator that decides which role to bring up next. ### §38.5 Shutdown ordering -Shutdown flips `running` to `false`, sends a `Shutdown` transition -message (which wakes the role driver if it is parked), and aborts -every spawned task in order — the role driver first, the divergence -watcher, the peer-facing server, the client-facing server. Peer -replication tasks observe `running` at the top of their loop and exit -cleanly; the gRPC servers run their tonic shutdown protocol before -the abort takes effect. +Shutdown cancels the node-wide `CancellationToken`, which every spawned task +selects on (§31). Lifecycle is RAII: dropping `ClusterNode` triggers the cancel +and then *awaits* the tasks cooperatively rather than aborting them — the +replication driver tears down its child sessions, the drivers exit their select +loops, and the gRPC servers run their tonic graceful-shutdown protocol. No +`running: AtomicBool` flag and no abort-in-order step is involved. --- @@ -2592,8 +2660,8 @@ the abort takes effect. When the configuration has no `[cluster]` section, `ClusterNode` does not invoke the supervisor at all. It constructs a single client-facing -`Server` task with a hardcoded `Role::Leader` posture, no `RoleFlag`, -no Node gRPC server, no `Term`, no `Vote`, no `Quorum`, no peer tasks. +`Server` task with a hardcoded `Role::Leader` posture, no role watch, +no Node gRPC server, no `Term`, no `Vote`, no `RaftNode`, no peer tasks. The deployment is identical in observable behaviour to a single-node cluster but pays none of the supervisor's coordination cost. This is the configuration `roda` ships with by default. @@ -2601,13 +2669,13 @@ the configuration `roda` ships with by default. ### §39.2 Single-node cluster (`peers.len() == 1`) A cluster whose configuration lists exactly one peer (itself) runs -the full supervisor infrastructure: a `RoleFlag`, a `Term`, a `Vote`, -a single-slot `Quorum`. The node boots as `Leader` (§38.3) and the -replication loop has zero peers to fan out to; every `on_commit` hook -firing immediately advances the cluster watermark since there is no -one to wait for. This mode exercises the cluster code path against a -trivial cluster shape and is the one used by most cluster integration -tests. +the full supervisor infrastructure: a role watch, a `Term`, a `Vote`, and the +`RaftNode` (with a peer-less quorum). The node boots as `Leader` (§38.3) and the +replication push loop has zero peers to fan out to; the cluster-commit driver +(§30.3) advances cluster-commit purely from the leader's own snapshot-index +advances, since there is no one else to wait for. This mode exercises the +cluster code path against a trivial cluster shape and is the one used by most +cluster integration tests. ### §39.3 Multi-node cluster diff --git a/docs/load.md b/docs/load.md index 683f229..f2850fb 100644 --- a/docs/load.md +++ b/docs/load.md @@ -1,6 +1,18 @@ # Load Test Report -Generated by the **Load Test** workflow on a CCX33 server (Hetzner, 8 dedicated vCPU), 1,000,000 accounts, deposit operations, release build with WAL persistence enabled. +The tables below come from the **single-node `ledger` load bins** — `load`, `load_latency`, and +`load_wasm` — run manually on a CCX33 server (Hetzner, 8 dedicated vCPU), 1,000,000 accounts, deposit +operations, release build with WAL persistence enabled. Reproduce with +`cargo run -p ledger --release --bin load|load_latency|load_wasm` (or `scripts/dev/ledger-load.sh`, +which runs `load_latency`). All three run `LedgerConfig::bench()`, whose +`transaction_count_per_segment` is 10,000,000 (vs. the default profile's 50,000). + +> The control-crate `--group load` scenarios (`scripts/dev/load.sh` → +> `cargo run --package control --bin scenario -- run-all --group load`: `load_peak`, `load_spike`, +> `load_sustained_2min`, …) are a **separate cluster deposit suite**, not the single-node runs reported +> here. + +> **Re-benchmark recommended** (predates ADR-022 existence checks + ADR-027 index hook on the hot path). - **Throughput** — sustained tx/s with submit-side latency (`load`). - **Latency under load** — end-to-end latency to each pipeline stage (`load_latency --wait-level`). @@ -94,9 +106,9 @@ Generated by the **Load Test** workflow on a CCX33 server (Hetzner, 8 dedicated ``` -## Latency Under Load +## Latency Under Load (single node) -Each probe submits one deposit and times it until the selected pipeline index reaches it, repeated 1000x per scenario across five load levels. `--wait-level` picks the stage: +These figures are single-node (`load_latency`): no consensus, no replication. Each probe submits one deposit and times it until the selected pipeline index reaches it, repeated 1000x per scenario across five load levels. `--wait-level` picks the stage: | Level | Stage | |---|---| @@ -105,6 +117,11 @@ Each probe submits one deposit and times it until the selected pipeline index re | commit | fdatasync'd to disk — durable | | snapshot | applied to the snapshot/indexer — readable | +> **Cluster latency.** A fifth wait level, **cluster-commit** (quorum-committed across the cluster), has +> no single-node equivalent and is **not** measured by these bins. It is sampled server-side via the +> `LatencyProbe.Probe` RPC (`crates/proto/proto/latency.proto`, `latency-probe` feature) and exercised +> by the `cluster_load_latency` bin (`crates/cluster/src/bin/cluster_load_latency.rs`). + ### wait-level: compute ```text diff --git a/docs/wasm-runtime.md b/docs/wasm-runtime.md index d7df412..ca90378 100644 --- a/docs/wasm-runtime.md +++ b/docs/wasm-runtime.md @@ -2,9 +2,9 @@ roda-ledger lets you extend the transaction set at runtime with sandboxed WebAssembly functions. A registered function becomes a first-class `Operation::Function`: it is sequenced, executed atomically by the Transactor, and produces normal `TxEntry` records in the WAL alongside every other transaction. -This document covers everything about writing, registering, invoking, and operating programmable functions: ABI, lifecycle, storage layout, recovery guarantees, and gRPC surface. +This document covers everything about writing, registering, invoking, and operating programmable functions: ABI, lifecycle, host API, programmable KV state, storage layout, recovery guarantees, and gRPC surface. -The design is specified in [ADR-014](./adr/0014-wasm-function-registry.md). +The design spans four ADRs: [ADR-014](./adr/0014-wasm-function-registry.md) (the function registry + `execute`), [ADR-022](./adr/0022-account-layouts-and-program-defined-accounts.md) (account existence, flags, linked accounts), [ADR-023](./adr/0023-programmable-state.md) (the typed KV store, constants, and the `register` phase), and [ADR-026](./adr/0026-roda-wasm-abi.md) (the official `roda-wasm-abi` guest SDK). --- @@ -13,11 +13,12 @@ The design is specified in [ADR-014](./adr/0014-wasm-function-registry.md). - [When to use it](#when-to-use-it) - [Function ABI](#function-abi) - [Host API](#host-api) +- [Programmable KV state](#programmable-kv-state) +- [Named constants & the register phase](#named-constants--the-register-phase) - [Return values & rollback](#return-values--rollback) - [Writing a function](#writing-a-function) - - [Rust](#rust) - - [AssemblyScript](#assemblyscript) - - [Hand-written WAT](#hand-written-wat) + - [The `roda-wasm-abi` guest SDK](#the-roda-wasm-abi-guest-sdk) + - [Appendix: hand-written ABI](#appendix-hand-written-abi) - [Registering a function](#registering-a-function) - [gRPC](#grpc) - [Rust library](#rust-library) @@ -48,7 +49,7 @@ If your operation is expressible as one of `Deposit`, `Withdrawal`, or `Transfer ## Function ABI -Every registered function must export **one** symbol named `execute` with a fixed signature: +Every registered function must export a symbol named `execute` with a fixed signature: ``` execute(i64, i64, i64, i64, i64, i64, i64, i64) -> i32 @@ -62,11 +63,17 @@ execute(i64, i64, i64, i64, i64, i64, i64, i64) -> i32 The fixed arity eliminates any need for linear memory pointer / length passing: account ids, amounts, rates, flags, timestamps all fit into 8 `i64` slots. +A function **may** also export a second symbol named `register` with signature `() -> ()`. It is optional and defines the module's named constants; see [Named constants & the register phase](#named-constants--the-register-phase). + --- ## Host API -A function can only call **three** host imports. All three live in the `ledger` module: +A function imports its host verbs from the WASM module named `ledger`. There are **eleven** verbs across three groups: balances (ADR-014), accounts & flags (ADR-022), and the typed KV store + constants (ADR-023). A module imports only the verbs it uses — there is **no import allow-list**; importing a verb the module never calls is harmless, and importing a name that is *not* one of the eleven fails at instantiation (the function then surfaces as `INVALID_OPERATION`). + +The names and signatures below are the raw WASM imports. The official [`roda-wasm-abi`](#the-roda-wasm-abi-guest-sdk) crate exposes safe Rust wrappers over all of them (its wrapper for `get_balance` is named `balance`). + +### Balances ``` (import "ledger" "credit" (func (param i64 i64))) @@ -74,51 +81,134 @@ A function can only call **three** host imports. All three live in the `ledger` (import "ledger" "get_balance" (func (param i64) (result i64))) ``` -| Host call | Signature | Effect | +| Host verb | Signature | Effect | |-----------|-----------|--------| -| `credit(account_id, amount)` | `(u64, u64) -> ()` | Decreases `account_id`'s balance by `amount`. Cannot fail individually. | -| `debit(account_id, amount)` | `(u64, u64) -> ()` | Increases `account_id`'s balance by `amount`. Cannot fail individually. | -| `get_balance(account_id)` | `(u64) -> i64` | Reads the current balance (signed). | +| `credit(account_id, amount)` | `(u64, u64) -> ()` | Decreases `account_id`'s balance by `amount`. | +| `debit(account_id, amount)` | `(u64, u64) -> ()` | Increases `account_id`'s balance by `amount`. | +| `get_balance(account_id)` | `(u64) -> i64` | Reads the current balance (signed). Absent account reads `0`. | Convention in roda-ledger: **credit subtracts, debit adds.** A `Transfer { from, to, amount }` is `credit(from, amount)` + `debit(to, amount)` — credits move value *out* of an account, debits move value *in*. Same convention applies to WASM functions. -**Zero-sum invariant.** The ledger verifies `sum(credits) == sum(debits)` after the function returns. If the function emits an unbalanced set of credits / debits, the transaction is rejected and rolled back with `Status::ZERO_SUM_VIOLATION`. +### Accounts & flags (ADR-022) + +``` +(import "ledger" "linked_account" (func (param i64 i32) (result i64))) +(import "ledger" "get_flag" (func (param i64 i32) (result i32))) +(import "ledger" "has_flag" (func (param i64 i32 i32) (result i32))) +(import "ledger" "set_flag" (func (param i64 i32 i32))) +``` + +| Host verb | Signature | Effect | +|-----------|-----------|--------| +| `linked_account(account_id, type_id)` | `(u64, u16) -> u64` | Get-or-create the sub-account linked to `account_id` under `type_id` (ADR-022 §6); returns the child account id. A first call lazily opens a `PROGRAMMED` account and emits `AccountOpened` + `AccountLinked` followers. | +| `get_flag(account_id, lane)` | `(u64, u8) -> u8` | Read the flag byte at `lane` (0..=7) of `account_id`. Lane 0 is the status lane. | +| `has_flag(account_id, lane, value)` | `(u64, u8, u8) -> bool` | True iff `account_id`'s `lane` byte equals `value`. | +| `set_flag(account_id, lane, value)` | `(u64, u8, u8) -> ()` | Set `account_id`'s `lane` byte to `value`. WASM may write any lane, including the status lane 0 (ADR-022 §6). | + +`type_id`, `lane`, and `value` cross the WASM boundary as `i32` (the host narrows them to `u16` / `u8`). + +### KV store & constants (ADR-023) -**Overdraft policy.** Host credits / debits never fail individually, and the ledger does not enforce negative-balance checks inside functions. If you need overdraft protection, call `get_balance` and return a non-zero status to trigger rollback: +``` +(import "ledger" "kv_get" (func (param i32 i32 i32 i32) (result i64))) +(import "ledger" "kv_set" (func (param i32 i32 i32 i32 i64))) +(import "ledger" "kv_register_constant" (func (param i32))) +(import "ledger" "kv_get_constant" (func (param i32) (result i32))) +``` + +| Host verb | Signature | Effect | +|-----------|-----------|--------| +| `kv_get(k0, k1, k2, k3)` | `([u32; 4]) -> i64` | Read the integer value at the four-component key; `0` if absent (or non-integer). | +| `kv_set(k0, k1, k2, k3, value)` | `([u32; 4], i64) -> ()` | Set the key to a signed integer value. | +| `kv_register_constant(name_ptr)` | `(*const u8) -> ()` | Register a constant by null-terminated UTF-8 name (create-if-absent). **Only callable from `register`** — calling it elsewhere fails the tx with `PROHIBITED_HOST_CALL`. | +| `kv_get_constant(name_ptr)` | `(*const u8) -> u32` | Resolve a registered constant's id, for use as a key component. An unknown name fails the tx with `CONSTANT_NOT_FOUND` (the host stops execution rather than return a bogus id). | + +See [Programmable KV state](#programmable-kv-state) and [Named constants & the register phase](#named-constants--the-register-phase) below. + +**Account-existence rule.** Unlike pre-ADR-022, host verbs are **existence-aware**. `credit`, `debit`, `get_flag`, `has_flag`, and `set_flag` operate only on an account whose status lane is non-zero (an account that has been opened — `Operation::OpenAccount`, or lazily via `linked_account`). A balance/flag verb on an *unopened* account sets `ACCOUNT_NOT_FOUND` and **fails the whole transaction** (full rollback). It is not a silent no-op. Open accounts up front, or create program sub-accounts with `linked_account`, before crediting/debiting them. + +**Zero-sum invariant.** The ledger verifies `sum(credits) == sum(debits)` after the function returns. If the function emits an unbalanced set of credits / debits, the transaction is rejected and rolled back with `ZERO_SUM_VIOLATION`. + +**Overdraft policy.** The ledger does not enforce negative-balance checks inside functions — balance arithmetic saturates and a credit may legitimately drive a balance negative. If you need overdraft protection, read the balance and return a non-zero status to trigger rollback: ```rust -if get_balance(sender) < amount as i64 { - return 1; // INSUFFICIENT_FUNDS +if balance(sender) < amount as i64 { + return Status::fail(1); // INSUFFICIENT_FUNDS } ``` -No randomness, no wall clock, no I/O, no WASM threads, no atomics. The ABI is deliberately small so every execution is deterministic. +No randomness, no wall clock, no I/O, no WASM threads, no atomics — every host verb is deterministic, so the ABI stays replayable on followers. --- -## Return values & rollback +## Programmable KV state -The `execute` return value is the transaction's final status: +ADR-023 adds a single typed key→value map that a function reads and writes **atomically with its balance effects**, replicated through the same WAL. It is intentionally minimal: one map, no scopes, no registers, no tree. -- `0` — the host-side credits and debits become part of the WAL transaction; the zero-sum invariant is verified; if it holds, the transaction commits. -- any non-zero value — the transaction is **fully rolled back**: every credit / debit the function applied is reversed in the same way a failed built-in operation would be. The `TxMetadata` record is still written, tagged with the function's CRC32C, and its `status` field carries the returned `u8` so auditors can distinguish unsuccessful named operations by exact reason. +- **Key**: four `u32` components — `[u32; 4]`. Shorter logical keys zero-pad the trailing slots; structure (a "namespace" component, an account component, …) is convention inside those four slots. Build one with the SDK's [`key!`](#the-roda-wasm-abi-guest-sdk) macro. +- **Value**: a signed `i64`. +- **Absent read**: `kv_get` on a key that was never set returns `0` (also `0` if the stored value is somehow non-integer). There is no separate "exists" probe. +- **Mutation & rollback**: each `kv_set` records an undo entry and logs a packed `KvEntry` follower (folded into the trailer CRC, like a credit/debit). If the transaction fails, the whole map is restored; followers and recovery reapply `KvEntry` records without re-running the module. + +A key component may also carry a **constant id** rather than a plain integer (see below); the host tags it so the packed key remembers it was a constant. + +--- + +## Named constants & the register phase -Host failures (linking, instantiation, traps) surface as `Status::INVALID_OPERATION` (numeric `5`). +A module is **stateless** — it keeps nothing in WASM globals between calls. To give a key component a stable, named meaning, a module declares **named constants** and resolves them by name at execute time; the host owns the name→id map. -Standard status values used by the ledger: +This is what the optional second export, `register`, is for: -| Value | Name | -|-------|------| -| `0` | `NONE` (success) | -| `1` | `INSUFFICIENT_FUNDS` | -| `2` | `ACCOUNT_NOT_FOUND` | -| `3` | `ZERO_SUM_VIOLATION` | -| `4` | `ENTRY_LIMIT_EXCEEDED` | -| `5` | `INVALID_OPERATION` | -| `6` | `ACCOUNT_LIMIT_EXCEEDED` | -| `7` | `DUPLICATE` | -| `8..=127` | reserved for future standard reasons | -| `128..=255` | user-defined | +- A module *may* export `register` with signature `() -> ()` (alongside the required `execute`). It is optional; a module with no constants omits it. +- The host calls `register` **once per instantiation** (and after any re-instantiation), in a dedicated **Register phase**. The only host verb callable in that phase is `kv_register_constant` — any balance / flag / KV-data verb called during `register` fails the tx with `PROHIBITED_HOST_CALL`. Symmetrically, `kv_register_constant` is rejected outside `register`. +- `register` runs at **registration time**, inside the same `FunctionRegistration` transaction that installs the module. If `register` returns non-zero (e.g. a prohibited call), the registration transaction rolls back and the install is reverted — the module is registered atomically with its constants, or not at all. +- In `execute`, resolve a constant by name with `kv_get_constant("NAME")`, which returns the `u32` id to use as a key component. An unknown name fails the tx with `CONSTANT_NOT_FOUND`. Constant names are bounded to **32 bytes** (`KV_CONSTANT_NAME_MAX`); a longer name fails with `CONSTANT_NAME_TOO_LONG`. + +```rust +// register the name once … +register!(|| { + kv_register_constant(c"counter"); +}); + +// … then resolve it by name on every call (no global caches the id) +execute!(|p| { + let k = key!(kv_get_constant(c"counter"), p.account(0)); + kv_set(k, kv_get(k) + 1); + Status::OK +}); +``` + +Constants and KV cells are checkpointed together in the per-segment `kv_snapshot_{N}` file (see [Storage layout](#storage-layout-on-disk)). + +--- + +## Return values & rollback + +The `execute` return value is the transaction's final status: + +- `0` — the host-side credits and debits become part of the WAL transaction; the zero-sum invariant is verified; if it holds, the transaction commits. +- any non-zero value — the transaction is **fully rolled back**: every credit / debit the function applied is reversed in the same way a failed built-in operation would be. The `TxMetadata` record is still written, tagged with the function's CRC32C, and its `fail_reason` field carries the returned `u8` so auditors can distinguish unsuccessful named operations by exact reason. + +Host failures (linking, instantiation, traps) surface as `INVALID_OPERATION` (numeric `5`). + +Standard status values are defined by `FailReason` in `crates/storage/src/entities.rs`. The guest SDK does not name them — it only offers `Status::OK` (0) and `Status::fail(code)` — but a function returning these codes is interpreted by the ledger as the corresponding reason: + +| Value | Name | Notes | +|-------|------|-------| +| `0` | `NONE` (success) | | +| `1` | `INSUFFICIENT_FUNDS` | | +| `2` | `ACCOUNT_NOT_FOUND` | set by a balance/flag verb on an unopened account | +| `3` | `ZERO_SUM_VIOLATION` | | +| `4` | `ENTRY_LIMIT_EXCEEDED` | | +| `5` | `INVALID_OPERATION` | also the catch-all for traps / link / instantiation failures | +| `6` | — | **retired** (was `ACCOUNT_LIMIT_EXCEEDED`; the id allocator panics on exhaustion instead) | +| `7` | `DUPLICATE` | | +| `8` | `PROHIBITED_HOST_CALL` | a verb called in the wrong phase (ADR-023 §6) | +| `9` | `CONSTANT_NOT_FOUND` | `kv_get_constant` on an unregistered name | +| `10` | `CONSTANT_NAME_TOO_LONG` | constant name exceeds `KV_CONSTANT_NAME_MAX` (32 bytes) | +| `11..=127` | reserved for future standard reasons | | +| `128..=255` | user-defined | | Returning `128..=255` is how functions report domain-specific errors without polluting the standard status namespace. @@ -126,10 +216,15 @@ Returning `128..=255` is how functions report domain-specific errors without pol ## Writing a function -### Rust +### The `roda-wasm-abi` guest SDK + +The supported way to author a module in Rust is the [`roda-wasm-abi`](../crates/roda-wasm-abi) crate (ADR-026). It wraps the whole guest ABI — the fixed 8×`i64` → `i32` calling convention, every host verb, the `key!` builder, and named constants — behind the `execute!` / `register!` macros and a typed `Params`. It is `#![no_std]`, allocator-free, and dependency-free, so it compiles cleanly to `wasm32-unknown-unknown`. ```toml # Cargo.toml +[dependencies] +roda-wasm-abi = "0.1" + [lib] crate-type = ["cdylib"] @@ -139,118 +234,129 @@ lto = true strip = true ``` +A balanced transfer — debit `account(0)`, credit `account(2)`, by `amount(1)` (the real `examples/transfer`): + ```rust // src/lib.rs +use roda_wasm_abi::{credit, debit, execute, Status}; + +execute!(|p| { + let amount = p.amount(1); + debit(p.account(0), amount); + credit(p.account(2), amount); + Status::OK +}); +``` + +The `execute!` body receives a [`Params`] with typed accessors — `p.account(i)`, `p.amount(i)`, `p.get(i)` (raw `i64`) — and returns anything convertible to `Status`: a `Status`, a `Result<(), u8>`, or `()` (unconditional commit). Return `Status::OK` to commit (subject to the zero-sum check) or `Status::fail(code)` to roll the whole transaction back. + +A stateful example — a KV counter scoped under a named constant (the real `examples/counter`): + +```rust +use roda_wasm_abi::{ + credit, debit, execute, key, kv_get, kv_get_constant, kv_register_constant, kv_set, register, + Status, +}; + +register!(|| { + kv_register_constant(c"counter"); +}); + +execute!(|p| { + let account = p.account(0); + let k = key!(kv_get_constant(c"counter"), account); + let next = kv_get(k) + 1; + kv_set(k, next); + + debit(0, next as u64); // into the system account + credit(account, next as u64); // out of the caller's account + Status::OK +}); +``` + +The SDK's safe verb wrappers (`crates/roda-wasm-abi/src/{account,kv}.rs`) mirror the host signedness: `balance` (the wrapper over `get_balance`), `credit`, `debit`, `linked_account`, `get_flag` / `has_flag` / `set_flag`, and `kv_get` / `kv_set` / `kv_register_constant` / `kv_get_constant`. + +Compile: + +```bash +rustup target add wasm32-unknown-unknown +cargo build --release --target wasm32-unknown-unknown +# produces target/wasm32-unknown-unknown/release/.wasm +``` + +For tests and tooling, the host-only `tools` feature (`roda-wasm-abi = { version = "0.1", features = ["tools"] }`) compiles module source to wasm in-process via `roda_wasm_abi::tools::compile_to_wasm`. + +### Appendix: hand-written ABI + +You do not need this for Rust — prefer the SDK above. It documents the raw ABI for other toolchains (AssemblyScript, hand-written WAT) and for understanding what the macros emit. Hand-written modules are the form used throughout the engine's own test/bench suite. + +The contract is unchanged: export `execute` with `(i64 × 8) -> i32`, optionally export `register` with `() -> ()`, and import host verbs from module `ledger`. + +Raw `extern "C"` in Rust (equivalent to the macro-based transfer above): + +```rust #[link(wasm_import_module = "ledger")] unsafe extern "C" { fn credit(account_id: u64, amount: u64); fn debit(account_id: u64, amount: u64); - fn get_balance(account_id: u64) -> i64; } -/// Transfer `amount` from `from` to `to`, taking a `fee_bps` (basis -/// points) fee into `fee_acct`. -/// -/// Params: -/// param0: sender -/// param1: receiver -/// param2: fee_acct -/// param3: amount -/// param4: fee_bps (e.g. 50 = 0.5%) #[unsafe(no_mangle)] pub extern "C" fn execute( - param0: i64, param1: i64, param2: i64, param3: i64, - param4: i64, _p5: i64, _p6: i64, _p7: i64, + p0: i64, p1: i64, p2: i64, _p3: i64, + _p4: i64, _p5: i64, _p6: i64, _p7: i64, ) -> i32 { - let sender = param0 as u64; - let receiver = param1 as u64; - let fee_acct = param2 as u64; - let amount = param3 as u64; - let fee_bps = param4 as u64; - let fee = amount * fee_bps / 10_000; - unsafe { - if get_balance(sender) < (amount + fee) as i64 { - return 1; // INSUFFICIENT_FUNDS - } - credit(sender, amount + fee); - debit(receiver, amount); - debit(fee_acct, fee); + debit(p0 as u64, p1 as u64); + credit(p2 as u64, p1 as u64); } 0 } ``` -Compile: - -```bash -cargo build --target wasm32-unknown-unknown --release -# produces target/wasm32-unknown-unknown/release/.wasm -``` - -### AssemblyScript +AssemblyScript: ```typescript // assembly/index.ts @external("ledger", "credit") declare function credit(account_id: u64, amount: u64): void; - @external("ledger", "debit") declare function debit(account_id: u64, amount: u64): void; -@external("ledger", "get_balance") -declare function get_balance(account_id: u64): i64; - export function execute( - param0: i64, param1: i64, param2: i64, param3: i64, - param4: i64, _p5: i64, _p6: i64, _p7: i64, + p0: i64, p1: i64, p2: i64, _p3: i64, + _p4: i64, _p5: i64, _p6: i64, _p7: i64, ): i32 { - const sender = param0; - const receiver = param1; - const feeAcct = param2; - const amount = param3; - const feeBps = param4; - const fee = amount * feeBps / 10_000; - - if (get_balance(sender) < (amount + fee)) { - return 1; // INSUFFICIENT_FUNDS - } - credit(sender, amount + fee); - debit(receiver, amount); - debit(feeAcct, fee); + debit(p0, p1); + credit(p2, p1); return 0; } ``` -Compile: - ```bash asc assembly/index.ts -o build/function.wasm --optimize --runtime stub ``` -### Hand-written WAT - -Useful for minimal-size tests and examples. Every WAT snippet in the test suite follows this template: +Hand-written WAT — the form used by the engine's tests and the `load_wasm` generator: ```wat (module (import "ledger" "credit" (func $credit (param i64 i64))) (import "ledger" "debit" (func $debit (param i64 i64))) - (import "ledger" "get_balance" (func $get_balance (param i64) (result i64))) (func (export "execute") (param i64 i64 i64 i64 i64 i64 i64 i64) (result i32) - ;; credit(param0, param1) - local.get 0 local.get 1 call $credit + ;; debit(param0, param1) + local.get 0 local.get 1 call $debit - ;; debit(param2, param1) - local.get 2 local.get 1 call $debit + ;; credit(param2, param1) + local.get 2 local.get 1 call $credit i32.const 0)) ``` -Compile via the `wat` crate (already a dependency in test / bench paths): +Compile via the `wat` crate (a dependency in test / bench paths): ```rust let bytes = wat::parse_str(WAT)?; @@ -335,6 +441,8 @@ message Function { } ``` +On the wire `params` is a `repeated int64`, but the Rust `Operation::Function.params` type is a **fixed `[i64; 8]`** — the server zero-pads a short list and truncates anything beyond 8 slots to land on that fixed arity. + ```bash grpcurl -d '{ "function": { @@ -382,19 +490,21 @@ let result = client .await?; ``` -Every `Operation::Function` produces a normal transaction in the WAL with a `TxMetadata.tag` of the form: +Every `Operation::Function` produces a normal transaction in the WAL with an 8-byte `TxMetadata.tag` built by `build_wasm_tag`: ``` -b"fnw\n" ++ crc32c[0..4] (8 bytes total) +[ b'f', b'n', b'w', b'\n', crc[0], crc[1], crc[2], crc[3] ] // crc = crc32c.to_le_bytes() ``` -`roda-ctl unpack` renders it as: +i.e. a 4-byte literal prefix `fnw\n` followed by the binary's CRC32C in **little-endian** byte order. + +How `roda-ctl unpack` renders it depends on the CRC bytes. `encode_tag` (`storage/src/entities.rs`) trims trailing NULs and prints the tag as UTF-8 *only if the whole thing is valid UTF-8*; otherwise it dumps all 8 bytes as 16 lowercase hex digits. Because the CRC bytes are usually not valid UTF-8, the typical render is the hex form — the `fnw\n` prefix is then visible as its hex bytes `666e770a`: ```json -{"type": "TxMetadata", "tx_id": 441001, "tag": "fnw\n4a2f1c3d", ...} +{"type": "TxMetadata", "tx_id": 441001, "tag": "666e770a3d1c2f4a", ...} ``` -The CRC32C identifies the exact binary that executed — cross-reference it with the `FunctionRegistered` WAL record or with a `ListFunctions` response to resolve the name / version. +(here CRC `0x4a2f1c3d` little-endian is `3d 1c 2f 4a`). The embedded CRC32C identifies the exact binary that executed — cross-reference it with the `FunctionRegistered` WAL record or a `ListFunctions` response to resolve the name / version. --- @@ -438,6 +548,8 @@ data/ ├── snapshot_000002.crc ├── function_snapshot_000002.bin # function-registry snapshot (same trigger) ├── function_snapshot_000002.crc +├── kv_snapshot_000002.bin # KV state + interned constants (same trigger, ADR-023) +├── kv_snapshot_000002.crc └── functions/ ├── fee_transfer_v1.wasm # binary under its version ├── fee_transfer_v2.wasm # replaced version (override) @@ -446,7 +558,7 @@ data/ Function binaries are written atomically (temp file + rename). Unregister truncates the file to 0 bytes — it is **not** deleted, preserving the audit trail. -The function snapshot is emitted on the same `snapshot_frequency` trigger as the balance snapshot, so recovery always finds a paired `snapshot_{N}.bin` + `function_snapshot_{N}.bin` at the same segment boundary. +The function snapshot is emitted on the same `snapshot_frequency` trigger as the balance snapshot, so recovery always finds a paired `snapshot_{N}.bin` + `function_snapshot_{N}.bin` at the same segment boundary. The **KV snapshot** (`kv_snapshot_{N}.bin` + `.crc`, ADR-023) is written on that same seal trigger — it holds the full programmable KV map plus the interned constants (`id → name`) as-of that segment, in one file. --- @@ -461,10 +573,12 @@ Registration is durable **before** `register_function` returns. The call blocks **Recovery on clean restart or crash** proceeds as follows: 1. Load the latest `function_snapshot_{N}.bin`. For each record with `crc32c != 0`, read `functions/{name}_v{version}.wasm` and compile it into the runtime. -2. Replay every `FunctionRegistered` WAL record in segments after the snapshot: - - `crc32c != 0` → load the referenced version (replaces any older handler). - - `crc32c == 0` → unload the handler. -3. Resume normal operation. +2. Seed the KV map and constant registry from the paired `kv_snapshot_{N}.bin` at the same segment boundary (ADR-023 §7). +3. Replay every WAL record in segments after the snapshot: + - `FunctionRegistered`, `crc32c != 0` → load the referenced version (replaces any older handler). + - `FunctionRegistered`, `crc32c == 0` → unload the handler. + - `KvEntry` / `KvConstant` → reapply to the KV map / constant registry (no re-execution). +4. Resume normal operation. A failure to read a binary or install a handler during recovery is **non-recoverable**: the server aborts startup rather than continue with a registry that diverges from the WAL. The CRC32C embedded in every `FunctionRegistered` record lets recovery detect silent disk corruption before anything transactional runs. @@ -476,9 +590,9 @@ The runtime is intentionally narrow: - **No randomness.** No host API exposes a PRNG. - **No wall clock.** Functions do not see time; tag timestamps are decided by the host. -- **No I/O, no network, no filesystem.** Only the 3 ledger host calls are available. +- **No I/O, no network, no filesystem.** Only the `ledger` host verbs are available, and every one of them is deterministic. - **No threads, no atomics.** Functions run single-threaded. -- **No persistent memory.** A function has no state that survives between invocations: every call runs against a fresh wasmtime instance (internally cached by the engine for speed; no durable state is carried). +- **No persistent memory.** A function has no WASM-global state that survives between invocations: it resolves named constants by name each call (`kv_get_constant`) and reads/writes durable state only through the KV verbs and balances. The host owns the name→id map; the module stays stateless. - **Sandboxed.** A trap or infinite loop cannot corrupt ledger state — the transaction is rolled back and the handler is left intact for subsequent calls. This is the property we need for future Raft replication: the leader executes the WASM function, and followers apply the resulting entries directly — no re-execution, no divergence. @@ -489,30 +603,30 @@ This is the property we need for future Raft replication: the leader executes th The runtime is tuned for low per-call overhead on the hot path. -- **Per-call cost**: one `HashMap::get` on the per-Transactor caller cache, one `TypedFunc::call`, two host imports per credit / debit crossing. +- **Per-call cost**: one `HashMap::get` on the per-Transactor caller cache, one `TypedFunc::call`, plus one host crossing per verb the body invokes. - **Cache invalidation** is *per-name*: registering `foo` does not evict the cached entry for `bar`. Each cached entry stores the `update_seq` it was verified at; the next lookup either short-circuits (seq unchanged) or does a shared-registry read to reconcile just that one name. - **One wasmtime `Engine`** per ledger (shared via `Arc`). - **One `Linker`** with host imports wired exactly once at ledger startup. -- **One `Store`** per Transactor, long-lived across calls. Function state is carried in host `TransactorState`, not in WASM-visible globals. -- **Instantiation** happens once per `(name, crc)` pair on first lookup; the resulting `TypedFunc` is cached and reused. +- **One `Store`** per Transactor, long-lived across calls. The transactor state (`Computer`) is reached through the store's `WasmStoreData`, not via WASM-visible globals. +- **Instantiation** happens once per `(name, crc)` pair on first lookup; the resulting `execute` (and optional `register`) `TypedFunc` is cached and reused. -Empirical numbers from the current build (Apple M-series, release build): +Indicative numbers (Apple M-series, release build). These predate the ADR-022/023 (#110) programmable-state rewrite — treat them as ballpark, not current; re-run the benches below for live figures: | Benchmark | Native `Deposit` | WASM `Function` (same effect) | |-----------|------------------|-------------------------------| -| `TransactorRunner::process_direct` (1 tx) | ~132 ns/op | ~335 ns/op | -| `TransactorRunner::process_direct_batch` (1000) | ~133 ns/op | ~286 ns/op | -| End-to-end `--wait` load test TPS, 1M accounts | ~819 k | ~814 k | +| `transaction_runner_bench` (1 tx) | ~132 ns/op | ~335 ns/op | +| `transaction_runner_bench` (1000-batch) | ~133 ns/op | ~286 ns/op | +| End-to-end `--wait` load test TPS, 1M accounts | ~819 k | ~814 k | -The per-op overhead at the Transactor level is ~200 ns — pure host-crossing cost. At the pipeline level (gRPC → WAL commit → response) it is invisible: the WAL commit path dominates. +The per-op overhead at the Transactor level is roughly a couple hundred ns — pure host-crossing cost. At the pipeline level (gRPC → WAL commit → response) it is invisible: the WAL commit path dominates. Run the comparison yourself: ```bash -cargo bench --bench transaction_runner_bench # native -cargo bench --bench transaction_runner_bench_wasm # WASM -cargo run --release --bin load -- --wait --duration 30 -cargo run --release --bin load_wasm -- --wait --duration 30 +cargo bench -p ledger --bench transaction_runner_bench # native +cargo bench -p ledger --bench transaction_runner_bench_wasm # WASM +cargo run -p ledger --release --bin load -- --wait --duration 30 +cargo run -p ledger --release --bin load_wasm -- --wait --duration 30 ``` --- @@ -530,9 +644,10 @@ Validation runs at `register_function` before any disk write: | Module parses | validated by wasmtime | | Exports `execute` | required | | `execute` signature | exactly `(i64 × 8) -> i32` | -| `execute` host imports | only `ledger.credit` / `ledger.debit` / `ledger.get_balance` | +| `register` signature | if present, exactly `() -> ()` (the export is optional) | +| Host imports | **not** restricted by an allow-list — validation checks only the `execute` / `register` *signatures*. Any of the eleven `ledger` verbs may be imported; importing an unknown name fails later, at instantiation. | -A binary that fails any of these returns `InvalidArgument` from gRPC or `io::ErrorKind::InvalidData` / `InvalidInput` from the Rust API. Nothing is written to disk and no WAL record is produced. +A binary that fails any of these returns `InvalidArgument` from gRPC or `io::ErrorKind::InvalidData` / `InvalidInput` from the Rust API. Nothing is written to disk and no WAL record is produced. (A module that imports a name outside the `ledger` verb set passes validation but fails to instantiate at first invocation, surfacing as `INVALID_OPERATION`.) --- @@ -549,13 +664,22 @@ A function by this name is already loaded. Pass `override_existing = true` to re **Operations return `ZERO_SUM_VIOLATION` (status `3`)** The function's total credits did not match total debits. Every `credit(a, N)` needs a matching `debit(b, N)` somewhere else in the same invocation (or vice versa). -**Unexpected balances after a restart** -Inspect both snapshots at the last sealed segment id: +**Operations return `ACCOUNT_NOT_FOUND` (status `2`)** +The function called a balance or flag verb on an account that was never opened. Open it first (`Operation::OpenAccount`) or create the program sub-account with `linked_account` before crediting/debiting it (ADR-022). + +**Operations return `PROHIBITED_HOST_CALL` (status `8`)** +A verb was called in the wrong phase (ADR-023 §6): `kv_register_constant` outside `register`, or any balance / flag / KV-data verb *inside* `register`. Register constants only in `register`; do everything else in `execute`. + +**Operations return `CONSTANT_NOT_FOUND` (status `9`) or `CONSTANT_NAME_TOO_LONG` (status `10`)** +`kv_get_constant` resolved a name the module never registered in `register` (`9`), or a constant name exceeded 32 bytes (`10`). Ensure `register` declares every name `execute` resolves, and keep names ≤ 32 bytes. + +**Unexpected balances or KV state after a restart** +Inspect the snapshots at the last sealed segment id: ```bash -ls data/function_snapshot_*.bin +ls data/function_snapshot_*.bin data/kv_snapshot_*.bin roda-ctl unpack data/wal_000NNN.bin ``` -You should see a `FunctionRegistered` record for every function that should be loaded and a matching snapshot file at a later or equal segment id. +You should see a `FunctionRegistered` record for every function that should be loaded and matching `function_snapshot_{N}` / `kv_snapshot_{N}` files at a later or equal segment id. **`Snapshot: read_function(name vN) failed` during startup** The WAL references a binary that is missing from `data/functions/`. This is non-recoverable. Either restore the binary from backup, or manually remove the stale `FunctionRegistered` record from the WAL using `roda-ctl` (at your own risk). @@ -565,5 +689,9 @@ The WAL references a binary that is missing from `data/functions/`. This is non- ## See also - [ADR-014 — WASM Function Registry and Function Operation Execution](./adr/0014-wasm-function-registry.md) +- [ADR-022 — Account Layouts and Program-Defined Accounts](./adr/0022-account-layouts-and-program-defined-accounts.md) +- [ADR-023 — Programmable State (typed KV store + constants)](./adr/0023-programmable-state.md) +- [ADR-026 — `roda-wasm-abi` guest SDK](./adr/0026-roda-wasm-abi.md) +- [`roda-wasm-abi` crate](../crates/roda-wasm-abi) — the official guest SDK, with `examples/transfer` and `examples/counter`. - [Architecture](./03-architecture.md) — for context on the Transactor / WAL / Snapshot pipeline. - [API](./02-api.md) — full API reference for the ledger service.