diff --git a/Makefile b/Makefile index eba0f81..262736a 100644 --- a/Makefile +++ b/Makefile @@ -58,7 +58,8 @@ doc: ## Generates & checks documentation .PHONY: book book: ## Builds the book & serves documentation site - mdbook serve --open docs + @echo "Docs are authored in docs/external/src and ingested by 0xMiden/docs." + @echo "Preview them from the docs repo after running its ingestion workflow." # --- testing ------------------------------------------------------------------------------------- diff --git a/bin/node/docker/docker-compose.yml b/bin/node/docker/docker-compose.yml index 9b1505d..008332b 100644 --- a/bin/node/docker/docker-compose.yml +++ b/bin/node/docker/docker-compose.yml @@ -6,7 +6,7 @@ services: dockerfile: bin/node/docker/Dockerfile ports: - "57292:57292" # gRPC - - "9091:9090" # Health checks + command: ["miden-note-transport-node-bin", "--host", "0.0.0.0", "--database-url", "/app/data/node.db"] environment: - OTEL_ENABLED=true - JSON_LOGGING=true @@ -15,7 +15,6 @@ services: - OTEL_SERVICE_NAME=miden-note-transport-node - RUST_LOG=INFO - DOCKER_ENV=true - - DATABASE_URL=/app/data/node.db volumes: - node_data:/app/data depends_on: diff --git a/docs/external/src/_category_.yml b/docs/external/src/_category_.yml new file mode 100644 index 0000000..54817e0 --- /dev/null +++ b/docs/external/src/_category_.yml @@ -0,0 +1,3 @@ +label: "Note Transport" +position: 5 +collapsed: true diff --git a/docs/external/src/design.md b/docs/external/src/design.md new file mode 100644 index 0000000..22bc0a9 --- /dev/null +++ b/docs/external/src/design.md @@ -0,0 +1,107 @@ +--- +sidebar_position: 2 +title: Design +--- + +# Design + +The note transport node is intentionally small: it accepts note bytes, indexes them by note tag, and returns matching notes to clients. + +## Note flow + +1. A sender creates a private note in a Miden transaction. +2. After the note data is available locally, the sender calls `SendNote` with a serialized note header and note details. +3. The transport node parses the header, extracts the note ID and tag, and stores the header and details in SQLite. +4. A recipient calls `FetchNotes` for one or more tags and receives matching notes with a cursor. +5. The recipient stores the returned cursor and uses it on the next fetch. + +The transport node does not connect to a Miden node and does not know whether a note has been committed on-chain. Clients still need to import fetched notes and sync against the Miden network. + +## Stored data + +The transport node stores: + +- note ID, derived from the serialized header; +- note tag, derived from the serialized header; +- serialized header bytes; +- serialized details bytes; +- creation timestamp; +- `seq`, a monotonic SQLite `AUTOINCREMENT` value. + +The note ID column is unique. Re-sending the same note ID is rejected by the database instead of creating a duplicate row. + +## Cursor pagination + +`FetchNotes` uses a `seq` cursor: + +```protobuf +message FetchNotesRequest { + repeated fixed32 tags = 1; + fixed64 cursor = 2; +} + +message FetchNotesResponse { + repeated TransportNote notes = 1; + fixed64 cursor = 2; +} +``` + +The server returns notes matching any requested tag with `seq > cursor`, ordered by ascending `seq`, up to the server batch size. The response cursor is the highest `seq` returned. A client should persist that value and send it on the next request. + +Current limits: + +- A request may include up to 128 tags. +- A response returns up to 500 notes. +- There is no client-specified `limit` field in the protobuf API. + +The multi-tag query runs in one database snapshot. This avoids a race where separate per-tag queries could advance the cursor past a note inserted between queries. + +## Legacy cursor handling + +Earlier designs used timestamp cursors. Existing clients may have stored timestamp-sized cursor values. The node treats cursor values above `1_000_000_000_000` as legacy timestamp cursors and resets the effective query cursor to `0`. + +This lets upgraded clients recover instead of waiting for `seq` to reach an old timestamp-sized value. + +## Streaming + +`StreamNotes` opens a server-side stream for one tag: + +```protobuf +message StreamNotesRequest { + fixed32 tag = 1; + fixed64 cursor = 2; +} + +message StreamNotesUpdate { + repeated TransportNote notes = 1; + fixed64 cursor = 2; +} +``` + +Internally, a background task polls SQLite every 500 ms for new notes matching active subscriptions and forwards updates through bounded channels. + +The current server implementation does not use the request cursor to initialize subscription state. Use `FetchNotes` for durable catch-up and cursor persistence, then use streaming only as a live update channel. + +## Storage and retention + +The node uses SQLite and embedded migrations. File-backed databases use a larger connection pool. In-memory databases use a single connection because SQLite `:memory:` databases are isolated per connection. + +Notes older than the configured retention period are removed by a maintenance task. + +## Block context + +The current protobuf API does not include commitment block number, note metadata, or inclusion proof fields. The transport node stores only `header` and `details`. + +This means the node cannot tell a client which block committed a fetched note. Clients must reconcile fetched notes with chain state themselves. The client-side lookback workaround and the proposed transport-level block context are tracked separately in [0xMiden/note-transport-service#68](https://github.com/0xMiden/note-transport-service/issues/68). + +## What the node does not do + +The node does not: + +- validate note contents against chain state; +- connect to a Miden node; +- attach commitment block context; +- attach note inclusion proofs; +- inspect or decrypt note details; +- authenticate senders or recipients; +- guarantee delivery after the retention period. diff --git a/docs/external/src/index.md b/docs/external/src/index.md new file mode 100644 index 0000000..3462f1d --- /dev/null +++ b/docs/external/src/index.md @@ -0,0 +1,53 @@ +--- +sidebar_position: 1 +title: Note Transport +description: "Off-chain relay service for private note delivery on Miden." +pagination_prev: null +--- + +# Note Transport + +The Miden note transport service is an off-chain relay for private note delivery. It gives senders a place to publish serialized private notes and gives recipients a way to fetch notes that match the tags they monitor. + +Private note contents are not published on-chain. The chain stores note commitments, while the full note data must reach the recipient through another channel. Note transport is the standard network service for that off-chain delivery path. + +## Start here + + + + How the node stores notes, assigns cursors, routes by tag, and handles current protocol boundaries. + + + CLI flags, Docker Compose, telemetry, storage, ports, retention, and production cautions. + + + Request and response shapes for send, fetch, stream, stats, plus the recommended client sync pattern. + + + +## API surface + +| RPC | Use it for | Current behavior | +| --- | --- | --- | +| `SendNote` | Publish one transported note. | The `header` must decode as a Miden `NoteHeader`; `details` are stored as opaque bytes. | +| `FetchNotes` | Durable catch-up by tag. | Returns notes for one or more tags using a server-assigned `seq` cursor. | +| `StreamNotes` | Live updates for one tag. | Use it after a fetch cycle; current subscriptions do not initialize from the request cursor. | +| `Stats` | Basic operational counts. | Returns aggregate note and tag counts. Per-tag statistics are defined in protobuf but not populated yet. | + +## Transport model + +- **Private payload delivery.** The Miden chain stores note commitments. Note transport carries the full private note data that recipients need to import locally. +- **Tag-based routing.** Notes are indexed by the 32-bit `NoteTag` embedded in note metadata. The node has no account registry or recipient identity model. +- **Client-owned privacy policy.** The node parses note headers only. Clients decide which tags to monitor and whether note details should be encrypted before sending. +- **Temporary mailbox.** Notes are retained for the configured retention window. Delivery is best-effort and clients must persist fetch cursors. + +## Current boundaries + +- **No chain-state validation.** The node does not connect to a Miden node and does not prove that a stored note was committed on-chain. +- **No block context yet.** The current API does not attach commitment block numbers, note metadata, or inclusion proofs to fetched notes. This is tracked in [0xMiden/note-transport-service#68](https://github.com/0xMiden/note-transport-service/issues/68). +- **Duplicate notes are rejected.** SQLite stores note IDs with a uniqueness constraint. Sending the same note twice fails instead of creating duplicate rows. +- **Cursor values are server-owned.** Fetch pagination uses the monotonic SQLite `seq` value returned by the server. Clients should persist returned cursors, not fabricate them. + +## Current implementation + +The current node implementation is a Rust gRPC service backed by SQLite. It stores each note with a monotonic `seq` value assigned at insert time, uses that value for `FetchNotes` pagination, and can export traces and metrics through OpenTelemetry. diff --git a/docs/external/src/operators.md b/docs/external/src/operators.md new file mode 100644 index 0000000..c85cefd --- /dev/null +++ b/docs/external/src/operators.md @@ -0,0 +1,123 @@ +--- +sidebar_position: 3 +title: Operators +--- + +# Operators + +This page covers running a note transport node. + +## Build from source + +From the repository root: + +```bash +cargo install --path bin/node --locked +``` + +This installs the `miden-note-transport-node-bin` binary. + +## Run the node + +The default configuration binds to localhost and stores notes in an in-memory SQLite database: + +```bash +miden-note-transport-node-bin +``` + +For a reachable node with persistent storage: + +```bash +miden-note-transport-node-bin \ + --host 0.0.0.0 \ + --port 57292 \ + --database-url /var/lib/miden-note-transport/node.db \ + --retention-days 30 +``` + +## CLI flags + +| Flag | Default | Description | +| --- | --- | --- | +| `--host` | `127.0.0.1` | Address to bind to. | +| `--port` | `57292` | gRPC port. | +| `--database-url` | `:memory:` | SQLite database URL or file path. Use a file path for persistence. | +| `--retention-days` | `30` | How long to retain notes before cleanup. | +| `--max-note-size` | `512000` | Maximum note details size in bytes. | +| `--max-connections` | `4096` | Maximum concurrent gRPC connections. | +| `--request-timeout` | `4` | Per-request timeout in seconds. | + +The CLI flags above are parsed as command-line arguments. They are not currently read from `DATABASE_URL` or similarly named environment variables. + +## Telemetry and logging + +Telemetry is configured through environment variables: + +| Variable | Default | Description | +| --- | --- | --- | +| `OTEL_ENABLED` | `false` | Enables OpenTelemetry export when set to `true`. | +| `OTEL_TRACES_ENDPOINT` | `http://localhost:4317` | OTLP endpoint for trace and metric export. | +| `JSON_LOGGING` | `false` | Emits JSON logs when set to `true`. | +| `RUST_LOG` | `INFO` | Standard Rust tracing filter. | + +Example: + +```bash +OTEL_ENABLED=true \ +OTEL_TRACES_ENDPOINT=http://otel-collector:4317 \ +JSON_LOGGING=true \ +RUST_LOG=INFO \ +miden-note-transport-node-bin --host 0.0.0.0 --database-url /var/lib/miden-note-transport/node.db +``` + +## Docker Compose + +The repository includes a Docker Compose setup for the node plus telemetry services: + +```bash +make docker-node-up +``` + +This starts: + +- note transport node; +- OpenTelemetry Collector; +- Tempo; +- Prometheus; +- Grafana. + +Use: + +```bash +make docker-node-down +``` + +to stop the stack. + +The Compose node service passes `--database-url /app/data/node.db` and mounts `/app/data` on the `node_data` volume, so note storage survives container restarts. + +## Ports + +| Port | Service | +| --- | --- | +| `57292` | Note transport gRPC API. | +| `4317` | OTLP gRPC receiver in the collector. | +| `4318` | OTLP HTTP receiver in the collector. | +| `3000` | Grafana. | +| `9090` | Prometheus. | +| `3200` | Tempo. | + +The note transport node exposes gRPC health through the same gRPC server, not a separate HTTP health port. + +## Database behavior + +Use a file-backed SQLite path for production-like deployments. The default `:memory:` database is useful for local testing but loses all notes on restart. + +The node runs embedded migrations at startup. The current schema stores note IDs with a uniqueness constraint and uses a monotonic `seq` column for pagination. + +## Operational cautions + +- Treat debug logs as sensitive. Note IDs and tags can be correlated with user activity. +- Configure a retention period that matches the expected offline window for your users. +- Monitor request errors. Duplicate note IDs or invalid note headers are rejected. +- Use `FetchNotes` for durable catch-up. Streaming is best used as a live update channel after a fetch cycle. diff --git a/docs/external/src/users.md b/docs/external/src/users.md new file mode 100644 index 0000000..70c09a4 --- /dev/null +++ b/docs/external/src/users.md @@ -0,0 +1,156 @@ +--- +sidebar_position: 4 +title: Users +--- + +# Users + +This page covers integrating with the note transport gRPC API. + +## API surface + +The service is defined in `proto/proto/miden_note_transport.proto`. + +```protobuf +service MidenNoteTransport { + rpc SendNote(SendNoteRequest) returns (SendNoteResponse); + rpc FetchNotes(FetchNotesRequest) returns (FetchNotesResponse); + rpc StreamNotes(StreamNotesRequest) returns (stream StreamNotesUpdate); + rpc Stats(google.protobuf.Empty) returns (StatsResponse); +} +``` + +## Send a note + +`SendNote` stores one note: + +```protobuf +message SendNoteRequest { + TransportNote note = 1; +} + +message SendNoteResponse {} + +message TransportNote { + bytes header = 1; + bytes details = 2; +} +``` + +`header` must be a serialized Miden `NoteHeader`. The node parses it to extract the note ID and tag. `details` is opaque to the transport node and may contain encrypted note details. + +The server rejects: + +- requests without a note; +- headers that cannot be parsed as `NoteHeader`; +- details larger than the configured `--max-note-size`; +- duplicate note IDs. + +## Fetch notes + +`FetchNotes` returns notes for one or more tags: + +```protobuf +message FetchNotesRequest { + repeated fixed32 tags = 1; + fixed64 cursor = 2; +} + +message FetchNotesResponse { + repeated TransportNote notes = 1; + fixed64 cursor = 2; +} +``` + +Use this flow: + +1. Start with `cursor = 0`. +2. Send all tags the client wants to check, up to 128 tags. +3. Import or process the returned notes. +4. Persist the response `cursor`. +5. Repeat with the stored cursor. + +The response cursor is the highest server-side `seq` value returned in that response. Never fabricate cursor values; use values returned by the server. + +The server batch size is 500 notes. If a response contains many notes, call `FetchNotes` again with the returned cursor until the response is empty or smaller than the batch size. + +## Stream notes + +`StreamNotes` provides a server-side stream for one tag: + +```protobuf +message StreamNotesRequest { + fixed32 tag = 1; + fixed64 cursor = 2; +} + +message StreamNotesUpdate { + repeated TransportNote notes = 1; + fixed64 cursor = 2; +} +``` + +Use streaming as a live update channel. For durable sync, first call `FetchNotes` and persist its cursor. + +Current behavior to account for: + +- The protobuf request includes a `cursor`, but the current server implementation does not seed subscription state from that field. +- Subscriptions are per tag. +- The streamer polls for updates every 500 ms. +- If a subscriber cannot keep up with the bounded channel, the subscription is dropped. + +On reconnect, run `FetchNotes` with your persisted cursor before opening a new stream. + +## Stats + +`Stats` returns aggregate counts: + +```protobuf +message StatsResponse { + uint64 total_notes = 1; + uint64 total_tags = 2; + repeated TagStats notes_per_tag = 3; +} + +message TagStats { + fixed32 tag = 1; + uint64 note_count = 2; + google.protobuf.Timestamp last_activity = 3; +} +``` + +The current server returns `total_notes` and `total_tags`. Per-tag statistics are not populated yet. + +## Client sync pattern + +A typical client should: + +1. Configure a note transport endpoint. +2. Track the note tags it needs to monitor. +3. Fetch notes during sync using the stored transport cursor. +4. Import fetched notes into the client. +5. Sync with the Miden node to reconcile note commitments. +6. Persist the returned transport cursor only after the fetched notes have been handled successfully. + +The transport node does not provide commitment block numbers or inclusion proofs. Clients must still handle chain-state reconciliation. The block-context improvement is tracked in [0xMiden/note-transport-service#68](https://github.com/0xMiden/note-transport-service/issues/68). + +## Troubleshooting + +### Notes do not appear + +- Check that the sender actually called `SendNote`. +- Check that the recipient is fetching the same tag stored in the note header. +- Check whether the note expired under the node retention policy. +- Reset the local transport cursor to `0` if client state is suspected to be ahead of the server. + +### Duplicate send fails + +The database stores note IDs uniquely. Sending the same note twice is rejected instead of producing two stored rows. + +### Streaming misses notes + +Use `FetchNotes` for catch-up. Streaming is not a replacement for durable cursor sync in the current implementation. + +### Large notes are rejected + +The `--max-note-size` setting applies to the note details size. Increase it on the operator side only if the deployment is prepared to accept larger payloads.