Add source-owned note transport docs

Makefile

@@ -58,7 +58,8 @@ doc: ## Generates & checks documentation
 
 .PHONY: book
 book: ## Builds the book & serves documentation site
-	mdbook serve --open docs
+	@echo "Docs are authored in docs/external/src and ingested by 0xMiden/docs."
+	@echo "Preview them from the docs repo after running its ingestion workflow."
 
 # --- testing -------------------------------------------------------------------------------------
 

bin/node/docker/docker-compose.yml

@@ -6,7 +6,7 @@ services:
       dockerfile: bin/node/docker/Dockerfile
     ports:
       - "57292:57292"  # gRPC
-      - "9091:9090"  # Health checks
+    command: ["miden-note-transport-node-bin", "--host", "0.0.0.0", "--database-url", "/app/data/node.db"]
     environment:
       - OTEL_ENABLED=true
       - JSON_LOGGING=true
@@ -15,7 +15,6 @@ services:
       - OTEL_SERVICE_NAME=miden-note-transport-node
       - RUST_LOG=INFO
       - DOCKER_ENV=true
-      - DATABASE_URL=/app/data/node.db
     volumes:
       - node_data:/app/data
     depends_on:

docs/external/src/_category_.yml

@@ -0,0 +1,3 @@
+label: "Note Transport"
+position: 5
+collapsed: true

docs/external/src/design.md

@@ -0,0 +1,107 @@
+---
+sidebar_position: 2
+title: Design
+---
+
+# Design
+
+The note transport node is intentionally small: it accepts note bytes, indexes them by note tag, and returns matching notes to clients.
+
+## Note flow
+
+1. A sender creates a private note in a Miden transaction.
+2. After the note data is available locally, the sender calls `SendNote` with a serialized note header and note details.
+3. The transport node parses the header, extracts the note ID and tag, and stores the header and details in SQLite.
+4. A recipient calls `FetchNotes` for one or more tags and receives matching notes with a cursor.
+5. The recipient stores the returned cursor and uses it on the next fetch.
+
+The transport node does not connect to a Miden node and does not know whether a note has been committed on-chain. Clients still need to import fetched notes and sync against the Miden network.
+
+## Stored data
+
+The transport node stores:
+
+- note ID, derived from the serialized header;
+- note tag, derived from the serialized header;
+- serialized header bytes;
+- serialized details bytes;
+- creation timestamp;
+- `seq`, a monotonic SQLite `AUTOINCREMENT` value.
+
+The note ID column is unique. Re-sending the same note ID is rejected by the database instead of creating a duplicate row.
+
+## Cursor pagination
+
+`FetchNotes` uses a `seq` cursor:
+
+```protobuf
+message FetchNotesRequest {
+    repeated fixed32 tags = 1;
+    fixed64 cursor = 2;
+}
+
+message FetchNotesResponse {
+    repeated TransportNote notes = 1;
+    fixed64 cursor = 2;
+}
+```
+
+The server returns notes matching any requested tag with `seq > cursor`, ordered by ascending `seq`, up to the server batch size. The response cursor is the highest `seq` returned. A client should persist that value and send it on the next request.
+
+Current limits:
+
+- A request may include up to 128 tags.
+- A response returns up to 500 notes.
+- There is no client-specified `limit` field in the protobuf API.
+
+The multi-tag query runs in one database snapshot. This avoids a race where separate per-tag queries could advance the cursor past a note inserted between queries.
+
+## Legacy cursor handling
+
+Earlier designs used timestamp cursors. Existing clients may have stored timestamp-sized cursor values. The node treats cursor values above `1_000_000_000_000` as legacy timestamp cursors and resets the effective query cursor to `0`.
+
+This lets upgraded clients recover instead of waiting for `seq` to reach an old timestamp-sized value.
+
+## Streaming
+
+`StreamNotes` opens a server-side stream for one tag:
+
+```protobuf
+message StreamNotesRequest {
+    fixed32 tag = 1;
+    fixed64 cursor = 2;
+}
+
+message StreamNotesUpdate {
+    repeated TransportNote notes = 1;
+    fixed64 cursor = 2;
+}
+```
+
+Internally, a background task polls SQLite every 500 ms for new notes matching active subscriptions and forwards updates through bounded channels.
+
+The current server implementation does not use the request cursor to initialize subscription state. Use `FetchNotes` for durable catch-up and cursor persistence, then use streaming only as a live update channel.
+
+## Storage and retention
+
+The node uses SQLite and embedded migrations. File-backed databases use a larger connection pool. In-memory databases use a single connection because SQLite `:memory:` databases are isolated per connection.
+
+Notes older than the configured retention period are removed by a maintenance task.
+
+## Block context
+
+The current protobuf API does not include commitment block number, note metadata, or inclusion proof fields. The transport node stores only `header` and `details`.
+
+This means the node cannot tell a client which block committed a fetched note. Clients must reconcile fetched notes with chain state themselves. The client-side lookback workaround and the proposed transport-level block context are tracked separately in [0xMiden/note-transport-service#68](https://github.com/0xMiden/note-transport-service/issues/68).
+
+## What the node does not do
+
+The node does not:
+
+- validate note contents against chain state;
+- connect to a Miden node;
+- attach commitment block context;
+- attach note inclusion proofs;
+- inspect or decrypt note details;
+- authenticate senders or recipients;
+- guarantee delivery after the retention period.

docs/external/src/index.md

@@ -0,0 +1,53 @@
+---
+sidebar_position: 1
+title: Note Transport
+description: "Off-chain relay service for private note delivery on Miden."
+pagination_prev: null
+---
+
+# Note Transport
+
+The Miden note transport service is an off-chain relay for private note delivery. It gives senders a place to publish serialized private notes and gives recipients a way to fetch notes that match the tags they monitor.
+
+Private note contents are not published on-chain. The chain stores note commitments, while the full note data must reach the recipient through another channel. Note transport is the standard network service for that off-chain delivery path.
+
+## Start here
+
+<CardGrid cols={3}>
+  <Card title="Design" href="./design" eyebrow="Architecture">
+    How the node stores notes, assigns cursors, routes by tag, and handles current protocol boundaries.
+  </Card>
+  <Card title="Operators" href="./operators" eyebrow="Run a node">
+    CLI flags, Docker Compose, telemetry, storage, ports, retention, and production cautions.
+  </Card>
+  <Card title="Users" href="./users" eyebrow="gRPC API">
+    Request and response shapes for send, fetch, stream, stats, plus the recommended client sync pattern.
+  </Card>
+</CardGrid>
+
+## API surface
+
+| RPC | Use it for | Current behavior |
+| --- | --- | --- |
+| `SendNote` | Publish one transported note. | The `header` must decode as a Miden `NoteHeader`; `details` are stored as opaque bytes. |
+| `FetchNotes` | Durable catch-up by tag. | Returns notes for one or more tags using a server-assigned `seq` cursor. |
+| `StreamNotes` | Live updates for one tag. | Use it after a fetch cycle; current subscriptions do not initialize from the request cursor. |
+| `Stats` | Basic operational counts. | Returns aggregate note and tag counts. Per-tag statistics are defined in protobuf but not populated yet. |
+
+## Transport model
+
+- **Private payload delivery.** The Miden chain stores note commitments. Note transport carries the full private note data that recipients need to import locally.
+- **Tag-based routing.** Notes are indexed by the 32-bit `NoteTag` embedded in note metadata. The node has no account registry or recipient identity model.
+- **Client-owned privacy policy.** The node parses note headers only. Clients decide which tags to monitor and whether note details should be encrypted before sending.
+- **Temporary mailbox.** Notes are retained for the configured retention window. Delivery is best-effort and clients must persist fetch cursors.
+
+## Current boundaries
+
+- **No chain-state validation.** The node does not connect to a Miden node and does not prove that a stored note was committed on-chain.
+- **No block context yet.** The current API does not attach commitment block numbers, note metadata, or inclusion proofs to fetched notes. This is tracked in [0xMiden/note-transport-service#68](https://github.com/0xMiden/note-transport-service/issues/68).
+- **Duplicate notes are rejected.** SQLite stores note IDs with a uniqueness constraint. Sending the same note twice fails instead of creating duplicate rows.
+- **Cursor values are server-owned.** Fetch pagination uses the monotonic SQLite `seq` value returned by the server. Clients should persist returned cursors, not fabricate them.
+
+## Current implementation
+
+The current node implementation is a Rust gRPC service backed by SQLite. It stores each note with a monotonic `seq` value assigned at insert time, uses that value for `FetchNotes` pagination, and can export traces and metrics through OpenTelemetry.

docs/external/src/operators.md

@@ -0,0 +1,123 @@
+---
+sidebar_position: 3
+title: Operators
+---
+
+# Operators
+
+This page covers running a note transport node.
+
+## Build from source
+
+From the repository root:
+
+```bash
+cargo install --path bin/node --locked
+```
+
+This installs the `miden-note-transport-node-bin` binary.
+
+## Run the node
+
+The default configuration binds to localhost and stores notes in an in-memory SQLite database:
+
+```bash
+miden-note-transport-node-bin
+```
+
+For a reachable node with persistent storage:
+
+```bash
+miden-note-transport-node-bin \
+  --host 0.0.0.0 \
+  --port 57292 \
+  --database-url /var/lib/miden-note-transport/node.db \
+  --retention-days 30
+```
+
+## CLI flags
+
+| Flag | Default | Description |
+| --- | --- | --- |
+| `--host` | `127.0.0.1` | Address to bind to. |
+| `--port` | `57292` | gRPC port. |
+| `--database-url` | `:memory:` | SQLite database URL or file path. Use a file path for persistence. |
+| `--retention-days` | `30` | How long to retain notes before cleanup. |
+| `--max-note-size` | `512000` | Maximum note details size in bytes. |
+| `--max-connections` | `4096` | Maximum concurrent gRPC connections. |
+| `--request-timeout` | `4` | Per-request timeout in seconds. |
+
+The CLI flags above are parsed as command-line arguments. They are not currently read from `DATABASE_URL` or similarly named environment variables.
+
+## Telemetry and logging
+
+Telemetry is configured through environment variables:
+
+| Variable | Default | Description |
+| --- | --- | --- |
+| `OTEL_ENABLED` | `false` | Enables OpenTelemetry export when set to `true`. |
+| `OTEL_TRACES_ENDPOINT` | `http://localhost:4317` | OTLP endpoint for trace and metric export. |
+| `JSON_LOGGING` | `false` | Emits JSON logs when set to `true`. |
+| `RUST_LOG` | `INFO` | Standard Rust tracing filter. |
+
+Example:
+
+```bash
+OTEL_ENABLED=true \
+OTEL_TRACES_ENDPOINT=http://otel-collector:4317 \
+JSON_LOGGING=true \
+RUST_LOG=INFO \
+miden-note-transport-node-bin --host 0.0.0.0 --database-url /var/lib/miden-note-transport/node.db
+```
+
+## Docker Compose
+
+The repository includes a Docker Compose setup for the node plus telemetry services:
+
+```bash
+make docker-node-up
+```
+
+This starts:
+
+- note transport node;
+- OpenTelemetry Collector;
+- Tempo;
+- Prometheus;
+- Grafana.
+
+Use:
+
+```bash
+make docker-node-down
+```
+
+to stop the stack.
+
+The Compose node service passes `--database-url /app/data/node.db` and mounts `/app/data` on the `node_data` volume, so note storage survives container restarts.
+
+## Ports
+
+| Port | Service |
+| --- | --- |
+| `57292` | Note transport gRPC API. |
+| `4317` | OTLP gRPC receiver in the collector. |
+| `4318` | OTLP HTTP receiver in the collector. |
+| `3000` | Grafana. |
+| `9090` | Prometheus. |
+| `3200` | Tempo. |
+
+The note transport node exposes gRPC health through the same gRPC server, not a separate HTTP health port.
+
+## Database behavior
+
+Use a file-backed SQLite path for production-like deployments. The default `:memory:` database is useful for local testing but loses all notes on restart.
+
+The node runs embedded migrations at startup. The current schema stores note IDs with a uniqueness constraint and uses a monotonic `seq` column for pagination.
+
+## Operational cautions
+
+- Treat debug logs as sensitive. Note IDs and tags can be correlated with user activity.
+- Configure a retention period that matches the expected offline window for your users.
+- Monitor request errors. Duplicate note IDs or invalid note headers are rejected.
+- Use `FetchNotes` for durable catch-up. Streaming is best used as a live update channel after a fetch cycle.