Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
983 lines (844 loc) · 73.7 KB

ARCHITECTURE.md

File metadata and controls

983 lines (844 loc) · 73.7 KB
Raw

Stenion Architecture

The technical shape of the system: how the code is organized, how data flows from the chain to a score on the dashboard, and how it's deployed. For how a score is calculated, see METHODOLOGY.md; for how to add a protocol, see CONTRIBUTING.md.

Monorepo layout

Stenion is a pnpm workspaces monorepo. Each directory is an internal package; the adapters import @stenion/core's Adapter interface as a real typed dependency.

/core        — @stenion/core        Adapter interface + RiskFactorType taxonomy + shared types
/adapters    — @stenion/adapters    one file per protocol (blend.ts, kinetic.ts), each an Adapter
/db          — @stenion/db          Postgres layer: pg pool, typed Store, raw-SQL migrations
/indexer     — @stenion/indexer     scheduler that runs adapters on an interval, writes to Postgres
/api         — @stenion/api         standalone REST server (legacy — see "Why @stenion/api exists")
/dashboard   — @stenion/dashboard   Next.js site + the deployed API routes + the cron-trigger route

TypeScript is configured in four layers (see CLAUDE.md for the rationale):

  • tsconfig.base.json — shared compiler settings only (target, strict, etc.).

  • tsconfig.node.json — extends base, adds nodeNext module/resolution. Backend packages (core, db, indexer, api, adapters) extend this.

  • tsconfig.check.json — extends the Node config, adds noEmit + allowImportingTsExtensions. A backend package's own tsconfig.json extends this, so its sources and its *.test.ts are typechecked as one project; the package emits from a sibling tsconfig.build.json that excludes tests. Test files import with explicit .ts extensions because Node's runner needs them under type stripping, and tsc only permits that when it isn't emitting — hence the split.

    The direction matters. Editors resolve a file through the nearest tsconfig.json, so that config is the one that must include the tests. Excluding them there (and typechecking via a separately-named config) leaves test files in no project at all: the CLI passes, because it was pointed at the right file explicitly, while the editor falls back to an inferred project and underlines every .ts import. All four backend packages (core, adapters, db, indexer) use this split.

    One package needs more: indexer. Everywhere else a tested module is a leaf with no relative imports of its own, so the question never arises. indexer/src/cycle.ts is the first module that is both imported by a test and importing siblings (./retry, ./alerts). Node's type-stripping ESM loader resolves a test's import graph literally, so the source must say ./retry.ts; the emitted CommonJS must say ./retry.js. indexer/tsconfig.build.json therefore adds allowImportingTsExtensions + rewriteRelativeImportExtensions (TS 5.7+), which rewrites the extension on emit and lets one source satisfy both. Without it the choice is a test that cannot load the module or a build that cannot emit it. Scoped to that package on purpose — leaf-only tested modules remain the simpler default.

  • dashboard has its own Next.js-generated config (bundler resolution) — it does not extend the Node config. It needs no split: it's already noEmit and sets the flag directly.

What each package does

@stenion/core — the contract everything else agrees on. Defines the Adapter<TRawData> interface (fetchRawData → computeRiskFactors → score), the RiskFactorType enum (the fixed five-factor *Safety taxonomy), and the shared result types. Adding a factor here is a breaking change felt by every adapter, so it's deliberately small and stable. Carries ADAPTER_INTERFACE_VERSION as a seam for future breaking changes.

It also owns the pieces of the rulebook that must not differ between adapters, in core/src/scoring.ts: scoreFactors() (the weighted mean — an adapter's score() delegates to it and must never reimplement it, or two protocols end up on two rulebooks) and freshnessWindow() with STALE_CEILING_SECONDS. Per-protocol input reading stays in the adapters; nothing in this file reaches for chain data.

@stenion/adapters — one file per protocol, each a class implementing Adapter. An adapter reads a protocol's on-chain state (Soroban RPC + Horizon), reduces it into the five *Safety factors using the formulas in METHODOLOGY.md, and produces a weighted safetyScore. Currently BlendAdapter and KineticAdapter. Adapters throw on failure; they never swallow errors.

One adapter can serve several markets. BlendAdapter takes a BlendPool — slug, display name, pool contract, mark, links, deployment label — and the module exports BLEND_POOLS, the list the indexer iterates. Every Blend market runs the same pool wasm (all three registered pools report code hash a41fc53d…, and the V2 pool factory's is_pool returns true for each), so a further market is a config entry and no new scoring code. Nothing on BlendPool is a threshold, a weight or a formula — a per-pool rulebook would break METHODOLOGY.md ground rule 1 — and the identity fields all come from the pool the instance was given, so an adapter cannot publish one pool's contractId beside another pool's numbers. Targeting, not aggregation: each pool is its own ranked entry, and the three live Blend pools span 30 points (54, 50 and 24) on identical contract code.

A row in protocols is therefore not always a protocol. Four targets, two protocols: Blend's Fixed pool, Kinetic, and the YieldBlox and Etherfuse pools on Blend V2. That distinction is carried in the data, not left to the reader — see deployment_host / deployment_label below.

@stenion/db — the single, typed storage layer, shared by both the indexer (writes) and the dashboard/API (reads) so there's no duplicated connection logic. Exposes a lazy singleton pg Pool (getPool/closePool), a createStore(pool) factory with all read/write methods, env loading, and the persisted RunRecord type. Three tables — two that hold the product, and one that holds no product data at all:

  • protocols — one row per scored market (slug PK, name, chain, adapter class name) plus its identity: logo (a root-relative path into the dashboard's own public/ tree — we host every mark, never hotlink), contract_id (the raw Soroban address the score is derived from, so a reader can check it in an explorer; the explorer itself is chosen in dashboard/app/lib/explorer.ts, not per adapter), site_url / docs_url, and the deployment pair deployment_host / deployment_label (migration 0006). All are nullable, because "publishes no mark", "publishes no docs" and "runs on its own contracts" are real answers the UI renders deliberately rather than papering over with a placeholder. Upserted at indexer startup from adapter metadata — and overwritten every cycle, so these are maintainer-managed; a future protocol self-service flow needs separate precedence-taking columns, not edits to these.

    The deployment pair is written and read together and published as one deployedOn object ({ host, label }, or null), on both the leaderboard and the detail response — the only identity field besides logo that the board carries, because it is what a row is rather than verification detail a reader looks up afterwards. deployment_host is a display name, not a protocols.id, and there is deliberately no foreign key: Stenion's blend row is itself one Blend market, so a reference to it would claim the pool runs on that entry rather than on the host protocol's contract. A half-populated pair maps to null rather than to a partial object.

  • risk_scores — append-only history. safety_score is promoted to its own numeric column (it's what the registry ranks on); the five factors live in one jsonb column (displayed, not ranked, and growing the taxonomy then needs no migration). methodology_version records which rulebook produced the score — see below. A DB-level CHECK enforces the ok/failed discriminated union.

    operational_state (migration 0007) is one more jsonb column, stamped per run like the factors and for the same reason: it is a live reading, not identity, and it is only meaningful next to the run whose inputs it was read alongside. It records which user operations a market's own contracts were refusing — and it is published beside the score, never folded into it (METHODOLOGY.md). It rides both the leaderboard and the detail response, on the same reasoning as deployedOn. Nullable, and not part of the CHECK: a failed run read nothing, and a row written before the column existed has none. Null means "not read", never "unrestricted".

    Deploy order for 0007 is the reverse of 0002/0003/0006's hazard, and it matters. Those migrations had to stay writable by the already-deployed indexer. This one is read by the new Store queries, which name operational_state in both SELECTs — so the migration must run before the code that reads it is promoted, or the leaderboard and detail routes 500 on a missing column. The old indexer keeps writing happily against the migrated schema in the meantime (it just leaves the column null), so migrate-then-deploy is safe in both directions; deploy-then-migrate is not.

  • api_rate_limits — one token bucket per public-API client, and the odd one out: it is infrastructure, not data. It exists in Postgres only because serverless has no shared memory, so there is nowhere else every instance can see (createRateLimiter, deliberately not part of Store — the domain layer has no business knowing about it). The key is a salted hash of the client IP, never the address, so the table cannot become a log of who reads the API; rows idle for an hour are pruned. Nothing here is read by any scoring or serving path. See "Caching and rate limits".

Methodology versioning. The rulebook is at v1, and versioning starts there: every stored row carries methodology_version = 1, and no second version exists yet. Development-era history under earlier iterations of the rules was discarded rather than migrated — the reasoning, and what does and doesn't warrant a bump, are in METHODOLOGY.md. Mechanically: a scoring change that makes old scores non-comparable bumps METHODOLOGY_VERSION in @stenion/core; the indexer stamps it onto every run. History is never backfilled — risk_scores keeps only outputs (score + factor map), never the raw on-chain inputs, so an old row genuinely cannot be recomputed under new rules. The version is surfaced on the protocol detail and on each history point so the dashboard marks the break rather than rendering an unexplained step change. Migrations that add such a column must stay writable by the currently deployed indexer: main keeps running the old code until it's promoted, and both share one Neon database. That is why 0002 shipped the column with a DEFAULT 1 and enforced only the ok half of its CHECK; 0004 drops the default and tightens the CHECK to the full union, now that the deployed indexer names the column explicitly on both arms. The column is therefore required rather than defaulted: a future writer that bumps METHODOLOGY_VERSION and forgets it fails loudly instead of being silently stamped with the old version — a mis-stamp that could never be repaired, since the raw inputs are not stored.

Store also exposes listRecentRuns(protocolId, limit) — status/error/runAt for the newest N runs of one protocol, newest first. It exists for the indexer's consecutive-failure alerting, which derives a streak from this history rather than persisting a counter (see the indexer section). It is deliberately narrower than HistoryEntry: the streak logic needs only whether a run failed, what it said, and when — never a score or a factor map.

Migrations are raw .sql files plus a ~40-line runner (db/src/migrate.ts) — no ORM. Alerting added no migration and no new table: that was the point of deriving the streak.

@stenion/indexer — the scheduler. On an interval it runs every adapter through a small toTarget<T>() wrapper (which hides each adapter's TRawData so a heterogeneous adapter list can share one typed run loop), wraps each run in try/catch, and writes the outcome — score + factors, or a failed marker — to Postgres. It exports runIndexerCycle() (one cycle, used by the cron route) and guards its standalone loop behind require.main === module so importing it doesn't start the loop.

The package is four modules, split along lines worth preserving. src/cycle.ts holds the run loop (runCycle, toTarget) and is a pure function of its arguments — it takes the targets, the Store to write to, and its retry/alerting behaviour as an argument, reaching for no env, no pool, and no config. src/retry.ts and src/alerts.ts are pure leaf modules (below). src/index.ts is the process entry point: env loading, pool construction, the interval, and the require.main guard — and it is the only place config and the run loop meet. The split exists because the error model is the part most worth testing and least exercised in production, and the entry point cannot be imported from a test at all — its require.main guard and extensionless relative imports are both CommonJS-only, which Node's ESM type-stripping loader rejects. Keep new run-loop logic in cycle.ts.

Retry and failure alerting. The indexer used to be deliberately dumb — one interval, no retries, no alerting — so a transient RPC blip recorded a failed run silently and nobody found out until they looked. It now retries and notifies. This makes failures louder and rarer; it does not change what a failure is. Adapters still throw, the indexer still catches, and a run that ultimately fails is still recorded as failed — a protocol that is genuinely down still shows as down.

  • Bounded retry against a wall-clock deadline, not a fixed schedule. src/retry.ts's withRetry takes an absolute deadline and never runs past it: each attempt is capped at whatever time is actually left, and a retry starts only if the remaining budget covers the backoff plus an attempt worth making. The attempt count and delays are the ceiling; the deadline is the guarantee. This is deliberate — a fixed schedule only stays inside 60s if you know how long an attempt takes, and nothing does: every RPC call in both adapters is a bare await with no AbortSignal, and Node's fetch has no default timeout.

  • The 60s ceiling is the binding constraint. maxDuration is capped at 60 on Vercel's Hobby tier and cannot be raised, and a cycle killed mid-flight is worse than one that fails cleanly — it can leave one protocol scored and the other neither scored nor recorded as failed. The run loop's budget is STENION_CYCLE_BUDGET_MS, default 42s, leaving room for cold start, pool connect, upserts, streak queries and a 3s-capped alert POST inside 60s.

  • Targets run through a bounded worker pool, and the budget is no longer divided between them. STENION_CYCLE_CONCURRENCY (default 2) targets are in flight at once, pulled from a shared cursor rather than run in fixed batches. Each one's deadline is the end of the budget minus one full attempt reserved for every wave still queued behind it (targetDeadline in indexer/src/cycle.ts).

    Why the division rule was removed. It used to be now + remaining / targetsLeft, which made every target's deadline a function of how many targets existed: 21s each at two, 14s at three (already below the 15s attempt timeout), and 10.5s at four — below the healthy fetch duration of two protocols that already work. Registering a pool could therefore fail protocols that were fine the day before. That is not "adding a pool makes things slower", it is a bug, and no allocation scheme fixes it: at concurrency 1 the registry cannot give three targets one full attempt each inside 42s, whatever the rule.

    What the new rule guarantees. Every target gets at least one full 15s attempt at any feasible target count — against a slowest healthy fetch of 6.1s measured on the deployed function (12.5s was the worst ever seen on a developer machine, and even that fits) — and usually far more, because queuedAfter is read when a worker picks a target up, so a target that finished early has already shortened the queue and the next one inherits the slack. A target still cannot eat the queue's last chance to be looked at, which is the guarantee the old even division was really buying.

    The ceiling, as arithmetic rather than folklore:

    ceil(targetCount / STENION_CYCLE_CONCURRENCY) * STENION_ATTEMPT_TIMEOUT_MS <= STENION_CYCLE_BUDGET_MS

    At the shipped defaults that holds to four targets and fails at five. cycleFeasibility() checks it every cycle and at indexer startup, and logs a [budget] warning naming the numbers and the levers (raise concurrency, lower the attempt timeout, or shard). It warns and runs rather than refusing — taking the whole registry down because someone registered a fifth pool is worse than running five protocols imperfectly and saying so. Past the ceiling it degrades to whole attempts on a first-come basis with the tail failing cleanly (DeadlineExceededError) and going visibly stale on /api/v1/health, rather than squeezing every target into a length at which none of them can succeed.

  • Concurrency is bounded because the peak is what costs. Both adapters are strictly sequential internally — every RPC and Horizon call is a bare await — so one target in flight is exactly one request in flight, and STENION_CYCLE_CONCURRENCY is the peak simultaneous load Stenion puts on the shared, rate-limited public RPC. A Promise.allSettled over every target would make that peak grow with every pool registered, which is the wrong dial to leave unbounded. Total request volume per cycle is unchanged whatever the concurrency — roughly 13 (Blend Fixed, 3 reserves), 23 (YieldBlox, 8) and 22 (Kinetic, 4).

    It ships at 1, not 2. What concurrency changes is the request rate, and the estimate made here before deploying — "~2.3/s to ~4.5/s" — was wrong, because it divided the request count by developer-machine durations. See the incident note below: 2 was deployed, measured, and reverted the same day.

  • RESOLVED (2026-08-25): concurrency 2 drew 429s from the public RPC. Kept in full rather than deleted, because it is the reason this document now says measure the deployed function everywhere it used to say compute from timings — and because anyone raising STENION_CYCLE_CONCURRENCY should have to read it first.

    STENION_RPC_URL is mainnet.sorobanrpc.com: the free, shared, keyless public endpoint, whose rate limit is unpublished and not ours to raise.

    What happened. #68 shipped the worker pool at concurrency 2. Within one cycle, Blend — the target that runs behind the concurrent pair — began failing with Request failed with status code 429, recorded only after all three retry attempts were exhausted. Measured from risk_scores via /api/v1/protocol/:id:

    Window blend yieldblox kinetic total
    34 cycles before the deploy (sequential) 0/34 0/34 0/34 0/102
    8 cycles after, with manual curls adding load 4/8 1/8 1/8 6/24 (25%)
    8 cycles after, scheduled only — no manual triggers 4/8 0/8 0/8 4/24 (17%)

    The clean window is the load-bearing one: no manual triggers, ordinary 5-minute cadence, and Blend still failed half its cycles against a baseline of zero. (One cycle 429'd all three targets at once, but that was in the contaminated window, so it is reported and not leaned on.)

    Root cause: rate, not peak. Peak in-flight only went 1 → 2, which is nothing in absolute terms. But the deployed function is 2-3x faster than the developer machine the original estimate was computed from, so the same requests are compressed into a third of the time: wave 1 issues roughly 45 requests (YieldBlox ~23, Kinetic ~22) inside ~4 seconds — about 11 requests/second — and Blend's ~13 land immediately behind them. The target that fails is the one running behind the burst. The pre-deploy estimate was wrong in the same direction as the failures, which is the whole lesson: a public endpoint limits rate, and rate is exactly what a duration estimate gets wrong when the durations come from the wrong machine.

    The fix, applied. Both halves together, not either alone:

    • STENION_CYCLE_CONCURRENCY default 2 → 1. Removes the burst entirely. It does not reinstate the bug #68 fixed: budget division is gone independently of concurrency, so at 1 each target still gets the budget less a reservation rather than a shrinking even share.
    • STENION_ATTEMPT_TIMEOUT_MS default 15s → 10s. Needed because of the first: at 1 worker a 15s timeout makes cycleFeasibility infeasible at three targets (3 × 15s = 45s > 42s) and warn on every cycle. 10s is justified by the measurement rather than guessed — nothing healthy exceeds 6.1s deployed — and makes a sequential cycle feasible to four targets (4 × 10s = 40s ≤ 42s), so #65's Etherfuse needed no further config change.

    Verified against the real defaults: 3 targets 30,000ms silent, 4 targets 40,000ms silent, 5 targets 50,000ms warns. Alerting is asserted byte-identical at concurrency 1 and 2.

    Now standing at four. #65 registered Etherfuse, so cycleFeasibility was re-run against the loaded config and the real registry rather than against the arithmetic above: 4 targets, concurrency 1, attemptTimeoutMs 10,000, budgetMs 42,000 → requiredMs 40,000, feasible, no warning. The next registration is the one that trips it — a fifth target needs 50,000ms and warns every cycle, so it arrives with a budget or concurrency decision attached, measured against the deployed function rather than argued from arithmetic.

    Local-dev caveat: a developer machine has been seen taking 12.5s on YieldBlox, which now exceeds the 10s cap, so a local pnpm indexer may time out and retry where it used to succeed first time. Raise STENION_ATTEMPT_TIMEOUT_MS in .env if that bites — production is the case the default is sized for.

  • Targets are ordered slowest-first (orderByLatency in indexer/src/index.ts: YieldBlox → Kinetic → Blend). Under a worker pool, longest-processing-time-first minimises the makespan: a slow target started last is a slow target nothing can overlap with. This is the opposite of the old fastest-first order, which was correct only because the old division rule gave the first target the tightest share and later ones the inherited slack. A target not in the list sorts first, i.e. an unmeasured pool is assumed to be the slowest and gets the most generous slot — so BLEND_POOLS stays the single list to edit when adding a market.

  • Per-target durationMs and whole-cycle totalMs ride on the cycle summary, which the cron route spreads into its JSON response. That is deliberate: the budget arithmetic above can only be validated on Vercel's path to the RPC, and a developer machine's path is not that. curling the cron route returns the real measurements.

    Measured from the deployed function on 2026-08-25, five cycles at the shipped defaults (3 targets, concurrency 2), read from the cron route's own response:

    Target Wave durationMs range Median
    Kinetic 1 4,788–6,100 5,001
    YieldBlox 1 3,792–4,314 3,956
    Blend Fixed 2 2,271–2,628 2,408
    Cycle — 6,461–7,322 ~6,850

    (Cycle range over the four clean cycles; a fifth spent 9,504ms because one target exhausted three attempts before failing. Ranges are durationMs/totalMs as returned by POST /api/cron/run-indexer — retries and backoff included, DB write excluded.)

    These are two to three times faster than the developer-machine figures they replace (which were Blend 6.0–7.5s, Kinetic 7.7–10.5s, YieldBlox 8.1–12.5s, 24.5–26.9s sequential, measured 2026-08-19). Vercel's path to the RPC is simply not a laptop's, which is why the issue insisted on measuring from the deployed function rather than trusting arithmetic over local timings. Two consequences worth stating: the whole cycle uses under a fifth of its 42s budget, and the 15s attempt timeout is now roughly 2.5x the slowest healthy fetch rather than barely above it.

    The 3-target case is measured. The 4-target case still is not. The fourth target now exists — #65 registered Etherfuse — but nothing above is evidence about it: the numbers here were captured at three targets, and, see the rate-limit note below, the cost of a cycle is not only its duration. The feasibility ceiling of four targets is a statement about the attempt timeout fitting in the budget, not a measured result, and must not be described as proven until a deployed cycle has run it.

    One thing to watch on the first deployed cycles. Etherfuse's fetchRawData was observed locally at 8.0s and 12.2s in two consecutive runs against the shared public RPC — the second past the 10s attempt timeout. Local timings are exactly what the #68 incident says never to reason from, so this is written down as the thing to check in the cron route's per-target durationMs, not as a claim about production or as grounds to move a knob.

  • The attempt timeout is soft. It races the attempt against a timer, abandoning the in-flight work rather than cancelling it. That bounds the observed attempt duration, which is what the budget needs, and is harmless under serverless where the socket dies with the invocation. True cancellation needs an AbortSignal threaded through Adapter.fetchRawData — a breaking interface change, tracked in ROADMAP.md.

  • Transient and permanent failures are deliberately not distinguished. Every adapter failure is a bare new Error(string) with no typed error and no preserved status code, so the only available classifier is regex over message text — which drifts silently when a message is reworded, and drifts toward retrying nothing. It also buys little: the structural failures (a missing storage key, a malformed decode) throw fast, while the slow failures are exactly the transient ones, so classification would save budget precisely where budget is not at risk. The wall-clock deadline protects the case that matters. A typed PermanentAdapterError in core is the clean path if this is ever wanted; it is in ROADMAP.md, not guessed at here.

  • Alerting fires after N consecutive failures (STENION_ALERT_THRESHOLD, default 4 ≈ 20 minutes at the 5-minute cadence), POSTed to STENION_ALERT_WEBHOOK_URL as a plain fetch — no dependency, and unset means alerting is simply off. Both arms are edge-triggered: failing fires at exactly N so a six-hour outage is one message rather than seventy-two, and a recovered message follows when the protocol scores again. The recovery half is what makes silence after an alert unambiguous; resolving that ambiguity by re-alerting every cycle would need dedup state this deliberately doesn't have. An alert names the protocol, the streak length, how long it has been going, the latest error verbatim, and every distinct message in the streak — four identical errors and four different ones usually mean "the protocol changed" versus "the RPC provider is flaky".

  • The rendered message is capped at 2,000 characters (MAX_MESSAGE_CHARS). Discord rejects a longer content with a 400 rather than truncating it, and the case that reaches the limit is the worst one available: an RPC-wide outage takes out every target, they all cross the threshold on the same cycle, and their alerts batch into one POST. The render is one block per alert, so the body scales linearly with target count — two protocols with four distinct Soroban HostError messages each measured ~2,500 characters, and three of the same is ~3,700. Without the cap, the alert for the biggest possible outage is the one that silently never arrives. The structured alerts array is never truncated, so nothing is lost for a machine consumer.

    The third target moved where truncation starts. Measured with a moderate ~150-character error message, the same four-distinct-errors scenario renders 1,958 characters across two targets — it fit — and 2,944 across three. Outages that used to arrive whole now arrive marked truncated. The cap is doing its job either way; what changed is how often a reader sees the marker.

  • Verifying delivery without waiting for a real outage: pnpm smoke:alert-webhook drives the real path — a seeded failure streak through runCycle, decideAlert, formatAlert and the real webhookNotifier — at a live webhook URL, reporting the HTTP status and body that webhookNotifier itself discards. It uses an in-memory store (Postgres is untouched) and an obviously fake protocol id, so a message landing in a shared channel cannot be mistaken for a real outage. --dry-run prints the payload without sending; --mode failing|recovered picks one arm. Confirmed against a live Discord webhook on 2026-08-19: both arms accepted, HTTP 204.

Where the streak lives: derived, not counted. The indexer is invoked per-cycle by an external scheduler, so there is no long-running process to hold a counter. The streak is read back out of risk_scores each cycle (Store.listRecentRuns, a bounded walk of the (protocol_id, run_at DESC) index the leaderboard's LATERAL joins already use). A persisted counter would be a second source of truth that can disagree with the history it describes — insertRunRecord failure is caught and logged rather than fatal, so a counter could increment beside a row that never landed, and the alert would claim a streak the database cannot show you. Derivation cannot desynchronize, because it is the history.

The predicate is "count failed rows from the newest backwards until the first non-failed one", and it is deliberately not "no ok run in the last N". The two agree on a populated table and disagree catastrophically on an empty one: a protocol with no history at all satisfies the second immediately, so a freshly-truncated risk_scores would page someone on the first cycle. Counting backwards yields 0 on an empty history, so an alert requires N rows that actually exist and actually failed, and a newly-added protocol is protected by the same arithmetic with no special case. This stopped being hypothetical on 2026-08-19, when risk_scores was truncated — the streak derivation now genuinely starts from an empty table. Both cases are asserted in indexer/src/alerts.test.ts and indexer/src/cycle.test.ts.

What this does NOT cover: a total database outage. If Postgres is unreachable, no run row is written, so no streak advances and no alert fires — and the streak query would fail too. That surfaces as the cron route returning 500 (prepare() throws before the loop), not as a webhook message. Reading "the indexer alerts on failure" as including "the database is gone" would be wrong. Alerting on infrastructure failure as well as protocol failure is a separate feature, deliberately not folded in here.

@stenion/dashboard — a Next.js 15 (App Router) site, and the actual deployment target. It's three things in one Vercel project:

  1. The public site (homepage, registry, on-site methodology, on-site API docs, about, per-protocol detail pages). Data pages are async Server Components that read @stenion/db's Store in-process — no HTTP hop.

    The registry's search/filter/sort state lives entirely in query params (?q=…&status=…&sort=…), never in component state, so a filtered view is linkable and survives a reload. The page renders from those params on the server; the control (components/registry-controls.tsx) is a real <form method="get"> that only changes the URL and never holds or filters the list. That is what keeps every reason, summary and status phrase in the server-rendered HTML for find-in-page and indexing. The ordering itself is pure functions in app/lib/registry-query.ts — separated from the JSX so the rule that unscored entries never enter the ranked ordering is a testable value rather than a rendering habit.

    /coverage/:id is a second kind of detail page: one protocol we assessed and do not score, served entirely from the static app/lib/coverage.ts and never through getProtocolDetail. Routing it under /protocol/:id would either render ids GET /api/v1/protocol/:id 404s on, or make that function return a second scoreless shape — the dashboard-vs-API divergence app/lib/api.ts exists to prevent, in the two forms it can take. Its one live read is the dedupe check: if the board has since scored the id, it redirects to /protocol/:id, and it fails open on a database error (the page is static and stays true during an outage). /coverage with no id redirects to /registry?status=not-scored.

    Rendered docs (/methodology, /docs/api) are a second, separate kind of page: they read a repo-root markdown file at request time and render it through components/markdown-doc.tsx, so the file stays the single source of truth and is readable both on GitHub and on the site. Each such route needs an outputFileTracingIncludes entry in next.config.mjs, because the file lives outside the dashboard directory and would otherwise be missing from the serverless bundle — a failure that is invisible in next dev, where the file is simply on disk. MarkdownDoc adds heading anchors, wraps tables in their own scroll container, gives code fences a copy button, and rewrites repo-relative links to the GitHub source except for files that are themselves rendered here (app/lib/site.ts's RENDERED_DOC_ROUTES), which stay on-site.

    The protocol page's score-history chart is a client component drawing hand-rolled SVG (no charting library) over the history array the detail response already carries — it adds no endpoint and no query. All of its judgment about what counts as a discontinuity lives in the pure, framework-free app/lib/score-series.ts so it can be tested against fixtures; the component only draws what that returns. The rule it enforces is that a break in the line means the score is unknown here — a failed run, an indexing gap wider than 3× the measured cadence, or a methodology-version change. None of the three is ever drawn through, and a failed run is never rendered as a zero.

  2. The public API, as Route Handlers: GET /api/v1/protocols, GET /api/v1/coverage, GET /api/v1/protocol/:id, GET /api/v1/health.

  3. A secret-gated cron-trigger route (POST /api/cron/run-indexer) that runs one indexer cycle.

@stenion/api — a standalone node:http REST server. Not deployed — see below.

Data flow

  Soroban RPC + Horizon              (trustless on-chain sources)
          │
          ▼
   Adapter.fetchRawData()            raw protocol state (per-adapter shape)
          │
          ▼
   Adapter.computeRiskFactors()      → the five *Safety factors (shared taxonomy)
          │
          ▼
   Adapter.score()                   → weighted safetyScore (0–100)
          │
          ▼
   Indexer (runIndexerCycle)         try/catch per adapter, one row per run
          │
          ▼
   Postgres  (@stenion/db)           protocols + risk_scores (append-only history)
          │
          ├──────────────┐
          ▼              ▼
   Dashboard pages   API routes      dashboard reads the Store in-process;
   (Server         (/api/v1/*)       routes read the same Store for external
    Components)                       consumers (wallets, third parties)

The key invariant: the dashboard's own pages and the public API routes both go through the same Store methods (listProtocolsWithLatestScore, getProtocolDetail), so the JSON contract and what the site renders can't drift apart. Nothing is ever recomputed at read time — the indexer owns scoring; readers only shape stored rows.

Staleness model: the displayed safetyScore is always the latest ok run (null if never scored); the newest run of any status is surfaced separately as lastRunAt/lastRunStatus. A registry that's honest about freshness beats one with holes on a failed cycle.

That honesty has to survive the trip to the screen, so the UI never leaves a failed run as nothing but an older timestamp. dashboard/app/lib/format.ts's freshness() turns the pair into a tone, a short label, and a full explanation; the registry row carries an accent rule plus a pill and caption, and the protocol page carries a notice with both timestamps. Freshness never borrows the score bands — safe/warn/danger mean risk level, so a stale marker in amber or red would report a pipeline fault as a verdict on the protocol. freshnessPillClass uses the accent and the neutrals instead, and format.test.ts asserts that mechanically rather than leaving it to review attention.

Deploy architecture

One Vercel project = the dashboard. The indexer and the standalone API are not deployed as separate services. Everything runs from the single Next.js app:

  • API → Next.js Route Handlers inside the dashboard (app/api/v1/protocols, app/api/v1/coverage, app/api/v1/protocol/[id], app/api/v1/health). The scored routes use the same Store methods and JSON as the original standalone API — a transport change, not a rewrite. The coverage route combines the static coverage module with live leaderboard ids solely for deduplication. The health route reports indexer freshness and nothing else — see "The health endpoint" below. CORS (access-control-allow-origin: *) is set on these four routes only, for future browser/wallet/third-party clients reading public, payment-blind data. /api/v1/* is the only public API surface — see "API versioning" below. All four are rate limited; three of them are CDN-cached and health deliberately is not — see "Caching and rate limits" below.

  • Indexer → triggered by POST /api/cron/run-indexer, which calls runIndexerCycle() once. The route is secret-gated (Authorization: Bearer <CRON_SECRET>, compared with crypto.timingSafeEqual); if CRON_SECRET is unset it refuses to run, so it's never open. No CORS on this route, and no rate limiting — it's authenticated and internal, and limiting it could only ever block a scheduled run.

  • Scheduling is external — a cron-job.org job POSTs to the cron route every 5 minutes with Authorization: Bearer <CRON_SECRET>. The route itself is stateless about cadence: it runs exactly one cycle per request, so the interval is entirely the caller's.

    The schedule is not in version control. It lives in the cron-job.org dashboard — there is no workflow file, no vercel.json crons entry, and no other scheduling config in this repo. Changing the cadence, pausing indexing, or rotating the target URL is done in that service's UI, not in a PR. If indexing has stopped, check there before looking for a bug in this repo.

    Why not Vercel Cron: the Hobby tier caps scheduled functions at once per day, which is far too slow for live scoring — 5-minute freshness is the product. Upgrading to Pro for cron alone isn't justified pre-funding, so an external scheduler hits the same secret-gated route instead. This is a deliberate choice, not an oversight: the route is a plain authenticated HTTP endpoint, so swapping cron-job.org for Vercel Cron (or anything else) later is a scheduler change only, with no code change.

Build wiring: the dashboard's build script compiles the workspace deps (core → db → adapters → indexer) before next build, because those packages resolve via their dist/ output. next.config.mjs marks pg and @stellar/stellar-sdk as serverExternalPackages (kept as runtime requires, not webpack-bundled) and pins outputFileTracingRoot to the repo root so workspace-dep tracing is correct. On Vercel: Root Directory = dashboard, Build Command = pnpm run build.

The workspace packages themselves are not externalised — they are bundled into the serverless functions, and therefore minified, which renames classes and functions. Nothing that is persisted or published may be derived from a runtime identifier (constructor.name, fn.name): those values are correct under node --test and next dev and wrong in the only environment that writes the data. ProtocolMetadata.adapterRef is a hardcoded literal for exactly this reason — see CONTRIBUTING.md.

Tests: pnpm test at the root, fanning out to whichever packages define one, and run by CI on every PR. There is no test framework dependency — tests are *.test.ts files run by Node's built-in test runner (node --test) against native TypeScript stripping, which is why CI and .nvmrc pin Node 24 (the floor is 22.18). Coverage is deliberately narrow: pure logic whose important cases live data can't reach.

Three things follow from strip-only mode and are worth knowing before writing a test:

  • A .ts test file must import with an explicit .ts extension.
  • It cannot value-import a TypeScript enum from source — RiskFactorType included, since Node rejects enum as unstrippable syntax. Import the enum's type and use its string values, or import it from a package's built dist/ (plain JS, so the enum is fine there).
  • Type-only imports must be written import type. Stripping is syntactic: it cannot tell that Adapter is an interface, so a combined import { Adapter, freshnessWindow } survives into the running module and fails to resolve against @stenion/core's CommonJS output, which has no runtime Adapter. This bites any module a test imports, not just the test file itself.

The worked examples:

  • core/src/scoring.test.ts — scoreFactors, the weighted mean every protocol's score passes through. Several assertions parse METHODOLOGY.md and the RiskFactorType enum as text rather than restating their numbers, so the rule that code and the methodology may not drift is enforced mechanically instead of by review attention.
  • adapters/blend.test.ts / adapters/kinetic.test.ts — computeRiskFactors against synthetic raw state. computeRiskFactors is a pure function of already-decoded on-chain data, so every methodology rule is reachable without RPC. This is where methodology v2's oracleSafety is pinned: every live pool but YieldBlox prices fresh and bounded, so a live run exercises neither the disabled-bound path nor K2's inert-breaker path — the two the rulebook exists to catch.
  • adapters/snapshot.test.ts — the same adapters against frozen mainnet captures in adapters/fixtures/. This asks a different question from the synthetic suites: not "does the code match the rulebook" but "did a refactor move a published number on real data". It is the only coverage that would notice a decode or fixed-point scaling regression, because the synthetic builders use convenient values (b_rate = 1.0, one decimals value, round balances) and real pools do not — dropping the b_rate multiplication entirely is an exact identity under a unit rate and passes all 71 synthetic tests, while failing here.
  • db/src/store.test.ts — the row → response mapping (toHistoryEntry, toProtocolDetail, toLeaderboardEntry), extracted from the query methods so the public JSON contract can be tested without Postgres. Covers the ok/failed union and the staleness model — neither of which the live site exercises, since no run has ever failed.
  • db/src/store.integration.test.ts — the SQL itself (the two LATERAL joins, NULLS LAST ranking, the shape CHECK). Skipped unless STENION_TEST_DATABASE_URL is set, so CI and contributor PRs never need database credentials. See CONTRIBUTING.md.
  • dashboard/app/api/_http.test.ts — the response envelope: status, content type, and CORS. These fail only in a third party's browser, never on our own pages (which read the Store in-process), so nothing else would catch a regression.
  • dashboard/app/api/_cache.test.ts — the cache TTL policy, and specifically the invariant that a cached response can never hide a newer indexer run by more than the 10s floor. That promise is arithmetic over a clock and is unobservable everywhere we can look: locally there is no cache, and on Vercel a violation looks like a correct-shaped JSON body that is quietly minutes old. Nothing goes red, so the bound is asserted here or nowhere.
  • dashboard/app/api/_rate-limit.test.ts — client identity, config parsing, refusal headers, and the per-instance deny memo. Every branch decides whether to refuse someone, and none of it runs in normal operation: the first time it matters is either an abuse incident or an integrator's launch, and a mistake that pools clients together makes the second look like the first.
  • db/src/rate-limit.test.ts — what a token balance means, including the wait an integrator is told to back off by. Production runs this at tokens = 59.9997 and tokens = -0.0001; the interesting cases are exactly the ones a real request never lands on. The refill itself is Postgres arithmetic and is not covered here.
  • indexer/src/cycle.test.ts — the run loop's error model, against a deliberately throwing adapter and an in-memory Store. The contract is that an adapter throws, the indexer records a failed run and continues; risk_scores has never held a failed row (1,683 rows as of 2026-08-16, and truncated on 2026-08-19), so nothing about this path is evidenced by it having run in production. Also covers retry inside the loop (a transient failure clearing on a later attempt; an exhausted retry still recording failed with the adapter's own message), the budget rule (targetDeadline never dropping a target below one full attempt at any feasible target count, and cycleFeasibility holding at four targets and failing at five), the worker pool (peak in-flight bounded, results ordered by registration however completion is ordered, a failure isolated to its own target while another is mid-flight), and alerting against a seeded failure streak — including the case that matters most now, that an empty history raises nothing on the first cycle.
  • indexer/src/retry.test.ts — the backoff schedule and the deadline, driven by a fake clock so the timing is asserted rather than waited on. Pins the two properties the design rests on: it never runs past its deadline, and it rejects with the last error rather than swallowing it.
  • indexer/src/alerts.test.ts — streak counting, the edge-triggered fire/recover decision, and the message text. The first assertions are the empty- and near-empty-history guarantee: a fresh risk_scores and a first-ever failed cycle must both raise nothing.
  • dashboard/app/lib/format.test.ts — the freshness descriptor and its colour mapping, on the same footing as the score-series tests below and for the same reason: no run has ever failed, so every string a reader would see on a failed run exists only here. One assertion is a rule rather than a behaviour — that no fault state is dressed in a score-band colour.
  • dashboard/app/lib/score-series.test.ts — the score-history series builder. As of 2026-08-14 risk_scores held 527 rows and not one failed run, so the failed-run path had to be proven against fixtures rather than by looking at the page.

Environment variables (all on the one Vercel project, Production + Preview): DATABASE_URL (Neon pooled), STENION_RPC_URL, STENION_HORIZON_URL, CRON_SECRET, and optionally STENION_ALERT_WEBHOOK_URL (failure/recovery alerts; unset = alerting off) and STENION_CYCLE_CONCURRENCY (targets in flight at once; default 2). The retry and threshold knobs — STENION_RETRY_ATTEMPTS, STENION_RETRY_BASE_DELAY_MS, STENION_ATTEMPT_TIMEOUT_MS, STENION_CYCLE_BUDGET_MS, STENION_ALERT_THRESHOLD — all have defaults and only need setting to override them; every one is documented in .env.example. Locally, every package reads these from a single repo-root .env via a walk-up loader.

Caching and rate limits

The public API had neither, deliberately, until it was deployed and about to be pitched to wallet integrators. Both exist for one reason: the whole system runs on Neon's free tier, and an aggressive client could exhaust it. The data behind these routes only changes every ~5 minutes, so caching is nearly free; rate limiting covers the client that defeats the cache.

Neither changed the JSON. Existing consumers see the same bodies with a Cache-Control header added, plus a 429 status that did not exist before.

What is actually doing the caching: Vercel's CDN, driven by a Cache-Control header the route handlers set per response. Not an in-process cache — the API is serverless, each invocation is its own process, so a module-level cache would miss on every cold start and hold a different answer per warm instance. Being honest about the limits of that:

  • The CDN caches per edge region, so the origin sees roughly one request per TTL per region with traffic, not one globally.
  • The cache key includes the query string, so ?anything=1 is a fresh key and a guaranteed miss. The cache reduces cost for well-behaved clients; it does not protect the database from a hostile one. That is the rate limiter's job, and it is why the limiter has to be accurate rather than decorative.

The TTL, and why it is computed per response. lastRunAt / lastRunStatus are how a consumer knows whether our data is stale (see "Staleness model" above). A fixed TTL of N seconds serves a body claiming "the last run succeeded at T" for up to N seconds after a later run has already failed — the cache would be lying in exactly the field that exists to stop us lying about freshness. Shortening N bounds that window; it does not remove it.

So the TTL is derived from the data in the body (dashboard/app/api/_cache.ts): cache until the earliest moment the next indexer run could plausibly land, and no further.

GET /api/v1/health is the other, and it goes the opposite way: no-store, never cached at all. The reasoning above works because the thing a cache could hide changes only when a run lands, so expiring before the next run closes the hole. Health does not behave that way — staleness advances with the wall clock, so a body built at 29 minutes stale and cached for even 45 seconds is still being served, saying healthy with a 200, after the true answer has become degraded with a 503. The window is small; the endpoint is the one whose entire purpose is to be believed about freshness, and a health check that can be stale is a contradiction rather than a tradeoff. The 503s must not be cached either, for a stronger reason: the CDN cache key is the URL, so a cached 503 would go on being served to everyone after the pipeline recovered, turning a resolved incident into an ongoing one. What it costs is one database round trip per request, uncushioned. Measured from a dev machine on 2026-08-22, warm: 0.4–1.1s for listRunHealth against 1.1–1.2s for the leaderboard query, both dominated by network round trip rather than by the query — so the uncached route is no more expensive per request than the cached one, and it is bounded above by the rate limiter.

GET /api/v1/coverage is the deliberate exception. Its published records are static and normally change only with a deploy, and its body intentionally has no lastRunAt from which to derive a deadline. It therefore uses a fixed 3,600-second shared-cache TTL. The route still reads live leaderboard ids as a defensive dedupe guard; the one-hour bound limits how long a forgotten reciprocal cleanup could leave an entry cached after it becomes scorable. A deployment replaces the static records and invalidates the old deployment's cache.

Constant Value Why
INDEXER_INTERVAL_SECONDS 300 The cron-job.org cadence; observed median run_at spacing is 4m59s.
CYCLE_JITTER_SECONDS 45 run_at is stamped when a protocol's turn begins, not when cron fires, so its spacing shifts by however much the protocols ahead of it sped up or slowed down — bounded by STENION_CYCLE_BUDGET_MS (default 42s). Concurrency only shrinks this (targets sharing a wave start together), so it stays a safe upper bound.
MAX_TTL_SECONDS 45 Blast radius, not load. See below.
MIN_TTL_SECONDS 10 Floor, so the moment around a landing run isn't an uncached hole every client stampedes through.

The deadline is lastRunAt + 300 − 45 = lastRunAt + 255, clamped into [10, 45]. On a leaderboard response every protocol's lastRunAt counts and the tightest deadline wins, because any of them landing changes the body. A null or unparseable lastRunAt collapses to the floor.

Why the ceiling is 45 and not 255. It is not a load number. A continuously-requested route hits the origin 1/TTL times per second whatever the traffic is, so 45s is ~1.3 origin requests per minute per region and 255s would be ~0.24 — a difference of nothing to Neon. It is a blast-radius number: INDEXER_INTERVAL_SECONDS is an assumption about a schedule that lives in the cron-job.org dashboard and not in this repo, so nothing here fails if someone changes the cadence. The ceiling caps what a wrong assumption costs at 45s of staleness instead of a full cycle's worth.

The guarantee this buys. A cached response can hide a newer run for at most MIN_TTL_SECONDS (10s) — asserted mechanically over every run age in _cache.test.ts, because the property is invisible in every environment we can look at: locally there is no cache, and on Vercel a failure looks like a correct-shaped JSON body that is quietly a few minutes old. Nothing goes red. On top of that bound, the CDN's own Age header makes the residual window visible rather than merely small: a consumer that cares can subtract it.

Two deliberate omissions:

  • No stale-while-revalidate. It is the standard fix for the stampede at expiry, and it works by serving a body past its deadline — precisely the masking above. The stampede it would prevent is bounded by the rate limiter and by this project's traffic; the staleness it would reintroduce is not bounded by anything.
  • max-age=0 for private caches. A copy in someone's browser is one we cannot see, cannot expire, and gain nothing from — the shared tier already absorbs the load — and it would put a response's real age beyond what Age reports.

Errors and 404s are no-store. A cached 500 outlives the outage that caused it, and a cached 404 would keep 404ing for a protocol added in the next cycle.

The rate limiter. A token bucket, one row per client, in Postgres (api_rate_limits, migration 0005; db/src/rate-limit.ts). Postgres and not memory for the reason above: an in-memory counter is per-instance, so N warm instances would allow N × the intended rate while reporting that the limit was enforced. False confidence is worse than no limiter.

  • The limits: 60 requests/minute sustained, 60 of burst, per client. Sized against what is actually counted — cache misses. A wallet polling every 5 seconds produces roughly one miss per TTL because the CDN serves everything in between, so 60 is an order of magnitude above any legitimate integrator. What it does bite is the case it is for: a client defeating the cache with a varying query string, where every request is a database query. That client is capped at ~1 query/second instead of unbounded.
  • A cache hit does not count, and this is structural rather than a policy choice: a hit never invokes the function. It is also the right policy — the limit protects the database, and a hit costs the database nothing. The consequence, stated plainly: the documented limit is not a cap on total requests. A client polling a cached endpoint can exceed it all day.
  • Per client IP, taken from x-real-ip (Vercel sets it, single-valued) falling back to the first x-forwarded-for hop. Behind a shared NAT — a corporate office, a mobile carrier — everyone shares one bucket. That is survivable here only because of the previous point: NAT'd browser traffic overwhelmingly hits the CDN, so a thousand users behind one address still generate roughly one miss per TTL between them. The case that would genuinely break is a thousand NAT'd clients each cache-busting, which is indistinguishable from the abuse this is meant to stop.
  • Not an IP log. The stored key is a salted SHA-256 prefix, never the address (STENION_RATE_LIMIT_SALT — set it in production, or the hash is reversible by enumerating IPv4). Rows idle for an hour are pruned opportunistically on ~1 in 256 served requests, so the table stays proportional to active clients.
  • A 429 carries Retry-After (seconds), X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset (unix epoch seconds), and Cache-Control: no-store. The last is load-bearing: the CDN keys on URL, not client, so a cacheable 429 would be replayed to every other client that asked next — one scraper's limit becoming everyone's outage. Those headers ship on refusals only: a 200 is shared-cached and served to many clients, so an X-RateLimit-Remaining baked into it would be one client's balance, frozen, replayed to everybody — wrong for every reader including the one it came from.
  • It fails open. If the limiter's own query throws — table missing because the migration has not run, pool exhausted, Neon down — the request is allowed and the error is logged. A broken guard rail must not become a broken API, and it means migration 0005 and the deploy can land in either order.
  • The cron trigger is not rate limited. It is secret-gated and internal; limiting it could only ever block a scheduled run.

What this does not protect against. A distributed attack. The limiter is per-client-key, so a thousand hosts each staying under the limit are a thousand clients as far as it is concerned. Stopping that is a network-edge job (Vercel's firewall), not an application one. It also trusts the platform's proxy headers — true on Vercel, which overwrites them, but a proxy that passed a client-supplied x-forwarded-for through would let a client split itself across unlimited buckets. That is a property of the deployment, not something this code can detect.

What it costs. One database round trip per cache miss. Deliberate — it is the price of a counter that is genuinely shared. A per-instance memo short-circuits clients that have already been refused, so a flood costs one write per block window rather than one per request; that memo may only ever refuse faster, never allow, which is what makes a per-instance structure sound there when it isn't for the counter itself.

The health endpoint

GET /api/v1/health answers one question — is the indexer still producing data? — in a form a machine can act on.

Why it exists. Scoring silently stopping is the worst failure mode this system has, and it is worst precisely because nothing goes red. The site keeps serving last-known scores, every page renders, no route 500s, and the numbers quietly age. Before this endpoint the only ways to notice were to query Neon by hand or to eyeball a timestamp in the UI and compare it against a clock. The indexer's failure webhook (see "Retry and failure alerting") covers a different case: it fires when an adapter fails repeatedly, and it cannot fire at all if the cron simply stops arriving, because nothing runs to notice. This endpoint is the outside-in check that catches that.

Staleness is measured from the last successful run. A failed run produced no score. Measuring from lastRunAt would let an adapter that fails reliably every five minutes report as perfectly fresh forever — inverting the endpoint's purpose. So staleMinutes is the age of the newest ok run and nothing else.

lastRunAt / lastRunStatus are still published per protocol, because read together with the above they say where the problem is:

lastRunAt lastSuccessfulRunAt What it means
fresh fresh Working.
fresh stale The cron is arriving; this adapter is failing. Isolated.
stale stale The cron is not arriving. Infrastructure.
null null Registered but never indexed.

Three states, not a boolean. "Unhealthy" conflates one adapter failing (a code bug in one file) with nothing succeeding anywhere (the pipeline is down). Those have different owners and different first moves, and a single flag forces an operator to open the body to find out which — which is what having a status is supposed to avoid.

status Meaning HTTP
healthy Every protocol scored successfully within the threshold. 200
degraded Some current, some not — or all stale but inside the down window. 503
down Nothing current anywhere, past the down window. The cron itself looks dead. 503

Both non-healthy states answer 503, so an uptime monitor catches either without parsing the body; the distinction between them is for the human who then opens it. 503 rather than 500 because nothing errored — the route queried successfully and is telling the truth about a pipeline that is behind, which is exactly what 503 means. A genuine 500 on this route means we could not find out, and keeping those on separate codes is what lets a monitor tell "the indexer is stopped" from "the health check is broken".

The thresholds, both configurable, defaults in dashboard/app/api/_health.ts:

Setting Default Why
STENION_HEALTH_STALE_MINUTES 30 Six missed cycles at the ~5-minute cadence. Sits above STENION_ALERT_THRESHOLD (4 cycles, ~20 min) so the webhook mentions a problem before the monitor goes red.
STENION_HEALTH_DOWN_MULTIPLIER 2 60 minutes with not one successful run anywhere before blaming the cron itself.

30 is sized against what already exists rather than picked round. The indexer's alert threshold is 4 consecutive cycles, on the stated reasoning that "a score 20 minutes stale is not an emergency — false pages are how people learn to ignore alerts". A 503 an uptime monitor consumes is a louder signal than that webhook, so it must sit at a higher bar; 30 > 20 gives the intended escalation order. The floor is set by the cadence: under ~10 minutes, one slow cycle, a cold start, or a single cron-job.org misfire would read as an outage.

The down window exists because "everything is stale at once" is weaker evidence than it sounds. Every adapter shares Soroban RPC and Horizon, so a broad upstream outage takes them all out together while our own infrastructure is fine — and calling that "the cron is dead" sends an operator to the wrong place. With four targets today (one adapter serving three of them), "all of them" is a small sample. Doubling the window costs nothing operationally, because degraded is already 503 and a monitor has already fired; all it buys is the confidence to name which thing broke. The window is measured from the freshest success across the registry — the most generous reading available, so the endpoint can never report worse than the truth.

A fresh failure alone is not unhealthy. A protocol whose newest run failed but whose newest success is four minutes old reports healthy. That is deliberate. The indexer already retries (default 3 attempts) before recording a failure at all, but it is still one cycle, and the data it protects is current — turning red there pages someone about a blip. Repeated failure is not missed: an adapter that keeps failing stops producing successful runs and crosses the threshold on its own. Sustained failure and staleness are the same event seen at different times, so staleness alone catches it with the transient case filtered out for free. A consumer who does want to act on any single failed cycle reads lastRunStatus, which is why it is published.

One query, no fan-out. Store.listRunHealth() reuses the same two LEFT JOIN LATERAL subqueries the leaderboard uses over the existing (protocol_id, run_at DESC) index — the ok side just selects run_at instead of the score. No schema change was needed; the row that answers this was already one column away. Nothing score-derived is selected: no safety_score, no factors jsonb, so a freshness probe cannot fail because a score was malformed, and no adapter error text is republished on an unauthenticated endpoint. A health check that fans out per protocol gets slower in proportion to how much there is to report on, and a probe that times out under load is indistinguishable from the outage it exists to detect.

A cold Neon can make this probe time out, and that is left alone deliberately. Measured on 2026-08-22, the first query against a Neon instance scaled to zero took ~20s; Vercel's default route timeout is 10s, so such a probe returns a platform 504 rather than this route's own 503. Raising maxDuration to chase a nicer body is not worth it: Neon only goes cold when nothing has touched it for a long while, which on this deployment means the indexer has stopped — precisely the case where the verdict is "unhealthy" regardless. A monitor treats 504 and 503 identically, so the answer survives; only the body is lost, and buying it back would make every healthy probe wait longer.

An empty registry reports down, not healthy. Vacuous truth is the wrong answer for a probe: "nothing is stale because there is nothing" is a database migrated but never indexed, or one pointed at the wrong connection string.

The policy — what stale means, the three states, the HTTP mapping — lives in dashboard/app/api/_health.ts, a leaf module with no imports at all, for the same reason _cache.ts and _http.ts are: every interesting state here is one production has never produced (risk_scores has never held a failed row), so it is asserted in _health.test.ts or it is asserted nowhere.

API versioning

The public API is versioned in the URL. The documented, canonical paths are:

Endpoint Returns
GET /api/v1/protocols The leaderboard: every protocol + its latest score.
GET /api/v1/coverage Assessed protocols and markets Stenion does not score.
GET /api/v1/protocol/:id One protocol's detail, factors, and run history.
GET /api/v1/health Indexer freshness per protocol + one overall status.

The consumer-facing reference is API.md, rendered on the site at /docs/api. This section owns the policy; that document owns the contract as an integrator meets it — request and response examples, the ok/failed history union, the staleness model, error shapes, and the observable caching/rate-limit headers. Its examples are captured from the live production API rather than written from the types, deliberately: a doc written from db/src/store.ts would reproduce the type rather than the truth. Re-capture them when a response shape changes — and note that what a client actually observes is not always what a route sets (Vercel's CDN consumes s-maxage, so a 200 reaches the client as Cache-Control: public, max-age=0 plus Age).

The policy:

  • Additive changes stay on v1. A new field in the response — a sixth *Safety factor, an extra piece of metadata — does not break a client that ignores fields it doesn't know about, so it ships on v1. Consumers should parse defensively and tolerate unknown fields.
  • Breaking changes get a v2. Renaming a field, removing one, changing a type or the meaning of an existing value, or restructuring the envelope — anything that can break a client reading the documented shape — goes to a new version path, with v1 left serving its existing contract until it's deliberately retired.

Note that a methodology change (a formula, threshold, or weight) is not an API version change: safetyScore is still a 0–100 number with the same meaning, so the scores move but the contract doesn't. Methodology changes are versioned in METHODOLOGY.md, not in the URL. A change to the taxonomy — a renamed or removed factor — is breaking, and would need a v2.

No unversioned paths. The pre-versioning paths /api/protocols and /api/protocol/:id are gone — they 404. They existed briefly as transitional aliases during the /v1 move and were removed once a repo-wide sweep confirmed nothing referenced them. Every public API path carries a version segment; there is no unversioned surface to fall back to.

The cron trigger is not versioned. POST /api/cron/run-indexer is internal plumbing, not a public contract — it's secret-gated, has no CORS, and its only caller is our own cron-job.org schedule. Versioning it would imply a compatibility promise we don't make. It stays at /api/cron/*, and a /api/v1/cron/* path deliberately does not exist.

Why @stenion/api exists but isn't deployed

@stenion/api was the original public API — a bare node:http server built before the deploy architecture consolidated onto one Vercel project. Bare node:http doesn't fit Vercel's serverless model, and running the API as a separate service from the dashboard is more moving parts for a solo, pre-funding project to operate. So the two endpoints were re-homed as Next.js Route Handlers in the dashboard (same Store methods, identical JSON contract).

The package is kept in the tree — as the reference for the original bare-Node implementation and in case a standalone API service is ever wanted again — but it is legacy and not deployed. The live API is the dashboard's routes.

It is not at parity, deliberately. It has no caching and no rate limiting, and those were not back-ported when the dashboard routes gained them. Both are deployment concerns rather than API concerns: the cache is a Cache-Control header that only means something with a CDN in front, and the rate limiter's counter lives in Postgres specifically because serverless has no shared memory — a single long-lived Node process has memory, so paying a database round trip per request there would be the wrong trade. Whoever revives it owns both decisions afresh; the JSON contract and the versioned paths are what must not change. The one rule that does carry over is a property of the data rather than the transport: whatever caches it must not mask lastRunAt/lastRunStatus. The header comment in api/src/index.ts says all of this at the point someone would actually read it.