Seller pricing from the live 402, a per-payer spend ceiling, a higher routing tier, and a wallet-trend alarm

.github/trigger-deploy

@@ -1 +1 @@
-1786112186
+1786117290

.github/trigger-test

@@ -1 +1 @@
-1786111133
+1786117290

.github/workflows/deploy.yml

@@ -1012,6 +1012,15 @@ jobs:
       - name: Boot /supported guard (a dead facilitator costs ONE rail, not every paid route — the 2026-08-01 Celo outage; probe-driven drop, fail-open on total blindness, escape hatch — offline)
         run: node scripts/test-supported-guard.js
 
+      - name: "Live 402 quote (a priceless row learns its price from the challenge; unpriceable never becomes $0; PUT/PATCH/DELETE never probed — offline)"
+        run: node scripts/test-x402-live-quote.js
+
+      - name: "Buyer wallet trend (a self-funding wallet that FALLS is unexplained; a transient in-flight dip is not — offline)"
+        run: node scripts/test-buyer-balance-trend.js
+
+      - name: "External spend guard (per-payer debt ceiling: an UNSETTLED upstream spend keeps counting, so verify-then-fail-to-settle cannot drain the wallet — offline)"
+        run: node scripts/test-external-spend-guard.js
+
       - name: "Facilitator diagnostics (an HTML error page must name itself: block vs rate limit vs origin failure, and never touch the response — offline)"
         run: node scripts/test-facilitator-diagnostics.js
 

.github/workflows/heartbeat.yml

@@ -260,6 +260,40 @@ jobs:
             gh issue close "$OPEN" --repo "$GITHUB_REPOSITORY" --comment "Recovered: balance back above the low-water mark at $(date -u +%FT%TZ)."
           fi
 
+      # The spending wallet should never go DOWN, so a fall is worth more than
+      # a floor. Everything that spends from it also settles into it
+      # (SELF_FUNDING_SLUGS), and every execution tier charges more than it can
+      # spend, so barring a manual withdrawal the balance only rises. A
+      # low-water alarm fires after the money is gone; this fires on the first
+      # unexplained dollar. A withdrawal trips it too, deliberately: the alarm's
+      # job is to say "this wallet fell and nobody told me", and a human who
+      # withdrew can close the issue in one click.
+      - name: Upstream buyer wallet trend (a self-funding wallet that FALLS is unexplained)
+        if: always()
+        continue-on-error: true
+        run: |
+          TREND=$(curl -s --max-time 15 "$PROD/api/gateway-status" | jq -r '.upstreamBuyer.trend // "unknown"' 2>/dev/null || echo unknown)
+          echo "upstream buyer trend: $TREND"
+          TITLE="Upstream buyer wallet is DRAINING (unexplained fall)"
+          OPEN=$(gh issue list --repo "$GITHUB_REPOSITORY" --state open --search "in:title \"$TITLE\"" --json number --jq '.[0].number // empty')
+          if [ "$TREND" = "draining" ]; then
+            if [ -z "$OPEN" ]; then
+              gh issue create --repo "$GITHUB_REPOSITORY" --title "$TITLE" --body "The x402 upstream spending wallet has fallen below its high-water mark across several consecutive reads.
+
+          That wallet is SELF-FUNDING: every tool that spends from it also settles into it, and every execution tier charges more than it can spend (worst case +\$0.005 / +\$0.01 / +\$0.05 / +\$0.30 per call). Its balance should only rise. A sustained fall means one of:
+
+          1. A manual withdrawal - close this issue if that was you.
+          2. Upstream spend whose revenue never arrived: a buyer's payment verified and then failed to settle, which is the drain the per-payer ceiling in src/external-spend-guard.js bounds. Check /__operator/stats and the route-execute receipts.
+          3. Something we do not understand, which is why this alarm exists.
+
+          Balances are deliberately not published on /api/gateway-status; read the wallet directly to see the number."
+            else
+              gh issue comment "$OPEN" --repo "$GITHUB_REPOSITORY" --body "Still falling at $(date -u +%FT%TZ)."
+            fi
+          elif [ "$TREND" = "ok" ] && [ -n "$OPEN" ]; then
+            gh issue close "$OPEN" --repo "$GITHUB_REPOSITORY" --comment "Recovered: the wallet set a new high-water mark at $(date -u +%FT%TZ)."
+          fi
+
       # Settlement freshness — the alarm that did not exist on 2026-08-07.
       #
       # The daily paid canary stopped buying on 2026-08-02 and reported success

CLAUDE.md

@@ -332,6 +332,62 @@ with `res.statusCode === 200`. (`node_modules/@x402/express/dist/esm/index.mjs`.
   which is real but lives on the stdio npm package). Mutation-tested: removing
   `request_tool` from the listing fails 2 assertions, a fake tool name fails 1, a
   fake route fails 1.
+- **Canary gate + settlement freshness alarm (2026-08-07):** the daily paid canary
+  stopped buying on **2026-08-02** and reported success every run for five days. Its
+  gate asked GitHub for the last SUCCESSFUL RUN, but a run whose gate SKIPS the buy
+  also concludes green, so every skip refreshed the window the next gate read and it
+  ratcheted permanently shut (measured across 40 runs: not one scheduled run bought
+  after the gate shipped; every real purchase came from a manual dispatch, which
+  bypasses the gate via `if: github.event_name == 'schedule'`). Nothing paged, because
+  skipping is not a failure — the ONLY surface that noticed was `/status`, reporting
+  the settlement component stale. **The gate now asks PRODUCTION when a canary last
+  BOUGHT** (`/api/status` settlement observation, written only by a canary that ran),
+  requiring fresh AND operational; unreachable status or a missing observation proceeds
+  with the buy, and every `jq` read carries a fallback because jq exits non-zero on a
+  non-JSON body and `set -e` would fail the gate. The canary job's `if` gained
+  `!cancelled()`: a job-level `if` with no status function still carries the implicit
+  `success()` on `needs`, so a FAILED gate would have SKIPPED the buy — the opposite of
+  what the comment beside it claimed, and never verified. **`heartbeat.yml` now pages on
+  a stale settlement observation** and self-heals once per episode by dispatching the
+  canary on FIRST detection only (a dispatch always buys; page rather than loop if
+  buying is genuinely broken). Proven end-to-end 2026-08-07: alarm fired → dispatched →
+  found a real failure → opened issues; then the 14:17 UTC SCHEDULED run bought (first
+  since 08-02) and the recovery branch closed its own issue. `scripts/test-canary-coverage.js`
+  locks the class: the gate must read `/api/status` and must NOT read `gh run list`,
+  every jq read must have a fallback, and the `if` must carry a status function.
+- **Facilitator failure diagnostics (`src/facilitator-diagnostics.js`, 2026-08-07):**
+  15 settle failures across Base/Solana/Polygon/Arbitrum all logged 200 characters of
+  `<html><head><title>Coinbase</title>…` — `@x402/core`'s `responseExcerpt` truncates an
+  error body at 200 chars, and on an HTML page that budget is spent entirely on markup.
+  A facilitator outage and an edge REFUSING OUR EGRESS were indistinguishable, and those
+  need opposite responses (wait vs build the fifth relay — Yahoo/Nasdaq/Sei/Nodely are
+  the existing four, and Nodely 403s Railway's IP outright). A global-fetch wrapper,
+  scoped to registered facilitator hosts and non-2xx non-JSON responses only, reads the
+  body BEFORE the vendor truncates it, strips markup, and classifies: cloudflare
+  challenge/block, access denied, rate limited, origin error behind the edge, gateway
+  timeout — keeping `cf-ray`/`server`/`retry-after`. It **clones** before reading
+  (consuming the body would break settlement), swallows every internal failure, and logs
+  once at boot so a silent failure to install is visible immediately. **Errors are also
+  LABELLED with the facilitator that threw them** (`labelFacilitatorErrors`): the failure
+  hooks log the chain and never the client, so Solana/Polygon/Arbitrum failures read as
+  Coinbase's words though the boot log routes those to PayAI and only Base to CDP —
+  clients are tried in order, so the surfacing error is the FIRST tried, not the chain's
+  owner. The label is **PREFIXED, never substituted**: `isPreBroadcastSettleRejection`
+  matches `settle failed (402)` as a substring, so replacing the message would silently
+  break the fallback's safety classification. `scripts/test-facilitator-diagnostics.js`
+  (30 assertions, offline, in CI).
+- **Redis has REAL coverage in CI (2026-08-07):** nothing had ever connected to a redis.
+  `test-shared-limit.js` injects a fake store on purpose (it proves "two callers share
+  one counter", and a fake proves that exactly), which left the CLIENT path untested —
+  so a redis 4→6 bump arrived with a green CI that could not have caught a client
+  regression, the same worthless green as the tesseract 5→7 trap. Prod is **NOT**
+  in-memory (verified against Railway: `REDIS_URL` and `RATE_LIMIT_REPLICAS` are set,
+  and the shared limiter FAILS CLOSED). The test job now runs a `redis:7-alpine` service
+  container and `scripts/test-redis-integration.js` drives the real client (cap-of-1,
+  over-limit decrement, refund flooring, cache round trip). It asserts `degraded === false`
+  so it cannot pass via the fail-closed path with no server, and it **exits 1 rather than
+  skipping** when `REDIS_URL` is absent — a skipped integration test is why this went
+  untested at all.
 - **Marketplace latency / snapshot caching (`src/x402-economy.js`):** `GET /marketplace`
   (and `/api/x402-economy`) render from `x402EconomySnapshot()` — a ~500ms on-chain read
   (EIP-3009 USDC settlements on Base via CDP SQL). It is **stale-while-revalidate**: a fresh

scripts/test-buyer-balance-trend.js

@@ -0,0 +1,78 @@
+// The spending wallet should never go down, so a FALL is the signal.
+//
+// Everything that spends from it settles into it (SELF_FUNDING_SLUGS), and every
+// execution tier charges more than it can spend. Barring a manual withdrawal the
+// balance is monotonically non-decreasing. A low-water alarm fires after the
+// money is gone; this fires on the first unexplained dollar.
+//
+// The hard part is the TRANSIENT dip that settlement ordering guarantees: we pay
+// the seller during the handler and collect afterwards. An alarm that cannot
+// tell that from a drain would page on every healthy call.
+import { noteBuyerBalance } from "../src/tools/blockscout-kit.js";
+
+let pass = 0, fail = 0;
+const ok = (c, m) => { if (c) { pass++; console.log(`ok - ${m}`); } else { fail++; console.error(`FAIL - ${m}`); } };
+const fresh = (v) => noteBuyerBalance(v, { reset: true });
+
+{
+  ok(fresh(10) === "ok", "the first read is a baseline, never an alarm");
+  ok(noteBuyerBalance(10.3) === "ok", "a rise is healthy - this is what a self-funding wallet does");
+  ok(noteBuyerBalance(10.6) === "ok", "…and keeps re-baselining upward");
+}
+
+{
+  // THE FALSE POSITIVE THIS MUST NOT HAVE. Settlement runs after the handler,
+  // so the balance dips while a call is in flight and recovers when revenue
+  // lands. Paging on that would make the alarm useless within a day.
+  fresh(10);
+  ok(noteBuyerBalance(9.7) === "ok", "a dip inside tolerance is an in-flight call, not a drain");
+  ok(noteBuyerBalance(10.4) === "ok", "and it recovers when the buyer's payment settles");
+}
+
+{
+  // THE REAL DRAIN. A fall past tolerance, sustained across consecutive reads.
+  fresh(10);
+  ok(noteBuyerBalance(7) === "ok", "one big fall is not yet an alarm - a single read could be an in-flight max-tier call");
+  ok(noteBuyerBalance(6.9) === "ok", "two is still not");
+  ok(noteBuyerBalance(6.8) === "draining", "three consecutive reads below the high-water mark is a drain");
+}
+
+{
+  // A slow bleed sits INSIDE tolerance on every individual read. If a
+  // within-tolerance read cleared the counter, a wallet losing $0.40 a read
+  // would never alarm - it would just quietly empty.
+  fresh(10);
+  noteBuyerBalance(6.5); // past tolerance, counter 1
+  noteBuyerBalance(6.4); // counter 2
+  ok(noteBuyerBalance(9.6) === "ok",
+    "a read within tolerance of the high-water mark does not itself alarm");
+  fresh(10);
+  ok([9.6, 9.55, 9.5].map((v) => noteBuyerBalance(v)).every((s) => s === "ok"),
+    "…and a genuinely small wobble never alarms on its own");
+}
+
+{
+  // Recovery must clear it, or one bad afternoon pages forever.
+  fresh(10);
+  noteBuyerBalance(6); noteBuyerBalance(5.9);
+  ok(noteBuyerBalance(5.8) === "draining", "draining while it is falling");
+  ok(noteBuyerBalance(11) === "ok", "a new high clears the alarm and re-baselines");
+  ok(noteBuyerBalance(10.9) === "ok", "and the counter really was reset, not merely masked");
+}
+
+{
+  ok(fresh(NaN) === "unknown", "an unreadable balance is unknown, never a drain");
+  ok(noteBuyerBalance(undefined) === "unknown", "and never throws on junk");
+}
+
+{
+  // A withdrawal looks exactly like a drain, and SHOULD: the alarm's job is to
+  // say "this wallet fell and nobody told me". A human who withdrew can close
+  // the issue; a silent fall is the one we must never miss.
+  fresh(50);
+  const seq = [20, 20, 20].map((v) => noteBuyerBalance(v));
+  ok(seq[2] === "draining", "a manual withdrawal alarms too - indistinguishable on purpose");
+}
+
+console.log(`\n${pass} passed, ${fail} failed`);
+process.exit(fail ? 1 : 0);

scripts/test-external-spend-guard.js

@@ -0,0 +1,147 @@
+// Offline tests for src/external-spend-guard.js.
+//
+// The hole it closes: settlement runs AFTER the handler, and the external
+// routing handler pays a third-party seller from our wallet. A payment that
+// VERIFIES and then fails to SETTLE leaves us out the upstream spend with the
+// buyer charged nothing. Self-dealt - one wallet listing the seller and buying
+// from it - every drained dollar returns to the attacker.
+import {
+  maySpend, noteSpend, resolveSpend, payerExposureUsd, exposureSnapshot, __reset,
+} from "../src/external-spend-guard.js";
+
+let pass = 0, fail = 0;
+const ok = (c, m) => { if (c) { pass++; console.log(`ok - ${m}`); } else { fail++; console.error(`FAIL - ${m}`); } };
+
+const A = "0xAbCdEf0123456789AbCdEf0123456789AbCdEf01";
+
+// --- the attack, in one block ------------------------------------------------
+{
+  __reset();
+  // An explicit ceiling, not the shipped default: this block is about the
+  // MECHANISM, and tying it to whatever the default happens to be today would
+  // make it pass for the wrong reason the next time a tier moves.
+  const CEIL = { maxUnsettledUsd: 0.75 };
+  // Call 1: allowed, we spend upstream.
+  ok(maySpend(A, 0.5, CEIL).ok, "a fresh payer may spend");
+  const h1 = noteSpend(A, 0.5);
+  ok(payerExposureUsd(A) === 0.5, "the spend counts as exposure while unresolved");
+
+  // Their payment FAILS to settle. This is the whole point: handler success is
+  // not revenue, and the exposure must survive it.
+  resolveSpend(h1, false);
+  ok(payerExposureUsd(A) === 0.5,
+    `an UNSETTLED spend keeps counting against the payer (got ${payerExposureUsd(A)})`);
+
+  // Call 2 from the same wallet is refused before we spend a second time.
+  const second = maySpend(A, 0.5, CEIL);
+  ok(second.ok === false, "a second call is refused while the first is unpaid - the drain stops at one");
+  ok(/has not settled/i.test(second.reason), `the refusal explains itself (got: ${second.reason})`);
+}
+
+{
+  // The honest buyer's path: settle, and exposure clears immediately.
+  __reset();
+  const h = noteSpend(A, 0.5);
+  resolveSpend(h, true);
+  ok(payerExposureUsd(A) === 0, "a SETTLED spend clears the exposure at once");
+  ok(maySpend(A, 0.5, { maxUnsettledUsd: 0.75 }).ok, "and the payer may immediately spend again - this is a debt ceiling, not a reputation");
+}
+
+{
+  // A wallet that pays reliably is never impeded, however many calls it makes.
+  __reset();
+  let everRefused = false;
+  for (let i = 0; i < 25; i++) {
+    if (!maySpend(A, 0.5).ok) everRefused = true;
+    resolveSpend(noteSpend(A, 0.5), true);
+  }
+  ok(!everRefused && payerExposureUsd(A) === 0 && maySpend(A, 0.5).ok,
+    "25 settled calls in a row are never refused and leave zero exposure - a good buyer never hits the ceiling");
+}
+
+// --- identity handling -------------------------------------------------------
+{
+  __reset();
+  const h = noteSpend(A.toLowerCase(), 0.5);
+  resolveSpend(h, false);
+  ok(payerExposureUsd(A.toUpperCase().replace("0X", "0x")) === 0.5,
+    "EVM addresses are case-insensitive, so a payer cannot reset their ledger by changing case");
+
+  // base58 / Stellar / Algorand are case-SENSITIVE: folding them merges
+  // distinct payers, the same rule src/payer.js enforces.
+  const s1 = "GDNJXCKW7ZM7GEEVP674TWPU26YJNBQ2FI4ZIPRKTPTNUEJMDHFJWWRL";
+  const h2 = noteSpend(s1, 0.4);
+  resolveSpend(h2, false);
+  ok(payerExposureUsd(s1.toLowerCase()) === 0,
+    "a base58/Stellar address is NOT case-folded - folding would merge distinct payers");
+}
+
+{
+  __reset();
+  // An unattributable payer (free mode, a rail whose payer we cannot read) is
+  // allowed: refusing would break every legitimate buyer on those rails, and a
+  // single call is still bounded by the tier cap.
+  const v = maySpend(null, 0.5);
+  ok(v.ok === true && /not attributable/i.test(v.reason),
+    "an unreadable payer is allowed and says why - the tier cap still bounds the call");
+  ok(noteSpend(null, 0.5) === null, "…and nothing is recorded for a payer we cannot name");
+}
+
+// --- ceiling arithmetic ------------------------------------------------------
+{
+  __reset();
+  ok(maySpend(A, 0.5, { maxUnsettledUsd: 0.5 }).ok, "exactly at the ceiling is allowed");
+  const h = noteSpend(A, 0.5);
+  resolveSpend(h, false);
+  ok(maySpend(A, 0.01, { maxUnsettledUsd: 0.5 }).ok === false,
+    "one cent past the ceiling is refused - the check is on the TOTAL, not the single call");
+}
+
+{
+  // An unresolved row must not bar a payer forever (a process restart, a
+  // response that never finished), but must not clear so fast a loop outruns it.
+  __reset();
+  const now = 1_000_000;
+  noteSpend(A, 0.5, now);
+  ok(payerExposureUsd(A, now + 60_000) === 0.5, "exposure stands a minute later");
+  ok(payerExposureUsd(A, now + 11 * 60_000) === 0, "an unresolved spend ages out after the stale window");
+}
+
+{
+  __reset();
+  const h = noteSpend(A, 0.25);
+  resolveSpend(h, false);
+  const snap = exposureSnapshot();
+  ok(snap.length === 1 && snap[0].unsettledUsd === 0.25 && snap[0].calls === 1,
+    `the operator view reports who owes upstream spend (got ${JSON.stringify(snap)})`);
+}
+
+{
+  // resolveSpend must never throw on junk - it runs inside a response
+  // finish handler, where an exception would break the response.
+  __reset();
+  let threw = null;
+  try { resolveSpend(null, true); resolveSpend({ payer: "nope", id: 9 }, true); resolveSpend(undefined, false); }
+  catch (e) { threw = e; }
+  ok(!threw, "resolving an unknown or missing handle is a no-op, never a throw");
+}
+
+// --- the coupling that would silently kill a tier ----------------------------
+// If the ceiling is ever smaller than the largest execution tier's underlying
+// cap, that tier is dead on arrival: a single legitimate call exceeds the
+// ceiling and EVERY payer is refused, honest ones included. Nothing else would
+// report this - the tier would simply never succeed - so it gets an assertion
+// rather than the comment it started as.
+{
+  __reset();
+  const { EXEC_TIERS } = await import("../src/tools/route-execute.js");
+  const { __config } = await import("../src/external-spend-guard.js");
+  const biggest = Math.max(...EXEC_TIERS.map((t) => t.underlyingMaxUsd));
+  ok(__config.DEFAULT_MAX_UNSETTLED_USD >= biggest,
+    `the unsettled ceiling ($${__config.DEFAULT_MAX_UNSETTLED_USD}) covers the largest tier's underlying cap ($${biggest}) - otherwise that tier can never run`);
+  ok(maySpend("0x1111111111111111111111111111111111111111", biggest).ok,
+    "a single largest-tier call is allowed for a payer with no exposure");
+}
+
+console.log(`\n${pass} passed, ${fail} failed`);
+process.exit(fail ? 1 : 0);