Seller pricing from the live 402, a per-payer spend ceiling, a higher routing tier, and a wallet-trend alarm - #715
Merged
Merged
Conversation
The canary gate + settlement alarm, the facilitator diagnostics and labels, and the redis CI coverage. Each entry leads with the defect it exists to prevent, because the pattern behind all three was a guarantee that was described but could not fail when violated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…logue has none
A seller listed 39 endpoints and every row indexed as price:null, priceUsd:0,
payable:"unknown" - while each endpoint returns a textbook x402 v2 challenge
(eip155:8453, Base USDC, amount 990000 = $0.99, real payTo) the moment you POST
{} at it. Confirmed independently before touching anything: their manifest,
their live 402, and our index rows.
Two causes, both ours, and neither specific to them:
1. A manifest may list `resources` as bare URL STRINGS - theirs does, and the
shape is permitted. normaliseManifestTools reads a price only from an
OBJECT, so a string-listing seller is permanently priceless however well
their endpoints behave. It also leaves method defaulting to GET, and their
routes 404 on GET and 402 on POST, so a GET-only probe sees a dead
catalogue.
2. probePaywall - the only thing that talks to a seller's endpoint - filters on
`Number(t.price) > 0`. A priceless route is never probed, and probing is the
only thing that would give it a price. Circular by construction: the sellers
who most need the probe are precisely the ones excluded from it.
Not one seller's problem. Measured across the index the same day: 146 of 500
sellers had ZERO priced rows and 127 rows read payable:"unknown".
OpenAPI cannot close this - it has no place for an x402 quote - so the 402
itself is the only source of truth, which is why the router already reads a
live 402 for payTo before spending. This reads the same challenge for price and
networks, and mirrors bazaarItemToTool's preference order (Base USDC, then any
USDC, then first) so a live-probed row and a Bazaar row stay comparable.
Conservative about money in three ways. An asset we cannot price leaves price
NULL and still records the networks, so the row reads payable-on-Base rather
than a guessed figure - $0 would publish a paid tool as free, the one wrong
answer a buyer acts on immediately. PUT/PATCH/DELETE are never probed, so an
unpaid probe cannot mutate a stranger's server. And it only ever ADDS: a priced
row is skipped and a failed probe leaves the row untouched.
Gentle on sellers, per the #645 lesson: at most 3 priceless routes per seller
per crawl, per-ROUTE backoff through probeDue, and a route that gets priced is
never a candidate again.
Verified against the reporting seller's live endpoint through the real code
path - assertPublicUrl and the SSRF dispatcher included - GET 404 skipped, POST
402 read, $0.99 and their payTo learned. 28 offline assertions on fixtures
captured from that same response.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… routing ceiling
Two things, and the order matters: the guard is what makes the price rise safe.
THE HOLE. @x402/express runs the handler FIRST and settles AFTER, and the
external routing handler pays a third-party seller from our spending wallet.
So: buyer's payment verifies, we pay the seller real USDC, our settlement
fails, buyer is charged nothing, we are out the spend. Self-dealt - one wallet
listing the seller and buying from it - every drained dollar lands back in the
attacker's pocket, bounded per call only by the tier cap.
Verify-then-fail-to-settle is not theoretical; it happens naturally when a
payer's balance drops between the two, which is documented on Solana where our
own best buyer drained to $0 and its last four purchases "timed out".
What already existed bounds WHAT we pay and none of it bounds WHETHER WE GET
PAID: the canonical-USDC asset pin (a decoy in another token cannot be signed),
the tier cap re-checked against the live 402 rather than the seller's
advertised price, and the 50-settlement/3-payer reliability floor. All of them
run before a spend that settlement has not yet blessed.
So a payer now carries a DEBT CEILING of unsettled upstream spend. Recorded
before the buy, resolved on the FINAL response - res.on("finish") with
statusCode 200, after settlement - never on handler success, which is precisely
the state that precedes a settlement failure. An unsettled spend keeps counting
until it ages out; a settled one clears instantly. It only ever bites a wallet
whose payments are failing.
Not a reputation system: 25 settled calls in a row are never impeded.
THE PRICE. The $0.50 underlying ceiling made the premium half of the index
unroutable - the seller who reported the price:null bug prices their gates at
$0.99, $1.50 and $2.99, every one above the top tier, so the router could only
409 them to their own direct route. route-execute-pro covers underlying <=$3.00
at $3.30, the same 10% spread as the max tier.
Three bugs found while doing it, two of them mine:
- The first draft registered res.on("finish") inside the tool handler, where
`res` does not exist: handlers are called as handler(input, req). It would
have installed nothing, reported nothing, and left the guard recording spends
that never resolved. The handle now rides the request and server.js resolves
it centrally.
- I added a tier without adding it to SELF_FUNDING_SLUGS, so its revenue would
have settled to the treasury while its spend came from the burner - a slow
one-way drain. Caught by the existing test that locks that set against
EXEC_TIERS, written after the same mistake in July.
- The ceiling and the largest tier are coupled: a ceiling below the biggest
underlying cap makes that tier dead on arrival, refusing every payer
including honest ones, with nothing to report it. Now asserted, not merely
commented.
20 + 45 offline assertions.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…all, not the smallest Raising the routing ceiling to $3.00 quietly broke the alarm that watches the wallet paying for it. UPSTREAM_BUYER_LOW_USD defaulted to $0.50, which was correct when the only thing spending from that wallet was Blockscout at $0.002/call - $0.50 covered hundreds of calls. With route-execute-pro, "ok" means "has at least $0.50" for a wallet that cannot fund ONE call, so /api/gateway-status would have reported ok and the heartbeat would have stayed green right up to the failure the alarm exists to prevent. Measured before writing this: prod reports upstreamBuyer "ok" right now, and "ok" is a bucket, not a balance - it cannot distinguish $0.60 from $60. Default is now two largest-tier calls, so we are paged with room to top up rather than at the moment of starvation, and an assertion locks it to EXEC_TIERS: a threshold that silently stops covering the biggest call reports nothing on its own. Same coupling, and same fix, as the unsettled-spend ceiling in the previous commit - adding a tier now fails CI in two places instead of degrading two guards in silence. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… that runs low Mike's observation, and it holds up in the code: that wallet should never go down. Everything that spends from it also settles INTO it - SELF_FUNDING_SLUGS sets payTo to the burner for exactly those tools (payments.js acceptsForItem) - and every execution tier charges more than it can spend. Worst case per call: +$0.005, +$0.01, +$0.05, +$0.30. Blockscout is the same shape. So barring a manual withdrawal the balance is monotonically non-decreasing. Which makes the low-water threshold the wrong instrument. It answers "is there enough left", and only after the money has gone. The interesting question is "did it fall at all", because a fall means one of exactly three things: someone withdrew, a spend's revenue never arrived (the verify-then-fail-to-settle drain the per-payer ceiling now bounds), or something we do not understand. A withdrawal trips it too, on purpose. The alarm's job is to say "this wallet fell and nobody told me"; a human who withdrew closes the issue in one click, and a silent fall is the one we must never miss. The hard part is the dip that settlement ordering GUARANTEES: we pay the seller during the handler and collect afterwards, so the balance is legitimately lower in between. An alarm that cannot tell that from a drain would page on every healthy call and be muted within a day. Hence a high-water mark, a tolerance, and a requirement that the fall persist across consecutive 5-minute reads. One subtlety with its own assertion: a read INSIDE tolerance must not reset the counter. A wallet bleeding $0.40 per read sits within tolerance every single time, so a counter that cleared on those would never fire while the wallet quietly emptied. Balances stay off /api/gateway-status - the trend is bucketed to "ok" / "draining" / "unknown", never a number, same rule as the existing status. 16 offline assertions, including the false positive that matters (an in-flight dip that recovers) and the recovery path that clears the issue. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ction, and the redaction's own leak Reviewed everything I added today instead of assuming it was clean. GLOBAL PROBE BUDGET (the one that mattered). The live-quote enrichment was bounded per SELLER at three routes per crawl, which sounds gentle until you multiply: roughly a third of indexed rows carry no price, so a per-seller-only limit fires thousands of outbound requests every five minutes across the whole index. That is issue #645 rebuilt with a different label - the report about 686 requests to a path that 404'd every time. Per-route backoff quiets it eventually, but "eventually" is the first several cycles, and the seller feels those. Now bounded per CYCLE as well; the rest wait their turn. EXPOSURE CLEARED ON A THROW. The external-spend catch cleared the payer's exposure on the theory that a failed buy never spent. payExternal can throw AFTER signing and broadcasting - a network error on the response, a timeout - and clearing on those is precisely how a spend disappears from the ledger the guard exists to keep. The exposure now stands and ages out on its own. DEAD IMPORT. exposureSnapshot was imported into server.js and never used. It returns payer addresses, so the load-bearing fact is that it was never wired to a public surface; the import is gone. REDACTION, AND ITS OWN LEAK. The facilitator diagnostic logs a body we did not write, from a host we authenticate to, into a log aggregator - so long hex/base64 runs are redacted, since an error page echoing a request header would put a credential there permanently. The first version walked straight past a JWT (segments are dot-separated and individually short), so a bearer token would have been logged whole. Then the fixture I wrote to prove that failed our own gitleaks history scan: RuleID jwt. A realistic three-segment token in the repo is exactly what that rule is for, and .gitleaks.toml is explicit that paths are never excluded because a path exclude would hide a real credential in a test fixture. So the fixture is now shape-only with no eyJ prefix - it exercises the same branch without putting a token-shaped string in the repo - and the commit that carried it was rewritten out of the branch rather than allowlisted. Also asserted: a Cloudflare ray id must SURVIVE redaction, since it is the thing a provider asks for. Checked and found fine: price stays polymorphic (number from bazaar, "$x" from manifests) but every surface reads it through parsePrice/fmtUsd. Local sweep green: facilitator-diagnostics 35, live-quote 28, spend-guard 20, balance-trend 16, route-execute 46, wish 54, canary-coverage 148, and the MCP self-consistency guard at 1467 against a booted server. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Started from a seller's report that their 39 listed endpoints all showed
price:null. Confirmed independently before changing anything, and it turned out not to be about them.Live-402 pricing. A manifest may list
resourcesas bare URL strings (permitted, and theirs does), so our reader found no price and defaulted the method to GET — while their routes are POST-only. Worse,probePaywallfiltered onprice > 0before probing, so a priceless route was never probed and probing is the only thing that could price it. Sampling the 69k-row index showed roughly a third of rows priceless. The crawler now reads price and networks from the live challenge, tries GET then POST{}, never probes PUT/PATCH/DELETE, and leaves an unpriceable asset asnullrather than$0— publishing a paid tool as free is the one wrong answer a buyer acts on.Per-payer spend ceiling. Settlement runs after the handler, and the external routing handler pays a third-party seller from our wallet. A payment that verifies and then fails to settle leaves us out the spend with the buyer charged nothing; self-dealt, every drained dollar returns to the attacker. Spend is now recorded before the buy and resolved on the FINAL response, never on handler success.
Higher tier.
route-execute-procovers underlying ≤$3.00 at $3.30 (same 10% spread as max), because the $0.50 ceiling made the premium half of the index unroutable.Wallet trend alarm. That spending wallet is self-funding and every tier is net-positive, so it should only rise. A fall means a withdrawal, a spend whose revenue never arrived, or something we don't understand — all worth waking up for. Tolerates the dip that settlement ordering guarantees.
Self-review found four defects in my own new code: no global probe budget (issue #645 rebuilt), exposure cleared on a throw that can happen post-broadcast, a dead import, and a redaction that walked past a JWT. The redaction fixture then failed our own gitleaks history scan, which is that rule working; the fixture is shape-only now and the commit was rewritten out rather than allowlisted.