Registration overhaul: SIWX Free endpoints, display filtering, schema detection, robustness#918
Merged
Conversation
The discovery package doesn't always populate advisory.inputSchema even when the 402 body contains a full bazaar schema. Before rejecting, check the raw paymentRequiredBody for a bazaar extension with schema data. This prevents false rejections for merchants like rep.memoryapi.org whose 402 bodies have complete schemas but whose OpenAPI extraction doesn't propagate to advisory.inputSchema.
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Two fixes for endpoints like api.northeastdealintel.com where 5 endpoints return 402 but only 1 registered: 1. registerResourcesFromDiscovery now passes resource.method to probeX402Endpoint so GET endpoints are probed with GET (not POST) 2. validateResource already falls back to bazaar schema (previous commit) which handles the missing inputSchema from discovery
onchainlu
changed the title
Previously the batch test probed ALL discovered endpoints (including health checks, webhooks, internal routes). For origins like api.northeastdealintel.com (41 endpoints, only 5 paid), this triggered rate limiting (503s) that caused even the valid paid endpoints to fail. Now only endpoints with authMode 'paid' or 'apiKey+paid' are probed.
Non-paid endpoints (health checks, webhooks, etc.) aren't probed but should still show a red X since they won't be registered. Sort order: failed probes → passed → non-probed at bottom.
When registerResourcesFromDiscovery registers multiple resources from the same origin concurrently, the connectOrCreate on the origin record can race — both try to create, second hits P2002 unique constraint. Fix: upsert the origin outside the transaction first, then connect (not connectOrCreate) inside the transaction.
- Failed (red X): probed but failed validation - Warning (yellow triangle): registered but has audit warnings - Verified (green check): passed with no warnings - SIWX (blue check): identity-gated, no payment - Skipped (grey dash, strikethrough): not x402, not probed Sort order: failed → warning → verified/SIWX → skipped
- Sort: errors → warnings → SIWX (blue) → verified (green) → skipped - Bazaar fallback: type-check paymentRequiredBody + try-catch - Origin upsert moved inside transaction to prevent race conditions - nonPaidUrls: only classify with evidence from authModeMap, not absence - Remove paidModes useMemo code smell (inline static set)
…king - Filter discovery to only show paid + SIWX endpoints (hide apiKey, unprotected) - Relabel SIWX badges as "Free" with green styling - Register SIWX endpoints to DB via registerSiwxResource() (x402Version: 0, no Accepts) - Update displayableResourceWhere, listOrigins, listOriginsWithResources, searchResources with OR clauses for SIWX — respects chain/address filters (SIWX excluded when filtering) - Allow SIWX-only origins to succeed registration - Replace SIWX warning with missing output schema warning in validateResource() - Add batch test progress tracking (Checking N/M endpoints...) - Include SIWX count in registrableResourceCount after batch test
onchainlu
changed the title
…d endpoints - Remove Advanced section (Custom Headers + Manual URLs) from register form - Clean up dead state/handlers/imports - Fix discovery filter: only include endpoints with a known registrable authMode (previously unclassified endpoints passed through, showing 36 non-paid endpoints for origins like api.northeastdealintel.com)
- Discovery filter now excludes only explicitly non-registrable (unprotected, apiKey). Unclassified endpoints (authMode absent) are kept — they may be paid but the discovery package didn't detect it (e.g. bazaar-only on rep.memoryapi.org). - Server-side batch test filter updated to match: probe unclassified + paid, skip SIWX/unprotected/apiKey. - stableflowers.dev: 3 unprotected excluded, 2 SIWX + 1 paid shown correctly. - northeastdealintel: 41 shown pre-probe, 5 pass post-probe (36 unclassified fail). - rep.memoryapi.org: 5 unclassified shown and probed (all should pass). - Add "(Not blocking)" to post-registration warnings dropdown.
Discovery package warns about missing schemas in the raw 402 body (bazaar extension), but the advisory may already have schemas from the OpenAPI spec. Drop SCHEMA_INPUT_MISSING when advisory.inputSchema exists and SCHEMA_OUTPUT_MISSING when advisory.outputSchema exists. Applied in both batch test (developer.ts) and registration (register-origin.ts) paths.
Server probes sequentially anyway, so 1 endpoint per request gives real-time "Checking 3/5 endpoints..." updates without changing total probe time.
- Replace 5 separate prompts (generic, error, warning, v1, schema) with one consolidated prompt that lists all errors AND warnings together. The setup prompt (no discovery) remains separate. - Consolidated prompt includes all failing URLs + their errors, all warnings, and missing schema URLs in a single copy-to-clipboard block. - Fix instructions now cover paid endpoints (402), free endpoints (security: []), request validation ordering, required params, and schemas. - Simplify DiscoveryFixHint: remove v1Migration/needsSetup/missingSchema props - Fix "Missing input schema" error: say "requestBody or parameter schema" instead of misleading "request/response schemas" - SETUP_PROMPT now mentions both paid (x402) and free (identity-gated) endpoints
Extract shared deduplicateWarnings() to discovery/utils. Applied at both merge points: batch test (developer.ts) and registration (resources.ts). Prevents duplicate warnings when probe and validation emit the same code.
When an origin has the same URL with different HTTP methods (e.g. POST and DELETE on /api/site/domain), concurrent registration hits a unique constraint race on the resources table. Catch P2002, look up the existing record, and return success. Found via stableupload.dev testing.
Sort resources before chunking: paid/apiKey+paid first, unclassified last. For origins like api.northeastdealintel.com (5 paid + 36 unclassified), paid endpoints are probed before rate limiting kicks in from probing non-paid endpoints that return 503.
onchainlu
changed the title
- Add "Free (Unprotected) Endpoints" section to discovery spec page explaining security: [] as the OpenAPI standard for declaring no-auth-required - Add endpoint classification summary table (paid, SIWX, free, unclassified) - Add "No valid x402 response" to Common Failure Reasons with fix guidance - Update registration error message to tell merchants about security: [] when endpoints fail probing — "If these endpoints are free, add security: [] to their OpenAPI definition to exclude them from probing"
- registerSiwxResource P2002 catch: always return success even if findUnique returns null (record deleted between race and lookup). The constraint violation proves the resource existed. - Consolidated prompt: early return with spec link when all issue arrays are empty (shouldn't be reachable but avoids misleading generic message).
Previously only errors had a collapsible section — warnings showed as yellow triangle icons with no explanation. Now warnings get their own "(Not blocking)" dropdown listing each endpoint and its warning messages, plus a fix prompt.
When a merchant server rate-limits a probe (429 or 503), retry up to 2 times with exponential backoff (1.5s, 3s). Fixes transient probe failures for origins like tiresapi.com where sequential probing of 13 endpoints triggers rate limiting on later endpoints.
When the user clicks "Register All", pass the batch test results (URL → advisory map) to the server. Endpoints with pre-tested advisories skip probing and use the advisory directly for registration. Endpoints not in the pre-tested set are probed as before. This avoids redundant probing and rate limiting — the batch test already validated these endpoints moments ago. Chain: handleRegisterAll → register(origin, preTestedResults) → registerFromOrigin mutation → registerResourcesFromDiscovery(preTestedAdvisories)
Matches the existing fire-and-forget pattern used for ownership verification. The origin row is already created inside upsertResource's transaction — the metadata upsert just enriches it with scraped title/favicon/OG. When multiple resources from the same origin register concurrently (especially with pre-tested advisories skipping probes), the metadata upsert can P2002 race. Safe to swallow since another concurrent call will succeed.
…data
The registerFromOrigin endpoint previously accepted pre-tested advisory
data (including payTo addresses) from the client. A malicious actor could
intercept the tRPC call and substitute payTo with their own address,
redirecting merchant payments.
Fix: probe results now stay server-side in Redis under a session ID.
The batch test caches results under probe-session:{sessionId}:{url}
with 5-min TTL. Registration reads from this cache — the client only
passes an opaque probeSessionId, never advisory data.
Also adds cross-origin URL validation in discovery: endpoints whose
resolved URL doesn't match the discovery origin are rejected, preventing
malicious OpenAPI specs from injecting absolute URLs to other origins.
Some merchants embed the full 402 payment body in a payment-required header, exceeding Node's default 16 KB header size limit. The discovery library silently drops these as network errors. Adds a direct HTTPS fallback probe (node:https with maxHeaderSize 128KB) that fires when the library's probe fails. Recovers the 402 body, merges OpenAPI schema data, and surfaces a HEADERS_OVERFLOW warning to the merchant advising them to reduce their header size. Before: war-tracker.com registered 9/16 endpoints (7 skipped) After: war-tracker.com registers 16/16 endpoints (7 with warnings)
…dation - Remove unsafe `as unknown as` double cast on payment options in directProbe402 fallback; filter + validate accepts before mapping - Extract duplicated SIWX registration logic into registerAsSiwx helper - Validate cached probe result structure before casting from Redis - Reuse URL object in fetch-discovery cross-origin check
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Overhaul of merchant registration — fixes schema detection, adds SIWX (Free) endpoint support, cleans up the UI, hardens robustness, fixes a payment redirection vulnerability, and recovers endpoints with oversized response headers.
Security: server-side probe cache
payToaddresses) previously round-tripped through the client. A malicious actor could intercept the tRPC call and substitute payment addresses.probe-session:{sessionId}:{url}with 5-min TTL. Registration reads from this cache; the client only passes an opaqueprobeSessionId.Oversized header fallback (HEADERS_OVERFLOW)
payment-requiredheader, exceeding Node's 16 KB limit. The discovery library silently drops these as network errors.node:https— a fallback probe withmaxHeaderSize: 128KBrecovers the 402 body, merges OpenAPI schema data, and surfaces aHEADERS_OVERFLOWwarning to the merchant.Schema detection
validateResource()— checks 402 body for bazaar extension before rejecting for missing input schemaDisplay filtering
unprotected,apiKey) from discovery listSIWX → Free
registerSiwxResource()— writes Resource withx402Version: 0,metadata: { authMode: 'siwx' }Probing
Warnings & prompts
deduplicateWarnings()utilitysecurity: []for free endpointsDocumentation
Robustness
registerSiwxResource— concurrent registrations caught and treated as successregisterSiwxResourceTest results (21 origins)
Batch-to-registration consistency: perfect across all 21 origins — every batch test failure maps to a registration skip/fail with identical error messages.
Files changed
probe-cache.tsprobe.tsregister-origin.tsresources.tsregisterSiwxResource(), bazaar schema fallback, output schema warningdeveloper.tspublic/resources.tsprobeSessionIdinstead ofpreTestedResults(security fix)fetch-discovery.tsuse-batch-test.tsuse-discovery.tsREGISTRABLE_AUTH_MODESfilter, pass sessionId not advisory datause-register-from-origin.tsform.tsxdiscovery-panel.tsxdiscovery-actions.tsxdeduplicate-warnings.tsmarkdown.ts