feat: opt-in event logging for Stitch tool call - #182
Merged
Conversation
…lobs
Adds an opt-in event logging system that captures every Stitch MCP call
made through the proxy or stitch-mcp tool, eagerly snapshotting HTML and
screenshot bytes before their downloadUrls expire.
Enable with STITCH_MCP_LOG=1. Output goes to .stitch-mcp/log/events.jsonl
and .stitch-mcp/log/blobs/<2>/<sha>.<ext>. Disabled by default; zero
overhead and zero on-disk writes when off.
- src/lib/log/blob-store: content-addressed put/fetch/has/get with sha256
dedup; never throws (Result type per typed-service-contract pattern).
- src/lib/log/capture: turns (request, response, duration) into
call.requested + call.completed/failed events. Generative tools
eagerly download HTML, screenshot, theme, designSystem; read tools
log only args-derived join keys (no asset downloads).
- src/lib/log/append: validate envelope + atomic appendFile of one line.
- src/commands/tool/steps/LogExecuteToolStep: swaps in for ExecuteToolStep
when log is on; reaches into the SDK client for the raw envelope and
captures even on transport-level throws (timeouts, network errors).
Uses 600s timeout to accommodate multi-minute generations.
- src/commands/proxy/LoggingCallToolHandler: overrides the SDK proxy's
tools/call handler with a capture-wrapped variant; never propagates
capture failures to the MCP client.
- scripts/log-probe{,-2}.ts + scripts/log-validate.ts: research scripts
that locked the wire schema from real Stitch responses and validated
the on-disk structure end-to-end before implementation.
72 new tests (TSC + TDD red-green-refactor), full suite 444 pass / 0 fail.
Two follow-ups from the bug-bash on the original commit:
1. Unclassified tools were silently skipped — `kindOf()` returned null and
the capture handler bailed early, so any tool not in GENERATIVE_TOOLS or
READ_TOOLS produced zero events. New tools added to the Stitch surface
would be invisible to the log until manually registered.
Fix: kindOf() now returns 'unknown' for any tool it doesn't recognize.
The handler logs call.requested + call.completed (or call.failed) with
the raw response stored as result_blob. CompletedUnknownPayloadSchema
is added to the discriminated union.
2. Read-tool capture only recorded what was *requested*, not what was
*returned*. list_screens responses (and similar) were unrecoverable
from the log alone. Now read calls also store the response as
result_blob, plus walk the response for `name: "projects/<id>"` and
`name: "screens/<id>"` patterns to populate returned_project_ids and
returned_screen_ids — keeping the events queryable without parsing
the blob.
Test updates: kindOf('unknown_tool') now expects 'unknown'; CompletedRead
requires result_blob; new CompletedUnknown schema test; the old
CAPTURE_UNKNOWN_TOOL test is rewritten to assert the new logged behavior.
list_projects test bumps put count 1→2 to account for the new result blob
and asserts returned_project_ids extraction.
Verified end-to-end against live Stitch: the unknown-tool path captured
list_design_systems (which had been completely missing from the log
before this change), and create_project's response correctly extracted
returned_project_ids: 1.
Walks through a realistic project lifecycle with STITCH_MCP_LOG=1 and inspects .stitch-mcp/log/ to confirm event capture across all paths: create_project → generate_screen_from_text → edit_screens → generate_variants → list_screens → list_design_systems (the unclassified tool that exercises the kind:'unknown' path). Inspection reports event counts, type breakdown, trace pairing, asset capture per generative call, and any failures. Useful both for one-off verification before merge and as a reproduction harness if event-logging behavior regresses later. Requires `jq`. Each run takes ~10–15 min due to live generative calls.
The capture handler suite loads real Stitch responses from .stitch-mcp/log-probe/ via probe(name). That directory is gitignored, so the 9 fixture-dependent tests fail in CI with ENOENT for everyone except the original author. Gate the 5 probe-dependent describe blocks (generative + read + failures + robustness) behind existsSync(PROBE) — describe.skip when missing, normal describe when present. The inline 'unknown tools' test is split into its own describe so it always runs in CI. Local: 31/31 pass (probe fixtures present). CI: 22 pass, 9 skip (probe fixtures absent). Follow-up: commit a scrubbed subset of probe artifacts to tests/fixtures/ so CI can run the full capture suite. Tracked separately.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds an opt-in capture system that records every Stitch MCP tool call made through
stitch-mcp toolor the proxy. Off by default, zero overhead and zero on-disk writes when off; enable withSTITCH_MCP_LOG=1.Output goes to
.stitch-mcp/log/:events.jsonl— one JSON envelope per event (call.requested+call.completed|call.failed)blobs/<2-char-shard>/<sha256>.<ext>— content-addressed payloads (args, raw responses, structured content, HTML, screenshots, theme, design system)Motivation
We have no record of what users actually generate through
stitch-mcp— args, prompts, returned screen IDs, asset bytes, or failures. That makes debugging "this generation came back wrong" effectively impossible after the fact, since Stitch's downloadUrls expire fast. The capture is opt-in and local-only, so it doesn't change behavior for anyone not setting the env var.What's logged
generate_screen_from_text,edit_screens,generate_variantsget_screen,list_screens,list_projects,get_project,create_projectreturned_project_ids/returned_screen_idslist_design_systems)isError: trueresponse, transport-level throws, or empty generative responsesraw_blob,error_textpreserved verbatimEager fetch matters: Stitch's
downloadUrls expire — if we only logged URLs, the bytes would be gone by the time anyone tried to replay.Architecture
src/lib/log/blob-store/— content-addressed put/fetch/has/get with sha256 dedup; never throws (Result type per the typed-service-contract pattern)src/lib/log/capture/— turns(request, response, duration)into eventssrc/lib/log/append.ts— validate envelope + atomicappendFileof one linesrc/commands/tool/steps/LogExecuteToolStep.ts— swaps in forExecuteToolStepwhen log is on; reaches into the SDK client for the raw envelope and captures even on transport-level throws (timeouts, network errors)src/commands/proxy/LoggingCallToolHandler.ts— overrides the SDK proxy'stools/callhandler with a capture-wrapped variant; never propagates capture failures to the MCP clientBug bash
Branch was bug-bashed end-to-end against live Stitch with
scripts/bug-bash-logging.sh. Two findings led to follow-up commits on this branch:kindOf()returnednulland capture bailed early, solist_design_systemsproduced zero events. Fixed in0f25bdc:kindOf()now returns'unknown'and the handler logs args + raw response.list_screenswas logged with justprojectId— no record of which screens came back. Fixed in0f25bdc: read calls now store the full response asresult_blobplus walk the response forname: "projects/<id>"/"screens/<id>"patterns to populatereturned_project_ids/returned_screen_ids.Final bug-bash run: 12 events / 30 blobs / 6 paired traces / 0 orphans across
create_project→generate_screen_from_text→edit_screens→generate_variants(3 children captured with parent linkage) →list_screens→list_design_systems.Side finding (unrelated to this PR)
list_screensreturns empty{}from Stitch even after the project has multiple screens. Surfaced by the bug-bash; logged here for follow-up but not addressed in this PR.Test plan
bun test— 559 pass, 6 skip, 0 fail (74 new tests for log/blob-store/capture/append + handlers)npx tsc --noEmit— zero new errorsbun run build— cleanSTITCH_MCP_LOG=1— all six tool calls captured with expected fields populated, asset URLs fetched before expiry, failure path captured raw response + error textper generative call. 100 generative calls ≈ 10 MB. Sha256 dedup means repeat queries are nearly free. No automatic
rotation/TTL — recommend follow-up if long sessions become a concern.