Skip to content

feat: opt-in event logging for Stitch tool call - #182

Merged
davideast merged 4 commits into
mainfrom
feat/event-logging
May 4, 2026
Merged

feat: opt-in event logging for Stitch tool call#182
davideast merged 4 commits into
mainfrom
feat/event-logging

Conversation

@davideast

@davideast davideast commented May 4, 2026

Copy link
Copy Markdown
Owner

Adds an opt-in capture system that records every Stitch MCP tool call made through stitch-mcp tool or the proxy. Off by default, zero overhead and zero on-disk writes when off; enable with STITCH_MCP_LOG=1.

Output goes to .stitch-mcp/log/:

  • events.jsonl — one JSON envelope per event (call.requested + call.completed | call.failed)
  • blobs/<2-char-shard>/<sha256>.<ext> — content-addressed payloads (args, raw responses, structured content, HTML, screenshots, theme, design system)

Motivation

We have no record of what users actually generate through stitch-mcp — args, prompts, returned screen IDs, asset bytes, or failures. That makes debugging "this generation came back wrong" effectively impossible after the fact, since Stitch's downloadUrls expire fast. The capture is opt-in and local-only, so it doesn't change behavior for anyone not setting the env var.

What's logged

Tool kind Examples Captured
generative generate_screen_from_text, edit_screens, generate_variants args, structured content, eagerly fetched HTML + screenshot bytes, theme, design system, parent/sibling screen relationships
read get_screen, list_screens, list_projects, get_project, create_project args, full result blob, extracted returned_project_ids / returned_screen_ids
unknown anything else (e.g. list_design_systems) args, full result blob — so a new tool added upstream is never silently invisible
failure any isError: true response, transport-level throws, or empty generative responses full raw response stored as raw_blob, error_text preserved verbatim

Eager fetch matters: Stitch's downloadUrls expire — if we only logged URLs, the bytes would be gone by the time anyone tried to replay.

Architecture

  • src/lib/log/blob-store/ — content-addressed put/fetch/has/get with sha256 dedup; never throws (Result type per the typed-service-contract pattern)
  • src/lib/log/capture/ — turns (request, response, duration) into events
  • src/lib/log/append.ts — validate envelope + atomic appendFile of one line
  • src/commands/tool/steps/LogExecuteToolStep.ts — swaps in for ExecuteToolStep when log is on; reaches into the SDK client for the raw envelope and captures even on transport-level throws (timeouts, network errors)
  • src/commands/proxy/LoggingCallToolHandler.ts — overrides the SDK proxy's tools/call handler with a capture-wrapped variant; never propagates capture failures to the MCP client

Bug bash

Branch was bug-bashed end-to-end against live Stitch with scripts/bug-bash-logging.sh. Two findings led to follow-up commits on this branch:

  1. Unclassified tools were silently skipped. kindOf() returned null and capture bailed early, so list_design_systems produced zero events. Fixed in 0f25bdc: kindOf() now returns 'unknown' and the handler logs args + raw response.
  2. Read-tool capture only recorded what was requested, not what was returned. list_screens was logged with just projectId — no record of which screens came back. Fixed in 0f25bdc: read calls now store the full response as result_blob plus walk the response for name: "projects/<id>" / "screens/<id>" patterns to populate returned_project_ids / returned_screen_ids.

Final bug-bash run: 12 events / 30 blobs / 6 paired traces / 0 orphans across create_projectgenerate_screen_from_textedit_screensgenerate_variants (3 children captured with parent linkage) → list_screenslist_design_systems.

Side finding (unrelated to this PR)

  • list_screens returns empty {} from Stitch even after the project has multiple screens. Surfaced by the bug-bash; logged here for follow-up but not addressed in this PR.

Test plan

  • bun test — 559 pass, 6 skip, 0 fail (74 new tests for log/blob-store/capture/append + handlers)
  • npx tsc --noEmit — zero new errors
  • bun run build — clean
  • Live bug-bash with STITCH_MCP_LOG=1 — all six tool calls captured with expected fields populated, asset URLs fetched before expiry, failure path captured raw response + error text
  • Disk growth measured: ~600 bytes per identical read call (full sha256 dedup), ~5 KB per distinct read call, ~100 KB
    per generative call. 100 generative calls ≈ 10 MB. Sha256 dedup means repeat queries are nearly free. No automatic
    rotation/TTL — recommend follow-up if long sessions become a concern.

davideast added 4 commits May 3, 2026 16:57
…lobs

Adds an opt-in event logging system that captures every Stitch MCP call
made through the proxy or stitch-mcp tool, eagerly snapshotting HTML and
screenshot bytes before their downloadUrls expire.

Enable with STITCH_MCP_LOG=1. Output goes to .stitch-mcp/log/events.jsonl
and .stitch-mcp/log/blobs/<2>/<sha>.<ext>. Disabled by default; zero
overhead and zero on-disk writes when off.

- src/lib/log/blob-store: content-addressed put/fetch/has/get with sha256
  dedup; never throws (Result type per typed-service-contract pattern).
- src/lib/log/capture: turns (request, response, duration) into
  call.requested + call.completed/failed events. Generative tools
  eagerly download HTML, screenshot, theme, designSystem; read tools
  log only args-derived join keys (no asset downloads).
- src/lib/log/append: validate envelope + atomic appendFile of one line.
- src/commands/tool/steps/LogExecuteToolStep: swaps in for ExecuteToolStep
  when log is on; reaches into the SDK client for the raw envelope and
  captures even on transport-level throws (timeouts, network errors).
  Uses 600s timeout to accommodate multi-minute generations.
- src/commands/proxy/LoggingCallToolHandler: overrides the SDK proxy's
  tools/call handler with a capture-wrapped variant; never propagates
  capture failures to the MCP client.
- scripts/log-probe{,-2}.ts + scripts/log-validate.ts: research scripts
  that locked the wire schema from real Stitch responses and validated
  the on-disk structure end-to-end before implementation.

72 new tests (TSC + TDD red-green-refactor), full suite 444 pass / 0 fail.
Two follow-ups from the bug-bash on the original commit:

1. Unclassified tools were silently skipped — `kindOf()` returned null and
   the capture handler bailed early, so any tool not in GENERATIVE_TOOLS or
   READ_TOOLS produced zero events. New tools added to the Stitch surface
   would be invisible to the log until manually registered.

   Fix: kindOf() now returns 'unknown' for any tool it doesn't recognize.
   The handler logs call.requested + call.completed (or call.failed) with
   the raw response stored as result_blob. CompletedUnknownPayloadSchema
   is added to the discriminated union.

2. Read-tool capture only recorded what was *requested*, not what was
   *returned*. list_screens responses (and similar) were unrecoverable
   from the log alone. Now read calls also store the response as
   result_blob, plus walk the response for `name: "projects/<id>"` and
   `name: "screens/<id>"` patterns to populate returned_project_ids and
   returned_screen_ids — keeping the events queryable without parsing
   the blob.

Test updates: kindOf('unknown_tool') now expects 'unknown'; CompletedRead
requires result_blob; new CompletedUnknown schema test; the old
CAPTURE_UNKNOWN_TOOL test is rewritten to assert the new logged behavior.
list_projects test bumps put count 1→2 to account for the new result blob
and asserts returned_project_ids extraction.

Verified end-to-end against live Stitch: the unknown-tool path captured
list_design_systems (which had been completely missing from the log
before this change), and create_project's response correctly extracted
returned_project_ids: 1.
Walks through a realistic project lifecycle with STITCH_MCP_LOG=1 and
inspects .stitch-mcp/log/ to confirm event capture across all paths:
create_project → generate_screen_from_text → edit_screens →
generate_variants → list_screens → list_design_systems (the unclassified
tool that exercises the kind:'unknown' path).

Inspection reports event counts, type breakdown, trace pairing, asset
capture per generative call, and any failures. Useful both for one-off
verification before merge and as a reproduction harness if event-logging
behavior regresses later.

Requires `jq`. Each run takes ~10–15 min due to live generative calls.
The capture handler suite loads real Stitch responses from
.stitch-mcp/log-probe/ via probe(name). That directory is gitignored, so
the 9 fixture-dependent tests fail in CI with ENOENT for everyone except
the original author.

Gate the 5 probe-dependent describe blocks (generative + read + failures
+ robustness) behind existsSync(PROBE) — describe.skip when missing,
normal describe when present. The inline 'unknown tools' test is split
into its own describe so it always runs in CI.

Local: 31/31 pass (probe fixtures present).
CI:    22 pass, 9 skip (probe fixtures absent).

Follow-up: commit a scrubbed subset of probe artifacts to tests/fixtures/
so CI can run the full capture suite. Tracked separately.
@davideast
davideast merged commit db1adf1 into main May 4, 2026
1 check passed
@davideast
davideast deleted the feat/event-logging branch May 4, 2026 03:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant