SDK/shared/ingest: tool results with isError: true must count as errors (status error, category tool_error)

[marselsel]

Spec review request — SDK/shared/ingest: tool results with isError: true must count as errors (status: error, new error_category: tool_error).

Why now: on the Commerce for Agents project 55 tool_call events carried output_content.isError = true while the dashboard showed a 0 % error rate and get_errors was empty — every metric derived from status is blind to the errors the model actually sees. Note the release order in the spec (shared + ingest before the SDK), and the second finding that thrown error messages are stored without PII stripping.

---

Tool-Result Errors Count as Errors

Status: Proposed (2026-08-26) · Packages: sdk, shared, ingest, docs · Version: @yavio/sdk 0.4.0 (behavioural change in metrics)

Problem

A tool that answers the model with { isError: true, content: [{ type: "text", text: "This offer reference has expired…" }] } reports an error — that is exactly what the MCP CallToolResult.isError flag is for (tool execution errors are reported inside the result, not as protocol errors). The proxy only knows one kind of error: a thrown exception.

src/server/proxy.ts (wrapToolCallback):

Path	Today
handler returns (any result)	status: "success", isError survives only inside output_content
handler throws	status: "error", error_category: "unknown", error_message from the exception

Consequences, measured on project Commerce for Agents (2026-08-24 → 26): 55 tool_call events with output_content.isError = true, all status = success; dashboard error rate 0 %, get_errors empty. Every dashboard number derived from status (lib/queries/overview.ts, tool-detail.ts, errors.ts, MCP get_errors) is blind to the errors the model actually sees.

Decision

A result with isError === true is an error. It gets its own category so "the handler told the model something went wrong" (stale reference, invalid postal code, missing field) can be told apart from "the handler crashed".

Field	Result-level error (isError === true)	Thrown (unchanged)
status	error	error
error_category	tool_error (new)	unknown
error_message	first content[] item of type text, trimmed, PII-stripped, clamped to 500 chars; absent if no text	exception message, PII-stripped, clamped to 500 chars
latency_ms, is_retry	as today	as today
input_keys / input_types / input_values / client meta	captured as on the success path (same captureInput gate)	as today (captureInputOnError)
output_content	captured (the model saw it; the text explains the error)	—
intent_signals	captured	captured
widget token injection	still performed — the result is returned to the client unchanged	—

Only the literal boolean true counts. "true", 1 or a missing flag are not errors (MCP types isError as boolean).

error_message is stripped through stripPii() on both paths — today the thrown message is stored raw (events.ts buildToolCallEvent passes data.errorMessage through untouched). A thrown Error("customer max@example.com not found") currently lands verbatim.

Not configurable. A flag to keep the old behaviour would only preserve a wrong number.

Changes

Package	Change
shared	packages/shared/src/events.ts: error_category enum + "tool_error".
ingest	No code change; deploy after the shared bump and before the SDK release — the validator (schema-validator.ts, IngestEvent.safeParse) rejects unknown categories per event (YAVIO-2100), so an SDK shipped first would lose every tool-error event.
sdk	proxy.ts: status/error_category/error_message from the result on the success path (helper toolResultError(result)); events.ts: errorCategory union + "tool_error", error_message PII-stripped and clamped; README "Captured data" section.
dashboard	No query change (all use status). tool-detail.ts and errors.ts render error_category raw, so tool_error appears without a label map change. Consider a tooltip: tool_error = the handler returned isError: true.
docs / specs	.specs/metrics/events.md tool_call table: error_category values + tool_error; error_message note "also from isError results". Docs site: SDK captured-data page.

Compatibility

• Error rates rise for every customer on upgrade — this is the correction of an under-count, not a regression. Release note must say so ("error rate now includes tool-reported errors").
• Alerts or thresholds customers built on status will fire more. Minor version bump (0.4.0), changelog entry.
• Old SDKs keep sending success; nothing on the server side breaks.

Tests

src/__tests__/proxy.test.ts

1. Result { isError: true, content: [{type:"text", text:"boom"}] } → status: error, error_category: tool_error, error_message: "boom", inputs and output captured, widget token still injected (non-serverOnly).
2. Result with isError: true and no text content → error without error_message.
3. isError: false, isError: "true", no flag → success.
4. Message clamp at 500 chars; max@example.com in the text → [EMAIL_REDACTED] (both paths).
5. Thrown error message with an email → stripped (regression for the raw pass-through).

packages/shared schema test: tool_error accepted. Ingest integration: a batch with error_category: tool_error is accepted (207 without rejections).

Open Questions

• Should tool_error results count toward is_retry differently? Today a repeat call of the same tool is a retry regardless of status — keep.
• Category refinement (e.g. mapping -32602 validation text to validation) — not now; the SDK cannot see JSON-RPC errors raised by the MCP SDK before the handler.

---

Spec file: .specs/sdk/tool-result-errors.md (drafted 2026-08-26 from findings on the Commerce for Agents project; not yet committed). Please review the decisions, then implement per the spec or comment here.

[marselsel]

Spec review (second Claude session, 2026-08-26) — verdict: agree, ship it. Three additions below.

Verified against code and live data (count-only queries):

• proxy.ts:186 sets status: "success" for any returned result; only a throw becomes error (:240-242, category unknown, message raw). events.ts:133 confirms error_message is the one free-text field that bypasses stripPii — input_keys/input_values/output_content/intent_signals all go through it.
• Live: Commerce for Agents 497 tool_calls, 0 status = error, 59 output_content.isError = true (spec said 55; grown since). Preisvergleich Prod: 68 calls, 0 status errors, 14 isError — a 21 % error rate the dashboard shows as 0 %. Shipal Prod: 4542 calls with outputValues: false, so today the flag is unknowable there — SDK/shared/ingest: tool results with isError: true must count as errors (status error, category tool_error) #75 fixes that too, since status is derived regardless of output capture. Worth stating in the spec as a benefit.
• ClickHouse error_category is LowCardinality(Nullable(String)) (0002_events_table.sql:28) — no migration. The ingest Zod enum (packages/shared/src/events.ts:77) is the only gate; the shared → ingest → SDK order in the spec is right.
• PostHog parity holds: mcp-analytics/src/modules/tracing.ts isToolResultError() is result.isError === true (literal true, same rule) → $mcp_is_error + a sibling $exception.

Add to the spec:

1. A blind spot SDK/shared/ingest: tool results with isError: true must count as errors (status error, category tool_error) #75 does not close — say so. McpServer runs validateToolInput() before executeToolHandler() (mcp.js:126-127 in the installed SDK) and its catch converts every failure into an isError result via createToolError() (mcp.js:135-142). So when the model sends bad arguments, the model sees an error but our proxy (which wraps the callback) never runs — no tool_call event exists at all, before or after SDK/shared/ingest: tool results with isError: true must count as errors (status error, category tool_error) #75. PostHog sees these because it wraps the low-level tools/call handler. Yavio already has exactly that wrapper in intent.ts (wrapCallHandler), but it is only installed when intent capture is enabled. Suggest a follow-up issue: move status detection to the protocol layer so validation errors, disabled and not-found tools are counted. The Open Questions wording ("cannot see JSON-RPC errors raised before the handler") is slightly off — they are not JSON-RPC errors, they are isError results produced above the callback.
2. is_retry is dead — "as today" means "always 0". The SDK never sets isRetry (events.ts:134 is the only consumer, proxy.ts never passes it), ingest does not derive it, and capture.retries is read by nothing — yet .specs/metrics/events.md:22 and 07-concepts/01-events.mdx:40 define a semantic. Not SDK/shared/ingest: tool results with isError: true must count as errors (status error, category tool_error) #75's job to fix, but the table row should not imply it works.
3. Thrown handler errors are also delivered to the client as isError results (createToolError), so unknown vs tool_error is a server-side distinction only. Put that in the tooltip/docs: "tool_error = handler returned isError: true; unknown = handler threw (client saw the same shape)."

Minor: the 500-char clamp for error_message sits below the ingest field limit (2048) — fine; reuse the MAX_INTENT_LENGTH precedent as a named constant rather than a second literal. And note in the release notes that every consumer pins ^0.3.0/^0.2.0, so on 0.x the 0.4.0 error-rate change is opt-in per app — nobody's dashboard jumps without an explicit bump.

[marselsel]

Author response (2026-08-26): accepted, spec updated (.specs/sdk/tool-result-errors.md).

• New section Known blind spot: pre-handler failures (validateToolInput() → createToolError()) produce no tool_call event at all; follow-up opened as SDK: count tool calls that fail before the handler (argument validation, disabled/unknown tool) — status detection at the protocol layer #77 (status detection at the protocol layer, reusing the wrapCallHandler idea). Open Questions wording fixed (they are isError results, not JSON-RPC errors).
• is_retry row now states it is never set today and must not be described as working.
• Tooltip/docs wording added: tool_error = handler returned isError: true; unknown = handler threw (client sees the same shape via createToolError).
• MAX_ERROR_MESSAGE_LENGTH = 500 as a named constant next to MAX_INTENT_LENGTH; note on the ^0.3.0 pins (0.4.0 is opt-in per app); live numbers updated (59 / 14 of 68 / Shipal now measurable); cross-spec rule with SDK: per-tool capture overrides (inputValues/outputValues/intent) + warn-once for tracking calls outside a wrapped tool #74 on outputValues: false.