Skip to content

Show the command that ran, its exit code, bounded output and honest source labels (CF-5, first part) - #287

Merged
BunsDev merged 3 commits into
mainfrom
cf5-evidence
Oct 18, 2026
Merged

BunsDev merged 3 commits into
mainfrom
cf5-evidence

Conversation

@BunsDev

@BunsDev BunsDev commented Oct 18, 2026

Copy link
Copy Markdown
Contributor

Part of #278 (CF-5): evidence projection and honest labels. It builds on #282 (CF-1) and #284 (CF-4). It adds no artifact store, executor or new open action, so #278 stays open for the parts listed at the end.

Native evidence, as observed on 2026-10-18

  • Claude Code 2.1.302 (claude -p, Haiku 5.5, a live WebSearch turn). The tool returns plain text with a Links: [...] list, and the reply's text block has no citations field. Every source is a link the model wrote.
  • codex-cli 0.163.0 (generated protocol). agentMessage.memoryCitation is the only structured citation. Web-search results are opaque.

What changes

  • shared/evidence.ts:
    • memoryEvidence turns Codex memory citations into native evidence. Malformed entries are dropped: no path, a negative or fractional line, or an end before the start.
    • boundOutput keeps the last 256 KiB of output on a UTF-8 boundary and reports the total.
  • Contract:
    • message.done.evidence.
    • tool.start.tool.exec: the command exactly as reported, and its cwd when reported.
    • Tool progress exitCode: a number, null for "the CLI said there is none", absent for "not reported".
    • Tool progress outputBytes and outputTruncated.
  • Codex: keeps the real command and cwd, the exit code (null for a declined command) and memory citations. Output is bounded as it streams: in the test, a 3 MiB stream went from 584 ms to 76 ms, and no update exceeds the preview.
  • Claude: a Bash call keeps its command. No cwd is claimed, because Bash keeps its own directory between calls, and no exit code is reported or invented.
  • Runner: bounds every tool's output before it is saved or sent, keeping the outcome and the total.
  • UI:
    • An expanded tool shows the command, cwd and "Exit code 2" / "No exit code" / "Exit code not reported".
    • Output past the preview says "Showing the last 256 KB of 3.0 MB."
    • Patch tools say they show the patch as the turn applied it, not the working tree now.
    • Codex memory citations show as "From Codex memory" (display only).
    • A "Sources:" list the model wrote is now "Links in the reply", with a note that Modex has not checked them.
  • Docs: new docs/conversation-evidence.md, the contract doc and status.

Evidence

Base fb76aac, head 64d7fc0 (three signed commits).

  • Red first:
    • test/evidence.test.ts: 4 of 6 failed on assertions once the pure module existed; nothing produced evidence, exec, exit codes or bounded output.
    • e2e/conversation-evidence.spec.ts: all 4 failed on the old UI. The log is kept.
  • pnpm test at 64d7fc0: the first stage ran 1,534 tests, with 1,518 passed, 0 failed and 16 skipped (skips that already exist). The second stage passed 21 of 21.
  • Focused e2e at 64d7fc0, after pnpm build and the Store client build: 37 of 37 passed, covering conversation-evidence, sources (one label assertion updated), typed-turns, app, agents, approvals, event-journal and store-demo. The full e2e suite runs in the required CI job.

Not covered here (#278 stays open)

  • Provider annotation ranges with Unicode offsets: neither CLI supplies offset annotations today.
  • Open or copy actions for memory-citation files, through a validated main handler.
  • Structured web-search results.
  • Comparing a turn patch with the current working tree beyond the label.
  • A retained full log beyond the 256 KiB preview.
  • New owned-artifact actions (traversal, symlink and cross-project tests belong with them).

…d output

shared/evidence.ts: Codex memory citations become native evidence on
the reply (malformed entries are dropped), and output is bounded to its
last 256 KiB on a UTF-8 boundary with the total kept. Through the
contract, a tool carries the command exactly as reported and its cwd
when reported (Codex; Claude's Bash keeps its own directory, so none is
claimed), and its exit code: a number, null when the CLI said there is
none, absent when it reported nothing (Claude). Codex bounds output as
it streams, so no update carries a whole log (a 3 MiB stream went from
584 ms to 76 ms in the test), and the runner bounds every tool's output
before it is saved or sent.

Refs #278

Coven-Harness: Claude Code / claude-opus-5-5
An expanded tool shows the command that ran, where, and "Exit code 2",
"No exit code" or "Exit code not reported". Output past the preview
says "Showing the last 256 KB of 3.0 MB." A patch says it is the patch
as this turn applied it, not the working tree now. Codex memory
citations show as "From Codex memory" with file, lines and note
(display only). A "Sources:" list the model wrote is now labelled
"Links in the reply", with a note that Modex has not checked them.

e2e/conversation-evidence.spec.ts failed on all 4 tests before this
and passes now, with sources.spec.ts.

Refs #278

Coven-Harness: Claude Code / claude-opus-5-5
docs/conversation-evidence.md: what each CLI provides natively (live
Claude WebSearch turn: no citations, only links in the text; Codex:
memory citations), what the transcript shows and what it does not
claim, and what is not done yet.

Refs #278

Coven-Harness: Claude Code / claude-opus-5-5
@vercel

vercel Bot commented Oct 18, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
modex Ready Ready Preview Oct 18, 2026 9:46am UTC

Request Review

@github-code-quality

Copy link
Copy Markdown

Code coverage overview

Languages: TypeScript

TypeScript / code-coverage-agent

main fb76aac cf5-evidence 64d7fc0 +/-
62% 62% 0%
View coverage of the 8 most impacted files.
File main fb76aac cf5-evidence 64d7fc0 +/-
apps/desktop/sr...ation-events.ts 99% 98% -1%
apps/desktop/sr...ckends/codex.ts 96% 96% 0%
apps/desktop/sr...ngine/runner.ts 99% 99% 0%
apps/desktop/sr...kends/claude.ts 97% 97% 0%
apps/desktop/sr...sktop-client.ts 99% 99% 0%
apps/desktop/sr...videnceCard.tsx 0% 0% 0%
apps/desktop/sr...shared/types.ts 100% 100% 0%
apps/desktop/sr...red/evidence.ts 0% 100% +100%

ℹ️ Calculations represent line coverage. Results may change as calculations are updated.

@BunsDev
BunsDev merged commit d3711c6 into main Oct 18, 2026
13 checks passed
@BunsDev
BunsDev deleted the cf5-evidence branch October 18, 2026 10:10
BunsDev added a commit that referenced this pull request Oct 18, 2026
Coven-Harness: Claude Code / claude-opus-5-5
BunsDev added a commit that referenced this pull request Oct 18, 2026
Coven-Harness: Claude Code / claude-opus-5-5
BunsDev added a commit that referenced this pull request Oct 18, 2026
The merge is clean. CF-5 adds the Bash command to Claude's tool.start
and bounds tool output in the runner, beside CF-2's stop, drain and
withdrawal code.

Refs #275

Coven-Harness: Claude Code / claude-opus-5-5

This branch was successfully deployed

1 active deployment
Preview — 64d7fc0f Deployed Oct 18, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant