feat: add synthetic context-scoring shadow experiment

[snowopsdev]

What changed

This adds an offline experiment that shows which pieces of made-up context would be kept or dropped. The answers are scripted, and the original context stays untouched. It doesn’t measure real accuracy or savings.

• Compares asking about all pieces in one request with asking about each piece separately.
• Gives each piece its own relevance question and clear criteria, with instructions to treat the sample text as data.
• Keeps uncertain pieces. Missing or invalid answers, cancelled requests, or responses from the wrong model mean everything is retained.
• Keeps the expected results separate from the questions being scored.
• Reports savings only when dated prices and complete token counts are available. Otherwise, savings stay unknown. Malformed cost inputs make the cost unknown instead of failing the run.
• Uses the repository’s pinned jev-1.13.0 constant and tracks the experiment’s questions under their own version.

Run pnpm experiment:context-shadow for JSON, or add --format markdown for a readable report. Usage errors exit with status 2.

Closes #42
Refs #3
Follows PR #30.

Second review round (a1a654b)

• Imports the model pin and plain-data helpers from src/contract instead of copying them, so a future pin change can’t miss this example.
• Hardens the input snapshot: the task is read once, chunks are read by index, and cost inputs are copied as JSON data. Stops sending a request after a same-tick cancellation. Cost is unknown when the two layouts report different baselines.
• Adds 19 tests, for example for recall below 100%, adapter errors, missing usage, nested cost snapshots, answer validation and cancellation of the remaining requests. The exact question text is now pinned in tests.
• Clarifies the cost model: baseline token counts are observed, and counterfactual counts are modeled until a paired live run. It also notes that this kit’s combined request does not pack chunks into shared state, so it does not test the state-size concern. That layout remains for the live study.
• The standalone overview.html now embeds overview.svg instead of copying it.

sample-result.json, the questions, criteria, untrusted-data note, thresholds, versions and request bodies are unchanged by this round.

Experiment flow

flowchart TD
    A["Synthetic task and context chunks"] --> B["Build both Noul request layouts"]
    A --> H["Original context stays intact with caller"]
    B --> C["Scripted fake adapter — no provider call"]
    C --> D{"Complete usable evidence?"}
    D -->|Yes| E["Proposed keep/drop sets; uncertainty retained"]
    D -->|No| F["Unavailable: retain all; propose no drops"]
    E --> G["JSON or Markdown report"]
    F --> G
    L["Evaluation labels and cost observations"] -.->|Report only| G

Loading

Synthetic example

The probabilities shown are scripted values recorded in the sample result. See the experiment guide for the request, evidence, and cache-accounting boundaries.

Verdict impact

Verdict impact: None. This doesn’t change how proposed actions are reviewed or who decides whether they can run. The proposal-review questions, decision table and AI model stay the same.

This only adds an offline experiment using made-up examples. It doesn’t change the app, contact an AI service, or remove anything from an agent’s context. Testing with live data and deciding whether to use this in a working agent are still future steps.

Verification

• pnpm typecheck: passes locally on a1a654b.
• pnpm test: 339 of 339 pass locally on a1a654b. The test calls the CLI in process and compares its exact JSON with sample-result.json, so an outdated sample fails pnpm test.
• Commits are signed: GitHub marks 22a523b, 1873005 and a1a654b Verified. The merge from main (5c24184) was committed and signed by GitHub.
• No provider call, credential, execution path, or new verdict added.
• Decision table and questions: the proposal-review decision table and question set v4 are unchanged. The example-local relevance question set moved to context-relevance-v2 in 1873005 because the combined request’s state changed; the wording didn’t change. The example guide records why.

Also checked locally: pnpm build, the secret scan and whitespace checks pass. The SVG parses, and the visual was inspected in a browser; no automated test checks the diagrams. Tests used fake responses only, and no live AI calls or Codex comparison ran.

Pending: the checks workflow for a1a654b needs maintainer approval, so CI (including the full-history secret scan) hasn’t run on any commit after 22a523b. The Vercel preview needs approval from the deployment team.

[vercel]

@snowopsdev is attempting to deploy a commit to the 0xBuns Team on Vercel.

A member of the Team first needs to authorize it.

[BunsDev]

PR #43 — feat: add synthetic context-scoring shadow experiment

Intent

This PR adds a fully offline, scripted demonstration of how a context-relevance scorer would decide which context chunks to keep or drop before sending them to a proposer. No live AI calls are made. The caller's original context is never modified. Results are explicitly labeled "synthetic demonstration." The work closes issue #42, advances roadmap phase 4, and follows the pattern established by PR #30.

The two request layouts compared are:

• fan_out — one Noul request containing all chunks, with one question per chunk inside it
• per_chunk — one Noul request per chunk, each with a single relevance question

---

Observed evidence

What is solid

• No live transport is possible. The ScoringAdapter type locks kind to "scripted_fake", and runLayout hard-rejects any adapter that claims a different kind (unsupported_provenance). There is literally no code path to a real provider.
• Evaluation labels are rigorously excluded from adapter inputs. Both code and tests confirm the relevant boolean never appears in any serialized Noul body. The test "both layouts preserve chunk identity and send only Noul request fields" explicitly asserts wire.includes('"relevant":') === false.
• Caller context immutability is tested. snapshotInput defensively copies all fields before the first await. A test mutates the input after the experiment starts and verifies the artifact records the original values.
• Failure modes are thorough. Cancellation (pre-aborted, mid-flight, micro-task-queued post-response), malformed answers, out-of-range probabilities, wrong model, unsupported provenance, duplicate chunk IDs, and missing cost evidence are all handled and individually tested.
• Cost arithmetic is conservative. Aggregate cached-token counts explicitly cannot produce a cost estimate; all three segment observations (prefix, context, suffix) plus per-layout baseline and counterfactual bindings are required. Unknown inputs yield status: "unknown", not zero.
• Markdown injection is prevented. markdownInline strips newlines and escapes structural characters; the test verifies \n## forged heading doesn't become a real heading.
• Commit is signed Verified (noted in PR verification section).

---

Concrete bugs and risks

1. Label mismatch in report.ts — minor but confusing
The Markdown renderer calls:

layoutMarkdown("All chunks, one Noul per chunk", "fan_out", ...)

The fan_out layout sends ONE request with all questions inside it. The label "one Noul per chunk" implies separate requests. The per_chunk label is "One request per chunk." A reader comparing the two labels will likely mis-parse which layout is which. Consider "All chunks, one request" vs. "One request per chunk."

2. fan_out layout body includes chunk texts twice — possibly unintentional
In buildCalls for fan_out, state.chunks is populated with all chunk {id, text} pairs AND each question's instructions.context_chunk also contains {id, text} for that chunk. Every chunk's text appears twice in the serialized body. If the intent was for the model to see all chunks in state while each question focuses on one, this is correct. If it's redundant payload, it inflates plannedRequestBytes and the token proxy. The design intent isn't documented inline.

3. validSegments indentation inconsistency — cosmetic
In experiment.ts, inside validSegments, the if statement following const record = ownDataRecord(row) is not indented to match its containing for block. TypeScript doesn't care, but it suggests the file wasn't fully auto-formatted before commit.

4. calls.indexOf(call) inside a loop — O(n²) on cancellation
When a call is cancelled or errors, the remaining calls are found via calls.slice(calls.indexOf(call) + 1). This is an O(n) search inside an O(n) loop, making cancellation propagation O(n²). For a demo with 3 chunks this is irrelevant. If this pattern is copied into a production path with large context arrays, it would need revisiting.

5. copyCostEvidence calls ownDataRecord(record.byLayout) once per layout iteration
Inside the flatMap over REQUEST_LAYOUTS, ownDataRecord(record.byLayout) is evaluated on each iteration rather than once before the loop. Harmless (2 layouts), but slightly redundant.

---

Missing context (not observable from this diff)

• CI checks are pending maintainer approval. The workflow at actions/runs/35860258915 hasn't run yet — type checking, full test suite, and secret scan are unverified on CI. Local checks passed on commit 4179407.
• Vercel preview is pending deployment team approval. No functional risk (this is an offline CLI experiment), but the preview hasn't been reviewed.
• The Noul contract and TypeSafe re-ranking pattern are referenced but not included in this diff; conformance to those contracts can't be verified here.
• No existing ContextScorer in the repo — this diff explicitly confirms that and doesn't add one. The integration decision remains gated on live measured results (roadmap items still open).

---

Recommended next steps

1. Unblock CI — approve the workflow run so type checking, tests, and the full-history secret scan execute on the actual head commit.
2. Clarify the fan_out Markdown label in report.ts — "All chunks, one request" is unambiguous alongside "One request per chunk."
3. Document or consolidate the dual chunk-text in fan_out — add a comment in buildCalls explaining why state.chunks and instructions.context_chunk both carry the text, or remove the redundancy if unintended.
4. Verify sample-result.json is regenerated after any future changes to demo.ts thresholds or question text — it's a committed snapshot and will silently drift otherwise. A CI step that runs pnpm experiment:context-shadow and diffs against the committed file would catch this.
5. Plan the live shadow phase — roadmap items "Review data egress and run a shadow experiment on synthetic or consented context" and "Make the integration decision from measured results" are still open; this PR's infrastructure is the right foundation for them.

[BunsDev]

PR #43 review — feat: add synthetic context-scoring shadow experiment (TypeSafeAI/jev-harness)

Intent

Adds a fully offline, example-local kit under examples/context-scoring-shadow/ that demonstrates how a context-relevance scorer would propose keep/drop sets before a proposer call. It compares two Noul request layouts — fan_out (one request, one question per chunk) and per_chunk (one request per chunk) — driven by a scripted fake adapter. No live provider call, no root export, no production ContextScorer, and the caller context is never mutated. Closes #42, refs #3, advances roadmap phase 4, follows PR #30. Blast radius is low: net-new files plus one package.json script and docs.

Observed evidence (verified against the diff)

• No live path exists. ScoringAdapter.kind is locked to 'scripted_fake'; runLayout rejects any other adapter (unsupported_provenance) and parseResponse rejects a response whose source is not 'scripted_fake' or whose model is not jev-1.13.0. A custom adapter claiming a 'live' source is rejected and cannot relabel the artifact (test confirms).
• Evaluation labels are excluded from the wire. The synthetic relevant label never appears in any serialized body; the test asserts the wire contains no relevant field and no cacheObservations/aggregateCachedInputTokens.
• Caller immutability. snapshotInput deep-copies task/chunks and structuredClones cacheObservations before the first await; tests mutate the input mid-run and confirm the artifact keeps the originals. ownDataRecord builds via Object.create(null), avoiding prototype-pollution through ids like 'constructor'.
• Fail-closed defaults. Cancellation (pre-abort, mid-flight, post-response microtask), malformed/out-of-range answers, wrong model, missing evidence, duplicate ids, and aggregate-only cache data all yield an 'unavailable' layout / 'unknown' cost and retain every chunk — never a spurious drop.
• Conservative cost. estimateCost requires dated price assumptions + per-segment (prefix/context/suffix) baseline and counterfactual observations bound to the exact keep/drop sets + per-request token usage; otherwise it returns 'unknown', never zero. The bundled demo (aggregate only) is correctly 'unknown'.

Concrete bugs and risks

1. fan_out duplicates each chunk's text, which skews the headline size comparison. In buildCalls the fan_out body carries every chunk in state.chunks AND repeats {id,text} inside each question's instructions.context_chunk, so every chunk text is serialized twice. Because plannedRequestBytes / plannedRequestTokenProxy are the very metrics this experiment exists to compare across layouts, the duplication inflates fan_out relative to per_chunk (which carries each text once). This is a methodological correctness issue, not just cosmetics: either the duplication is intentional (document why) or remove it so the layout size comparison is apples-to-apples. [experiment.ts buildCalls]
2. Misleading Markdown label. report.ts renders fan_out as 'All chunks, one Noul per chunk' — the phrase 'one ... per chunk' reads like the per_chunk layout. Suggest 'All chunks in one request' alongside the existing 'One request per chunk'. [report.ts renderMarkdownReport]
3. O(n^2) cancellation propagation. runLayout uses calls.slice(calls.indexOf(call)+1) inside the per-call loop (three sites): an O(n) scan inside an O(n) loop. Irrelevant for 3 chunks, but this file is explicitly framed as the template for a future production scorer, so track the loop index before it is copied into a hot path. [experiment.ts runLayout]
4. Minor cleanups. copyCostEvidence recomputes ownDataRecord(record.byLayout) once per layout inside the flatMap; validSegments has an inconsistently-indented if. Cosmetic only.
5. Verify on CI: LayoutCostObservations is imported as a type in experiment.ts but I do not see it referenced in the body — under tsc noUnusedLocals that would error. Local checks are reported as passing, so either lint permits it or I am missing a use; confirm on CI.

Methodology caveat (not a bug): results report recall against synthetic labels but not precision — a scorer that keeps everything scores recall 1.0. Fine for a demo, but any go/no-go decision must not read recall alone (the roadmap already says 'not token counts alone').

Missing context (cannot be verified from this diff)

• CI is pending maintainer approval (actions/runs/35860258915); type-check, full test suite, and full-history secret scan have NOT run on head 4179407. Vercel preview is pending deploy-team approval. Only local checks are claimed.
• The Noul contract and TypeSafe re-ranking pattern are external (docs.typesafe.ai); conformance of the wire shape cannot be checked here.
• sample-result.json is a committed snapshot and nothing in the diff regenerates or verifies it, so it can drift silently from demo.ts.
• Head is a fork (snowopsdev) and this is a non-atomic set of observed revisions, not a frozen snapshot.

Recommended next steps

1. Approve/unblock the CI workflow so tsc, tests, and the secret scan run on the actual head commit — the main open verification gap.
2. Resolve the fan_out text duplication: remove it or add an inline comment justifying it, and ensure the size metrics compare like for like.
3. Rename the fan_out Markdown label to remove the 'per chunk' ambiguity.
4. Add a CI step that runs pnpm experiment:context-shadow and diffs against sample-result.json to catch snapshot drift.
5. Carry these caveats into the live shadow phase (roadmap items still open): data-egress review, precision alongside recall, and cost derived from measured segment observations rather than byte/token proxies.

Note: an existing PR comment by BunsDev covers much of this ground; this review independently confirms those points and additionally elevates the fan_out size-comparison skew (item 1) and the recall-only methodology caveat. This is an inert analysis — no files were changed and nothing was posted, committed, or merged.

[snowopsdev]

Addressed the feedback from both reviews in 1873005de9e10074e3a9b594042cdb3f708a9c52:

• Removed the duplicate chunk text from the combined request. Shared state now holds the task and untrusted-data note; each question carries its own target chunk once. This follows the Noul structured-instructions example. The guide explains both layouts and the limits of the size comparison.
• Advanced the example's question version to context-relevance-v2 because the supplied context changed. The Jev model, relevance wording, criteria, thresholds, and proposal-review questions are unchanged. The sample result has been regenerated.
• Renamed the Markdown headings to “All chunks in one request” and “One request per chunk.”
• Added a regression check that each chunk's text appears once and that byte/token proxies match the serialized requests. Another test runs the CLI and compares its exact output with the committed sample; it runs in the existing pnpm test CI job and fails if the snapshot drifts. Markdown output is checked too.
• Replaced the cancellation index searches with the current loop index, moved the repeated cost-observation lookup outside the loop, fixed the indentation, and removed the unused type import. A new test checks cancellation midway through separate requests, including the requests that must not be sent.
• Documented that keeping everything gives perfect recall. Precision, independently labeled evaluation, retained context size, latency, measured cache costs, and an egress review remain requirements for future live work. The live-shadow and integration roadmap items stay open.

Local type checking, the full test suite, build, secret scan, and whitespace checks pass on this commit. Independent review found no remaining actionable issues. No live provider or Codex comparison ran.

GitHub marks the new commit signature Verified. The workflow for this commit needs maintainer approval before its checks and full-history secret scan can run. The Vercel preview also still needs deployment-team authorization.

Verdict impact: None.

[BunsDev]

Thanks @snowopsdev for the thoughtful contribution and for working through the review iterations. The latest version keeps this experiment offline and example-local, excludes labels from scorer inputs, retains all context on incomplete evidence, and clearly separates synthetic cost arithmetic from live measurements. The earlier payload duplication and artifact-drift concerns are addressed. Verified the updated branch with pnpm verify: boundary checks, typecheck, all 345 tests, production build, and secret checks pass. Hosted checks and secret scan also pass on this exact head. No proposal-review verdicts change; the live egress and integration gates remain open.