Add offline proposal validation (replay test) to ground confidence and filter unsupported proposals

[elecnix]

Summary

Add an offline proposal-validation step that empirically scores each proposal by replaying it against the conversation turn(s) it came from, and use that grounded score instead of (or alongside) the model's self-assigned confidence.

Today the analysis pipeline is generate → materialise → emit: session-overview synthesises proposals, each carries a model-assigned confidence, and that number is never tested. The proposal is shipped on the model's say-so. This issue proposes closing the loop with a generate → validate → keep/rank step that is feasible entirely offline using data we already store.

Motivation / evidence

Dogfooding finding (10-session corpus). The single most-wrong proposal in the run carried the highest confidence (95%) — it told the agent to "verify the git push target," when the push was always correct and the real defect was an unrelated gh pr create invocation missing --repo. Meanwhile, genuine single-shot corrections (e.g. "this repo does not use CircleCI") scored below the friction threshold and produced no proposal at all. In short: self-assigned confidence was anti-correlated with correctness.

Literature convergence. Every serious method in the automatic prompt/instruction-optimization line — APE, ProTeGi, OPRO, DSPy/MIPRO, TextGrad, and GEPA (arXiv:2507.19457) — shares the loop generate → validate against data → keep only if it measurably helps. pi-prospector is the outlier: it generates and emits with no validation. Separately, the experiential-agent and evaluation literature warns explicitly that self-rating is untrustworthy:

• Huang et al., "Large Language Models Cannot Self-Correct Reasoning Yet" (arXiv:2310.01798): self-generated judgements need external feedback.
• LLM-judge work (e.g. JudgeBench) documents verbosity/sycophancy bias and that self-verification has high recall but low precision at confirming correctness.
• ExpeL (arXiv:2308.10144) extracts insights by contrasting trajectories and filters the insight pool rather than trusting every extraction.
• Constitutional AI (arXiv:2212.08073) compiles each principle into a concrete critique template — exactly the mechanism this issue reuses.

Proposed change (clearly defined)

Introduce a new analyzer, proposal-validate, that runs after session-overview and consumes its summary nodes. For each materialised proposal it performs a replay test:

1. Resolve the proposal's originating turn(s). Follow the evidence trail: proposal ←produces– summary node –consumes→ classification node –anchors→ message. (See "Required prerequisite" below — proposals must pin the message id(s) they are about.)
2. Re-run the existing turn-pair-llm classifier on each originating turn, but with the candidate proposal text injected into the system context as a standing instruction the agent "already had." Use a different model than the one that generated the proposal (to avoid same-model self-agreement bias).
3. Compare classifications (with-rule vs. without-rule):
   • friction averted (friction_type: none / is_genuine_correction: false / sentiment improved) → supported;
   • no change or worse → unsupported.
4. Emit a validation node (new node kind) that consumes the source node and stores the comparison result and a derived validated_score ∈ [0,1].
5. Write the grounded result back onto the proposal: set validated_score, a validation_status (supported | unsupported | unvalidated), and the id of the validating validation node. Ranking and the proposals view use validated_score first, falling back to model confidence only when unvalidated.

Required prerequisite (sub-task)

Proposals are materialised from a session-level summary node that consumes many turns, so today a single proposal is not pinned to the specific turn(s) it concerns. Before validation can target a turn, session-overview must attach source_message_ids (the user-message ids of the high-signal turns that motivated each proposal) to every entry in its improvement_proposals array, and proposal-materializer must persist them (new proposals column or via anchors edges from the proposal). This is small and self-contained but is a hard dependency for replay.

Data-model changes

• New node kind validation added to the NodeKind TypeBox union in src/analyze/types.ts and to the analysis_nodes.node_kind documentation. Its content is a Type.Object({...}) schema (per AGENTS.md — no bare interfaces): { proposal_input_key, with_rule, without_rule, supported, validated_score, model_used }.
• proposals table gains validated_score REAL, validation_status TEXT NOT NULL DEFAULT 'unvalidated', validation_node_id TEXT. Clean v2 schema edit in src/db/schema.ts (no migration needed — recompute regenerates derived data).
• Proposal schema (improvement_proposals entries) gains source_message_ids: string[]; RawProposal/normalizeProposal in proposal-materializer.ts updated to carry it.
• New SQL lives only in db/queries.ts / db/analysis-queries.ts.

Identity / architecture fit

The validation node is content-addressed like every other node: its input_key derives from (analyzer = proposal-validate, version, config fingerprint incl. validator model, source set = the source node's output_key), and its output_key = H(input_key | content). This means:

• replays are idempotent and reproducible on any machine after a wipe, and
• prospect verify covers validation nodes for free,
• the result is auditable: "this proposal was kept because injecting it averted friction on message <id>," traceable by edges back to the transcript (preserves the Invariant in DESIGN.md §4 that every proposal traces to conversation evidence).

It also stays within the non-goal of no automatic editing (DESIGN.md §5): validation produces a grounded confidence, never an edit. The validator runs as the deterministic→LLM layering already in place (turn-pair-llm is reused).

CLI / UX

• New action/command prospect validate (and --prospect validate headless) that runs the proposal-validate analyzer over open proposals.
• prospect proposals ranks by validated_score (desc, nulls last) and labels each line replay-validated / model-rated so the human can tell the two apart.
• prospect proposals --full shows the with-rule vs. without-rule classification delta as part of the evidence block.

Caveats (call out in DESIGN.md)

The validator inherits the classifier's own blind spots — most notably that the classifier currently sees only message text, not tool calls/outputs (tracked separately). So:

• use a different model for validation than for generation;
• label the score "replay-validated", not "ground truth";
• keep it advisory and human-in-the-loop.

Even with these limits, an empirically-grounded score is strictly better than the self-rating that demonstrably failed on the corpus.

Acceptance criteria

• NodeKind union and schema include validation; prospect verify validates these nodes.
• session-overview attaches source_message_ids to each proposal; materializer persists them.
• proposal-validate analyzer: resolves originating turn(s), re-runs turn-pair-llm with/without the candidate rule via a configurable (distinct) validator model, emits a validation node, and writes validated_score / validation_status / validation_node_id onto the proposal.
• proposals ranking and --full view use and display the validated score and a replay-validated vs model-rated label.
• Validation nodes are content-addressed (input_key/output_key) and reproducible across a DB wipe.
• Tests (per AGENTS.md): unit tests for the with/without-rule comparison and score derivation; a component test (real SQLite + fixture JSONL + mocked LLM) proving a deliberately-wrong proposal is marked unsupported and a well-supported one supported, end to end.
• DESIGN.md: glossary entries for validation node kind and replay-validated confidence, plus the caveat above; ubiquitous language stays "proposal."

Out of scope

• Feeding tool-call arguments/outputs into the classifier (tracked separately; would improve both generation and validation but is independent).
• Cross-session proposal consolidation / dedup, contradiction detection, and rule retirement (separate "memory lifecycle" work).
• Any automatic editing of steering artifacts.

References

Citations below were independently verified; the broader prompt-optimization line (APE, ProTeGi, OPRO, DSPy/MIPRO, TextGrad) is named without per-paper ids to avoid propagating unverified identifiers.

• GEPA — Reflective Prompt Evolution — arXiv:2507.19457
• ExpeL — LLM Agents Are Experiential Learners — arXiv:2308.10144
• Constitutional AI — Harmlessness from AI Feedback (critique templates) — arXiv:2212.08073
• Huang et al. — Large Language Models Cannot Self-Correct Reasoning Yet — arXiv:2310.01798
• DSPy — Compiling Declarative LM Calls into Self-Improving Pipelines — arXiv:2310.03714
• JudgeBench — LLM-judge bias benchmark (named; id not independently verified)

---

Migrated from v2nic#6.

[elecnix]

Already implemented on main (verified by a code audit; the migrated issue predated this work). The proposal-validate analyzer performs the replay test: plan() resolves originating turns from source_message_ids (pinned by session-overview), re-runs the classifier with the candidate proposal injected using a distinct model tier (mid vs. the classifier's cheap), emits a content-addressed validation node (new NodeKind in types.ts), and writes validated_score/validation_status back onto the proposal (wired in framework.ts). Ranking + the --full view surface replay-validated vs model-rated. Covered by tests/unit/proposal-validate.test.ts and tests/component/proposal-validate.test.ts; documented in DESIGN.md. Closing as completed.

[elecnix]

Supporting literature: Scaling Test-Time Compute for Agentic Coding (arXiv:2604.16529), Apr 2026.

(Commenting on the issue rather than the PR: proposal-validate landed via v2nic#4, and that repo is archived and read-only.)

This issue's premise — that the synthesising model's self-rated confidence is not usable as a ranking signal — is the same premise behind the paper's selection mechanism. Instead of asking a model to score a rollout, it runs Recursive Tournament Voting: recursively narrowing a population of compact rollout summaries through small-group comparisons. Reported gains are large: Claude 4.5 Opus 70.9% → 77.6% on SWE-Bench Verified, 46.9% → 59.1% on Terminal-Bench v2.0.

SWE-Replay (arXiv:2601.22129) supplies the corroborating negative from the other direction: its LLM-as-a-Judge selector underperformed a cheap deterministic signal, 54.0% vs 60.0%, at additional cost. Asking a model how good something is remains the weak move in both papers.

Where our approach is arguably the stronger one. Replay validation is counterfactual, not comparative: we do not ask which proposal looks best, we inject the candidate rule as a standing instruction the agent "already had" and check whether the friction actually flips. That grounds a proposal in isolation, which no amount of comparison can do — a tournament over a field of uniformly bad proposals still returns a winner.

Where theirs is stronger, and what to steal. RTV can rank a field where every candidate is individually plausible; our supported / unsupported / unvalidated buckets cannot separate within a bucket. Comparative ranking layered on top of replay validation would fix that without giving up the counterfactual grounding — and the SWE-Replay result is the caution about how to build it: structured comparison, not a second model asked for a confidence number.

Related, from the same paper's sequential half (Parallel-Distill-Refine): it conditions new rollouts on summaries distilled from prior attempts. We already compute something close to that distillation for cross-session contrast and then discard it after synthesis.