Advisor gate outcome efficacy and hosted-call savings evaluation (replacement for #101)

[bearmug]

Purpose

Replace the original binary-gate retraining ticket with an outcome-focused evaluation contract.

The implementation already contains calibrated and trajectory-aware binary-gate machinery. The remaining question is whether the gate improves real agentic outcomes while reducing unnecessary hosted Advisor calls.

Scope

Evaluate the current gate against representative agentic traces and a deterministic baseline, stratified by phase and task class.

Measure:

• Advisor-call reduction and hosted-token reduction;
• false escalation and missed-review rates;
• task completion/regression outcomes;
• performance by quick edit, implementation, debug, review, architecture, and safety-sensitive tasks;
• latency and failure behavior.

Policy

• Evaluation must be offline/replayable where possible.
• No automatic model or policy promotion based on a single aggregate score.
• Safety-sensitive routes remain fail-closed and deterministic.
• Report confidence intervals or sample counts for each slice.
• Keep the controller and final review semantics unchanged during evaluation.

Acceptance criteria

• A reproducible baseline-versus-gate report exists.
• Per-phase confusion and escalation-rate metrics are reported.
• Hosted-call/token savings are measured, not inferred.
• Guard slices cover safety-sensitive, failure, and trivial-task cases.
• The report ends with an explicit promote/hold/reject recommendation.
• Tests cover artifact loading, replay determinism, and metric accounting.

Non-goals

• No broad retraining without evidence from the evaluation.
• No new remote worker or routing subsystem.
• No automatic production promotion in this ticket.