feat(eval): conformal judge intervals — distribution-free uncertainty bounds (v0.29.0)

[konjoinfinity]

Konjo sprint — v0.29.0

First sprint of a new Discovery cycle. The v0.24–v0.28 researched sweep shipped fully (#15–#19, all merged); a fresh researcher pass surfaced this as the High-impact / Low-complexity next step, rebalancing toward the judge-eval track after two inference sprints.

Why

kairu/reliability.py ships Cronbach α, ICC(2,1), and Fleiss κ — all retrospective, distribution-parametric estimators describing a judges × criteria matrix after the fact. None answers the deployment question: given this judge's score, what's a defensible interval around the true quality, with a coverage guarantee that holds regardless of the judge's score distribution?

What

New module kairu/conformal.py — split conformal prediction (Vovk; applied to LLM judges by Sheng et al., EMNLP 2025, arXiv:2509.18658), pure stdlib + kairu.ensemble:

• conformal_quantile(residuals, alpha) — finite-sample-corrected (1−α) quantile via the conformal rank ceil((n+1)(1−α)). Returns +inf (no finite bound) when the calibration set is too small for the requested α — the honest answer, not a fabricated number.
• calibrate_interval(prediction, calibration_pairs, *, alpha, score_range) → ConformalInterval — [score−q, score+q] with the paper's two judge-specific refinements: ordinal boundary clamp (never claims impossible scores) and the lower-bias midpoint point estimate (pulled inside the range near a boundary). Covers the reference with probability ≥ 1−α for any exchangeable new pair.
• conformal_from_ensemble(result, calibration_pairs, …) — bridges EnsembleResult.median_aggregate.

reliability.py and EnsembleResult are untouched — this is purely additive, so all prior eval tests stay green.

Quality gates

• 763 passed, 4 HF-gated skipped (+17 new).
• New tests/test_conformal.py at 100% module coverage — including an empirical coverage test verifying ~90% of held-out references fall within 90%-level intervals on synthetic exchangeable data (the actual conformal guarantee, not just self-consistency). Edge cases pinned: finite-sample rank arithmetic, +inf on undersized calibration, full-range clamp, degenerate range / empty-calibration / bad-α rejection.
• ruff lint + format clean; all functions radon grade A; 199 lines (< 500).

Reuse note

The quantile follows kairu's existing sorted-list / nearest-rank percentile pattern. No HF/torch — MockModel/ensemble_evaluate exercise every path offline. Backlog from the same Discovery sweep (logged in PLAN.md for future cycles): IRT judge discrimination, entropy-driven adaptive γ, dark-current datasheet, radix-tree KV dedup.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TmSqgYqcYfzPnkpDVR5vdw

---

Generated by Claude Code