Chore/accuracy score outcome#140
Merged
Merged
Conversation
Replaces the silent filter_map drop in evaluate_parametric / evaluate_in_context / evaluate_conflict with explicit per-row ScoreOutcome reporting. Engines whose prefill returns None (Apollo on store-miss, future RetrievalEngine impls, any opaque failure) now appear as Skipped rows in the result vector instead of disappearing — the accuracy JSON's row count finally equals the prompt count. Schema changes: - New ScoreOutcome enum (Served, SkippedEmptyPrompt, SkippedRetrievalMiss, SkippedBackendUnavailable, SkippedInternalError). Flat-tagged serde, exhaustive (no #[non_exhaustive]), variants mirror the typed EngineError enum the trait extraction will land. - PromptScore and ConflictScore gain an `outcome` field; their score-payload fields (predicted_top1, top1_match, bits_per_token, followed_context, parametric_fallback) become Option<T>. Correlated optionality enforced by served() / skipped() constructors with debug_assert guards. - StrategySplit gains *_served and *_served_rate per axis as required-companion fields to *_match_rate. Match-rate is computed over the served subset only (counting skips as zero punishes engines for honest reporting; see ROADMAP). - format_strategy_split emits a conditional second line with served denominators only when any axis skipped. Engines that never skip print byte-identical output to pre-change. - CLI summary in accuracy_cmd.rs gains the same conditional format via fmt_served_summary; verbose mode marks skipped rows with `·`. Discoverability-lag fix folded in: - EngineKind::supported_names() returns the canonical list (was hard-coded six-engine string in two bench error sites; now nine engines, single source of truth). Two tests enforce that the list stays in sync with the EngineKind variant set. Interim known issue (documented in the PromptScore docstring and the ROADMAP entry): the JSON's ffn_backend column reports the backend passed at construction, not the one actually used. For engines whose trait method dispatches to multiple internal paths with different FFN usage (markov_residual's CPU vs *_via_executor paths), the reported value may not reflect which backend actually executed. Downstream consumers should not condition on ffn_backend for those engines until the sibling trait extraction lands; see ROADMAP "P0 — sibling trait extraction for non-K/V engines" for the fix. Predecessor: the ROADMAP entry is in this commit; it justifies the interim taxonomy choice and lays out the six-touchpoint refactor that the schema fix is preparing the ground for. Tests: 611 -> 611 (parity, no regressions) on the lib suite; 219 + 3 on the CLI; clippy clean across both crates. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.