test(benchmark): fail loud when a committed report file is missing#60
Merged
Conversation
The threshold-t2 / bm25 / bm25-threshold reproducibility tests silently skipTest'ed on a missing committed JSON — a deleted or renamed report read as 'reproducibility verified' when nothing ran (cross-model review finding on #48). A missing committed report is a broken contract pin: now self.fail with the exact regenerate command, matching the baseline test's existing behavior. Negative-control verified (hidden file FAILS, not skips). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ction adjacency with #59) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Cross-model review of #48 flagged: the threshold-t2 / bm25 / bm25-threshold committed-report reproducibility tests silently
skipTestwhen the committed JSON is absent — so a deleted/renamed report reads as "reproducibility verified" when nothing ran. The baseline test already fails loud; the other three now match.Change
tests/test_retrieval_benchmark.py: threeskipTest→self.failwith the exact regenerate command.### Fixed.No scoring, metric, or report change.
Validation
Full suite 443 passed / 1 skipped (the remaining skip is the unrelated portable-retriever stdlib invariant) · ruff clean · negative control: hiding
retrieval-v1-lexical-bm25.jsonmakes the test FAIL (not skip); restoring it passes.🤖 Generated with Claude Code