Repository navigation
cli/cli set A: Claude 5.5 results and the cross-repo answer to #221 - #225
Merged
Merged
Conversation
… omp/swe3, cli/cli dataset Set A of the cross-repo check in #221: the three Claude 5.5 models on the 21 cli-cli-v1 tasks, omp/swe3 on Amazon Bedrock, judged by openai.gpt-5.6-sol at high reasoning effort. Only run-summary.json, metrics.json and eval.json are committed, as for v2. claude-opus-5-5 85.62 $2.18/task 21/21 scored claude-sonnet-5-5 82.50 $0.91/task 21/21 scored claude-haiku-5-5 80.82 $0.09/task 19/21 scored Haiku 5.5 failed two high-tier tasks by writing its artifacts outside the output directory the prompt named, as it did once on v2. auth-clipboard-config-default went to claude-code/cli/ on both attempts; skill-install-nested-discovery went to cli-v1/, and its retry wrote nothing. Both count as failures and were not judged. Their artifacts are kept under do-not-include/claude-haiku-5-5-misplaced-artifacts/, outside the harness scan, and the README there records what happened. cli/cli results stay in their own output scope and do not feed the v2 frontier, charts or vended models.json.
…ts doc Score the 21 cli-cli-v1 tasks with the swe-router rubric (Haiku 5.5 and Opus 5.5, three repeats each) and replay routing among the three Claude 5.5 models two ways: columns built from cli/cli itself (leave-one-out) and columns built from v2 alone (zero-shot). Add a v2 replay limited to the same three models so the two repositories compare directly. eval_swe_router.py gains --columns-from and --columns-rubric-inputs: build the bucket columns from another dataset's results and rubric pass, then route this dataset's tasks against them. Absolute buckets only, and no --holdout, since the routed tasks are never in the columns. docs/results-cli-cli-v1.md answers the four questions. Rankings hold; every model scores 3 to 8 points higher on cli/cli; v2's columns transfer and route a little more cautiously than cli/cli's own. On cli/cli the router sends 17 to 21 tasks to Haiku 5.5, and its two wrong-folder failures cost most of the quality, because route.py reports completion per bucket but never uses it to choose. Phase 1 should make the router see failures.
8 tasks
A rubric pass scores one dataset's tasks, so two datasets need two
files. The v2 passes were named only after their judge; rename them to
swe-router-rubric-mcp-gateway-registry-v2-claude-{opus,haiku}-5-5, the
pattern the cli/cli passes already use, and update the references in
swe-router-rubric-evaluation.md.
run-swe-router-headless.py --mode rubric now defaults --out to
docs/metrics/swe-router-rubric-<dataset scope>-<model slug>.json, so a
pass over a new dataset, or by a second judge, never overwrites an
existing file. Floor mode keeps its fixed name.
run-swe-router-headless.py --mode rubric now writes docs/metrics/swe-router-rubric-<dataset scope>-<model slug>.json when --out is not given, so a pass over a new dataset, or by a second judge, never overwrites an existing file. Floor mode keeps its fixed name. Update swe-router-rubric-evaluation.md for the renamed v2 files.
4 of 5 tasks
aarora79
force-pushed
the
bench/cli-cli-set-a
branch
from
October 14, 2026 23:49
46d3b8c to
d852093
Compare
3 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Set A of the cross-repo check in #221: Claude Haiku 5.5, Sonnet 5.5 and Opus 5.5 on the 21 cli/cli (Go) tasks, omp/swe3 on Amazon Bedrock, judged by
openai.gpt-5.6-solat high effort. The full write-up is docs/results-cli-cli-v1.md.Answers to #221
route.pyignores completion rate, so Haiku's failures go unseen. Phase 1 should fix that.Changes
run-summary.json,metrics.jsonandeval.jsonfor the three models undercli-cli-v1/, as for v2. Haiku's misplaced artifacts go underdo-not-include/claude-haiku-5-5-misplaced-artifacts/, outside the harness scan, with an entry in that folder's README.eval_swe_router.py:--columns-fromand--columns-rubric-inputsbuild bucket columns from another dataset, for the zero-shot replay. Two new tests; the full suite passes (642 tests).docs/metrics/.docs/results-cli-cli-v1.md, linked from AGENTS.md.Test plan
uv run python -m unittest discover -s tests(642 tests, OK)ruff format/ruff checkon changed Python;banditclean