Skip to content

cli/cli set A: Claude 5.5 results and the cross-repo answer to #221 - #225

Merged
aarora79 merged 4 commits into
mainfrom
bench/cli-cli-set-a
Oct 14, 2026
Merged

aarora79 merged 4 commits into
mainfrom
bench/cli-cli-set-a

Conversation

@aarora79

Copy link
Copy Markdown
Owner

Summary

Set A of the cross-repo check in #221: Claude Haiku 5.5, Sonnet 5.5 and Opus 5.5 on the 21 cli/cli (Go) tasks, omp/swe3 on Amazon Bedrock, judged by openai.gpt-5.6-sol at high effort. The full write-up is docs/results-cli-cli-v1.md.

Model v2 score cli/cli score cli/cli failure = 0 cli/cli cost (21 tasks)
Haiku 5.5 72.69 80.82 (19/21) 73.12 $1.99
Sonnet 5.5 79.49 82.50 82.50 $19.11
Opus 5.5 81.16 85.62 85.62 $45.88

Answers to #221

  • Rankings carry over overall. Within a tier the gaps shrink to a few points.
  • Score levels shift up 3 to 8 points on cli/cli, at lower cost per task.
  • Haiku is still the deciding question. It scores within 2 points of Sonnet where it finishes, but it failed 2 of 21 tasks by writing artifacts outside the folder the prompt named.
  • The rubric transfers zero-shot. Columns built from v2 alone route cli/cli tasks a little more cautiously than cli/cli's own columns. Routing on cli/cli loses quality for another reason: the router sends 17 to 21 tasks to Haiku, and route.py ignores completion rate, so Haiku's failures go unseen. Phase 1 should fix that.

Changes

  • Results: run-summary.json, metrics.json and eval.json for the three models under cli-cli-v1/, as for v2. Haiku's misplaced artifacts go under do-not-include/claude-haiku-5-5-misplaced-artifacts/, outside the harness scan, with an entry in that folder's README.
  • eval_swe_router.py: --columns-from and --columns-rubric-inputs build bucket columns from another dataset, for the zero-shot replay. Two new tests; the full suite passes (642 tests).
  • Rubric passes on cli/cli (Haiku 5.5 and Opus 5.5, three repeats) and six replays under docs/metrics/.
  • docs/results-cli-cli-v1.md, linked from AGENTS.md.

Test plan

  • uv run python -m unittest discover -s tests (642 tests, OK)
  • ruff format / ruff check on changed Python; bandit clean
  • Every number in the results doc checked against its source JSON

… omp/swe3, cli/cli dataset

Set A of the cross-repo check in #221: the three Claude 5.5 models on the
21 cli-cli-v1 tasks, omp/swe3 on Amazon Bedrock, judged by
openai.gpt-5.6-sol at high reasoning effort. Only run-summary.json,
metrics.json and eval.json are committed, as for v2.

  claude-opus-5-5    85.62  $2.18/task  21/21 scored
  claude-sonnet-5-5  82.50  $0.91/task  21/21 scored
  claude-haiku-5-5   80.82  $0.09/task  19/21 scored

Haiku 5.5 failed two high-tier tasks by writing its artifacts outside
the output directory the prompt named, as it did once on v2.
auth-clipboard-config-default went to claude-code/cli/ on both attempts;
skill-install-nested-discovery went to cli-v1/, and its retry wrote
nothing. Both count as failures and were not judged. Their artifacts are
kept under do-not-include/claude-haiku-5-5-misplaced-artifacts/, outside
the harness scan, and the README there records what happened.

cli/cli results stay in their own output scope and do not feed the v2
frontier, charts or vended models.json.
…ts doc

Score the 21 cli-cli-v1 tasks with the swe-router rubric (Haiku 5.5 and
Opus 5.5, three repeats each) and replay routing among the three Claude
5.5 models two ways: columns built from cli/cli itself (leave-one-out)
and columns built from v2 alone (zero-shot). Add a v2 replay limited to
the same three models so the two repositories compare directly.

eval_swe_router.py gains --columns-from and --columns-rubric-inputs:
build the bucket columns from another dataset's results and rubric pass,
then route this dataset's tasks against them. Absolute buckets only, and
no --holdout, since the routed tasks are never in the columns.

docs/results-cli-cli-v1.md answers the four questions. Rankings hold;
every model scores 3 to 8 points higher on cli/cli; v2's columns
transfer and route a little more cautiously than cli/cli's own. On
cli/cli the router sends 17 to 21 tasks to Haiku 5.5, and its two
wrong-folder failures cost most of the quality, because route.py
reports completion per bucket but never uses it to choose. Phase 1
should make the router see failures.
@aarora79 aarora79 added the documentation Improvements or additions to documentation label Oct 14, 2026
A rubric pass scores one dataset's tasks, so two datasets need two
files. The v2 passes were named only after their judge; rename them to
swe-router-rubric-mcp-gateway-registry-v2-claude-{opus,haiku}-5-5, the
pattern the cli/cli passes already use, and update the references in
swe-router-rubric-evaluation.md.

run-swe-router-headless.py --mode rubric now defaults --out to
docs/metrics/swe-router-rubric-<dataset scope>-<model slug>.json, so a
pass over a new dataset, or by a second judge, never overwrites an
existing file. Floor mode keeps its fixed name.
run-swe-router-headless.py --mode rubric now writes
docs/metrics/swe-router-rubric-<dataset scope>-<model slug>.json when
--out is not given, so a pass over a new dataset, or by a second judge,
never overwrites an existing file. Floor mode keeps its fixed name.
Update swe-router-rubric-evaluation.md for the renamed v2 files.
@aarora79
aarora79 force-pushed the bench/cli-cli-set-a branch from 46d3b8c to d852093 Compare October 14, 2026 23:49
@aarora79
aarora79 merged commit 5a1e05e into main Oct 14, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant