Skip to content

swe-router 0.2.0: route on a six-question rubric - #226

Merged
aarora79 merged 3 commits into
bench/cli-cli-set-afrom
swe-router-phase1
Oct 14, 2026
Merged

aarora79 merged 3 commits into
bench/cli-cli-set-afrom
swe-router-phase1

Conversation

@aarora79

Copy link
Copy Markdown
Owner

Summary

The skill now answers the six questions in rubric.md about the task, 0-2 each, and route.py turns the answers into a model. Four answers sum to the task's bucket (its difficulty level: easy, moderate, hard, very hard); blast_radius and verification set the required quality. route.py picks the cheapest permitted model whose mean score on benchmark tasks in that bucket clears it. Before this, the developer's model chose a floor and a tier in its own words.

Stacked on #225: merge that first and this retargets to main.

How it works, with diagrams: docs/swe-router-how-it-works.md (and .html). What the rubric scores file is and who reads it: docs/swe-router-rubric-scores.md (and .html).

Measured effect

Replaying the router over the 21 v2 tasks, each routed from columns that leave that task out, against claude-opus-5 on every task:

Router Cost saved Quality change Below the required quality
Previous skill 70.5% -5.06 9
Rubric 75.3% -3.75 4
Rubric + failure rule (this PR) 74.7% -3.22 4

On cli/cli with columns built from cli/cli itself, the failure rule moves quality (failure = 0) from -12.50 to -10.44. It cannot foresee a failure the benchmark never recorded: zero-shot cli/cli routing from v2 columns is unchanged, because Haiku 5.5 finished every hard v2 task. Sources: docs/metrics/swe-router-rubric-replays/*-completion.json.

Changes

  • vend/swe-router/route.py: --rubric and --confidence; reads routing-policy.json; skips models that failed a benchmark task in the bucket (min_completion_in_bucket, default 1.0); reads the next harder bucket when the benchmark has none for the task's; low confidence returns unsure. --tier/--floor kept for one release. A malformed policy fails with a message.
  • models.json schema 2.0: score_by_difficulty and completion_by_difficulty, built by build_vended_models.py from the v2 Opus 5.5 rubric pass, with buckets recomputed from the stored scores using the shipped policy. Provenance names the rubric pass.
  • rubric.md and routing-policy.json move into vend/swe-router/. install.sh ships both and keeps an edited routing-policy.json on upgrade, as it keeps allowed-models.txt.
  • SKILL.md rewritten around the rubric (version 0.2.0); it names no measured model, and a test holds that.
  • README.md: the rubric, every model's per-bucket columns (checked against models.json by a test), the replay results.
  • eval_swe_router.py --min-completion to replay the failure rule.

Test plan

  • uv run python -m unittest discover -s tests: 669 tests OK, including a check that route.py and swe_router_rubric.py agree on bucket and required quality for all 729 score combinations
  • build_vended_models.py --check passes
  • ruff, bandit, pre-commit, bash -n install.sh
  • install.sh --ref swe-router-phase1 into a scratch directory: seven files installed; a re-run kept an edited routing-policy.json and route.py honoured it
  • Mermaid diagrams in the two explainers render on GitHub

The skill no longer asks the developer's model for a quality floor and a
tier in its own words. It answers the six questions in rubric.md, 0-2
each, and route.py does the rest: four answers sum to a difficulty
bucket (easy, moderate, hard, very hard), and blast radius and
verification set the required quality. route.py then picks the cheapest
permitted model whose mean score on benchmark tasks in that bucket
clears it.

route.py
- --rubric '<six scores as JSON>' and --confidence; reads
  routing-policy.json beside it (--policy overrides).
- Skips a model that failed a benchmark task in the bucket
  (min_completion_in_bucket, default 1.0). A column mean leaves failures
  out, which let Haiku 5.5's failures through on cli/cli.
- Reads the next harder bucket when the benchmark has none for the task's
  bucket; v2 has no easy task.
- Low confidence returns status "unsure": stay on the current model.
- --tier/--floor still work for one release.
- A malformed policy fails with a message, not a traceback.

models.json (schema 2.0) adds score_by_difficulty and
completion_by_difficulty. build_vended_models.py builds them from the
dataset's Opus 5.5 rubric pass, recomputing each task's bucket from its
stored scores with the shipped policy, and records the pass in
provenance.rubric_scores.

rubric.md and routing-policy.json move from benchmarks/config/swe-router/
into vend/swe-router/, so the skill's copy is the only copy. install.sh
ships both and keeps an edited routing-policy.json on upgrade, as it
keeps allowed-models.txt.

SKILL.md is rewritten around the rubric and names no measured model, so
it no longer goes stale when the data changes; a test holds that. The
README documents the rubric, prints every model's per-bucket columns
(checked against models.json by a test), and reports the replays: on v2
with all 23 models, 74.7% saved for -3.22 points against always
claude-opus-5, where the previous skill gave 70.5% and -5.06.

eval_swe_router.py gains --min-completion to replay the failure rule.
docs/swe-router-how-it-works.{md,html} and
docs/swe-router-rubric-scores.{md,html} explain the three stages and the
rubric scores file.
@aarora79 aarora79 added the enhancement New feature or request label Oct 14, 2026
@aarora79
aarora79 force-pushed the bench/cli-cli-set-a branch from 46d3b8c to d852093 Compare October 14, 2026 23:49
The slide, its README screenshot and the ASCII copy still showed the
0.1 flow: a quality floor and a tier table. They now show a difficulty
bucket per benchmark task, rubric.md shipping beside route.py, and the
developer's task answering four questions for a bucket and two for the
bar. The PNG is re-rendered from the HTML at the same 1144x645.

The README's install section says seven files and that an edited
routing-policy.json survives an upgrade, describes how the skill picks a
model, and indexes the new explainers and the evidence behind the rubric.
route.py needs only the Python standard library, but it does need
python3 3.8 or later on PATH. Without it the skill could only fail the
first time it fired.

- install.sh checks for python3 3.8+ before downloading anything. If it
  is missing it exits 1, says how to install one per platform, and
  writes nothing.
- route.py reports an older Python as a JSON error with exit code 2
  instead of a traceback, for anyone who copies the files by hand.
- SKILL.md: if python3 is missing or route.py will not start, say so and
  recommend nothing; never fall back to doing the arithmetic in prose.
- Both READMEs state the requirement.

Tests cover the installer refusing with a stub python3 (no network
needed: the check runs before any download) and route.py's message on a
simulated Python 3.7. route.py itself runs on Python 3.8 and 3.9.
@aarora79
aarora79 merged commit 1da169b into bench/cli-cli-set-a Oct 14, 2026
aarora79 added a commit that referenced this pull request Oct 14, 2026
The changes reviewed and approved in #226, which merged into its stacked
base branch rather than main. The tree is identical to #226's head.

The skill answers the six questions in rubric.md about the task, and
route.py turns the answers into a difficulty bucket (from four answers)
and a required quality (from blast radius and verification), then picks
the cheapest permitted model whose mean benchmark score in that bucket
clears it. It skips any model that failed a benchmark task in the
bucket, reads the next harder bucket when the benchmark has none for the
task's, and keeps the developer on their model when confidence is low.
--tier/--floor still work for one release.

models.json moves to schema 2.0 with score_by_difficulty and
completion_by_difficulty, built from the v2 Opus 5.5 rubric pass.
rubric.md and routing-policy.json move into vend/swe-router/; the
installer ships both, keeps an edited policy on upgrade, and refuses to
install without python3 3.8+. SKILL.md names no measured model. The
workflow slide, its PNG, the ASCII diagram and both READMEs describe the
new flow, and docs/swe-router-how-it-works and
docs/swe-router-rubric-scores explain it.

On the 21 v2 tasks against claude-opus-5 on everything: 74.7% saved for
-3.22 points and 4 tasks below the bar, where the previous skill gave
70.5%, -5.06 and 9.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant