Repository navigation
swe-router 0.2.0: route on a six-question rubric - #226
Merged
Merged
Conversation
The skill no longer asks the developer's model for a quality floor and a
tier in its own words. It answers the six questions in rubric.md, 0-2
each, and route.py does the rest: four answers sum to a difficulty
bucket (easy, moderate, hard, very hard), and blast radius and
verification set the required quality. route.py then picks the cheapest
permitted model whose mean score on benchmark tasks in that bucket
clears it.
route.py
- --rubric '<six scores as JSON>' and --confidence; reads
routing-policy.json beside it (--policy overrides).
- Skips a model that failed a benchmark task in the bucket
(min_completion_in_bucket, default 1.0). A column mean leaves failures
out, which let Haiku 5.5's failures through on cli/cli.
- Reads the next harder bucket when the benchmark has none for the task's
bucket; v2 has no easy task.
- Low confidence returns status "unsure": stay on the current model.
- --tier/--floor still work for one release.
- A malformed policy fails with a message, not a traceback.
models.json (schema 2.0) adds score_by_difficulty and
completion_by_difficulty. build_vended_models.py builds them from the
dataset's Opus 5.5 rubric pass, recomputing each task's bucket from its
stored scores with the shipped policy, and records the pass in
provenance.rubric_scores.
rubric.md and routing-policy.json move from benchmarks/config/swe-router/
into vend/swe-router/, so the skill's copy is the only copy. install.sh
ships both and keeps an edited routing-policy.json on upgrade, as it
keeps allowed-models.txt.
SKILL.md is rewritten around the rubric and names no measured model, so
it no longer goes stale when the data changes; a test holds that. The
README documents the rubric, prints every model's per-bucket columns
(checked against models.json by a test), and reports the replays: on v2
with all 23 models, 74.7% saved for -3.22 points against always
claude-opus-5, where the previous skill gave 70.5% and -5.06.
eval_swe_router.py gains --min-completion to replay the failure rule.
docs/swe-router-how-it-works.{md,html} and
docs/swe-router-rubric-scores.{md,html} explain the three stages and the
rubric scores file.
aarora79
force-pushed
the
bench/cli-cli-set-a
branch
from
October 14, 2026 23:49
46d3b8c to
d852093
Compare
aarora79
force-pushed
the
swe-router-phase1
branch
from
October 14, 2026 23:49
310e891 to
613d3e1
Compare
The slide, its README screenshot and the ASCII copy still showed the 0.1 flow: a quality floor and a tier table. They now show a difficulty bucket per benchmark task, rubric.md shipping beside route.py, and the developer's task answering four questions for a bucket and two for the bar. The PNG is re-rendered from the HTML at the same 1144x645. The README's install section says seven files and that an edited routing-policy.json survives an upgrade, describes how the skill picks a model, and indexes the new explainers and the evidence behind the rubric.
route.py needs only the Python standard library, but it does need python3 3.8 or later on PATH. Without it the skill could only fail the first time it fired. - install.sh checks for python3 3.8+ before downloading anything. If it is missing it exits 1, says how to install one per platform, and writes nothing. - route.py reports an older Python as a JSON error with exit code 2 instead of a traceback, for anyone who copies the files by hand. - SKILL.md: if python3 is missing or route.py will not start, say so and recommend nothing; never fall back to doing the arithmetic in prose. - Both READMEs state the requirement. Tests cover the installer refusing with a stub python3 (no network needed: the check runs before any download) and route.py's message on a simulated Python 3.7. route.py itself runs on Python 3.8 and 3.9.
3 tasks done
aarora79
added a commit
that referenced
this pull request
Oct 14, 2026
The changes reviewed and approved in #226, which merged into its stacked base branch rather than main. The tree is identical to #226's head. The skill answers the six questions in rubric.md about the task, and route.py turns the answers into a difficulty bucket (from four answers) and a required quality (from blast radius and verification), then picks the cheapest permitted model whose mean benchmark score in that bucket clears it. It skips any model that failed a benchmark task in the bucket, reads the next harder bucket when the benchmark has none for the task's, and keeps the developer on their model when confidence is low. --tier/--floor still work for one release. models.json moves to schema 2.0 with score_by_difficulty and completion_by_difficulty, built from the v2 Opus 5.5 rubric pass. rubric.md and routing-policy.json move into vend/swe-router/; the installer ships both, keeps an edited policy on upgrade, and refuses to install without python3 3.8+. SKILL.md names no measured model. The workflow slide, its PNG, the ASCII diagram and both READMEs describe the new flow, and docs/swe-router-how-it-works and docs/swe-router-rubric-scores explain it. On the 21 v2 tasks against claude-opus-5 on everything: 74.7% saved for -3.22 points and 4 tasks below the bar, where the previous skill gave 70.5%, -5.06 and 9.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The skill now answers the six questions in
rubric.mdabout the task, 0-2 each, androute.pyturns the answers into a model. Four answers sum to the task's bucket (its difficulty level: easy, moderate, hard, very hard);blast_radiusandverificationset the required quality.route.pypicks the cheapest permitted model whose mean score on benchmark tasks in that bucket clears it. Before this, the developer's model chose a floor and a tier in its own words.Stacked on #225: merge that first and this retargets to
main.How it works, with diagrams:
docs/swe-router-how-it-works.md(and.html). What the rubric scores file is and who reads it:docs/swe-router-rubric-scores.md(and.html).Measured effect
Replaying the router over the 21 v2 tasks, each routed from columns that leave that task out, against
claude-opus-5on every task:On cli/cli with columns built from cli/cli itself, the failure rule moves quality (failure = 0) from -12.50 to -10.44. It cannot foresee a failure the benchmark never recorded: zero-shot cli/cli routing from v2 columns is unchanged, because Haiku 5.5 finished every hard v2 task. Sources:
docs/metrics/swe-router-rubric-replays/*-completion.json.Changes
vend/swe-router/route.py:--rubricand--confidence; readsrouting-policy.json; skips models that failed a benchmark task in the bucket (min_completion_in_bucket, default 1.0); reads the next harder bucket when the benchmark has none for the task's; low confidence returnsunsure.--tier/--floorkept for one release. A malformed policy fails with a message.models.jsonschema 2.0:score_by_difficultyandcompletion_by_difficulty, built bybuild_vended_models.pyfrom the v2 Opus 5.5 rubric pass, with buckets recomputed from the stored scores using the shipped policy. Provenance names the rubric pass.rubric.mdandrouting-policy.jsonmove intovend/swe-router/.install.shships both and keeps an editedrouting-policy.jsonon upgrade, as it keepsallowed-models.txt.SKILL.mdrewritten around the rubric (version 0.2.0); it names no measured model, and a test holds that.README.md: the rubric, every model's per-bucket columns (checked againstmodels.jsonby a test), the replay results.eval_swe_router.py --min-completionto replay the failure rule.Test plan
uv run python -m unittest discover -s tests: 669 tests OK, including a check thatroute.pyandswe_router_rubric.pyagree on bucket and required quality for all 729 score combinationsbuild_vended_models.py --checkpassesbash -n install.shinstall.sh --ref swe-router-phase1into a scratch directory: seven files installed; a re-run kept an editedrouting-policy.jsonandroute.pyhonoured it