Repository navigation
swe-router phase 0: a task rubric, tested against today's skill - #223
Merged
Merged
Conversation
Phase 0 of replacing swe-router's two free-form judgments (a consequence floor and a complexity tier) with one zero-shot rubric. Nothing in the shipped skill changes yet; this adds the tooling to test the idea against the existing benchmark results, which are reused unchanged. benchmarks/config/swe-router/rubric.md: six dimensions scored 0-2 against the task's own repository -- hidden contract, silent invariant, coordinated surfaces, ambiguity, verification, blast radius -- each with what to look at, anchors for 0/1/2, a common mistake, and a required piece of evidence. Size is explicitly not a signal. benchmarks/config/swe-router/routing-policy.json: the numbers the model never sees. Blast radius sets the required quality (0/1/2 -> 60/70/75, +5 when nothing would catch a mistake), with each level's meaning tied to the artifact judge's 'strong' band. The four difficulty dimensions sum to a bucket: easy, moderate, hard, very hard. benchmarks/scripts/swe_router_rubric.py: the shared logic -- policy validation, the score-to-quality and score-to-bucket mapping, the prompt, strict parsing of the six scores, and consolidation of repeats (lower median per dimension, with per-repeat quality and bucket kept). run-swe-router-headless.py --mode rubric: runs the rubric pass in each task's cloned repository with repeats, reusing the existing cloning, agents and cost accounting, and writes JSON plus a markdown report. eval_swe_router.py --rubric-inputs: replays routing with each task's required quality as its floor and its bucket as its column, regrouping every model's existing task scores by bucket. With --holdout the columns exclude the task being routed. 27 new tests. A one-task smoke run with Haiku 5.5 on Bedrock scored pass-ssrf-allowlist-env-to-registry-container at bar 80, bucket hard, for $0.015 in 42 seconds.
Two rubric passes started side by side (one per judge model) cloned the same task into /tmp/router-attempt-N/ and collided on git's shallow.lock; either one's cleanup could also have deleted the other's checkout. The clone parent now carries the process id, extending the existing per-attempt guard to concurrent runs. Clone failures still propagate: they are environment faults, not failed judgments.
Run the rubric pass on the 21 mcp-gateway-registry-v2 tasks through Claude Code 2.1.296 on Amazon Bedrock, 3 repeats each: Haiku 5.5 ($0.84, $0.0135 a judgment) and Opus 5.5 ($20.23, $0.32 a judgment). Replay each, leave-one-out, against today's skill on today's 23-model data. No benchmark run is repeated or re-judged. Against claude-opus-5 on every task, with every model available: today's skill saves 70.5% for -5.06 quality and 9 tasks under floor; the rubric judged by Opus 5.5 saves 75.3% for -3.75 and 4 under floor. Bedrock only: 68.0% / -3.10 / 7 against 74.7% / -3.57 / 4. Judged by Haiku 5.5, Bedrock only: 78.7% / -3.84 / 5. With every model available Haiku 5.5's routing sent one hard task to a self-hosted model that failed it (-7.17 counting the failure as 0). The two judges agree within one bucket on all 21 tasks, and the difficulty sum separates tasks that needed more than Haiku 5.5 from those it handled at 0.82 (Haiku 5.5 judging) and 0.78 (Opus 5.5). Add relative buckets (quarters of the tasks' own difficulty sums, computed leave-one-out) and test them: they sort tasks better but leave columns of 3-12 tasks and route worse in 3 of 4 comparisons. The policy stays on absolute buckets; relative is kept as an option for larger task sets. Write the findings to docs/swe-router-rubric-evaluation.md. eval_swe_router.py also reports the score delta with a routed failure counted as 0, beside the paired measure that leaves such tasks out.
eval_swe_router.py --column-model regression: for the routed task, fit each model's score as a straight line in the rubric's difficulty sum over the other tasks (failed runs left out, as in the column means), predict its score at this task's sum, and let route.py pick as before. Every task informs every prediction, so no estimate rests on a thin column. On the 21 v2 tasks it helps in one cell (Haiku 5.5 judging, every model available: -3.40 with failures as 0, against -7.17 for absolute buckets) and hurts in the other three. The fitted lines explain it: only Haiku 5.5 loses a lot as difficulty rises (about 4 points per step, more than two standard errors from zero); the others lose 1 to 2. Single-task scores vary by about 8-11 points around each line, while the column methods differ by 1 to 2 points of mean quality, so these 21 tasks cannot tell them apart. The policy stays on absolute buckets; the line is an option. Record the result and the slopes in docs/swe-router-rubric-evaluation.md, and point the next step at more tasks (cli/cli) rather than more methods.
docs/swe-router-rubric-design.md walks through the rubric design tested in phase 0: what the model reads, the six dimensions, how routing-policy.json turns scores into a bar and a bucket, how route.py picks, which file each customer owns, and what remains before it ships. It marks the two parts that exist only in the evaluation so far: bucket names in route.py and the low-confidence fallback.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 0 of replacing
swe-router's two free-form judgments (a consequence floor and a complexity tier) with one rubric. The shipped skill is unchanged; nothing undervend/or.claude/skills/swe-router/is touched. This PR adds the tooling, runs it on the 21mcp-gateway-registry-v2tasks using the existing benchmark results, and reports the outcome in docs/swe-router-rubric-evaluation.md.The rubric
A model reads the task and its repository and scores six dimensions from 0 to 2, with evidence: hidden contract, silent invariant, coordinated surfaces, ambiguity, verification, blast radius (rubric.md). Code, not the model, turns the scores into a required quality (blast radius 0/1/2 → 60/70/75, +5 when nothing would catch a mistake) and a difficulty bucket (the sum of the other four), using routing-policy.json. Routing then picks the cheapest model whose measured score in that bucket's column clears the required quality.
Results
Rubric passes through Claude Code 2.1.296 on Amazon Bedrock, 21 tasks × 3 repeats: Haiku 5.5 $0.84 ($0.0135 a judgment), Opus 5.5 $20.23 ($0.32). Replayed leave-one-out against
claude-opus-5on every task:eval_swe_router.pynow reports.A fitted line per model (
--column-model regression)Instead of bucket columns, each model's score is fitted as a line in the difficulty sum over the other 20 tasks. It helps in one cell (Haiku 5.5 judging, all models: -3.40 with failures as 0, against -7.17) and hurts in the other three (Opus 5.5 judging: -5.72 and -5.02 against -3.75 and -3.57). Only Haiku 5.5 loses a lot as difficulty rises (about 4 points per step); single-task scores vary by about 8-11 points around each line, so these 21 tasks cannot tell the column methods apart. The policy stays on absolute buckets, and the next step is more tasks (cli/cli), not more methods.
Changes
benchmarks/config/swe-router/rubric.md,routing-policy.json: the rubric and the policy.benchmarks/scripts/swe_router_rubric.py: policy validation, the mappings, prompt, strict parsing, consolidation of repeats, relative quartiles.run-swe-router-headless.py --mode rubric: the rubric pass. The clone folder now includes the process id, so two passes can run side by side.eval_swe_router.py --rubric-inputs [--bucketing] [--column-model]: regroups existing scores by bucket per routed task, or predicts each model's score from a fitted line. Totals now include the failure = 0 measure.docs/metrics/swe-router-rubric-*.json,docs/metrics/swe-router-rubric-replays/,docs/swe-router-rubric-*.md: the data behind every number.Testing
test_swe_router_rubric.py, 1 intest_run_swe_router_headless.py); the fullbenchmarks/suite passes.Not in this PR
Phase 1, moving the rubric into the shipped skill, waits on the cli/cli zero-shot check: 21 more tasks on a second repository.