Skip to content

swe-router phase 0: a task rubric, tested against today's skill - #223

Merged
aarora79 merged 5 commits into
mainfrom
feat/swe-router-rubric-phase0
Oct 14, 2026
Merged

aarora79 merged 5 commits into
mainfrom
feat/swe-router-rubric-phase0

Conversation

@aarora79

@aarora79 aarora79 commented Oct 14, 2026 •

Copy link
Copy Markdown
Owner

Summary

Phase 0 of replacing swe-router's two free-form judgments (a consequence floor and a complexity tier) with one rubric. The shipped skill is unchanged; nothing under vend/ or .claude/skills/swe-router/ is touched. This PR adds the tooling, runs it on the 21 mcp-gateway-registry-v2 tasks using the existing benchmark results, and reports the outcome in docs/swe-router-rubric-evaluation.md.

The rubric

A model reads the task and its repository and scores six dimensions from 0 to 2, with evidence: hidden contract, silent invariant, coordinated surfaces, ambiguity, verification, blast radius (rubric.md). Code, not the model, turns the scores into a required quality (blast radius 0/1/2 → 60/70/75, +5 when nothing would catch a mistake) and a difficulty bucket (the sum of the other four), using routing-policy.json. Routing then picks the cheapest model whose measured score in that bucket's column clears the required quality.

Results

Rubric passes through Claude Code 2.1.296 on Amazon Bedrock, 21 tasks × 3 repeats: Haiku 5.5 $0.84 ($0.0135 a judgment), Opus 5.5 $20.23 ($0.32). Replayed leave-one-out against claude-opus-5 on every task:

Judgment Candidates Saved Quality Quality, failure = 0 Under floor
Today's skill all 23 70.5% −5.06 −5.06 9
Rubric, Opus 5.5 all 23 75.3% −3.75 −3.75 4
Rubric, Haiku 5.5 all 23 85.1% −3.64 −7.17 6
Today's skill Bedrock 68.0% −3.10 −3.10 7
Rubric, Opus 5.5 Bedrock 74.7% −3.57 −3.57 4
Rubric, Haiku 5.5 Bedrock 78.7% −3.84 −3.84 5
  • The two judges agree within one bucket on all 21 tasks.
  • The difficulty sum separates the 7 tasks that needed more than Haiku 5.5 from the 14 it handled at 0.82 (Haiku judging) and 0.78 (Opus).
  • Haiku 5.5's one routed failure, with self-hosted models allowed, is visible only in the failure = 0 column, which eval_swe_router.py now reports.
  • Relative buckets (quarters of the tasks' own difficulty sums) sorted tasks better but routed worse in 3 of 4 comparisons: quarters of 21 tasks leave columns too thin for steady means. The policy stays on absolute buckets.

A fitted line per model (--column-model regression)

Instead of bucket columns, each model's score is fitted as a line in the difficulty sum over the other 20 tasks. It helps in one cell (Haiku 5.5 judging, all models: -3.40 with failures as 0, against -7.17) and hurts in the other three (Opus 5.5 judging: -5.72 and -5.02 against -3.75 and -3.57). Only Haiku 5.5 loses a lot as difficulty rises (about 4 points per step); single-task scores vary by about 8-11 points around each line, so these 21 tasks cannot tell the column methods apart. The policy stays on absolute buckets, and the next step is more tasks (cli/cli), not more methods.

Changes

  • benchmarks/config/swe-router/rubric.md, routing-policy.json: the rubric and the policy.
  • benchmarks/scripts/swe_router_rubric.py: policy validation, the mappings, prompt, strict parsing, consolidation of repeats, relative quartiles.
  • run-swe-router-headless.py --mode rubric: the rubric pass. The clone folder now includes the process id, so two passes can run side by side.
  • eval_swe_router.py --rubric-inputs [--bucketing] [--column-model]: regroups existing scores by bucket per routed task, or predicts each model's score from a fitted line. Totals now include the failure = 0 measure.
  • docs/metrics/swe-router-rubric-*.json, docs/metrics/swe-router-rubric-replays/, docs/swe-router-rubric-*.md: the data behind every number.

Testing

  • 40 new tests (39 in test_swe_router_rubric.py, 1 in test_run_swe_router_headless.py); the full benchmarks/ suite passes.
  • ruff and Bandit are clean; the security-check gate passed.

Not in this PR

Phase 1, moving the rubric into the shipped skill, waits on the cli/cli zero-shot check: 21 more tasks on a second repository.

Phase 0 of replacing swe-router's two free-form judgments (a consequence
floor and a complexity tier) with one zero-shot rubric. Nothing in the
shipped skill changes yet; this adds the tooling to test the idea against
the existing benchmark results, which are reused unchanged.

benchmarks/config/swe-router/rubric.md: six dimensions scored 0-2 against
the task's own repository -- hidden contract, silent invariant,
coordinated surfaces, ambiguity, verification, blast radius -- each with
what to look at, anchors for 0/1/2, a common mistake, and a required piece
of evidence. Size is explicitly not a signal.

benchmarks/config/swe-router/routing-policy.json: the numbers the model
never sees. Blast radius sets the required quality (0/1/2 -> 60/70/75,
+5 when nothing would catch a mistake), with each level's meaning tied to
the artifact judge's 'strong' band. The four difficulty dimensions sum to
a bucket: easy, moderate, hard, very hard.

benchmarks/scripts/swe_router_rubric.py: the shared logic -- policy
validation, the score-to-quality and score-to-bucket mapping, the prompt,
strict parsing of the six scores, and consolidation of repeats (lower
median per dimension, with per-repeat quality and bucket kept).

run-swe-router-headless.py --mode rubric: runs the rubric pass in each
task's cloned repository with repeats, reusing the existing cloning,
agents and cost accounting, and writes JSON plus a markdown report.

eval_swe_router.py --rubric-inputs: replays routing with each task's
required quality as its floor and its bucket as its column, regrouping
every model's existing task scores by bucket. With --holdout the columns
exclude the task being routed.

27 new tests. A one-task smoke run with Haiku 5.5 on Bedrock scored
pass-ssrf-allowlist-env-to-registry-container at bar 80, bucket hard,
for $0.015 in 42 seconds.
Two rubric passes started side by side (one per judge model) cloned the
same task into /tmp/router-attempt-N/ and collided on git's shallow.lock;
either one's cleanup could also have deleted the other's checkout. The
clone parent now carries the process id, extending the existing
per-attempt guard to concurrent runs. Clone failures still propagate:
they are environment faults, not failed judgments.
Run the rubric pass on the 21 mcp-gateway-registry-v2 tasks through
Claude Code 2.1.296 on Amazon Bedrock, 3 repeats each: Haiku 5.5 ($0.84,
$0.0135 a judgment) and Opus 5.5 ($20.23, $0.32 a judgment). Replay each,
leave-one-out, against today's skill on today's 23-model data. No
benchmark run is repeated or re-judged.

Against claude-opus-5 on every task, with every model available: today's
skill saves 70.5% for -5.06 quality and 9 tasks under floor; the rubric
judged by Opus 5.5 saves 75.3% for -3.75 and 4 under floor. Bedrock only:
68.0% / -3.10 / 7 against 74.7% / -3.57 / 4. Judged by Haiku 5.5, Bedrock
only: 78.7% / -3.84 / 5. With every model available Haiku 5.5's routing
sent one hard task to a self-hosted model that failed it (-7.17 counting
the failure as 0).

The two judges agree within one bucket on all 21 tasks, and the
difficulty sum separates tasks that needed more than Haiku 5.5 from those
it handled at 0.82 (Haiku 5.5 judging) and 0.78 (Opus 5.5).

Add relative buckets (quarters of the tasks' own difficulty sums,
computed leave-one-out) and test them: they sort tasks better but leave
columns of 3-12 tasks and route worse in 3 of 4 comparisons. The policy
stays on absolute buckets; relative is kept as an option for larger task
sets. Write the findings to docs/swe-router-rubric-evaluation.md.

eval_swe_router.py also reports the score delta with a routed failure
counted as 0, beside the paired measure that leaves such tasks out.
@aarora79 aarora79 added the enhancement New feature or request label Oct 14, 2026
eval_swe_router.py --column-model regression: for the routed task, fit
each model's score as a straight line in the rubric's difficulty sum over
the other tasks (failed runs left out, as in the column means), predict
its score at this task's sum, and let route.py pick as before. Every task
informs every prediction, so no estimate rests on a thin column.

On the 21 v2 tasks it helps in one cell (Haiku 5.5 judging, every model
available: -3.40 with failures as 0, against -7.17 for absolute buckets)
and hurts in the other three. The fitted lines explain it: only Haiku 5.5
loses a lot as difficulty rises (about 4 points per step, more than two
standard errors from zero); the others lose 1 to 2. Single-task scores
vary by about 8-11 points around each line, while the column methods
differ by 1 to 2 points of mean quality, so these 21 tasks cannot tell
them apart. The policy stays on absolute buckets; the line is an option.

Record the result and the slopes in docs/swe-router-rubric-evaluation.md,
and point the next step at more tasks (cli/cli) rather than more methods.
docs/swe-router-rubric-design.md walks through the rubric design tested
in phase 0: what the model reads, the six dimensions, how routing-policy.json
turns scores into a bar and a bucket, how route.py picks, which file each
customer owns, and what remains before it ships. It marks the two parts
that exist only in the evaluation so far: bucket names in route.py and the
low-confidence fallback.
@aarora79
aarora79 merged commit 0c5dfbd into main Oct 14, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant