Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
139 changes: 139 additions & 0 deletions docs/release-notes/0.4.0.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,139 @@
# Release 0.4.0 - swe-router routes on a rubric, and a second repository

**October 2026**

Version bump: **MINOR** -- adds a second dataset (21 tasks from cli/cli, the GitHub CLI, in Go) with the three Claude 5.5 models run on it, and moves the `swe-router` skill to 0.2.0, which picks a model from a six-question task rubric. Every published benchmark score and cost from 0.3.0 stands unchanged.

---

## Upgrading from 0.3.0

This repo is cloned and run, not deployed, so upgrading is a pull plus a dependency sync.

```bash
git pull origin main
git checkout 0.4.0

cd benchmarks && uv sync && cd ..
cd self-hosted/vllm && uv sync && cd ../..
```

Anyone who installed `swe-router` should re-run its installer. The new `route.py` reads only the new `models.json`, so the seven files must come from the same release, which the installer guarantees:

```bash
curl -sL https://raw.githubusercontent.com/aarora79/agentic-coding-harness-benchmarks/0.4.0/vend/swe-router/install.sh | bash -s -- --ref 0.4.0
```

An edited `allowed-models.txt` is kept. The installer now needs `python3` 3.8 or later on PATH, and stops before downloading anything without it.

### Methodology changes

None. Numbers remain comparable with 0.3.0. No frontier JSON under [docs/metrics/](../metrics/) changed in this window, and the cli/cli runs live in their own output scope (`cli-cli-v1`), outside every v2 table and chart.

The router's way of choosing a model changed, which is a change to a skill, not to how a benchmark number is produced. See [New functionality](#new-functionality).

### Config changes

| File | Change | Action needed |
|------|--------|---------------|
| [vend/swe-router/routing-policy.json](../../vend/swe-router/routing-policy.json) | New in the skill (moved from `benchmarks/config/swe-router/`). Maps blast radius and verification to a required quality (60 / 70 / 75, +5 when nothing would catch a mistake), sets the difficulty bucket ranges, the 3-point tie band, and `min_completion_in_bucket` (1.0) | None. A platform team may edit it; the installer keeps an edited copy on upgrade. |
| [vend/swe-router/models.json](../../vend/swe-router/models.json) | Schema 1.0 to 2.0 across this window: 1.1 renamed `provenance.source_commit` to `source_sha256` ([#222](https://github.com/aarora79/agentic-coding-harness-benchmarks/pull/222)); 2.0 adds `score_by_difficulty`, `completion_by_difficulty` and `provenance.rubric_scores` ([#227](https://github.com/aarora79/agentic-coding-harness-benchmarks/pull/227)) | Re-install the skill. A 0.1 `route.py` refuses a schema-2 file. |
| [benchmarks/scripts/runner_config.py](../../benchmarks/scripts/runner_config.py), [benchmarks/config/runner.example.yaml](../../benchmarks/config/runner.example.yaml), [self-hosted/vllm/pricing.json](../../self-hosted/vllm/pricing.json) | Unchanged | None. |

---

## New datasets

- **cli-cli-v1** ([benchmarks/dataset/cli-cli-v1.yaml](../../benchmarks/dataset/cli-cli-v1.yaml), [#224](https://github.com/aarora79/agentic-coding-harness-benchmarks/pull/224)): 21 tasks from closed issues in [cli/cli](https://github.com/cli/cli), the GitHub CLI, written in Go. It tests whether what the harness learned on `mcp-gateway-registry-v2` (a Python service) holds on a codebase nobody tuned it on ([#221](https://github.com/aarora79/agentic-coding-harness-benchmarks/issues/221)). To limit training-data contamination, every fix merged on or after 2026-05-01 and every issue was opened in 2026; fixes shipped in v2.93.0 to v2.102.0. Tiers follow v2's mix: 5 trivial, 5 low, 6 medium, 5 high. Its scores never merge into a v2 table.

---

## Runs on the new dataset

The three Claude 5.5 models on Amazon Bedrock, omp `/swe3`, judged by `openai.gpt-5.6-sol` at high effort ([#225](https://github.com/aarora79/agentic-coding-harness-benchmarks/pull/225)). Source: each model's `run-summary.json` under `benchmarks/swe-benchmark-data/<model>/omp/swe3/cli-cli-v1/`; analysis in [docs/results-cli-cli-v1.md](../results-cli-cli-v1.md).

| Model | Score | Failure counted as 0 | Cost for 21 tasks | Scored |
|---|---:|---:|---:|:-:|
| `claude-opus-5-5` | 85.62 | 85.62 | $45.88 | 21/21 |
| `claude-sonnet-5-5` | 82.50 | 82.50 | $19.11 | 21/21 |
| `claude-haiku-5-5` | 80.82 | 73.12 | $1.99 | 19/21 |

The ranking matches v2. Every model scores higher here, by 3.0 (Sonnet) to 8.1 points (Haiku). Haiku 5.5 failed two high-tier tasks by writing its artifacts outside the folder the prompt named, as it did once on v2; the files are kept, unjudged, under [do-not-include/claude-haiku-5-5-misplaced-artifacts](../../benchmarks/swe-benchmark-data/do-not-include/README.md).

---

## New functionality

### swe-router 0.2.0: routing on a six-question rubric

The skill used to ask the developer's model for a quality floor and one of four tiers in its own words. It now answers the six questions in [rubric.md](../../vend/swe-router/rubric.md) about the task, 0 to 2 each, and `route.py` does the arithmetic. Four answers add up to the task's **bucket**, its difficulty level (easy, moderate, hard, very hard). `blast_radius` and `verification` set the **required quality**. `route.py` then takes the cheapest permitted model whose mean benchmark score on tasks in that bucket clears it, skips any model that failed a benchmark task in that bucket, and keeps the developer on their model when the rubric answers carry low confidence. `--tier` and `--floor` still work for one release.

Each benchmark task's bucket comes from a rubric pass by Claude Opus 5.5, three repeats per task, stored in [docs/metrics/swe-router-rubric-mcp-gateway-registry-v2-claude-opus-5-5.json](../metrics/swe-router-rubric-mcp-gateway-registry-v2-claude-opus-5-5.json). The skill never reads that file; `build_vended_models.py` turns it into the per-bucket columns in `models.json`.

Replayed over the 21 v2 tasks, each routed from columns that leave that task out, against `claude-opus-5` on every task:

| Router | Cost saved | Quality change | Tasks below the required quality |
|---|---:|---:|---:|
| swe-router 0.1 | 70.5% | -5.06 | 9 |
| swe-router 0.2.0 | 74.7% | -3.22 | 4 |

Sources: `today-all.json` and `opus-5-5-absolute-all-completion.json` in [docs/metrics/swe-router-rubric-replays/](../metrics/swe-router-rubric-replays/). On cli/cli, columns built from v2 alone routed the Go tasks with no adjustment and erred towards the stronger model.

How it works, with diagrams: [docs/swe-router-how-it-works.md](../swe-router-how-it-works.md). The rubric scores file: [docs/swe-router-rubric-scores.md](../swe-router-rubric-scores.md). The design and its phase 0 test: [docs/swe-router-rubric-design.md](../swe-router-rubric-design.md), [docs/swe-router-rubric-evaluation.md](../swe-router-rubric-evaluation.md).

[PR #223](https://github.com/aarora79/agentic-coding-harness-benchmarks/pull/223) (phase 0), [PR #227](https://github.com/aarora79/agentic-coding-harness-benchmarks/pull/227) (0.2.0, reviewed as [#226](https://github.com/aarora79/agentic-coding-harness-benchmarks/pull/226))

---

## What's changed

### Skills
- `swe-router` 0.2.0: `rubric.md` and `routing-policy.json` ship with the skill; `SKILL.md` names no measured model, so it no longer goes stale when the data changes; the installer refuses to install without `python3` 3.8+ (#227).
- `run-swe-router-headless.py --mode rubric` scores a dataset's tasks with the rubric and names its output after the dataset and the judge (#223, #225).
- `eval_swe_router.py` gains `--rubric-inputs`, `--columns-from` for zero-shot replays across datasets, and `--min-completion` (#223, #225, #227).

### Documentation and slides
- [docs/results-cli-cli-v1.md](../results-cli-cli-v1.md): the cross-repo answer for #221 (#225).
- The workflow slide, its README image and the ASCII diagram show the rubric flow; both READMEs describe it (#227).

---

## Bug fixes

- `models.json` provenance no longer breaks on a squash merge: it records a content hash of the source data and the latest run date instead of a commit (#222).
- Two rubric passes running side by side no longer share clone folders (#223).

---

## Closed issues

| Issue | Title | Closed by |
|-------|-------|-----------|
| #161 | Model router: an advisory skill that recommends which model to switch to (supersedes /swe-auto) | manual |

Issue #221 (calibrate the quality floors and check the ranking on a second repository) stays open: this release answers it for the three Claude 5.5 models, and the open-weight models remain to run.

---

## Pull requests included

| PR | Title |
|----|-------|
| [#227](https://github.com/aarora79/agentic-coding-harness-benchmarks/pull/227) | swe-router 0.2.0: route on a six-question rubric |
| [#226](https://github.com/aarora79/agentic-coding-harness-benchmarks/pull/226) | swe-router 0.2.0: route on a six-question rubric (reviewed here; merged to its stacked base and re-landed as #227) |
| [#225](https://github.com/aarora79/agentic-coding-harness-benchmarks/pull/225) | cli/cli set A: Claude 5.5 results and the cross-repo answer to #221 |
| [#224](https://github.com/aarora79/agentic-coding-harness-benchmarks/pull/224) | Add the cli/cli dataset for the cross-repo check (#221) |
| [#223](https://github.com/aarora79/agentic-coding-harness-benchmarks/pull/223) | swe-router phase 0: a task rubric, tested against today's skill |
| [#222](https://github.com/aarora79/agentic-coding-harness-benchmarks/pull/222) | Stamp models.json provenance from the data, not from git |

---

## Contributors

Thank you to everyone who contributed to this release:

- **Amit Arora** ([@aarora79](https://github.com/aarora79))

---

**Full Changelog:** [0.3.0...0.4.0](https://github.com/aarora79/agentic-coding-harness-benchmarks/compare/0.3.0...0.4.0)
1 change: 1 addition & 0 deletions docs/release-notes/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ Every tagged release, newest first. Versions follow [Semantic Versioning](https:

| Version | Date | Highlights |
|---------|------|------------|
| [0.4.0](0.4.0.md) | October 2026 | `swe-router` 0.2.0 picks a model from a six-question task rubric (74.7% saved for -3.22 points on v2, against 70.5% and -5.06 before); new cli/cli dataset (21 Go tasks) with Claude Opus, Sonnet and Haiku 5.5 run on it. |
| [0.3.0](0.3.0.md) | October 2026 | Claude Opus 5.5, Sonnet 5.5 and Haiku 5.5 on the omp / `/swe3` frontier (Haiku 5.5 at 72.69 for $0.14 a task); Kimi K3 on Bedrock; `strands` and `strands-harness` added as harnesses; omp can no longer run a different model than requested. |
| [0.2.0](0.2.0.md) | September 2026 | `codex` added as a fifth coding harness; MiniCPM5-2B on the omp / `/swe3` frontier at 42.59 for $0.40 a task; Qwen3.6-35B FP8 on one L40S, held off the frontier on cost basis. |
| [0.1.0](0.1.0.md) | September 2026 | First tagged release: the harness x model x dataset benchmark, 18 models on the omp / `/swe3` headline run, four datasets, three hosting paths. |
Expand Down
Loading