From 84967a044104e3639192885616e8a5a698c58edc Mon Sep 17 00:00:00 2001 From: Amit Arora Date: Thu, 15 Oct 2026 00:00:51 +0000 Subject: [PATCH] docs: add 0.4.0 release notes --- docs/release-notes/0.4.0.md | 139 +++++++++++++++++++++++++++++++++++ docs/release-notes/README.md | 1 + 2 files changed, 140 insertions(+) create mode 100644 docs/release-notes/0.4.0.md diff --git a/docs/release-notes/0.4.0.md b/docs/release-notes/0.4.0.md new file mode 100644 index 00000000..808f2ba0 --- /dev/null +++ b/docs/release-notes/0.4.0.md @@ -0,0 +1,139 @@ +# Release 0.4.0 - swe-router routes on a rubric, and a second repository + +**October 2026** + +Version bump: **MINOR** -- adds a second dataset (21 tasks from cli/cli, the GitHub CLI, in Go) with the three Claude 5.5 models run on it, and moves the `swe-router` skill to 0.2.0, which picks a model from a six-question task rubric. Every published benchmark score and cost from 0.3.0 stands unchanged. + +--- + +## Upgrading from 0.3.0 + +This repo is cloned and run, not deployed, so upgrading is a pull plus a dependency sync. + +```bash +git pull origin main +git checkout 0.4.0 + +cd benchmarks && uv sync && cd .. +cd self-hosted/vllm && uv sync && cd ../.. +``` + +Anyone who installed `swe-router` should re-run its installer. The new `route.py` reads only the new `models.json`, so the seven files must come from the same release, which the installer guarantees: + +```bash +curl -sL https://raw.githubusercontent.com/aarora79/agentic-coding-harness-benchmarks/0.4.0/vend/swe-router/install.sh | bash -s -- --ref 0.4.0 +``` + +An edited `allowed-models.txt` is kept. The installer now needs `python3` 3.8 or later on PATH, and stops before downloading anything without it. + +### Methodology changes + +None. Numbers remain comparable with 0.3.0. No frontier JSON under [docs/metrics/](../metrics/) changed in this window, and the cli/cli runs live in their own output scope (`cli-cli-v1`), outside every v2 table and chart. + +The router's way of choosing a model changed, which is a change to a skill, not to how a benchmark number is produced. See [New functionality](#new-functionality). + +### Config changes + +| File | Change | Action needed | +|------|--------|---------------| +| [vend/swe-router/routing-policy.json](../../vend/swe-router/routing-policy.json) | New in the skill (moved from `benchmarks/config/swe-router/`). Maps blast radius and verification to a required quality (60 / 70 / 75, +5 when nothing would catch a mistake), sets the difficulty bucket ranges, the 3-point tie band, and `min_completion_in_bucket` (1.0) | None. A platform team may edit it; the installer keeps an edited copy on upgrade. | +| [vend/swe-router/models.json](../../vend/swe-router/models.json) | Schema 1.0 to 2.0 across this window: 1.1 renamed `provenance.source_commit` to `source_sha256` ([#222](https://github.com/aarora79/agentic-coding-harness-benchmarks/pull/222)); 2.0 adds `score_by_difficulty`, `completion_by_difficulty` and `provenance.rubric_scores` ([#227](https://github.com/aarora79/agentic-coding-harness-benchmarks/pull/227)) | Re-install the skill. A 0.1 `route.py` refuses a schema-2 file. | +| [benchmarks/scripts/runner_config.py](../../benchmarks/scripts/runner_config.py), [benchmarks/config/runner.example.yaml](../../benchmarks/config/runner.example.yaml), [self-hosted/vllm/pricing.json](../../self-hosted/vllm/pricing.json) | Unchanged | None. | + +--- + +## New datasets + +- **cli-cli-v1** ([benchmarks/dataset/cli-cli-v1.yaml](../../benchmarks/dataset/cli-cli-v1.yaml), [#224](https://github.com/aarora79/agentic-coding-harness-benchmarks/pull/224)): 21 tasks from closed issues in [cli/cli](https://github.com/cli/cli), the GitHub CLI, written in Go. It tests whether what the harness learned on `mcp-gateway-registry-v2` (a Python service) holds on a codebase nobody tuned it on ([#221](https://github.com/aarora79/agentic-coding-harness-benchmarks/issues/221)). To limit training-data contamination, every fix merged on or after 2026-05-01 and every issue was opened in 2026; fixes shipped in v2.93.0 to v2.102.0. Tiers follow v2's mix: 5 trivial, 5 low, 6 medium, 5 high. Its scores never merge into a v2 table. + +--- + +## Runs on the new dataset + +The three Claude 5.5 models on Amazon Bedrock, omp `/swe3`, judged by `openai.gpt-5.6-sol` at high effort ([#225](https://github.com/aarora79/agentic-coding-harness-benchmarks/pull/225)). Source: each model's `run-summary.json` under `benchmarks/swe-benchmark-data//omp/swe3/cli-cli-v1/`; analysis in [docs/results-cli-cli-v1.md](../results-cli-cli-v1.md). + +| Model | Score | Failure counted as 0 | Cost for 21 tasks | Scored | +|---|---:|---:|---:|:-:| +| `claude-opus-5-5` | 85.62 | 85.62 | $45.88 | 21/21 | +| `claude-sonnet-5-5` | 82.50 | 82.50 | $19.11 | 21/21 | +| `claude-haiku-5-5` | 80.82 | 73.12 | $1.99 | 19/21 | + +The ranking matches v2. Every model scores higher here, by 3.0 (Sonnet) to 8.1 points (Haiku). Haiku 5.5 failed two high-tier tasks by writing its artifacts outside the folder the prompt named, as it did once on v2; the files are kept, unjudged, under [do-not-include/claude-haiku-5-5-misplaced-artifacts](../../benchmarks/swe-benchmark-data/do-not-include/README.md). + +--- + +## New functionality + +### swe-router 0.2.0: routing on a six-question rubric + +The skill used to ask the developer's model for a quality floor and one of four tiers in its own words. It now answers the six questions in [rubric.md](../../vend/swe-router/rubric.md) about the task, 0 to 2 each, and `route.py` does the arithmetic. Four answers add up to the task's **bucket**, its difficulty level (easy, moderate, hard, very hard). `blast_radius` and `verification` set the **required quality**. `route.py` then takes the cheapest permitted model whose mean benchmark score on tasks in that bucket clears it, skips any model that failed a benchmark task in that bucket, and keeps the developer on their model when the rubric answers carry low confidence. `--tier` and `--floor` still work for one release. + +Each benchmark task's bucket comes from a rubric pass by Claude Opus 5.5, three repeats per task, stored in [docs/metrics/swe-router-rubric-mcp-gateway-registry-v2-claude-opus-5-5.json](../metrics/swe-router-rubric-mcp-gateway-registry-v2-claude-opus-5-5.json). The skill never reads that file; `build_vended_models.py` turns it into the per-bucket columns in `models.json`. + +Replayed over the 21 v2 tasks, each routed from columns that leave that task out, against `claude-opus-5` on every task: + +| Router | Cost saved | Quality change | Tasks below the required quality | +|---|---:|---:|---:| +| swe-router 0.1 | 70.5% | -5.06 | 9 | +| swe-router 0.2.0 | 74.7% | -3.22 | 4 | + +Sources: `today-all.json` and `opus-5-5-absolute-all-completion.json` in [docs/metrics/swe-router-rubric-replays/](../metrics/swe-router-rubric-replays/). On cli/cli, columns built from v2 alone routed the Go tasks with no adjustment and erred towards the stronger model. + +How it works, with diagrams: [docs/swe-router-how-it-works.md](../swe-router-how-it-works.md). The rubric scores file: [docs/swe-router-rubric-scores.md](../swe-router-rubric-scores.md). The design and its phase 0 test: [docs/swe-router-rubric-design.md](../swe-router-rubric-design.md), [docs/swe-router-rubric-evaluation.md](../swe-router-rubric-evaluation.md). + +[PR #223](https://github.com/aarora79/agentic-coding-harness-benchmarks/pull/223) (phase 0), [PR #227](https://github.com/aarora79/agentic-coding-harness-benchmarks/pull/227) (0.2.0, reviewed as [#226](https://github.com/aarora79/agentic-coding-harness-benchmarks/pull/226)) + +--- + +## What's changed + +### Skills +- `swe-router` 0.2.0: `rubric.md` and `routing-policy.json` ship with the skill; `SKILL.md` names no measured model, so it no longer goes stale when the data changes; the installer refuses to install without `python3` 3.8+ (#227). +- `run-swe-router-headless.py --mode rubric` scores a dataset's tasks with the rubric and names its output after the dataset and the judge (#223, #225). +- `eval_swe_router.py` gains `--rubric-inputs`, `--columns-from` for zero-shot replays across datasets, and `--min-completion` (#223, #225, #227). + +### Documentation and slides +- [docs/results-cli-cli-v1.md](../results-cli-cli-v1.md): the cross-repo answer for #221 (#225). +- The workflow slide, its README image and the ASCII diagram show the rubric flow; both READMEs describe it (#227). + +--- + +## Bug fixes + +- `models.json` provenance no longer breaks on a squash merge: it records a content hash of the source data and the latest run date instead of a commit (#222). +- Two rubric passes running side by side no longer share clone folders (#223). + +--- + +## Closed issues + +| Issue | Title | Closed by | +|-------|-------|-----------| +| #161 | Model router: an advisory skill that recommends which model to switch to (supersedes /swe-auto) | manual | + +Issue #221 (calibrate the quality floors and check the ranking on a second repository) stays open: this release answers it for the three Claude 5.5 models, and the open-weight models remain to run. + +--- + +## Pull requests included + +| PR | Title | +|----|-------| +| [#227](https://github.com/aarora79/agentic-coding-harness-benchmarks/pull/227) | swe-router 0.2.0: route on a six-question rubric | +| [#226](https://github.com/aarora79/agentic-coding-harness-benchmarks/pull/226) | swe-router 0.2.0: route on a six-question rubric (reviewed here; merged to its stacked base and re-landed as #227) | +| [#225](https://github.com/aarora79/agentic-coding-harness-benchmarks/pull/225) | cli/cli set A: Claude 5.5 results and the cross-repo answer to #221 | +| [#224](https://github.com/aarora79/agentic-coding-harness-benchmarks/pull/224) | Add the cli/cli dataset for the cross-repo check (#221) | +| [#223](https://github.com/aarora79/agentic-coding-harness-benchmarks/pull/223) | swe-router phase 0: a task rubric, tested against today's skill | +| [#222](https://github.com/aarora79/agentic-coding-harness-benchmarks/pull/222) | Stamp models.json provenance from the data, not from git | + +--- + +## Contributors + +Thank you to everyone who contributed to this release: + +- **Amit Arora** ([@aarora79](https://github.com/aarora79)) + +--- + +**Full Changelog:** [0.3.0...0.4.0](https://github.com/aarora79/agentic-coding-harness-benchmarks/compare/0.3.0...0.4.0) diff --git a/docs/release-notes/README.md b/docs/release-notes/README.md index 0bacf105..ed15076f 100644 --- a/docs/release-notes/README.md +++ b/docs/release-notes/README.md @@ -4,6 +4,7 @@ Every tagged release, newest first. Versions follow [Semantic Versioning](https: | Version | Date | Highlights | |---------|------|------------| +| [0.4.0](0.4.0.md) | October 2026 | `swe-router` 0.2.0 picks a model from a six-question task rubric (74.7% saved for -3.22 points on v2, against 70.5% and -5.06 before); new cli/cli dataset (21 Go tasks) with Claude Opus, Sonnet and Haiku 5.5 run on it. | | [0.3.0](0.3.0.md) | October 2026 | Claude Opus 5.5, Sonnet 5.5 and Haiku 5.5 on the omp / `/swe3` frontier (Haiku 5.5 at 72.69 for $0.14 a task); Kimi K3 on Bedrock; `strands` and `strands-harness` added as harnesses; omp can no longer run a different model than requested. | | [0.2.0](0.2.0.md) | September 2026 | `codex` added as a fifth coding harness; MiniCPM5-2B on the omp / `/swe3` frontier at 42.59 for $0.40 a task; Qwen3.6-35B FP8 on one L40S, held off the frontier on cost basis. | | [0.1.0](0.1.0.md) | September 2026 | First tagged release: the harness x model x dataset benchmark, 18 models on the omp / `/swe3` headline run, four datasets, three hosting paths. |