Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ One command per model. The `/benchmark` skill runs the pre-flight checks, the ha
/benchmark provider=bedrock model=claude-opus-5 dataset=dataset/mcp-gateway-registry-v2.yaml agent=omp
```

`agent` names the coding agent that drives the task and defaults to `claude`. Five harnesses are supported: `claude` (Claude Code), `pi`, `omp` (oh-my-pi), `kiro` (kiro-cli) and `codex` (OpenAI Codex, `codex exec --json`). The run charted below used `omp` on the `/swe3` skill, which is the default skill. The same flow runs headless from [`run-e2e-benchmark.sh`](benchmarks/scripts/run-e2e-benchmark.sh) (`--provider bedrock|litellm|vllm --model ... --dataset ... --agent claude|pi|omp|kiro|codex --skill swe2|swe3`). Repeat across your model list, then the generators plot the frontier.
`agent` names the coding agent that drives the task and defaults to `claude`. Six harnesses are supported: `claude` (Claude Code), `pi`, `omp` (oh-my-pi), `kiro` (kiro-cli), `codex` (OpenAI Codex, `codex exec --json`) and `strands` (an agent built on the [Strands Agents](https://strandsagents.com) SDK). The run charted below used `omp` on the `/swe3` skill, which is the default skill. The same flow runs headless from [`run-e2e-benchmark.sh`](benchmarks/scripts/run-e2e-benchmark.sh) (`--provider bedrock|litellm|vllm --model ... --dataset ... --agent claude|pi|omp|kiro|codex|strands --skill swe2|swe3`). Repeat across your model list, then the generators plot the frontier.

![Cost vs. quality Pareto frontier, omp harness on /swe3](docs/images/cost-quality-omp-swe3.png)

Expand Down Expand Up @@ -91,7 +91,7 @@ Install notes, the file-by-file breakdown and the measured results: **[vend/swe-

18 models over 21 tasks. Each task comes from a closed issue in a real repository, pinned to the release before the fix shipped, so the defect is present in the tree the agent clones. An independent judge scores every run 0-100, and the tasks split across four complexity tiers. Earlier runs on other harnesses and datasets stay published as background. They use different task sets, so their scores do not merge into the headline table.

**[All results](docs/results-index.md)** · [Headline run (omp, /swe3, v2)](docs/harness-omp-swe3.md) · [Which model for which task?](docs/model-selection-by-complexity.md) · [Cost per task methodology](docs/cost-per-task-methodology.md) · [v1 dataset](docs/results-swe3.md) · [/swe2 multi-agent](docs/results-swe2.md) · [Cross-harness comparison](docs/agentic-coding-swe-comparison-swe3.md)
**[All results](docs/results-index.md)** · [Headline run (omp, /swe3, v2)](docs/harness-omp-swe3.md) · [Strands harness](docs/harness-strands-swe3.md) · [Which model for which task?](docs/model-selection-by-complexity.md) · [Cost per task methodology](docs/cost-per-task-methodology.md) · [v1 dataset](docs/results-swe3.md) · [/swe2 multi-agent](docs/results-swe2.md) · [Cross-harness comparison](docs/agentic-coding-swe-comparison-swe3.md)

### Does routing pay for itself?

Expand Down Expand Up @@ -158,6 +158,7 @@ Where to read more, by topic:
| [benchmarks/docs/path-self-hosted-vllm.md](benchmarks/docs/path-self-hosted-vllm.md) | Path 3 setup: self-hosting a model on vLLM and pointing the harness at it. |
| [docs/kiro-cli-setup.md](docs/kiro-cli-setup.md) | The kiro-cli harness: install, sign-in, headless use, the Bedrock-managed-only constraint, and how its Kiro-credit spend is calculated. Results: [harness-kiro-cli-swe3.md](docs/harness-kiro-cli-swe3.md). |
| [docs/codex-setup.md](docs/codex-setup.md) | The codex harness: installing it, wiring it to Amazon Bedrock, the Responses-API constraint that decides which models it can reach, and a table of its known failure modes. |
| [docs/strands-setup.md](docs/strands-setup.md) | The Strands harness: installing the optional SDK group, how the runner builds the agent, and the five places it departs from Strands as it ships to finish long tasks on a vLLM endpoint. Results: [harness-strands-swe3.md](docs/harness-strands-swe3.md). |
| [docs/faq/](docs/faq/) | FAQ, one file per question. Wiring each harness to a model, tested on real endpoints: [codex](docs/faq/wiring-codex-to-models.md), [omp](docs/faq/wiring-omp-to-models.md), [Claude Code](docs/faq/wiring-claude-code-to-models.md) -- an open-weight model on Bedrock, one you serve on vLLM, or a model on Bedrock directly. |
| [benchmarks/docs/end-to-end-self-hosted-run.md](benchmarks/docs/end-to-end-self-hosted-run.md) | The full manual run-book for an end-to-end self-hosted benchmark. |
| [self-hosted/vllm/README.md](self-hosted/vllm/README.md) | Standing up a vLLM server: install, tensor parallelism, tool-call parsers, and the serving-config reference. |
Expand Down
2 changes: 2 additions & 0 deletions benchmarks/scripts/gen_agent_report.py
Original file line number Diff line number Diff line change
Expand Up @@ -64,13 +64,15 @@
# omp's own name is oh-my-pi; "omp" is the binary. Spell both out so the
# generated page names the project a reader can go and find.
"omp": "oh-my-pi (omp)",
"strands": "Strands Agents",
}

# Harnesses with an install/configuration page in docs/. Linked from the report
# so a reader who wants to reproduce the run knows where the setup lives.
HARNESS_SETUP_DOCS = {
"omp": ("omp setup", "omp-setup.md"),
"kiro-cli": ("kiro-cli setup", "kiro-cli-setup.md"),
"strands": ("Strands setup", "strands-setup.md"),
}

# Short per-harness code that (with the skill) suffixes chart filenames
Expand Down
5 changes: 4 additions & 1 deletion benchmarks/scripts/runner_config.py
Original file line number Diff line number Diff line change
Expand Up @@ -219,7 +219,10 @@
# suffix (e.g. "[1m]"). The /swe skill strips both to name its artifact folder,
# so the harness must derive the same slug to find the artifacts the skill wrote.
# Matches a leading "<region>.<vendor>." such as "us.anthropic." or "eu.meta.".
_BEDROCK_PREFIX_RE = re.compile(r"^[a-z]{2}\.[a-z0-9-]+\.")
# The region is a two-letter code or one of Bedrock's longer cross-region
# prefixes ("apac.", "us-gov.", "global."); a bare "<vendor>.<model>" such as
# "moonshotai.kimi-k2-thinking" has no region and is left alone (issue #205).
_BEDROCK_PREFIX_RE = re.compile(r"^(?:[a-z]{2}|apac|us-gov|global)\.[a-z0-9-]+\.")
# Matches a trailing bracketed suffix such as "[1m]".
_MODEL_SUFFIX_RE = re.compile(r"\[[^\]]*\]$")
# Matches a trailing Anthropic date+version stamp such as "-20251001-v1:0", so a
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
"fixed-in-1.24.0"
],
"model": "global.anthropic.claude-sonnet-5-5",
"model_slug": "global.anthropic.claude-sonnet-5-5",
"model_slug": "claude-sonnet-5-5",
"agent": "strands",
"skill": "swe3",
"provider": "bedrock",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
"fixed-in-1.29.0"
],
"model": "global.anthropic.claude-sonnet-5-5",
"model_slug": "global.anthropic.claude-sonnet-5-5",
"model_slug": "claude-sonnet-5-5",
"agent": "strands",
"skill": "swe3",
"provider": "bedrock",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
"fixed-in-1.26.0"
],
"model": "global.anthropic.claude-sonnet-5-5",
"model_slug": "global.anthropic.claude-sonnet-5-5",
"model_slug": "claude-sonnet-5-5",
"agent": "strands",
"skill": "swe3",
"provider": "bedrock",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
"fixed-in-1.24.0"
],
"model": "global.anthropic.claude-sonnet-5-5",
"model_slug": "global.anthropic.claude-sonnet-5-5",
"model_slug": "claude-sonnet-5-5",
"agent": "strands",
"skill": "swe3",
"provider": "bedrock",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
"fixed-in-1.0.21"
],
"model": "global.anthropic.claude-sonnet-5-5",
"model_slug": "global.anthropic.claude-sonnet-5-5",
"model_slug": "claude-sonnet-5-5",
"agent": "strands",
"skill": "swe3",
"provider": "bedrock",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
"fixed-in-1.0.22"
],
"model": "global.anthropic.claude-sonnet-5-5",
"model_slug": "global.anthropic.claude-sonnet-5-5",
"model_slug": "claude-sonnet-5-5",
"agent": "strands",
"skill": "swe3",
"provider": "bedrock",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
"fixed-in-1.0.20"
],
"model": "global.anthropic.claude-sonnet-5-5",
"model_slug": "global.anthropic.claude-sonnet-5-5",
"model_slug": "claude-sonnet-5-5",
"agent": "strands",
"skill": "swe3",
"provider": "bedrock",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
"fixed-in-1.28.0"
],
"model": "global.anthropic.claude-sonnet-5-5",
"model_slug": "global.anthropic.claude-sonnet-5-5",
"model_slug": "claude-sonnet-5-5",
"agent": "strands",
"skill": "swe3",
"provider": "bedrock",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
"fixed-in-v1.0.19"
],
"model": "global.anthropic.claude-sonnet-5-5",
"model_slug": "global.anthropic.claude-sonnet-5-5",
"model_slug": "claude-sonnet-5-5",
"agent": "strands",
"skill": "swe3",
"provider": "bedrock",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
"fixed-in-1.25.0"
],
"model": "global.anthropic.claude-sonnet-5-5",
"model_slug": "global.anthropic.claude-sonnet-5-5",
"model_slug": "claude-sonnet-5-5",
"agent": "strands",
"skill": "swe3",
"provider": "bedrock",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
"fixed-in-1.29.0"
],
"model": "global.anthropic.claude-sonnet-5-5",
"model_slug": "global.anthropic.claude-sonnet-5-5",
"model_slug": "claude-sonnet-5-5",
"agent": "strands",
"skill": "swe3",
"provider": "bedrock",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
"fixed-in-1.24.4"
],
"model": "global.anthropic.claude-sonnet-5-5",
"model_slug": "global.anthropic.claude-sonnet-5-5",
"model_slug": "claude-sonnet-5-5",
"agent": "strands",
"skill": "swe3",
"provider": "bedrock",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
"fixed-in-1.25.0"
],
"model": "global.anthropic.claude-sonnet-5-5",
"model_slug": "global.anthropic.claude-sonnet-5-5",
"model_slug": "claude-sonnet-5-5",
"agent": "strands",
"skill": "swe3",
"provider": "bedrock",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
"fixed-in-1.28.0"
],
"model": "global.anthropic.claude-sonnet-5-5",
"model_slug": "global.anthropic.claude-sonnet-5-5",
"model_slug": "claude-sonnet-5-5",
"agent": "strands",
"skill": "swe3",
"provider": "bedrock",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
"fixed-in-1.28.0"
],
"model": "global.anthropic.claude-sonnet-5-5",
"model_slug": "global.anthropic.claude-sonnet-5-5",
"model_slug": "claude-sonnet-5-5",
"agent": "strands",
"skill": "swe3",
"provider": "bedrock",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
"fixed-in-1.28.0"
],
"model": "global.anthropic.claude-sonnet-5-5",
"model_slug": "global.anthropic.claude-sonnet-5-5",
"model_slug": "claude-sonnet-5-5",
"agent": "strands",
"skill": "swe3",
"provider": "bedrock",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
"fixed-in-1.28.0"
],
"model": "global.anthropic.claude-sonnet-5-5",
"model_slug": "global.anthropic.claude-sonnet-5-5",
"model_slug": "claude-sonnet-5-5",
"agent": "strands",
"skill": "swe3",
"provider": "bedrock",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
"fixed-in-1.28.0"
],
"model": "global.anthropic.claude-sonnet-5-5",
"model_slug": "global.anthropic.claude-sonnet-5-5",
"model_slug": "claude-sonnet-5-5",
"agent": "strands",
"skill": "swe3",
"provider": "bedrock",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
"fixed-in-1.28.0"
],
"model": "global.anthropic.claude-sonnet-5-5",
"model_slug": "global.anthropic.claude-sonnet-5-5",
"model_slug": "claude-sonnet-5-5",
"agent": "strands",
"skill": "swe3",
"provider": "bedrock",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
"fixed-in-1.0.20"
],
"model": "global.anthropic.claude-sonnet-5-5",
"model_slug": "global.anthropic.claude-sonnet-5-5",
"model_slug": "claude-sonnet-5-5",
"agent": "strands",
"skill": "swe3",
"provider": "bedrock",
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"model": "global.anthropic.claude-sonnet-5-5",
"model_slug": "global.anthropic.claude-sonnet-5-5",
"model_slug": "claude-sonnet-5-5",
"agent": "strands",
"skill": "swe3",
"scope": "mcp-gateway-registry-v2",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
"fixed-in-1.24.0"
],
"model": "global.anthropic.claude-sonnet-5-5",
"model_slug": "global.anthropic.claude-sonnet-5-5",
"model_slug": "claude-sonnet-5-5",
"agent": "strands",
"skill": "swe3",
"provider": "bedrock",
Expand Down
15 changes: 15 additions & 0 deletions benchmarks/tests/test_runner_config.py
Original file line number Diff line number Diff line change
Expand Up @@ -319,6 +319,21 @@ def test_bedrock_prefix_stripped_without_suffix(self) -> None:
def test_other_region_and_vendor_prefix_stripped(self) -> None:
self.assertEqual(model_to_slug("eu.meta.llama3-70b"), "llama3-70b")

def test_long_cross_region_prefixes_stripped(self) -> None:
# Bedrock's longer cross-region prefixes (issue #205).
for model in (
"global.anthropic.claude-sonnet-5-5",
"apac.anthropic.claude-sonnet-5-5",
"us-gov.anthropic.claude-sonnet-5-5",
):
with self.subTest(model=model):
self.assertEqual(model_to_slug(model), "claude-sonnet-5-5")

def test_global_prefix_with_suffix_stripped(self) -> None:
self.assertEqual(
model_to_slug("global.anthropic.claude-opus-5[1m]"), "claude-opus-5"
)

def test_dated_haiku_folds_onto_short_slug(self) -> None:
# A dated Bedrock id must slug to the same short folder as its short name,
# so a re-run lands in the existing claude-haiku-4-5/ tree.
Expand Down
57 changes: 57 additions & 0 deletions docs/harness-strands-swe3.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# Results: Strands Agents harness (swe3)

Benchmark results for every model run under the **Strands Agents** coding agent with the **swe3** skill on `mcp-gateway-registry-v2`, generated from the committed `run-summary.json` files. Regenerate with `uv run scripts/gen_agent_report.py --harness strands --skill swe3 --repo mcp-gateway-registry-v2`. See [Strands setup](strands-setup.md) for install and configuration. Companion to the cross-harness comparison [agentic-coding-swe-comparison-swe3.md](agentic-coding-swe-comparison-swe3.md).

## Results by model

| Model | Mean score | Completed | Input | Output | Cache read | Cache write | Tokens processed† | Wall-clock | Run cost | Cost basis* |
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|
| claude-opus-5-5 | 81.21 | 21/21 | 1,660 | 980,873 | 74,748,022 | 2,483,522 | 78,214,077 | 184.8m | $51.69 | metered (Bedrock) |
| claude-sonnet-5-5 | 76.73 | 21/21 | 1,122 | 681,675 | 43,052,826 | 1,933,669 | 45,669,292 | 74.1m | $20.26 | metered (Bedrock) |
| claude-haiku-4-5 | 51.20 | 20/21 | 46,996 | 687,756 | 68,911,255 | 1,935,109 | 71,581,116 | 136.3m | $14.08 | metered (Bedrock) |
| minicpm5-2b | 45.28 | 21/21 | 3,245,301 | 689,816 | 156,682,768 | 0 | 160,617,885 | 167.4m | $6.93 | hardware-derived (g6e.4xlarge) |

\* **Cost basis differs by row and the dollars are NOT directly comparable.** _hardware-derived (throughput)_ (self-hosted vLLM): a rented GPU has no per-token bill, so cost is the model's blended cost-per-token -- the cheapest concurrency level of ITS OWN throughput sweep -- times the tokens this run processed. Each row is priced at the rate of the instance that model was actually served on, named in this column, at that instance's rate in self-hosted/vllm/pricing.json. **The instance differs by row, so a self-hosted dollar figure is the cost of that model's work on ITS OWN hardware, not a common basis**; comparing two self-hosted rows compares two model-plus-hardware pairings rather than the models alone. This prices the real work done, unlike a wall-clock estimate that would also charge idle agent-thinking time. _metered (Bedrock)_: a hosted API's real per-token bill, summed over the run. It is a metered invoice, not a hardware estimate, and (unlike the self-hosted rows) it benefits from Bedrock prompt caching. See [cost-per-task-methodology.md](cost-per-task-methodology.md).

† **Tokens processed** counts input + output + cache-read + cache-write -- all tokens the model actually processed, not just fresh input+output. On the Bedrock path a task often reports only ~2 fresh input tokens with the rest served from prompt cache, so counting input+output alone would understate the real work ~100x. (Self-hosted rows report their cache reuse via server-side Prometheus counters, folded in here where present.)

A task scoring 0 (missing/empty artifacts) is a model failure, excluded from the mean but counted in `Completed`. A model with 0 scored tasks did not complete any task under this harness.

## Charts

### Cost vs. quality (Pareto frontier)

![Cost vs quality, Strands Agents harness](images/cost-quality-strands-swe3.png)

### Quality by dimension (radar)

![Quality radar, Strands Agents harness](images/quality-radar-strands-swe3.png)

### Cost vs. accuracy (bubble area = tokens)

x = cost per task, y = mean score, bubble area = total tokens processed, color = hosting basis (metered Bedrock vs hardware-derived self-hosted -- NOT directly comparable as raw dollars; see the cost note above).

![Cost vs accuracy, Strands Agents harness](images/cost-accuracy-bubble-strands-swe3.png)

## Notes on these runs

This section is written by hand, below the generated part, and a regenerate does not keep it. Add it back after running the generator.

The rows used different versions of the Strands runner.

| Model | Run date | Path | Runner |
|---|---|---|---|
| claude-opus-5-5 | 2026-10-02 ([#195](https://github.com/aarora79/agentic-coding-harness-benchmarks/pull/195)) | Amazon Bedrock | Strands as first added in [#194](https://github.com/aarora79/agentic-coding-harness-benchmarks/pull/194): skill loaded through the `AgentSkills` plugin, no tool-output cap, no tool-name map |
| claude-haiku-4-5 | 2026-10-04 ([#206](https://github.com/aarora79/agentic-coding-harness-benchmarks/pull/206)) | Amazon Bedrock | the same runner as Opus 5.5 |
| claude-sonnet-5-5 | 2026-10-04 ([#203](https://github.com/aarora79/agentic-coding-harness-benchmarks/pull/203)) | Amazon Bedrock, `global.` cross-region profile | the same runner as Opus 5.5 |
| minicpm5-2b | 2026-10-04 ([#196](https://github.com/aarora79/agentic-coding-harness-benchmarks/issues/196)) | self-hosted vLLM, 131,072-token window | the runner after [#204](https://github.com/aarora79/agentic-coding-harness-benchmarks/pull/204), with the five changes listed in [strands-setup.md](strands-setup.md#where-the-runner-departs-from-strands-as-it-ships) |

Opus 5.5 scored 81.21 under Strands and 81.16 under omp on the same 21 tasks, and cost $2.46 a task under Strands against $4.44 under omp (`benchmarks/swe-benchmark-data/claude-opus-5-5/{strands,omp}/swe3/mcp-gateway-registry-v2/run-summary.json`). A strong model on Bedrock did not need the changes in #204.

MiniCPM5-2B did. Before #204, Strands lost task attempts on this model to rejected summaries, single large tool results, skipped skill loads and Claude Code tool names. The [model guide](../self-hosted/vllm/models/minicpm5-2b.md#measured-results-strands-swe3-mcp-gateway-registry-v2) compares the run with omp's on the same model: 45.28 over 21 tasks against 42.59 over 19.

Haiku 4.5 scored 51.20 over 20 of 21 tasks under Strands, against 56.18 over 21 under omp, at $0.70 a task against $0.76. Its one failure, `honor-cloud-provider-override-in-ui`, ran 973 turns and then crashed with `RecursionError: maximum recursion depth exceeded`, close to Python's default limit of 1,000 frames, with no artifacts written (`benchmarks/swe-benchmark-data/claude-haiku-4-5/strands/swe3/mcp-gateway-registry-v2/honor-cloud-provider-override-in-ui/metrics.json`). None of the changes in #204 address this.

Sonnet 5.5 has no omp run to compare against. Its results first landed in a folder named after the full profile id, `global.anthropic.claude-sonnet-5-5`, because the folder-name rule did not strip the `global.` prefix; [#205](https://github.com/aarora79/agentic-coding-harness-benchmarks/issues/205) fixed the rule and moved the folder to `claude-sonnet-5-5`.

Two sanity runs on the one-task `hello-world` dataset (`benchmarks/dataset/hello-world.yaml`) checked the harness when it was added in #194. They use a different dataset, so they stay out of the table: `claude-opus-4-8` scored 78.2, and `claude-haiku-4-5` wrote all six artifacts but was not judged.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/images/cost-quality-strands-swe3.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/images/quality-radar-strands-swe3.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Loading