Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .claude/skills/benchmark/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ Two optional inputs:
| `claude` | Claude Code, `claude -p`. The default. | every provider |
| `pi` | the pi coding agent, `pi -p --mode json` | `endpoint` and `bedrock` (it bundles the AWS SDK bedrock-runtime client) |
| `codex` | OpenAI Codex, `codex exec --json` | `endpoint` (via `OPENAI_BASE_URL` / `OPENAI_API_KEY`) and `bedrock` |
| `strands` | [Strands Agents SDK](../../../docs/strands-setup.md). No CLI binary: it runs in-process from `strands_agent_runner.py` | `endpoint` and `bedrock` |
| `kiro` | [kiro-cli](../../../docs/kiro-cli-setup.md), which drives Kiro's own managed models | `kiro` only, and it forces `--provider kiro` for you |

`claude` remains the default, so only ask if the user brings it up. Two things worth knowing if they do: `omp` is what the published frontier was measured with, and only the `claude` agent gets the `--stream` live trace (the others emit their own event stream or plain text).
Expand Down Expand Up @@ -177,6 +178,8 @@ aws sts get-caller-identity

**3a. Two CLIs must be installed: the chosen agent, and the judge.** The harness runs the agent binary to produce the artifacts, and the judge runs `codex exec` to score them. The agent binary follows `{agent}`: `claude -p`, `omp -p --mode json`, `pi -p --mode json`, `codex exec --json`, or `kiro-cli chat --no-interactive`. Confirm the one in play, plus codex:

`--agent strands` is the exception: it has no binary, because it runs in-process from the Strands Agents SDK. It needs an optional uv dependency group instead, and the orchestrator installs it (`uv sync --group strands`) when it is missing. See [docs/strands-setup.md](../../../docs/strands-setup.md). Only codex needs to be on PATH for that agent.

```bash
command -v {agent-binary} && command -v codex || echo "MISSING a required CLI"
```
Expand Down Expand Up @@ -229,7 +232,7 @@ Run the end-to-end script from `benchmarks/`. It re-runs every pre-flight check
```bash
cd benchmarks
./scripts/run-e2e-benchmark.sh --provider {provider} --model {model} --dataset {dataset} \
[--agent claude|pi|omp|codex|kiro] [--skill swe3|swe2] [--yes] [--count N] [--skip-judge]
[--agent claude|pi|omp|codex|strands|kiro] [--skill swe3|swe2] [--yes] [--count N] [--skip-judge]
```

Tell the user, before it runs:
Expand Down
4 changes: 3 additions & 1 deletion .claude/skills/onboard-benchmark/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,9 @@ Two facts decide everything downstream. **Ask; do not assume, and do not infer f
| Anthropic Messages (`POST /v1/messages`) | `claude` | Claude Code drives `ANTHROPIC_BASE_URL` and needs the Anthropic route |
| OpenAI Responses (`POST /v1/responses`) | `codex` | Recent codex speaks only the Responses API, not chat-completions |
| Both chat-completions and Anthropic | `omp` | It produced this repository's headline results |
| Native Amazon Bedrock, no gateway | any of `claude`, `pi`, `omp`, `codex` | Use `--provider bedrock` and ambient AWS credentials |
| Native Amazon Bedrock, no gateway | any of `claude`, `pi`, `omp`, `codex`, `strands` | Use `--provider bedrock` and ambient AWS credentials |

`strands` (the [Strands Agents SDK](../../../docs/strands-setup.md)) is a further option on either provider. Unlike the others it has no CLI binary, so there is nothing to install on PATH: it runs in-process and the orchestrator installs its optional dependency group with `uv sync --group strands`.

Two notes on `codex` as the harness. It is also the judge, so running `--agent codex` has one tool on both sides of the run; keep the judge model different from the model under test. And on a self-hosted vLLM endpoint it needs a Responses-safe tool-call parser (`qwen3_coder` or `hermes` are verified; seven others crash on the flat Responses tool shape, issue #183).

Expand Down
1 change: 1 addition & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -527,6 +527,7 @@ Read the doc that covers what you are about to do rather than rediscovering it.
- [docs/omp-setup.md](docs/omp-setup.md) -- installing `omp` (oh-my-pi), the harness behind the headline results, and the three ways it differs from `pi`. Read it before `--agent omp`: its config is YAML, it has no `--skill` flag, and it hangs on an inherited stdin.
- [docs/kiro-cli-setup.md](docs/kiro-cli-setup.md) -- kiro-cli's own sign-in and its credit-based cost basis.
- [docs/codex-setup.md](docs/codex-setup.md) -- installing `codex`, wiring it to Bedrock, and the constraint that decides which models `--agent codex` can reach: it speaks only the Responses API, which Bedrock serves for `openai.*` models alone. Its failure-mode table is the fast answer to a hung run, a `404 ... does not exist` on an endpoint, or a request retried five times. Per-path recipes: [docs/faq/wiring-codex-to-models.md](docs/faq/wiring-codex-to-models.md).
- [docs/strands-setup.md](docs/strands-setup.md) -- `--agent strands`, the Strands Agents SDK harness. Strands has no CLI, so the repo's [strands_agent_runner.py](benchmarks/scripts/strands_agent_runner.py) is the agent process; it needs the optional `uv sync --group strands` and reports token counts that the harness prices from the Bedrock price table, as for codex.
- [.claude/skills/vllm-setup/p5en-h200-cuda-fixes.md](.claude/skills/vllm-setup/p5en-h200-cuda-fixes.md) -- the one-time driver, NVMe and CUDA-JIT fixes an 8-GPU NVSwitch node needs. `setup-machine.sh` points here when it detects one, and vLLM fails at KV-cache init without them.
- [docs/getting-started.md](docs/getting-started.md) -- the guided tour from a fresh checkout to a first run, for anyone who wants the narrative rather than the reference pages above.
- [.claude/skills/onboard-benchmark/SKILL.md](.claude/skills/onboard-benchmark/SKILL.md) -- the `onboard-benchmark` skill, for a first run on **someone else's** infrastructure: their models behind a corporate gateway, their private or enterprise-hosted repositories, their own dataset. It covers what `benchmark` assumes is already true, and its first step is the one people skip -- confirming a single machine reaches both the model endpoint and the code host, since every task calls both.
Expand Down
21 changes: 12 additions & 9 deletions benchmarks/config/runner.example.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -100,18 +100,21 @@
# =============================================================================

# Coding agent that drives the task: "claude" (Claude Code, the default), "pi"
# (the pi coding agent), "omp" (oh-my-pi, a fork of pi), "kiro" (kiro-cli), or
# "codex" (OpenAI Codex, `codex exec --json`). The task, artifacts, and judge are
# identical for all of them; only the agent binary and invocation differ. All
# support provider=bedrock; all except kiro also support provider=endpoint (kiro
# drives Kiro's own managed models and cannot target a custom endpoint).
# (the pi coding agent), "omp" (oh-my-pi, a fork of pi), "kiro" (kiro-cli),
# "codex" (OpenAI Codex, `codex exec --json`), or "strands" (a Strands Agents SDK
# agent, run through scripts/strands_agent_runner.py). The task, artifacts, and
# judge are identical for all of them; only the agent binary and invocation
# differ. All support provider=bedrock; all except kiro also support
# provider=endpoint (kiro drives Kiro's own managed models and cannot target a
# custom endpoint). strands needs the optional dependency group: run
# `uv sync --group strands` first (docs/strands-setup.md).
# Usually left at the default and switched per run with --agent pi.
#
# Cost accounting differs by agent: claude/pi/omp report a billed cost directly,
# kiro reports credits (see kiro_dollars_per_credit), and codex reports only
# token counts, so its cost is derived from the local price table in
# scripts/bedrock_pricing.py. A codex model missing from that table yields a
# null cost rather than a misleading 0.
# kiro reports credits (see kiro_dollars_per_credit), and codex and strands
# report only token counts, so their cost is derived from the local price table
# in scripts/bedrock_pricing.py. A model missing from that table yields a null
# cost rather than a misleading 0.
agent: claude

# SWE skill: "swe3" (default) or "swe2". swe3 is SINGLE-AGENT: identical six
Expand Down
9 changes: 9 additions & 0 deletions benchmarks/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,15 @@ dev = [
"types-requests>=2.32.0",
"types-pyyaml>=6.0",
]
# Only agent=strands needs this: scripts/strands_agent_runner.py drives the task
# with the Strands Agents SDK. Kept out of the default install so the other
# harnesses do not pull it in. Install with `uv sync --group strands`. The
# [openai] extra is the client for OpenAI-compatible endpoints (vLLM, LiteLLM).
# Floor at 1.57: the vended shell/file_editor tools and the AgentSkills plugin
# the runner uses are not in older releases.
strands = [
"strands-agents[openai]>=1.57.1,<2",
]

# These scripts drive an external coding assistant (Claude Code) and score its
# artifacts; they do not import the model runtimes. Install them in their own
Expand Down
199 changes: 185 additions & 14 deletions benchmarks/scripts/bedrock_pricing.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
"""Amazon Bedrock price table and cost helpers for the benchmark harness.

Used by the codex agent path to derive total_cost_usd from token counts,
since codex exec does not report a billed cost itself.
Used by the codex and strands agent paths to derive total_cost_usd from token
counts, since neither reports a billed cost itself, and as the fallback for any
agent that reports none.

Provenance of the rates:
- Rates were read directly from https://aws.amazon.com/bedrock/pricing/ on the
Expand All @@ -10,7 +11,13 @@
(cross-region inference, global profile) Long Context Window (1M) table, the
tier codex exec uses when it routes through bedrock-mantle; the run token
counts confirmed it (all tasks exceeded 272K input tokens).
- Cache write is the 30-minute TTL rate.
- Cache write is the 30-minute TTL rate for the openai.* rows.
- The anthropic.* rows come from the AWS Price List API (service
AmazonBedrockFoundationModels, us-east-1, OnDemand) on the date in
``ANTHROPIC_PRICES_AS_OF``. Each model has a Regional row (plain key) and a
Global cross-region row (``global.`` key); cache write is the 5-minute TTL
rate. Re-read them with ``aws pricing get-products --region us-east-1
--service-code AmazonBedrockFoundationModels`` filtered on ``servicename``.

Not every model on Bedrock supports prompt caching. Qwen does not: a Converse
call carrying a ``cachePoint`` block is rejected with ``AccessDeniedException:
Expand All @@ -24,14 +31,16 @@

Public API:
cost_usd(model, input_tokens, output_tokens, cache_read_tokens, cache_write_tokens)
PRICES, PRICES_AS_OF
PRICES, PRICES_AS_OF, ANTHROPIC_PRICES_AS_OF
"""

from __future__ import annotations

import re

PRICES_AS_OF = "2026-09-11"
# The Anthropic rows were read from the AWS Price List API on this date.
ANTHROPIC_PRICES_AS_OF = "2026-09-29"

# USD per 1M tokens — Global CRIS, Long Context Window (1M), Standard tier.
# Sourced from https://aws.amazon.com/bedrock/pricing/ on PRICES_AS_OF.
Expand Down Expand Up @@ -65,34 +74,189 @@
"input": 0.1545,
"output": 0.6180,
},
# Anthropic Claude -- AWS Price List API, service AmazonBedrockFoundationModels,
# us-east-1, OnDemand, read on ANTHROPIC_PRICES_AS_OF. The plain key is the
# Regional rate, which every in-region and geo cross-region id pays
# (``us.``/``eu.``/``ap.``/bare -- all the committed runs use ``us.``). The
# ``global.``-prefixed key is the Global cross-region rate, 10% lower, used
# only for ``global.`` ids (see _rates). Cache write is the default 5-minute
# TTL rate, the TTL Claude Code and the Strands runner use.
# Claude Haiku 4.5 -- Regional.
"anthropic.claude-haiku-4-5": {
"input": 1.1,
"cache_write": 1.375,
"cache_read": 0.11,
"output": 5.5,
},
# Claude Haiku 4.5 -- Global cross-region.
"global.anthropic.claude-haiku-4-5": {
"input": 1.0,
"cache_write": 1.25,
"cache_read": 0.1,
"output": 5.0,
},
# Claude Opus 4.5 -- Regional.
"anthropic.claude-opus-4-5": {
"input": 5.5,
"cache_write": 6.875,
"cache_read": 0.55,
"output": 27.5,
},
# Claude Opus 4.5 -- Global cross-region.
"global.anthropic.claude-opus-4-5": {
"input": 5.0,
"cache_write": 6.25,
"cache_read": 0.5,
"output": 25.0,
},
# Claude Opus 4.6 -- Regional.
"anthropic.claude-opus-4-6": {
"input": 5.5,
"cache_write": 6.875,
"cache_read": 0.55,
"output": 27.5,
},
# Claude Opus 4.6 -- Global cross-region.
"global.anthropic.claude-opus-4-6": {
"input": 5.0,
"cache_write": 6.25,
"cache_read": 0.5,
"output": 25.0,
},
# Claude Opus 4.7 -- Regional.
"anthropic.claude-opus-4-7": {
"input": 5.5,
"cache_write": 6.875,
"cache_read": 0.55,
"output": 27.5,
},
# Claude Opus 4.7 -- Global cross-region.
"global.anthropic.claude-opus-4-7": {
"input": 5.0,
"cache_write": 6.25,
"cache_read": 0.5,
"output": 25.0,
},
# Claude Opus 4.8 -- Regional.
"anthropic.claude-opus-4-8": {
"input": 5.5,
"cache_write": 6.875,
"cache_read": 0.55,
"output": 27.5,
},
# Claude Opus 4.8 -- Global cross-region.
"global.anthropic.claude-opus-4-8": {
"input": 5.0,
"cache_write": 6.25,
"cache_read": 0.5,
"output": 25.0,
},
# Claude Opus 5 -- Regional.
"anthropic.claude-opus-5": {
"input": 5.5,
"cache_write": 6.875,
"cache_read": 0.55,
"output": 27.5,
},
# Claude Opus 5 -- Global cross-region.
"global.anthropic.claude-opus-5": {
"input": 5.0,
"cache_write": 6.25,
"cache_read": 0.5,
"output": 25.0,
},
# Claude Sonnet 5 -- Regional.
"anthropic.claude-sonnet-5": {
"input": 2.2,
"cache_write": 2.75,
"cache_read": 0.22,
"output": 11.0,
},
# Claude Sonnet 5 -- Global cross-region.
"global.anthropic.claude-sonnet-5": {
"input": 2.0,
"cache_write": 2.5,
"cache_read": 0.2,
"output": 10.0,
},
# Claude Fable 5.1 -- Regional.
"anthropic.claude-fable-5-1": {
"input": 11.0,
"cache_write": 13.75,
"cache_read": 0.275,
"output": 55.0,
},
# Claude Fable 5.1 -- Global cross-region.
"global.anthropic.claude-fable-5-1": {
"input": 10.0,
"cache_write": 12.5,
"cache_read": 0.25,
"output": 50.0,
},
# Claude Opus 5.5 -- Regional.
"anthropic.claude-opus-5-5": {
"input": 4.4,
"cache_write": 5.5,
"cache_read": 0.22,
"output": 22.0,
},
# Claude Opus 5.5 -- Global cross-region.
"global.anthropic.claude-opus-5-5": {
"input": 4.0,
"cache_write": 5.0,
"cache_read": 0.2,
"output": 20.0,
},
# Claude Sonnet 5.5 -- Regional.
"anthropic.claude-sonnet-5-5": {
"input": 2.2,
"cache_write": 2.75,
"cache_read": 0.22,
"output": 11.0,
},
# Claude Sonnet 5.5 -- Global cross-region.
"global.anthropic.claude-sonnet-5-5": {
"input": 2.0,
"cache_write": 2.5,
"cache_read": 0.2,
"output": 10.0,
},
}

_PER_1M = 1_000_000.0
_GLOBAL_PREFIX = "global."


def _normalize(model: str) -> str:
"""Return a model id reduced to its price-table key.

Three spellings of one model reach this module, and all must price the
Several spellings of one model reach this module, and all must price the
same: the inference-profile form (``us.openai.gpt-5.6-terra``), the raw
Bedrock id with its version suffix (``qwen.qwen3-coder-30b-a3b-v1:0``), and
the LiteLLM alias the harness records as ``--model``
(``qwen.qwen3-coder-30b-a3b-instruct``). Without the last two a priced
model still yields a null cost, which reads as a free run.
Bedrock id with its version suffix (``qwen.qwen3-coder-30b-a3b-v1:0``), the
LiteLLM alias the harness records as ``--model``
(``qwen.qwen3-coder-30b-a3b-instruct``), and the Anthropic forms with a
release date, a bare version or a context-window tag
(``anthropic.claude-haiku-4-5-20251001-v1:0``,
``anthropic.claude-opus-4-6-v1``, ``anthropic.claude-opus-4-8[1m]``).
Without these a priced model still yields a null cost, which reads as a
free run.

Args:
model: The model id in any of those spellings.

Returns:
The bare key: prefix, version suffix and ``-instruct`` removed.
The bare key: prefix, version suffix, release date, context tag and
``-instruct`` removed.
"""
clean = model
clean = re.sub(r"\[[^\]]*\]$", "", model)
for prefix in ("us.", "global.", "eu.", "ap."):
if clean.startswith(prefix):
clean = clean[len(prefix) :]
break
# Bedrock version suffixes: '-v1:0', ':0', '-v2:1'.
clean = re.sub(r"(-v\d+)?:\d+$", "", clean)
# Bedrock version suffixes: '-v1:0', ':0', '-v2:1', and a bare '-v1'.
clean = re.sub(r"(-v\d+)?(:\d+)?$", "", clean)
# Anthropic release dates: 'claude-haiku-4-5-20251001'.
clean = re.sub(r"-\d{8}$", "", clean)
if clean.endswith("-instruct"):
clean = clean[: -len("-instruct")]
return clean
Expand All @@ -104,10 +268,17 @@ def _rates(model: str) -> dict[str, float] | None:
Args:
model: The model id, in any spelling ``_normalize`` accepts.

A ``global.`` id is billed at the Global cross-region rate, so it prefers
the ``global.``-keyed row when the table has one; every other id, and a
``global.`` id with no such row, uses the plain key.

Returns:
The rate row, or None when the model is not in the table.
"""
return PRICES.get(_normalize(model)) or PRICES.get(model)
key = _normalize(model)
if model.startswith(_GLOBAL_PREFIX) and _GLOBAL_PREFIX + key in PRICES:
return PRICES[_GLOBAL_PREFIX + key]
return PRICES.get(key) or PRICES.get(model)


def cost_usd(
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/scripts/run-benchmark-batch.sh
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@
# --dataset dataset YAML, relative to benchmarks/ (required)
# --models comma-separated model ids, run in order (required)
# --tasks comma-separated task ids; default is every task in the dataset
# --agent coding agent: pi (default), claude, omp, kiro
# --agent coding agent: pi (default), claude, omp, kiro, codex, strands
# --skill swe3 (default) or swe2
# --aws-region region for the codex judge (default us-east-1). Needed even on
# the vllm path: the model is local but the judge is on Bedrock.
Expand Down
Loading
Loading