Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 58 additions & 21 deletions .claude/skills/benchmark/SKILL.md

Large diffs are not rendered by default.

297 changes: 297 additions & 0 deletions .claude/skills/onboard-benchmark/SKILL.md

Large diffs are not rendered by default.

3 changes: 2 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ When a task is unscoped, the source worth reading lives under `benchmarks/` and
├── docs/ # cross-cutting docs: results, comparisons, methodology, slides
│ ├── faq/ # FAQ: one file per question + an index (README.md) linking each
│ └── release-notes/ # release notes per version (newest first) + the versioning scheme
├── .claude/skills/ # repo skills (setup-machine, benchmark, swe/swe2/swe3, throughput, vllm-setup, security-check, swe-router, release-notes)
├── .claude/skills/ # repo skills (setup-machine, onboard-benchmark, benchmark, swe/swe2/swe3, throughput, vllm-setup, security-check, swe-router, release-notes)
└── .github/ # CI workflows and repo metadata
```

Expand Down Expand Up @@ -529,6 +529,7 @@ Read the doc that covers what you are about to do rather than rediscovering it.
- [docs/codex-setup.md](docs/codex-setup.md) -- installing `codex`, wiring it to Bedrock, and the constraint that decides which models `--agent codex` can reach: it speaks only the Responses API, which Bedrock serves for `openai.*` models alone. Its failure-mode table is the fast answer to a hung run, a `404 ... does not exist` on an endpoint, or a request retried five times. Per-path recipes: [docs/faq/wiring-codex-to-models.md](docs/faq/wiring-codex-to-models.md).
- [.claude/skills/vllm-setup/p5en-h200-cuda-fixes.md](.claude/skills/vllm-setup/p5en-h200-cuda-fixes.md) -- the one-time driver, NVMe and CUDA-JIT fixes an 8-GPU NVSwitch node needs. `setup-machine.sh` points here when it detects one, and vLLM fails at KV-cache init without them.
- [docs/getting-started.md](docs/getting-started.md) -- the guided tour from a fresh checkout to a first run, for anyone who wants the narrative rather than the reference pages above.
- [.claude/skills/onboard-benchmark/SKILL.md](.claude/skills/onboard-benchmark/SKILL.md) -- the `onboard-benchmark` skill, for a first run on **someone else's** infrastructure: their models behind a corporate gateway, their private or enterprise-hosted repositories, their own dataset. It covers what `benchmark` assumes is already true, and its first step is the one people skip -- confirming a single machine reaches both the model endpoint and the code host, since every task calls both.

**Running a benchmark:**

Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -144,6 +144,7 @@ Where to read more, by topic:
| [docs/getting-started.md](docs/getting-started.md) | Prerequisites and the setup sequence, from a fresh box to a first benchmark run. |
| [docs/repository-structure.md](docs/repository-structure.md) | What lives where in this repository. |
| [.claude/skills/setup-machine/SKILL.md](.claude/skills/setup-machine/SKILL.md) | **Start here on a new machine.** What `/setup-machine` inspects and installs, why each component is needed, where the vLLM venv lands on a small root disk, and what it deliberately does not do. |
| [.claude/skills/onboard-benchmark/SKILL.md](.claude/skills/onboard-benchmark/SKILL.md) | **Start here to benchmark your own code.** The `/onboard-benchmark` skill walks the first run on your own infrastructure: confirming one machine reaches both your model endpoint and your code host, picking the harness your gateway's API shape can drive, proving the agent and the judge with a real call, credentials for private and enterprise-hosted repositories, authoring a dataset from your closed issues, and one scored smoke task. |
| [docs/results-swe3.md](docs/results-swe3.md) / [docs/results-swe2.md](docs/results-swe2.md) | Full benchmark results per skill: task-by-task tables, per-model leaderboard, cost/quality frontier, hardware, and what the data says. |
| [.claude/skills/swe-router/SKILL.md](.claude/skills/swe-router/SKILL.md) | The `/swe-router` skill: how it sets a quality floor from the consequence of a change being wrong, picks the tier table to read it against, and selects the cheapest model that clears it. Advisory -- it recommends and stops. |
| [docs/swe-router-judged-inputs.md](docs/swe-router-judged-inputs.md) | What `omp` + `claude-opus-5` judged each of the 21 v2 tasks to need: floor, tier, the reasoning, and how much those moved across three identical runs. |
Expand Down
12 changes: 12 additions & 0 deletions benchmarks/config/runner.example.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,12 @@
# api_key str API key sent to the endpoint (provider=endpoint only).
# Local servers ignore the value; keep a placeholder
# like "local".
# api_key_env str Name of an environment variable holding the
# endpoint's API key (provider=endpoint only). When
# set it WINS over api_key, so a real token for a
# corporate model gateway never has to sit in this
# file. A named variable that is unset or empty fails
# at config load, not hours into a run.
# aws_region str AWS region for provider=bedrock (e.g. us-east-1).
# Falls back to AWS_REGION / AWS_DEFAULT_REGION when
# unset.
Expand Down Expand Up @@ -126,7 +132,13 @@ provider: endpoint

# provider=endpoint routing. Ignored when provider=bedrock.
endpoint: http://127.0.0.1:8000
# A self-hosted vLLM server ignores the key, so the placeholder is correct there.
api_key: local
# For a corporate model gateway that needs a REAL token, name the environment
# variable holding it instead and leave api_key alone. The token then never lands
# in this file (or in git), and a missing variable fails the run up front.
# export GATEWAY_API_KEY=...
# api_key_env: GATEWAY_API_KEY

# provider=bedrock routing. Uncomment (or pass --aws-region / export AWS_REGION)
# when running against Amazon Bedrock. Requires ambient AWS credentials.
Expand Down
123 changes: 123 additions & 0 deletions benchmarks/dataset/workshop-template.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
# =============================================================================
# SWE benchmark dataset - workshop template
# =============================================================================
#
# A starting point for benchmarking your OWN repositories. Copy it, do not edit
# it in place:
#
# cp dataset/workshop-template.yaml dataset/my-team.yaml
#
# Then replace every <REPLACE> placeholder. Validate before you run, which
# catches a typo in seconds instead of after a clone:
#
# uv run scripts/dataset_loader.py dataset/my-team.yaml
#
# Confirm the repositories are reachable at the refs you pinned, which catches a
# missing credential or a mistyped tag before any task runs:
#
# uv run scripts/preflight_check.py --dataset dataset/my-team.yaml --check-repos
#
# Then run one task end to end:
#
# ./scripts/run-e2e-benchmark.sh --provider vllm --model <your-model> \
# --dataset dataset/my-team.yaml --agent omp --skill swe3 --count 1 --yes
#
# The full field reference is in benchmarks/docs/harness-reference.md. Private
# repositories, GitHub Enterprise Server and internal certificate authorities are
# covered in docs/benchmark-your-own-repo.md.
#
# -----------------------------------------------------------------------------
# The one rule that decides whether a task measures anything
# -----------------------------------------------------------------------------
# `ref` must pin the release, tag or commit IMMEDIATELY BEFORE the one that
# fixed the issue. The harness clones the repository at that ref and asks the
# agent to do the work. Pin a ref that already contains the fix and the agent
# finds the change already made, so the task measures nothing and every model
# scores about the same on it.
#
# So for each task, find the closed issue, find the release that shipped its
# fix, and pin the release before that one.
#
# -----------------------------------------------------------------------------
# Two more things worth knowing
# -----------------------------------------------------------------------------
# * `ground_truth` is reviewer-only and is NEVER shown to the agent. It exists so
# a human can check the judge's verdict. Filling it in does not help or hint
# the model.
# * `output_scope` gives this dataset its own results folder. Set it whenever two
# datasets target the same repository, or the folder-level run-summary.json is
# rebuilt over both and silently blends two task sets into one mean.
# =============================================================================

schema_version: "1.0"
name: <REPLACE-dataset-slug> # kebab-case id, e.g. acme-platform
title: <REPLACE-Human-readable-title>
description: >
Real tasks drawn from closed issues in our own repositories, each pinned to the
release before its fix shipped.
created: <REPLACE-YYYY-MM-DD>

# Fallback ref for any task that does not set its own. Always set `ref` per task
# anyway: a shared default is what made the v1 dataset less useful than v2.
default_ref: main

# Own results folder, so this dataset never blends with another over the same repo.
output_scope: <REPLACE-dataset-slug>

# Documentary only. The harness records these in each run's metrics.json.
metrics:
- input_tokens
- output_tokens
- num_turns
- latency_seconds

complexity_levels: [trivial, low, medium, high]

tasks:
# ---------------------------------------------------------------------------
# Task 1. Start with something small so the first run finishes quickly and
# proves the wiring: a docs page, a config passthrough, one render condition.
# ---------------------------------------------------------------------------
- id: <REPLACE-task-slug> # stable slug; also the output folder name
repo: https://<REPLACE-git-host>/<org>/<repo>
ref: "<REPLACE-release-BEFORE-the-fix>"
complexity: trivial
tags: [<REPLACE-language>, <REPLACE-area>]
problem_issue_url: https://<REPLACE-git-host>/<org>/<repo>/issues/<n>
problem_statement: |
Describe the task the way a well-scoped ticket would: what to change, the
constraints that apply, and what "done" means. Write it so an agent can act
on it with nobody from your team present to answer questions.

Include anything a new engineer would need and would not guess: the file or
area to start in, the convention to follow, the behaviour to preserve.
clarifying_answers: |
Pre-supplied answers to the questions the skill would otherwise ask, so an
unattended run never stops to ask one. Optional but recommended.
1. Why this matters: <REPLACE>
2. Who is affected: <REPLACE>
3. Stack and constraints: <REPLACE>
4. Complexity: trivial
ground_truth:
approach: <REPLACE-how-it-was-actually-fixed> # never shown to the agent
expectations:
- <REPLACE-a-point-a-correct-design-must-cover>
reference_url: https://<REPLACE-git-host>/<org>/<repo>/pull/<n>

# ---------------------------------------------------------------------------
# Task 2. Then something with real substance, so the results tell you which
# model to use for which class of work rather than giving one average.
# Aim for 10 to 20 tasks spread across all four tiers before you trust a
# frontier; 5 or 6 per tier is what the published datasets use.
# ---------------------------------------------------------------------------
- id: <REPLACE-second-task-slug>
repo: https://<REPLACE-git-host>/<org>/<repo>
ref: "<REPLACE-release-BEFORE-the-fix>"
complexity: medium
tags: [<REPLACE-language>, <REPLACE-area>]
problem_issue_url: https://<REPLACE-git-host>/<org>/<repo>/issues/<n>
problem_statement: |
<REPLACE>
ground_truth:
approach: <REPLACE>
reference_url: https://<REPLACE-git-host>/<org>/<repo>/pull/<n>
Loading
Loading