Skip to content

Commit 114109e

Browse files
authored
Stamp models.json provenance from the data, not from git (#222)
build_vended_models.py stamped models.json with the commit that last touched the omp frontier JSON and that commit's date. A squash merge replaces the branch commit, so the stamp made on a PR no longer matched main and test_committed_models_json_matches_the_generator failed with every score still correct. It happened after #184, #210 and #219. Both fields now come from committed data, which a merge does not rewrite: - source_commit becomes source_sha256: the first 12 hex digits of the frontier JSON's SHA-256, the value `sha256sum <file> | cut -c1-12` prints. It changes only when the data does. - measured_on is the latest run_date among the run summaries of the models in the frontier, rather than a commit date in the merger's timezone. For the current data that is 2026-10-11, Sonnet 5.5's run. schema_version goes to 1.1 for the renamed field. route.py and the skill check only the major version, so both keep working. The generator no longer runs git, which removes its two subprocess calls. Tests: the git-based provenance test is replaced by checks that the hash matches the file bytes, changes with the data and ignores where the file lives, and that measured_on takes the latest frontier model's run date and is null when no summary records one. A simulated squash merge onto main, with a different commit, date and timezone, passes --check.
1 parent 1ec5326 commit 114109e

4 files changed

Lines changed: 115 additions & 98 deletions

File tree

‎benchmarks/scripts/build_vended_models.py‎

Lines changed: 57 additions & 64 deletions
Original file line numberDiff line numberDiff line change
@@ -31,8 +31,7 @@
3131
import argparse
3232
import json
3333
import logging
34-
import subprocess # nosec B404 - used with list args, no shell, hardcoded 'git'
35-
from datetime import date
34+
import hashlib
3635
from pathlib import Path
3736
from typing import Any
3837

@@ -53,8 +52,12 @@
5352
DEFAULT_OUT = _REPO_ROOT / "vend" / "swe-router" / "models.json"
5453

5554
# Bumped when the shape of models.json changes in a way a consumer would notice.
56-
# Consumers pin this; the skill refuses a major it does not know.
57-
SCHEMA_VERSION = "1.0"
55+
# Consumers pin this; the skill refuses a major it does not know. 1.1 replaced
56+
# provenance.source_commit (a git sha) with source_sha256 (a content hash) and
57+
# took measured_on from the run summaries instead of a commit date.
58+
SCHEMA_VERSION = "1.1"
59+
# provenance.source_sha256 length: enough to tell versions apart, short to read.
60+
SOURCE_HASH_CHARS = 12
5861

5962
# The judge is not recorded in the frontier JSON, so it is stated here. Keep in
6063
# step with codex_judge.py's default (JUDGE_MODEL overrides it per run).
@@ -77,78 +80,59 @@
7780
}
7881

7982

80-
def _git_commit(source: Path) -> str | None:
81-
"""Return the short sha of the commit that last changed ``source``.
83+
def _source_sha256(source: Path) -> str:
84+
"""Return the first 12 hex digits of the SHA-256 of ``source``'s bytes.
8285
83-
Deliberately not HEAD. HEAD moves with every commit to the repository, so
84-
stamping it would make the generated file differ after any unrelated change
85-
and turn the --check guard into noise. The commit that last touched the
86-
source is what actually identifies this version of the data.
86+
Identifies the exact version of the data from the file itself. An earlier
87+
version stamped the commit that last touched the file, which a squash merge
88+
replaces: the stamp made on the branch then no longer matched ``main``, and
89+
the committed-artifact check failed with every score still correct (after
90+
#184, #210 and #219). A content hash is the same on the branch and after the
91+
merge, and anyone can check it with ``sha256sum <file> | cut -c1-12``.
8792
8893
Args:
8994
source: The frontier JSON.
9095
9196
Returns:
92-
The abbreviated commit, or None outside a git checkout or for a file
93-
that has never been committed.
97+
A 12-character lowercase hex string.
9498
"""
95-
try:
96-
out = subprocess.run( # nosec B603 B607 - hardcoded 'git', list args, no shell
97-
[
98-
"git",
99-
"-C",
100-
str(source.parent),
101-
"log",
102-
"-1",
103-
"--format=%h",
104-
"--",
105-
source.name,
106-
],
107-
capture_output=True,
108-
text=True,
109-
timeout=10,
110-
check=False,
111-
)
112-
except (OSError, subprocess.SubprocessError):
113-
return None
114-
return out.stdout.strip() or None
99+
return hashlib.sha256(source.read_bytes()).hexdigest()[:SOURCE_HASH_CHARS]
115100

116101

117-
def _measured_on(source: Path) -> str:
118-
"""Return the date the source frontier was last modified, ISO format.
102+
def _latest_run_date(
103+
runs_dir: Path, harness: str, skill: str, dataset: str, models: set[str]
104+
) -> str | None:
105+
"""Return the latest ``run_date`` among the run summaries behind the frontier.
119106
120-
Uses the file's last git commit date when available, so regenerating without
121-
changing the data does not advance the measurement date. Falls back to the
122-
filesystem mtime.
107+
That is when the newest measurement in the file was taken, which is what a
108+
consumer means by "measured on". It comes from committed data, so it does
109+
not move when a commit is rewritten, and it does not depend on anyone's
110+
timezone.
123111
124112
Args:
125-
source: The frontier JSON.
113+
runs_dir: The swe-benchmark-data root.
114+
harness: Harness folder, e.g. "omp".
115+
skill: Skill folder, e.g. "swe3".
116+
dataset: Dataset scope folder.
117+
models: The model slugs the frontier JSON lists; other runs are ignored.
126118
127119
Returns:
128-
An ISO date string.
120+
An ISO date, or None when no matching summary records a ``run_date``.
129121
"""
130-
try:
131-
out = subprocess.run( # nosec B603 B607 - hardcoded 'git', list args, no shell
132-
[
133-
"git",
134-
"-C",
135-
str(source.parent),
136-
"log",
137-
"-1",
138-
"--format=%cs",
139-
"--",
140-
source.name,
141-
],
142-
capture_output=True,
143-
text=True,
144-
timeout=10,
145-
check=False,
122+
dates = []
123+
pattern = f"*/{harness}/{skill}/{dataset}/run-summary.json"
124+
for path in runs_dir.glob(pattern):
125+
summary = json.loads(path.read_text(encoding="utf-8"))
126+
if summary.get("model_slug") in models and summary.get("run_date"):
127+
dates.append(summary["run_date"])
128+
if not dates:
129+
logger.warning(
130+
"no run_date in any %s summary for the frontier's models; "
131+
"measured_on will be null",
132+
pattern,
146133
)
147-
if out.stdout.strip():
148-
return out.stdout.strip()
149-
except (OSError, subprocess.SubprocessError):
150-
pass
151-
return date.fromtimestamp(source.stat().st_mtime).isoformat()
134+
return None
135+
return max(dates)
152136

153137

154138
def _relative_or_name(source: Path, repo_root: Path) -> str:
@@ -243,7 +227,8 @@ def build(
243227
244228
Args:
245229
source: Path to a ``pareto-frontier-*.json``.
246-
repo_root: Repository root, used for the provenance commit.
230+
repo_root: Repository root, used to record the source path.
231+
runs_dir: The swe-benchmark-data root; defaults to the committed one.
247232
248233
Returns:
249234
The payload to write as models.json.
@@ -264,11 +249,19 @@ def build(
264249
bedrock = {m["model"] for m in data.get("bedrock_frontier", [])}
265250
self_hosted = {m["model"] for m in data.get("self_hosted_frontier", [])}
266251

252+
runs_dir = runs_dir if runs_dir is not None else DEFAULT_RUNS_DIR
267253
by_tier = per_tier_stats(
268-
runs_dir if runs_dir is not None else DEFAULT_RUNS_DIR,
254+
runs_dir,
255+
data.get("harness", ""),
256+
data.get("skill", ""),
257+
data.get("repo", ""),
258+
)
259+
measured_on = _latest_run_date(
260+
runs_dir,
269261
data.get("harness", ""),
270262
data.get("skill", ""),
271263
data.get("repo", ""),
264+
{m["model"] for m in models},
272265
)
273266
out_models = []
274267
for m in sorted(models, key=lambda x: -x["mean_score"]):
@@ -305,9 +298,9 @@ def build(
305298
"schema_version": SCHEMA_VERSION,
306299
"generated_by": "benchmarks/scripts/build_vended_models.py",
307300
"provenance": {
308-
"measured_on": _measured_on(source),
301+
"measured_on": measured_on,
309302
"source_file": _relative_or_name(source, repo_root),
310-
"source_commit": _git_commit(source),
303+
"source_sha256": _source_sha256(source),
311304
"harness": data.get("harness"),
312305
"skill": data.get("skill"),
313306
"dataset": data.get("repo"),

‎benchmarks/tests/test_build_vended_models.py‎

Lines changed: 54 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,9 @@
22

33
from __future__ import annotations
44

5+
import hashlib
56
import importlib.util
67
import json
7-
import subprocess
88
import sys
99
import tempfile
1010
import unittest
@@ -67,6 +67,19 @@ def _write(data: dict) -> Path:
6767
return Path(tmp.name)
6868

6969

70+
def _runs(dates: dict[str, str | None]) -> Path:
71+
"""Write one omp/swe3/v2 run summary per model, with the given run_date."""
72+
root = Path(tempfile.mkdtemp())
73+
for model, run_date in dates.items():
74+
folder = root / model / "omp" / "swe3" / "mcp-gateway-registry-v2"
75+
folder.mkdir(parents=True)
76+
summary: dict = {"model_slug": model, "tasks": []}
77+
if run_date:
78+
summary["run_date"] = run_date
79+
(folder / "run-summary.json").write_text(json.dumps(summary), "utf-8")
80+
return root
81+
82+
7083
class BuildTest(unittest.TestCase):
7184
def test_includes_every_model_not_only_the_frontier(self) -> None:
7285
# The skill filters to what the user's assistant offers before ranking,
@@ -119,7 +132,8 @@ def test_incomplete_runs_stay_visible(self) -> None:
119132
def test_provenance_carries_the_measurement_context(self) -> None:
120133
# A vended file has no reader who knows this repo, so the context
121134
# travels with the data.
122-
out = bvm.build(_write(_frontier()), _REPO_ROOT)
135+
runs = _runs({"big": "2026-10-01", "small": "2026-09-01"})
136+
out = bvm.build(_write(_frontier()), _REPO_ROOT, runs_dir=runs)
123137
p = out["provenance"]
124138
self.assertEqual(p["harness"], "omp")
125139
self.assertEqual(p["skill"], "swe3")
@@ -141,6 +155,40 @@ def test_empty_source_is_a_clear_error(self) -> None:
141155
with self.assertRaisesRegex(SystemExit, "no all_models"):
142156
bvm.build(_write(_frontier(all_models=[])), _REPO_ROOT)
143157

158+
def test_provenance_ignores_where_the_file_lives(self) -> None:
159+
# The squash-merge case: the same bytes must stamp the same provenance,
160+
# whatever commit or path they arrived under.
161+
runs = _runs({"big": "2026-10-01"})
162+
first = bvm.build(_write(_frontier()), _REPO_ROOT, runs_dir=runs)
163+
second = bvm.build(_write(_frontier()), _REPO_ROOT, runs_dir=runs)
164+
keys = ("measured_on", "source_sha256")
165+
self.assertEqual(
166+
[first["provenance"][k] for k in keys],
167+
[second["provenance"][k] for k in keys],
168+
)
169+
170+
def test_source_sha256_changes_with_the_data(self) -> None:
171+
runs = _runs({})
172+
before = bvm.build(_write(_frontier()), _REPO_ROOT, runs_dir=runs)
173+
after = bvm.build(_write(_frontier(note="new")), _REPO_ROOT, runs_dir=runs)
174+
self.assertNotEqual(
175+
before["provenance"]["source_sha256"],
176+
after["provenance"]["source_sha256"],
177+
)
178+
179+
def test_measured_on_is_the_latest_run_of_a_frontier_model(self) -> None:
180+
# "other" is not in the frontier JSON, so its later date is ignored.
181+
runs = _runs(
182+
{"big": "2026-10-01", "small": "2026-09-15", "other": "2026-12-31"}
183+
)
184+
out = bvm.build(_write(_frontier()), _REPO_ROOT, runs_dir=runs)
185+
self.assertEqual(out["provenance"]["measured_on"], "2026-10-01")
186+
187+
def test_measured_on_is_null_without_any_run_date(self) -> None:
188+
runs = _runs({"big": None, "small": None})
189+
out = bvm.build(_write(_frontier()), _REPO_ROOT, runs_dir=runs)
190+
self.assertIsNone(out["provenance"]["measured_on"])
191+
144192

145193
class CommittedArtifactTest(unittest.TestCase):
146194
"""The vended files are committed, so they are checked like any other input."""
@@ -194,35 +242,11 @@ def test_tier_scores_bracket_the_overall_mean(self) -> None:
194242
self.assertLessEqual(min(tiers), m["score"] + 0.01, m["model"])
195243
self.assertGreaterEqual(max(tiers), m["score"] - 0.01, m["model"])
196244

197-
def test_source_commit_tracks_the_frontier_not_head(self) -> None:
198-
# Stamping HEAD would make the file differ after every unrelated commit
199-
# and turn --check into noise. It must identify the data's version.
245+
def test_source_sha256_matches_the_committed_frontier_bytes(self) -> None:
246+
# What `sha256sum <file> | cut -c1-12` prints, so anyone can check it.
200247
payload = bvm.build(bvm.DEFAULT_SOURCE, _REPO_ROOT)
201-
stamped = payload["provenance"]["source_commit"]
202-
head = subprocess.run(
203-
["git", "-C", str(_REPO_ROOT), "rev-parse", "--short", "HEAD"],
204-
capture_output=True,
205-
text=True,
206-
check=False,
207-
).stdout.strip()
208-
last_touch = subprocess.run(
209-
[
210-
"git",
211-
"-C",
212-
str(_REPO_ROOT),
213-
"log",
214-
"-1",
215-
"--format=%h",
216-
"--",
217-
str(bvm.DEFAULT_SOURCE.relative_to(_REPO_ROOT)),
218-
],
219-
capture_output=True,
220-
text=True,
221-
check=False,
222-
).stdout.strip()
223-
self.assertEqual(stamped, last_touch)
224-
if head != last_touch:
225-
self.assertNotEqual(stamped, head)
248+
digest = hashlib.sha256(bvm.DEFAULT_SOURCE.read_bytes()).hexdigest()[:12]
249+
self.assertEqual(payload["provenance"]["source_sha256"], digest)
226250

227251
def test_vend_dir_holds_exactly_the_portable_files(self) -> None:
228252
# The portability contract: the skill must work in a directory holding

‎vend/swe-router/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -82,7 +82,7 @@ If it is firing when you do not want it, narrow the `Run it BEFORE…` sentence
8282

8383
**Self-hosted costs are measured under load.** Hosted figures come from a metered bill. Self-hosted figures come from the server's hourly price divided by throughput measured at a stated concurrency — a platform team serving a group of developers, not one person with an idle GPU. Both are a cost per task, and the skill ranks them together.
8484

85-
**These numbers move.** A fix to token accounting once changed `claude-opus-5` from $7.63 to $11.95 per task while every score stayed identical. Check `provenance.measured_on` and refetch if it looks old.
85+
**These numbers move.** A fix to token accounting once changed `claude-opus-5` from $7.63 to $11.95 per task while every score stayed identical. Check `provenance.measured_on`, the date of the newest run in the file, and refetch if it looks old. `provenance.source_sha256` identifies the exact data: it is the first 12 hex digits of the SHA-256 of the frontier file it was built from, so two copies with the same value hold the same numbers.
8686

8787
## How it decides
8888

‎vend/swe-router/models.json‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
11
{
2-
"schema_version": "1.0",
2+
"schema_version": "1.1",
33
"generated_by": "benchmarks/scripts/build_vended_models.py",
44
"provenance": {
5-
"measured_on": "2026-10-10",
5+
"measured_on": "2026-10-11",
66
"source_file": "docs/metrics/pareto-frontier-omp-swe3.json",
7-
"source_commit": "1c86108",
7+
"source_sha256": "72b1ed9ee222",
88
"harness": "omp",
99
"skill": "swe3",
1010
"dataset": "mcp-gateway-registry-v2",

0 commit comments

Comments
 (0)