Repository navigation
Stamp models.json provenance from the data, not from git - #222
Merged
Merged
Conversation
build_vended_models.py stamped models.json with the commit that last touched the omp frontier JSON and that commit's date. A squash merge replaces the branch commit, so the stamp made on a PR no longer matched main and test_committed_models_json_matches_the_generator failed with every score still correct. It happened after #184, #210 and #219. Both fields now come from committed data, which a merge does not rewrite: - source_commit becomes source_sha256: the first 12 hex digits of the frontier JSON's SHA-256, the value `sha256sum <file> | cut -c1-12` prints. It changes only when the data does. - measured_on is the latest run_date among the run summaries of the models in the frontier, rather than a commit date in the merger's timezone. For the current data that is 2026-10-11, Sonnet 5.5's run. schema_version goes to 1.1 for the renamed field. route.py and the skill check only the major version, so both keep working. The generator no longer runs git, which removes its two subprocess calls. Tests: the git-based provenance test is replaced by checks that the hash matches the file bytes, changes with the data and ignores where the file lives, and that measured_on takes the latest frontier model's run date and is null when no summary records one. A simulated squash merge onto main, with a different commit, date and timezone, passes --check.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
mainhas gone red after three results PRs (#184, #210, #219), each time with every score still correct.build_vended_models.pystampedvend/swe-router/models.jsonwith the commit that last touched the omp frontier JSON and that commit's date. A squash merge replaces the branch commit, so the stamp made on the PR no longer matchedmain, andtest_committed_models_json_matches_the_generatorfailed until someone rebuilt the file.This PR takes both provenance fields from committed data, which a merge does not rewrite.
source_commit: e.g.1c86108source_sha256: first 12 hex digits of the frontier JSON's SHA-256, e.g.72b1ed9ee222measured_on: commit date in the merger's timezonemeasured_on: latestrun_dateamong the frontier models' run summaries, e.g.2026-10-11source_sha256is exactly whatsha256sum docs/metrics/pareto-frontier-omp-swe3.json | cut -c1-12prints, so anyone can check it. It changes only when the data changes.Changes
benchmarks/scripts/build_vended_models.py:_source_sha256and_latest_run_datereplace_git_commitand_measured_on.schema_versiongoes to1.1. The generator no longer runs git, which removes its two subprocess calls.vend/swe-router/models.json: rebuilt. Onlyschema_version,measured_onand the renamed hash field change; every model entry is identical.vend/swe-router/README.md: explains both fields.benchmarks/tests/test_build_vended_models.py: the git-based test is replaced by checks that the hash matches the file bytes, changes with the data and does not depend on where the file lives, and thatmeasured_ontakes the latest frontier model's run date (ignoring other runs) and is null when no summary has one.Compatibility
route.pyand the skill check only the schema's major version, so1.1is accepted. A consumer that parsedprovenance.source_commitwill findsource_sha256instead.Testing
benchmarks/suite pass.build_vended_models.py --checkpasses, and ruff and Bandit are clean.origin/mainin a throwaway worktree, with a different commit and a committer date in the -11:00 timezone, then ran--check. It reported the file up to date; the old scheme fails in exactly that setup.Follow-up, not in this PR
.github/workflows/test.ymlchecks out withfetch-depth: 0, and its comment says that is so the generator can read the frontier's commit date. With this change the generator no longer reads git history. Whether to reduce the fetch depth, and correct the comment, is a CI change I have left for review.