Skip to content

Stamp models.json provenance from the data, not from git - #222

Merged
aarora79 merged 1 commit into
mainfrom
fix/models-json-provenance-from-data
Oct 13, 2026
Merged

aarora79 merged 1 commit into
mainfrom
fix/models-json-provenance-from-data

Conversation

@aarora79

Copy link
Copy Markdown
Owner

Summary

main has gone red after three results PRs (#184, #210, #219), each time with every score still correct. build_vended_models.py stamped vend/swe-router/models.json with the commit that last touched the omp frontier JSON and that commit's date. A squash merge replaces the branch commit, so the stamp made on the PR no longer matched main, and test_committed_models_json_matches_the_generator failed until someone rebuilt the file.

This PR takes both provenance fields from committed data, which a merge does not rewrite.

Field Before (from git) After (from data)
Data version source_commit: e.g. 1c86108 source_sha256: first 12 hex digits of the frontier JSON's SHA-256, e.g. 72b1ed9ee222
Date measured_on: commit date in the merger's timezone measured_on: latest run_date among the frontier models' run summaries, e.g. 2026-10-11

source_sha256 is exactly what sha256sum docs/metrics/pareto-frontier-omp-swe3.json | cut -c1-12 prints, so anyone can check it. It changes only when the data changes.

Changes

  • benchmarks/scripts/build_vended_models.py: _source_sha256 and _latest_run_date replace _git_commit and _measured_on. schema_version goes to 1.1. The generator no longer runs git, which removes its two subprocess calls.
  • vend/swe-router/models.json: rebuilt. Only schema_version, measured_on and the renamed hash field change; every model entry is identical.
  • vend/swe-router/README.md: explains both fields.
  • benchmarks/tests/test_build_vended_models.py: the git-based test is replaced by checks that the hash matches the file bytes, changes with the data and does not depend on where the file lives, and that measured_on takes the latest frontier model's run date (ignoring other runs) and is null when no summary has one.

Compatibility

route.py and the skill check only the schema's major version, so 1.1 is accepted. A consumer that parsed provenance.source_commit will find source_sha256 instead.

Testing

  • 27 generator tests and the full benchmarks/ suite pass. build_vended_models.py --check passes, and ruff and Bandit are clean.
  • Squash-merge simulation: I squashed this branch onto origin/main in a throwaway worktree, with a different commit and a committer date in the -11:00 timezone, then ran --check. It reported the file up to date; the old scheme fails in exactly that setup.
  • The security-check gate passed.

Follow-up, not in this PR

.github/workflows/test.yml checks out with fetch-depth: 0, and its comment says that is so the generator can read the frontier's commit date. With this change the generator no longer reads git history. Whether to reduce the fetch depth, and correct the comment, is a CI change I have left for review.

build_vended_models.py stamped models.json with the commit that last
touched the omp frontier JSON and that commit's date. A squash merge
replaces the branch commit, so the stamp made on a PR no longer matched
main and test_committed_models_json_matches_the_generator failed with
every score still correct. It happened after #184, #210 and #219.

Both fields now come from committed data, which a merge does not
rewrite:

- source_commit becomes source_sha256: the first 12 hex digits of the
  frontier JSON's SHA-256, the value `sha256sum <file> | cut -c1-12`
  prints. It changes only when the data does.
- measured_on is the latest run_date among the run summaries of the
  models in the frontier, rather than a commit date in the merger's
  timezone. For the current data that is 2026-10-11, Sonnet 5.5's run.

schema_version goes to 1.1 for the renamed field. route.py and the
skill check only the major version, so both keep working. The generator
no longer runs git, which removes its two subprocess calls.

Tests: the git-based provenance test is replaced by checks that the hash
matches the file bytes, changes with the data and ignores where the file
lives, and that measured_on takes the latest frontier model's run date
and is null when no summary records one. A simulated squash merge onto
main, with a different commit, date and timezone, passes --check.
@aarora79 aarora79 added the bug Something isn't working label Oct 13, 2026
@aarora79
aarora79 merged commit 114109e into main Oct 13, 2026
5 checks passed
@aarora79 aarora79 mentioned this pull request Oct 15, 2026
3 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant