Repository navigation
Commit 114109e
authored
Stamp models.json provenance from the data, not from git (#222)
build_vended_models.py stamped models.json with the commit that last
touched the omp frontier JSON and that commit's date. A squash merge
replaces the branch commit, so the stamp made on a PR no longer matched
main and test_committed_models_json_matches_the_generator failed with
every score still correct. It happened after #184, #210 and #219.
Both fields now come from committed data, which a merge does not
rewrite:
- source_commit becomes source_sha256: the first 12 hex digits of the
frontier JSON's SHA-256, the value `sha256sum <file> | cut -c1-12`
prints. It changes only when the data does.
- measured_on is the latest run_date among the run summaries of the
models in the frontier, rather than a commit date in the merger's
timezone. For the current data that is 2026-10-11, Sonnet 5.5's run.
schema_version goes to 1.1 for the renamed field. route.py and the
skill check only the major version, so both keep working. The generator
no longer runs git, which removes its two subprocess calls.
Tests: the git-based provenance test is replaced by checks that the hash
matches the file bytes, changes with the data and ignores where the file
lives, and that measured_on takes the latest frontier model's run date
and is null when no summary records one. A simulated squash merge onto
main, with a different commit, date and timezone, passes --check.1 parent 1ec5326 commit 114109e
4 files changed
Lines changed: 115 additions & 98 deletions
File tree
- benchmarks
- scripts
- tests
- vend/swe-router
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
31 | 31 | | |
32 | 32 | | |
33 | 33 | | |
34 | | - | |
35 | | - | |
| 34 | + | |
36 | 35 | | |
37 | 36 | | |
38 | 37 | | |
| |||
53 | 52 | | |
54 | 53 | | |
55 | 54 | | |
56 | | - | |
57 | | - | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
58 | 61 | | |
59 | 62 | | |
60 | 63 | | |
| |||
77 | 80 | | |
78 | 81 | | |
79 | 82 | | |
80 | | - | |
81 | | - | |
| 83 | + | |
| 84 | + | |
82 | 85 | | |
83 | | - | |
84 | | - | |
85 | | - | |
86 | | - | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
87 | 92 | | |
88 | 93 | | |
89 | 94 | | |
90 | 95 | | |
91 | 96 | | |
92 | | - | |
93 | | - | |
| 97 | + | |
94 | 98 | | |
95 | | - | |
96 | | - | |
97 | | - | |
98 | | - | |
99 | | - | |
100 | | - | |
101 | | - | |
102 | | - | |
103 | | - | |
104 | | - | |
105 | | - | |
106 | | - | |
107 | | - | |
108 | | - | |
109 | | - | |
110 | | - | |
111 | | - | |
112 | | - | |
113 | | - | |
114 | | - | |
| 99 | + | |
115 | 100 | | |
116 | 101 | | |
117 | | - | |
118 | | - | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
119 | 106 | | |
120 | | - | |
121 | | - | |
122 | | - | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
123 | 111 | | |
124 | 112 | | |
125 | | - | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
126 | 118 | | |
127 | 119 | | |
128 | | - | |
| 120 | + | |
129 | 121 | | |
130 | | - | |
131 | | - | |
132 | | - | |
133 | | - | |
134 | | - | |
135 | | - | |
136 | | - | |
137 | | - | |
138 | | - | |
139 | | - | |
140 | | - | |
141 | | - | |
142 | | - | |
143 | | - | |
144 | | - | |
145 | | - | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
146 | 133 | | |
147 | | - | |
148 | | - | |
149 | | - | |
150 | | - | |
151 | | - | |
| 134 | + | |
| 135 | + | |
152 | 136 | | |
153 | 137 | | |
154 | 138 | | |
| |||
243 | 227 | | |
244 | 228 | | |
245 | 229 | | |
246 | | - | |
| 230 | + | |
| 231 | + | |
247 | 232 | | |
248 | 233 | | |
249 | 234 | | |
| |||
264 | 249 | | |
265 | 250 | | |
266 | 251 | | |
| 252 | + | |
267 | 253 | | |
268 | | - | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
269 | 261 | | |
270 | 262 | | |
271 | 263 | | |
| 264 | + | |
272 | 265 | | |
273 | 266 | | |
274 | 267 | | |
| |||
305 | 298 | | |
306 | 299 | | |
307 | 300 | | |
308 | | - | |
| 301 | + | |
309 | 302 | | |
310 | | - | |
| 303 | + | |
311 | 304 | | |
312 | 305 | | |
313 | 306 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
| 5 | + | |
5 | 6 | | |
6 | 7 | | |
7 | | - | |
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
| |||
67 | 67 | | |
68 | 68 | | |
69 | 69 | | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
70 | 83 | | |
71 | 84 | | |
72 | 85 | | |
| |||
119 | 132 | | |
120 | 133 | | |
121 | 134 | | |
122 | | - | |
| 135 | + | |
| 136 | + | |
123 | 137 | | |
124 | 138 | | |
125 | 139 | | |
| |||
141 | 155 | | |
142 | 156 | | |
143 | 157 | | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
144 | 192 | | |
145 | 193 | | |
146 | 194 | | |
| |||
194 | 242 | | |
195 | 243 | | |
196 | 244 | | |
197 | | - | |
198 | | - | |
199 | | - | |
| 245 | + | |
| 246 | + | |
200 | 247 | | |
201 | | - | |
202 | | - | |
203 | | - | |
204 | | - | |
205 | | - | |
206 | | - | |
207 | | - | |
208 | | - | |
209 | | - | |
210 | | - | |
211 | | - | |
212 | | - | |
213 | | - | |
214 | | - | |
215 | | - | |
216 | | - | |
217 | | - | |
218 | | - | |
219 | | - | |
220 | | - | |
221 | | - | |
222 | | - | |
223 | | - | |
224 | | - | |
225 | | - | |
| 248 | + | |
| 249 | + | |
226 | 250 | | |
227 | 251 | | |
228 | 252 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
82 | 82 | | |
83 | 83 | | |
84 | 84 | | |
85 | | - | |
| 85 | + | |
86 | 86 | | |
87 | 87 | | |
88 | 88 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | | - | |
| 2 | + | |
3 | 3 | | |
4 | 4 | | |
5 | | - | |
| 5 | + | |
6 | 6 | | |
7 | | - | |
| 7 | + | |
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
| |||
0 commit comments