Repository navigation
benchmark: add kimi-k3 to the omp/swe3 frontier and propagate through the derived docs - #176
Merged
Merged
Conversation
… the derived docs PR #165 landed kimi-k3's omp/swe3/v2 run, throughput sweep and B300 pricing, but not the derived artifacts. This regenerates them and lands kimi-k3 on the frontier at 79.17 / $4.33 per task (now 17 models, 6 on the combined frontier). - Regenerate the frontier JSON, the cost-quality / bubble / radar charts, the omp report table, the cross-harness comparison, models.json, and the judged routing eval. - Add kimi-k3 to model-aliases.json and the shipped allowed-models.txt (the frontier grew 5 -> 6; a test asserts the allow-list equals the frontier), and bump the allow-list size assertion in test_route.py. - Update the hand-maintained numbers: the vend README tier tables and ranked block, model counts (16 -> 17), the root README, and the routing-eval headline (now 45.4% cheaper for 4.6% less quality, three models: qwen3.8-27b, claude-opus-5, kimi-k3).
… committed frontier
aarora79
force-pushed
the
benchmark/kimi-k3-omp-swe3-frontier
branch
from
September 8, 2026 16:21
6f64cfa to
2572222
Compare
… with a delta column plot_cost_quality.py draws a compact table of the on-frontier models in the lower-right of the scatter (inside the plot, above the legend, so it doesn't widen the canvas). Columns: model, quality, cost/task, and a delta column showing what each step up the frontier buys over the row below it -- the quality gained and the extra cost per task. Deltas are computed from the displayed rounded values so subtracting the columns reproduces them. Regenerate the omp/swe3 chart (light + dark); drawing-only, the frontier JSON and models.json are untouched.
aarora79
force-pushed
the
benchmark/kimi-k3-omp-swe3-frontier
branch
from
September 8, 2026 17:05
2572222 to
6288172
Compare
…3 in the presentation deck Swap the embedded cost-quality chart for the current one (17 models including kimi-k3, with the quality / cost / delta frontier table), add the kimi-k3 row to the combined-frontier table, and refresh the stale glm-5.3 figures and counts (16 -> 17 models; five -> six on the frontier; glm-5.3 81.27/$8.09 -> 80.11/$7.04). Regenerate the PDF from the updated HTML (30 pages, 960x540).
Same as the presentation deck: swap the embedded cost-quality chart for the current one (17 models including kimi-k3, with the quality / cost / delta frontier table), refresh the stale figures (16 -> 17 models; glm-5.3 81.27/$8.09 -> 80.11/$7.04, 1.6 -> 2.7 points off the top model), and regenerate the exec PDF (18 pages, 960x540).
aarora79
added a commit
that referenced
this pull request
Sep 8, 2026
…uash-merge The #176 squash-merge (76bd085) re-touched the frontier file, so the committed models.json's source_commit (549b1f8, a branch commit gone from main) no longer matched a fresh build, failing test_committed_models_json_matches_the_generator on main. Rebuild to re-stamp it to the frontier's current commit.
aarora79
added a commit
that referenced
this pull request
Sep 8, 2026
…ommand) render (#177) * docs: point the swe-router video embed at the updated (QR + install-command) render Swap the user-attachments URL in the README and both slide-deck HTMLs to the re-uploaded video, which shows the repo QR code on the first and last scenes and the real install command in the ship scene. The deck PDFs are intentionally unchanged: a static PDF export cannot play the <video> element, so a video-URL swap does not alter their rendering. * slides: add a watch-the-video link below the deck video window (HTML + PDF) On the video slide of both decks (presentation and exec), add a visible link to the MP4 below the player, so the PDF export -- where the <video> cannot play -- still points readers to the video. The link targets the stable media-assets release URL (auto-serves the latest render). Regenerate both PDFs (presentation 30pp, exec 18pp, 960x540). * fix(swe-router): re-stamp models.json source_commit after the #176 squash-merge The #176 squash-merge (76bd085) re-touched the frontier file, so the committed models.json's source_commit (549b1f8, a branch commit gone from main) no longer matched a fresh build, failing test_committed_models_json_matches_the_generator on main. Rebuild to re-stamp it to the frontier's current commit. * slides: remove the swe-router 'in use' placeholder slide from both decks The placeholder ('[ placeholder ] -- Screen recording ... replace before presenting') was a stand-in for the explainer video, which is now embedded on the video slide. Remove it from both decks and regenerate the PDFs (presentation 30 -> 29 pages, exec 18 -> 17).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lands kimi-k3 (from #165, which added the run data, throughput sweep and B300 pricing) on the published omp
/swe3frontier and propagates it through every derived artifact.Where kimi-k3 lands
79.17 / $4.33 per task, the 3rd-highest scorer (behind
claude-opus-582.83 andglm-5.380.11). It joins the cost/quality frontier, which grows from 5 to 6 models: it undercutsglm-5.3on cost and dominatesclaude-sonnet-5($4.67 / 76.97) and the opus-4.x points on both axes.What changed
pareto-frontier-omp-swe3.json, the cost-quality / cost-accuracy-bubble / quality-radar charts (light + dark),harness-omp-swe3.md(the token/cost/score table),agentic-coding-swe-comparison-swe3.md,vend/swe-router/models.json, and the judged routing eval (swe-router-eval-judged.json+.md).kimi-k3tomodel-aliases.jsonand to the shippedallowed-models.txt(the shipped list is the frontier by construction, enforced by a test), and bumped the allow-list size assertion intest_route.pyfrom 5 to 6.qwen3.8-27bx13,claude-opus-5x3,kimi-k3x1).All 425 benchmarks tests pass;
ruffclean.