Skip to content

benchmark: add kimi-k3 to the omp/swe3 frontier and propagate through the derived docs - #176

Merged
aarora79 merged 5 commits into
mainfrom
benchmark/kimi-k3-omp-swe3-frontier
Sep 8, 2026
Merged

aarora79 merged 5 commits into
mainfrom
benchmark/kimi-k3-omp-swe3-frontier

Conversation

@aarora79

@aarora79 aarora79 commented Sep 8, 2026

Copy link
Copy Markdown
Owner

Lands kimi-k3 (from #165, which added the run data, throughput sweep and B300 pricing) on the published omp /swe3 frontier and propagates it through every derived artifact.

Where kimi-k3 lands

79.17 / $4.33 per task, the 3rd-highest scorer (behind claude-opus-5 82.83 and glm-5.3 80.11). It joins the cost/quality frontier, which grows from 5 to 6 models: it undercuts glm-5.3 on cost and dominates claude-sonnet-5 ($4.67 / 76.97) and the opus-4.x points on both axes.

What changed

  • Regenerated (from the committed run-summaries): pareto-frontier-omp-swe3.json, the cost-quality / cost-accuracy-bubble / quality-radar charts (light + dark), harness-omp-swe3.md (the token/cost/score table), agentic-coding-swe-comparison-swe3.md, vend/swe-router/models.json, and the judged routing eval (swe-router-eval-judged.json + .md).
  • Skill data: added kimi-k3 to model-aliases.json and to the shipped allowed-models.txt (the shipped list is the frontier by construction, enforced by a test), and bumped the allow-list size assertion in test_route.py from 5 to 6.
  • Hand-maintained numbers: the four vend README tier tables and the ranked-tier block, the 16 -> 17 model counts, the root README frontier line, and the routing-eval headline: routing is now 45.4% cheaper for 4.6% less quality, using three models (qwen3.8-27b x13, claude-opus-5 x3, kimi-k3 x1).

All 425 benchmarks tests pass; ruff clean.

… the derived docs

PR #165 landed kimi-k3's omp/swe3/v2 run, throughput sweep and B300 pricing, but not the derived artifacts. This regenerates them and lands kimi-k3 on the frontier at 79.17 / $4.33 per task (now 17 models, 6 on the combined frontier).

- Regenerate the frontier JSON, the cost-quality / bubble / radar charts, the omp report table, the cross-harness comparison, models.json, and the judged routing eval.
- Add kimi-k3 to model-aliases.json and the shipped allowed-models.txt (the frontier grew 5 -> 6; a test asserts the allow-list equals the frontier), and bump the allow-list size assertion in test_route.py.
- Update the hand-maintained numbers: the vend README tier tables and ranked block, model counts (16 -> 17), the root README, and the routing-eval headline (now 45.4% cheaper for 4.6% less quality, three models: qwen3.8-27b, claude-opus-5, kimi-k3).
@aarora79 aarora79 added the documentation Improvements or additions to documentation label Sep 8, 2026
@aarora79
aarora79 force-pushed the benchmark/kimi-k3-omp-swe3-frontier branch from 6f64cfa to 2572222 Compare September 8, 2026 16:21
… with a delta column

plot_cost_quality.py draws a compact table of the on-frontier models in the lower-right of the scatter (inside the plot, above the legend, so it doesn't widen the canvas). Columns: model, quality, cost/task, and a delta column showing what each step up the frontier buys over the row below it -- the quality gained and the extra cost per task. Deltas are computed from the displayed rounded values so subtracting the columns reproduces them. Regenerate the omp/swe3 chart (light + dark); drawing-only, the frontier JSON and models.json are untouched.
@aarora79
aarora79 force-pushed the benchmark/kimi-k3-omp-swe3-frontier branch from 2572222 to 6288172 Compare September 8, 2026 17:05
…3 in the presentation deck

Swap the embedded cost-quality chart for the current one (17 models including kimi-k3, with the quality / cost / delta frontier table), add the kimi-k3 row to the combined-frontier table, and refresh the stale glm-5.3 figures and counts (16 -> 17 models; five -> six on the frontier; glm-5.3 81.27/$8.09 -> 80.11/$7.04). Regenerate the PDF from the updated HTML (30 pages, 960x540).
Same as the presentation deck: swap the embedded cost-quality chart for the current one (17 models including kimi-k3, with the quality / cost / delta frontier table), refresh the stale figures (16 -> 17 models; glm-5.3 81.27/$8.09 -> 80.11/$7.04, 1.6 -> 2.7 points off the top model), and regenerate the exec PDF (18 pages, 960x540).
@aarora79
aarora79 merged commit 76bd085 into main Sep 8, 2026
5 checks passed
aarora79 added a commit that referenced this pull request Sep 8, 2026
…uash-merge

The #176 squash-merge (76bd085) re-touched the frontier file, so the committed models.json's source_commit (549b1f8, a branch commit gone from main) no longer matched a fresh build, failing test_committed_models_json_matches_the_generator on main. Rebuild to re-stamp it to the frontier's current commit.
aarora79 added a commit that referenced this pull request Sep 8, 2026
…ommand) render (#177)

* docs: point the swe-router video embed at the updated (QR + install-command) render

Swap the user-attachments URL in the README and both slide-deck HTMLs to the re-uploaded video, which shows the repo QR code on the first and last scenes and the real install command in the ship scene.

The deck PDFs are intentionally unchanged: a static PDF export cannot play the <video> element, so a video-URL swap does not alter their rendering.

* slides: add a watch-the-video link below the deck video window (HTML + PDF)

On the video slide of both decks (presentation and exec), add a visible link to the MP4 below the player, so the PDF export -- where the <video> cannot play -- still points readers to the video. The link targets the stable media-assets release URL (auto-serves the latest render). Regenerate both PDFs (presentation 30pp, exec 18pp, 960x540).

* fix(swe-router): re-stamp models.json source_commit after the #176 squash-merge

The #176 squash-merge (76bd085) re-touched the frontier file, so the committed models.json's source_commit (549b1f8, a branch commit gone from main) no longer matched a fresh build, failing test_committed_models_json_matches_the_generator on main. Rebuild to re-stamp it to the frontier's current commit.

* slides: remove the swe-router 'in use' placeholder slide from both decks

The placeholder ('[ placeholder ] -- Screen recording ... replace before presenting') was a stand-in for the explainer video, which is now embedded on the video slide. Remove it from both decks and regenerate the PDFs (presentation 30 -> 29 pages, exec 18 -> 17).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant