Skip to content

benchmark(kimi-k3): Kimi K3 on Bedrock via omp/swe3, v2 dataset - #191

Merged
aarora79 merged 3 commits into
aarora79:mainfrom
shekharprateek:benchmark/kimi-k3-omp-swe3-v2
Sep 24, 2026
Merged

aarora79 merged 3 commits into
aarora79:mainfrom
shekharprateek:benchmark/kimi-k3-omp-swe3-v2

Conversation

@shekharprateek

@shekharprateek shekharprateek commented Sep 23, 2026 •

Copy link
Copy Markdown

Summary

  • Kimi K3 (moonshotai.kimi-k3) benchmarked on the v2 dataset (mcp-gateway-registry-v2, 21 tasks) via Path 2 (open-weight on Amazon Bedrock through the LiteLLM proxy), agent omp, skill swe3.
  • Mean score 72.87 over 21 completed tasks, 0 failures. Mean cost $3.87/task.
  • Model slug: kimi-k3-bedrock (distinguishes the Bedrock run from the vLLM results at kimi-k3/).
  • Cost derived from the Bedrock price table (PR fix(pricing): Kimi K3 Bedrock rates + cost fallback for omp/pi via LiteLLM #192) since the LiteLLM proxy does not relay pricing from the upstream.
  • Judge: openai.gpt-5.6-sol via codex exec on Amazon Bedrock at high reasoning effort.

Artifacts

43 files under benchmarks/swe-benchmark-data/kimi-k3-bedrock/omp/swe3/mcp-gateway-registry-v2/: per-task eval.json (judge scores) and metrics.json (cost + token usage), plus run-summary.json.

Prateek Sinha added 3 commits September 23, 2026 11:45
…ay-registry-v2

Mean score 71.9 over 20 completed tasks (1 model failure: lifecycle-workflow-webhooks).
Path 2 (open-weight on Amazon Bedrock via LiteLLM proxy), agent omp, skill swe3.
…ptured)

Re-run on the fix/kimi-k3-bedrock-pricing branch which derives cost from
the Bedrock price table when omp reports null via the LiteLLM proxy.

Run 2 vs run 1:
- Score: 72.87 (21/21, 0 failed) vs 71.90 (20/21, 1 failed)
- Cost: $3.87/task mean vs null
- lifecycle-workflow-webhooks passed this time (70.8, was the sole failure)
Rename moonshotai.kimi-k3 -> kimi-k3-bedrock so the folder name makes
the hosting path obvious. The vLLM results at kimi-k3/ are untouched.
@aarora79
aarora79 merged commit 4a3b79a into aarora79:main Sep 24, 2026
5 checks passed
aarora79 added a commit that referenced this pull request Oct 5, 2026
…nt Path 2 runs as metered (#210)

* Add Opus 5.5 and Kimi K3 on Bedrock to the omp/swe3 frontier

Regenerate the omp/swe3 v2 frontier JSON, the cost-quality, bubble and radar
charts (light and dark) and the omp report table over every omp v2 run on
disk. Two runs had landed since the last regeneration: claude-opus-5-5 from
#195 (81.16 at $4.44 a task) and kimi-k3-bedrock from #191 (72.87 at $3.87).

Opus 5.5 joins both the combined and the Bedrock frontiers. It scores higher
than claude-sonnet-5 (76.97 at $4.67) and glm-5.3 (80.11 at $7.04) for less,
so those two leave the frontier. kimi-k3-bedrock is dominated by qwen3.8-27b.

Add both models to swe-router's model-aliases.json.

* Rebuild swe-router's models.json and update the frontier claims

Rebuild vend/swe-router/models.json from the new omp/swe3 frontier (21
models). Rewrite the README and results-index sentences the new frontier
made wrong: claude-opus-5-5 (81.16 at $4.44) now sits behind claude-opus-5 in
place of glm-5.3, and replaces claude-sonnet-5 on the Bedrock frontier. Update
the model counts that describe the chart and models.json.

* Ship claude-opus-5-5 in place of glm-5.3 in swe-router's allow-list

The shipped allow-list is the combined cost/quality frontier, which
test_route enforces. Opus 5.5 (81.16, $4.44) replaced glm-5.3 (80.11, $7.04)
on that frontier. Update the comments that counted self-hosted entries and
explained why claude-sonnet-5 is absent.

* Count a Bedrock model reached through LiteLLM as metered on the charts

plot_cost_quality.py and gen_agent_report.py decided a run was metered only
when its provider was "bedrock". A Path 2 run, a Bedrock model reached through
the LiteLLM proxy, records "endpoint", so kimi-k3-bedrock was drawn and grouped
as self-hosted and its report row showed no cost.

Treat an endpoint run as metered when no throughput sweep prices it and it
reports a positive bill. An audit of all 57 endpoint run summaries finds two
such runs, kimi-k3-bedrock (omp) and qwen.qwen3-coder-30b-a3b-instruct (codex),
both Bedrock-billed; every self-hosted run is unchanged. kiro-cli credits stay
their own basis.

Regenerate the omp/swe3 charts, frontier JSON and report. kimi-k3-bedrock
(72.87 at $3.87) joins the Bedrock frontier and claude-opus-4-5 (66.32 at
$4.18) leaves it. The combined frontier is unchanged.

* Rebuild models.json and describe the new Bedrock frontier

Rebuild swe-router's models.json after kimi-k3-bedrock moved to the metered
basis. In the results index, list kimi-k3-bedrock on the Bedrock frontier in
place of claude-opus-4-5, and say Path 2 now has a published run.
@aarora79 aarora79 mentioned this pull request Oct 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants