Repository navigation
benchmark(kimi-k3): Kimi K3 on Bedrock via omp/swe3, v2 dataset - #191
Merged
aarora79 merged 3 commits intoSep 24, 2026
Merged
Conversation
added 3 commits
September 23, 2026 11:45
…ay-registry-v2 Mean score 71.9 over 20 completed tasks (1 model failure: lifecycle-workflow-webhooks). Path 2 (open-weight on Amazon Bedrock via LiteLLM proxy), agent omp, skill swe3.
…ptured) Re-run on the fix/kimi-k3-bedrock-pricing branch which derives cost from the Bedrock price table when omp reports null via the LiteLLM proxy. Run 2 vs run 1: - Score: 72.87 (21/21, 0 failed) vs 71.90 (20/21, 1 failed) - Cost: $3.87/task mean vs null - lifecycle-workflow-webhooks passed this time (70.8, was the sole failure)
Rename moonshotai.kimi-k3 -> kimi-k3-bedrock so the folder name makes the hosting path obvious. The vLLM results at kimi-k3/ are untouched.
aarora79
added a commit
that referenced
this pull request
Oct 5, 2026
…nt Path 2 runs as metered (#210) * Add Opus 5.5 and Kimi K3 on Bedrock to the omp/swe3 frontier Regenerate the omp/swe3 v2 frontier JSON, the cost-quality, bubble and radar charts (light and dark) and the omp report table over every omp v2 run on disk. Two runs had landed since the last regeneration: claude-opus-5-5 from #195 (81.16 at $4.44 a task) and kimi-k3-bedrock from #191 (72.87 at $3.87). Opus 5.5 joins both the combined and the Bedrock frontiers. It scores higher than claude-sonnet-5 (76.97 at $4.67) and glm-5.3 (80.11 at $7.04) for less, so those two leave the frontier. kimi-k3-bedrock is dominated by qwen3.8-27b. Add both models to swe-router's model-aliases.json. * Rebuild swe-router's models.json and update the frontier claims Rebuild vend/swe-router/models.json from the new omp/swe3 frontier (21 models). Rewrite the README and results-index sentences the new frontier made wrong: claude-opus-5-5 (81.16 at $4.44) now sits behind claude-opus-5 in place of glm-5.3, and replaces claude-sonnet-5 on the Bedrock frontier. Update the model counts that describe the chart and models.json. * Ship claude-opus-5-5 in place of glm-5.3 in swe-router's allow-list The shipped allow-list is the combined cost/quality frontier, which test_route enforces. Opus 5.5 (81.16, $4.44) replaced glm-5.3 (80.11, $7.04) on that frontier. Update the comments that counted self-hosted entries and explained why claude-sonnet-5 is absent. * Count a Bedrock model reached through LiteLLM as metered on the charts plot_cost_quality.py and gen_agent_report.py decided a run was metered only when its provider was "bedrock". A Path 2 run, a Bedrock model reached through the LiteLLM proxy, records "endpoint", so kimi-k3-bedrock was drawn and grouped as self-hosted and its report row showed no cost. Treat an endpoint run as metered when no throughput sweep prices it and it reports a positive bill. An audit of all 57 endpoint run summaries finds two such runs, kimi-k3-bedrock (omp) and qwen.qwen3-coder-30b-a3b-instruct (codex), both Bedrock-billed; every self-hosted run is unchanged. kiro-cli credits stay their own basis. Regenerate the omp/swe3 charts, frontier JSON and report. kimi-k3-bedrock (72.87 at $3.87) joins the Bedrock frontier and claude-opus-4-5 (66.32 at $4.18) leaves it. The combined frontier is unchanged. * Rebuild models.json and describe the new Bedrock frontier Rebuild swe-router's models.json after kimi-k3-bedrock moved to the metered basis. In the results index, list kimi-k3-bedrock on the Bedrock frontier in place of claude-opus-4-5, and say Path 2 now has a published run.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
moonshotai.kimi-k3) benchmarked on the v2 dataset (mcp-gateway-registry-v2, 21 tasks) via Path 2 (open-weight on Amazon Bedrock through the LiteLLM proxy), agentomp, skillswe3.kimi-k3-bedrock(distinguishes the Bedrock run from the vLLM results atkimi-k3/).openai.gpt-5.6-solviacodex execon Amazon Bedrock at high reasoning effort.Artifacts
43 files under
benchmarks/swe-benchmark-data/kimi-k3-bedrock/omp/swe3/mcp-gateway-registry-v2/: per-taskeval.json(judge scores) andmetrics.json(cost + token usage), plusrun-summary.json.