Repository navigation
benchmark: re-run 5 self-hosted models on omp /swe3 after the token-accounting fix (#157) - #174
Merged
Merged
Conversation
…(implementation + judge scores)
…implementation + judge scores)
M3 is a candidate for a future run, so record the serving configuration now rather than rediscovering it under time pressure. The published recipe says M3 needs a dedicated Docker image because support "has not yet shipped in a stable vLLM release". That is out of date for the 0.28.0 pinned here, which carries the config, the reasoning parser, the minimax_m3 tool-call parser and the MSA index-decode kernels natively. The guide names the files so the claim can be re-checked after an upgrade. Three things a first bring-up would otherwise hit: - --block-size 128 is mandatory for the MSA sparse attention and its index cache. vllm-serve.sh has no variable for it, so it rides in EXTRA_ARGS the way GLM-5.3's KV-cache dtype already does. - The FP8 build is MXFP8, a microscaling format, not the block FP8 used by M2.5 and GLM-5.3. Being block-quantized, it may hit the same tensor-parallel divisibility constraints that force M2.5 to TP=4, so the guide says to check config.json before trusting the recipe's TP=8. - On a p5en the node environment must be sourced first, or NCCL fails at distributed init before any weights load. The guide is marked as unverified throughout, because nothing here has been served yet. Every other guide in that directory records a measured configuration, and this one must not be mistaken for measurement until the model actually runs.
Read config.json, the HF API and the installed vLLM rather than relying on the recipe. Three things it got wrong or left out: - M3 is a VISION-LANGUAGE model. model_type is minimax_m3_vl: a CLIP vision tower plus a 60-layer MoE text model. Its dimensions live under text_config, not at the top level, which is why a naive config read returns nothing. The vision stack is dead weight for a text-only /swe3 run and should be noted before comparing memory or throughput against a text-only model. - TP=8 is legal, and the reason is not obvious. num_key_value_heads is 4, so the rule that forces M2.5 to TP=4 (kv_heads % TP == 0) fails. vLLM applies that test only when kv heads outnumber ranks; below that it replicates them, needing TP % kv_heads == 0, and 8 % 4 == 0. The MoE constraint passes easily too: MXFP8 blocks are [1,32], not [128,128], so 3072/8 = 384 divides cleanly. - Weights are 444 GB in MXFP8 (31 shards) and 854 GB in BF16 (59), so the FP8 build is the only practical choice on this node. Also sharpen the leading note to separate what was checked (dimensions, quantization, sizes, vLLM support, and the TP conclusion drawn from them) from what was not (that it serves at all, and every runtime number, including the proposed max-model-len and memory utilization).
…entation + judge scores)
…mplementation + judge scores)
… (implementation + judge scores)
…er the self-hosted re-run The re-run updated the 5 self-hosted models' run-summaries but not the derived artifacts, so the data-consistency tests failed (models.json no longer matched the recomputed tier means). - Regenerate docs/metrics/pareto-frontier-omp-swe3.json and the cost-quality charts from the fresh run-summaries (plot_cost_quality.py). - Rebuild vend/swe-router/models.json (build_vended_models.py) and the four README tier tables from it. - devstral-2-123b now passes 4 of 6 medium tasks (was 3), so update that completion assertion. Costs shifted with the token-accounting fix (glm-5.3 $8.09->$7.04, kimi-k2.7-code $3.13->$2.47, deepseek-v3.2 $2.61->$2.15), which reorders the cheapest-first tables.
…docs, charts and skill numbers Regenerate the omp report table (token counts, cost, scores), the cost-accuracy bubble and quality-radar charts, and the judged routing eval (json + md) from the fresh run-summaries, and rebuild models.json (measured_on is now the frontier's commit date). Update the hand-maintained numbers the re-run moved: the vend README ranked-tier block and its prose (glm-5.3 now tops trivial and no longer overtakes opus on medium), the SKILL.md overall table (kimi-k2.7-code) and the 'stay' worked example (now qwen3.8-27b, since glm-5.3 no longer clears the old floor of 75 on hard tasks), allowed-models.txt's glm comment, the root README glm and routing figures, and the swe-router-evaluation.md headline (44.7% cheaper for 4.8% less quality, three models).
aarora79
marked this pull request as ready for review
September 8, 2026 15:23
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #157 when complete. Draft: the run is still in progress, and commits land here per model as each finishes.
What this is
All 21
/swe3tasks re-run onmcp-gateway-registry-v2under omp for the five self-hosted models #157 names, on ap5en.48xlarge(8x H200). The extractor fix (#159) is merged, so these runs use the corrected token accounting rather than the truncatedagent_endsum.Per #157 all 21 tasks are re-run for every model, not only the 13 with a detectable signature, so no pre-fix and post-fix numbers are mixed inside one model's mean.
Progress
Two things reviewers should know
Stale scores blocked judging, and were cleared. The judge runs with
--no-overwrite, and every task still carried aneval.jsonfrom the pre-fix run: clearing stale artifacts rewrites the harness's own files but never removeseval.json, which only the judge writes. The first judge pass reported0 scored, 0 failed, 21 totaland would have paired new patches with old scores. All 105 stale files were deleted before scoring, so everyeval.jsonhere is freshly written against the patch beside it.NVLS is disabled on this node. Serving any TP=8 model failed at
ncclCommInitRankbecause NCCL could not bind NVLink SHARP multicast memory (CUDA 401), despite Fabric Manager reportingCompleted/Successon all 8 GPUs.NCCL_NVLS_ENABLE=0is the documented workaround (PR #173), and it drops in-network reduction. Wall-clock-derived cost per task from this run is therefore not directly comparable to earlier p5en runs where NVLS bound successfully. Treat the scores as comparable and the cost figures as needing that caveat.Not included
The six per-task artifacts and
omp-stream.jsonlare gitignored by design, so onlyeval.json,metrics.jsonandrun-summary.{json,md}are committed. That is the data #157 needs, but the patches and prose artifacts live only on the run instance.