Skip to content

benchmark: re-run 5 self-hosted models on omp /swe3 after the token-accounting fix (#157) - #174

Merged
aarora79 merged 10 commits into
mainfrom
benchmark/issue-157-omp-swe3-rerun
Sep 8, 2026
Merged

aarora79 merged 10 commits into
mainfrom
benchmark/issue-157-omp-swe3-rerun

Conversation

@aarora79

@aarora79 aarora79 commented Sep 8, 2026

Copy link
Copy Markdown
Owner

Closes #157 when complete. Draft: the run is still in progress, and commits land here per model as each finishes.

What this is

All 21 /swe3 tasks re-run on mcp-gateway-registry-v2 under omp for the five self-hosted models #157 names, on a p5en.48xlarge (8x H200). The extractor fix (#159) is merged, so these runs use the corrected token accounting rather than the truncated agent_end sum.

Per #157 all 21 tasks are re-run for every model, not only the 13 with a detectable signature, so no pre-fix and post-fix numbers are mixed inside one model's mean.

Progress

Model Tasks Judged Mean score Committed
deepseek-v3.2 21/21 21/21 61.66 yes
kimi-k2.7-code 21/21 21/21 pending verification yes
glm-5.3 in progress
minimax-m2.5 queued
devstral-2-123b queued

Two things reviewers should know

Stale scores blocked judging, and were cleared. The judge runs with --no-overwrite, and every task still carried an eval.json from the pre-fix run: clearing stale artifacts rewrites the harness's own files but never removes eval.json, which only the judge writes. The first judge pass reported 0 scored, 0 failed, 21 total and would have paired new patches with old scores. All 105 stale files were deleted before scoring, so every eval.json here is freshly written against the patch beside it.

NVLS is disabled on this node. Serving any TP=8 model failed at ncclCommInitRank because NCCL could not bind NVLink SHARP multicast memory (CUDA 401), despite Fabric Manager reporting Completed/Success on all 8 GPUs. NCCL_NVLS_ENABLE=0 is the documented workaround (PR #173), and it drops in-network reduction. Wall-clock-derived cost per task from this run is therefore not directly comparable to earlier p5en runs where NVLS bound successfully. Treat the scores as comparable and the cost figures as needing that caveat.

Not included

The six per-task artifacts and omp-stream.jsonl are gitignored by design, so only eval.json, metrics.json and run-summary.{json,md} are committed. That is the data #157 needs, but the patches and prose artifacts live only on the run instance.

@aarora79 aarora79 added the documentation Improvements or additions to documentation label Sep 8, 2026
aarora79 and others added 8 commits September 8, 2026 00:54
M3 is a candidate for a future run, so record the serving configuration
now rather than rediscovering it under time pressure.

The published recipe says M3 needs a dedicated Docker image because
support "has not yet shipped in a stable vLLM release". That is out of
date for the 0.28.0 pinned here, which carries the config, the reasoning
parser, the minimax_m3 tool-call parser and the MSA index-decode kernels
natively. The guide names the files so the claim can be re-checked after
an upgrade.

Three things a first bring-up would otherwise hit:

- --block-size 128 is mandatory for the MSA sparse attention and its
  index cache. vllm-serve.sh has no variable for it, so it rides in
  EXTRA_ARGS the way GLM-5.3's KV-cache dtype already does.
- The FP8 build is MXFP8, a microscaling format, not the block FP8 used
  by M2.5 and GLM-5.3. Being block-quantized, it may hit the same
  tensor-parallel divisibility constraints that force M2.5 to TP=4, so
  the guide says to check config.json before trusting the recipe's TP=8.
- On a p5en the node environment must be sourced first, or NCCL fails at
  distributed init before any weights load.

The guide is marked as unverified throughout, because nothing here has
been served yet. Every other guide in that directory records a measured
configuration, and this one must not be mistaken for measurement until
the model actually runs.
Read config.json, the HF API and the installed vLLM rather than relying
on the recipe. Three things it got wrong or left out:

- M3 is a VISION-LANGUAGE model. model_type is minimax_m3_vl: a CLIP
  vision tower plus a 60-layer MoE text model. Its dimensions live under
  text_config, not at the top level, which is why a naive config read
  returns nothing. The vision stack is dead weight for a text-only
  /swe3 run and should be noted before comparing memory or throughput
  against a text-only model.
- TP=8 is legal, and the reason is not obvious. num_key_value_heads is
  4, so the rule that forces M2.5 to TP=4 (kv_heads % TP == 0) fails.
  vLLM applies that test only when kv heads outnumber ranks; below that
  it replicates them, needing TP % kv_heads == 0, and 8 % 4 == 0. The
  MoE constraint passes easily too: MXFP8 blocks are [1,32], not
  [128,128], so 3072/8 = 384 divides cleanly.
- Weights are 444 GB in MXFP8 (31 shards) and 854 GB in BF16 (59), so
  the FP8 build is the only practical choice on this node.

Also sharpen the leading note to separate what was checked (dimensions,
quantization, sizes, vLLM support, and the TP conclusion drawn from
them) from what was not (that it serves at all, and every runtime
number, including the proposed max-model-len and memory utilization).
…er the self-hosted re-run

The re-run updated the 5 self-hosted models' run-summaries but not the derived artifacts, so the data-consistency tests failed (models.json no longer matched the recomputed tier means).

- Regenerate docs/metrics/pareto-frontier-omp-swe3.json and the cost-quality charts from the fresh run-summaries (plot_cost_quality.py).
- Rebuild vend/swe-router/models.json (build_vended_models.py) and the four README tier tables from it.
- devstral-2-123b now passes 4 of 6 medium tasks (was 3), so update that completion assertion.

Costs shifted with the token-accounting fix (glm-5.3 $8.09->$7.04, kimi-k2.7-code $3.13->$2.47, deepseek-v3.2 $2.61->$2.15), which reorders the cheapest-first tables.
…docs, charts and skill numbers

Regenerate the omp report table (token counts, cost, scores), the cost-accuracy bubble and quality-radar charts, and the judged routing eval (json + md) from the fresh run-summaries, and rebuild models.json (measured_on is now the frontier's commit date).

Update the hand-maintained numbers the re-run moved: the vend README ranked-tier block and its prose (glm-5.3 now tops trivial and no longer overtakes opus on medium), the SKILL.md overall table (kimi-k2.7-code) and the 'stay' worked example (now qwen3.8-27b, since glm-5.3 no longer clears the old floor of 75 on hard tasks), allowed-models.txt's glm comment, the root README glm and routing figures, and the swe-router-evaluation.md headline (44.7% cheaper for 4.8% less quality, three models).
@aarora79
aarora79 marked this pull request as ready for review September 8, 2026 15:23
@aarora79
aarora79 merged commit ad3424e into main Sep 8, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Re-run 5 self-hosted models on omp /swe3 after the token-accounting fix

1 participant