docs(qwen35): TP2 implementation record + knowledge-eval tooling (split 4/4 of #946) - #1006
Open
Ma1oneZhang wants to merge 4 commits into
Open
docs(qwen35): TP2 implementation record + knowledge-eval tooling (split 4/4 of #946)#1006Ma1oneZhang wants to merge 4 commits into
Ma1oneZhang wants to merge 4 commits into
Conversation
Signed-off-by: Ziyang Zhang <hafuhafu@qq.com>
…eval - tp-implementation.md: Phase 2a/2b + Step 3 batched-decode landing record - benchmarks/qwen35-27b-tp2-knowledge-eval.md: MMLU-Redux 94.09 vs 93.2, C-Eval 88.11 vs 90.5 (full runs, truncation rerun-merged); in band - scripts/eval_mc.py + eval_rerun_truncated.py: benchmark runner (OpenCompass/lm-eval recipe replicas over the chat completions API) - developer-onboarding: NCCL libnccl.so dlopen note Signed-off-by: Ziyang Zhang <hafuhafu@qq.com>
The MMLU-Redux extractor searched the whole completion for the first [ABCD], so any answer prefixed with "Answer: B" scored as the "A" in "Answer", silently marking correct B/C/D responses wrong (codex review on pegainfer-project#946). Reuse the SuperGPQA marker-preferring patterns restricted to A-D, and drop the now-dead MMLUREDUX_RE. Redo the MMLU-Redux numbers before citing them. Signed-off-by: Ziyang Zhang <hafuhafu@qq.com>
The 94.09 snapshot predates the answer-extraction fix; mark it as not citable until rerun with the marker-preferring extractor. Signed-off-by: Ziyang Zhang <hafuhafu@qq.com>
Contributor
Author
|
@codex please review |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Split 4/4 of #946, tracked in #1001. Independent of splits 1–3 (docs + bench tooling only; no runtime dependency).
scripts/eval_mc.pyMMLU-Redux extraction (absorbs the codex review comment from feat(qwen35): shard GDR state per TP rank + batched eager/graph decode under TP (+27B benchmark validation) #946): answers are extracted via the sharedsg_extract_labels(text, 'ABCD')marker logic instead of a first-capital-letter regex, matching the other benchmarks.Type of Change
Evidence
Checklist
docs/conventions/coding-style.md).CLAUDE.md).