docs(qwen35): TP2 implementation record + knowledge-eval tooling (split 4/4 of #946)

.gitignore

@@ -10,3 +10,4 @@ __pycache__/
 /docs/private/
 /profile/
 .claude/scheduled_tasks.lock
+/results/

docs/benchmarks/qwen35-27b-tp2-knowledge-eval.md

@@ -0,0 +1,54 @@
+# Qwen3.5-27B TP2 知识基准评测(官方分对比)
+
+> TL;DR:Qwen3.5-27B 在 pegainfer TP2(2× RTX 4090,batched eager decode)上跑知识基准,C-Eval 88.11(官方 90.5)、MMLU-Redux 94.09(官方 93.2),均在跨 harness 正常带内;MMLU-Pro / SuperGPQA 因运行时长原因仅完成抽样冒烟,未出最终分(见下文)。模型数值无 TP 引入的精度问题。
+>
+> 注:分数实测于 rebase 前的 f4c66780 分支(自研 Phase 1/2a 线);rebase 到 #870 后 logits golden gate 两侧一致通过,数值可迁移,但若正式引用请在本 PR 分支上复跑确认。
+>
+> 另注:**MMLU-Redux 94.09 是在旧抽取器下实测的,在复跑前不要引用**。旧抽取取全文首个 `[ABCD]`,"Answer: B" 会被算成 "Answer" 里的 A(codex review 发现);本 PR 已修复为 marker 优先 + 独立字母抽取,需以修复后的 `scripts/eval_mc.py` 在 exact head 上重测。
+
+## 环境
+
+- GPU:2× RTX 4090(48 GB 版本),`--tp-size 2 --cuda-graph false`(TP+CUDA Graph 仍 fail-closed)
+- 模型:Qwen/Qwen3.5-27B `fc05daec`,BF16,served-model-name `qwen35-27b-tp2`
+- 采样:temperature=0.0,top_p=1.0,chat completions(thinking 模式,即模板默认行为)
+- 评测器:`scripts/eval_mc.py`(自研,统一 `/v1/chat/completions` 并发 48),配方逐项复刻官方 harness:
+  - **C-Eval** = OpenCompass `ceval_gen`:52 学科 val split 全量 1346 题,dev split 5-shot,"答案: " 续写,首大写字母抽取
+  - **MMLU-Redux** = lm-eval `mmlu_redux_generative`:`fxmarty/mmlu-redux-2.0-ok` 57 学科 test 全量 5330 题,0-shot,marker/独立字母抽取(早期版本为全文首个 `[ABCD]`,会误吸 "Answer: X" 前缀,已修复待复跑)
+  - **MMLU-Pro** = lm-eval `mmlu_pro`:TIGER-Lab/MMLU-Pro test,validation split 5-shot CoT,`answer is (X)` 抽取
+  - **SuperGPQA** = OpenCompass `supergpqa_gen`:`m-a-p/SuperGPQA` train 26529 题,0-shot,"Answer: X" 字母/内容两层抽取
+- 启动命令:`LD_LIBRARY_PATH=<.venv>/nvidia/nccl/lib ./target/release/pegainfer --model-path <27B> --served-model-name qwen35-27b-tp2 --tp-size 2 --cuda-graph false --port 18082`
+- 结果原始数据:`results/qwen35-27b-tp2-eval/{ceval,mmlu_redux}_samples*.json`(本地,未入库)
+
+## 结果(截至 2026-08-20,评测按需要提前终止)
+
+| 基准 | 官方 | 实测 | n | 口径 | Δ 判定 |
+|---|---|---|---|---|---|
+| MMLU-Redux | 93.2 | **94.09** | 5330 全量 | 8192 cap + 截断重跑合并(32 条重跑,2 条仍截断) | **同带** |
+| C-Eval | 90.5 | **88.11** | 1346 全量 | 8192 cap + 截断重跑合并(48 条重跑,0 条残留) | **同带边缘**(-2.4pp,CI95 ±1.7pp) |
+| MMLU-Pro | 86.1 | — | 600/2000 中止 | 抽样 n=2000(cap 24576)跑到 30% 人工终止;100 题冒烟在 4096 cap 下 51% 截断 | 无最终分 |
+| SuperGPQA | 65.6 | — | 未正式跑 | 100 题冒烟:可完成子集 43 题对金标准确 27/43≈63% | 无最终分 |
+
+## 关键观察
+
+- **没有 TP 精度问题**:27B TP2 HF logits golden gate 全绿;C-Eval 非截断子集(1290/1346)准确率 90.2% ≈ 官方 90.5。C-Eval 的差距全部来自 thinking 长度上限被掐断的最难题,而非模型错算。
+- **MMLU-Redux 略高于官方**(+0.9pp):同带,说明 prompt/抽取/数值链路都对。
+- **thinking 长度是最大的系统变量**:thinking 模型在 C-Eval 上 ~4% 题需要 >8192 token,MMLU-Pro 上 >1/3 题在 4096 内收不住。官方 harness 的 max_tokens 未知(推测 ≥32k);本评测用 8192 首轮 + 32768 重跑合并来逼近。跨 harness ±1–2pp 属正常。
+- **吞吐前置条件**:此评测可行完全依赖 Step 3 的 batched eager TP decode 修复(此前 16 并发聚合仅 ~25 tok/s,全量不可行;修复后 ~450 tok/s @48 并发)。
+
+## 复现命令
+
+```bash
+# server(见上);评测(hf 镜像):
+HF_ENDPOINT=https://hf-mirror.com .venv/bin/python -u scripts/eval_mc.py ceval --max-tokens 8192 --concurrency 48 --out-dir results/qwen35-27b-tp2-eval
+HF_ENDPOINT=https://hf-mirror.com .venv/bin/python -u scripts/eval_mc.py mmlu_redux --max-tokens 8192 --concurrency 48 --out-dir results/qwen35-27b-tp2-eval
+# 截断样本重跑合并:
+.venv/bin/python -u scripts/eval_rerun_truncated.py ceval --max-tokens 32768 --concurrency 16
+# 抽样(--sample 按学科比例分层,seed 1337,n=2000 时 CI95 半宽 ±1.3pp):
+.venv/bin/python -u scripts/eval_mc.py mmlu_pro --sample 2000 --max-tokens 24576 --concurrency 48
+.venv/bin/python -u scripts/eval_mc.py supergpqa --sample 2000 --max-tokens 24576 --concurrency 48
+```
+
+## 下一步
+
+- 补齐 MMLU-Pro / SuperGPQA 抽样全量(各 2000 题,预估合计 ~10h,吞吐 ~450 tok/s 前提)。
+- 若要把 C-Eval 收敛到官方 ±1pp:换更大的 thinking 预算复跑全量(无合并),并确认 Qwen 官方 harness 的 prompt 模板与本评测是否一致。

docs/index.md

@@ -53,8 +53,8 @@ Organized by domain (model line / subsystem / playbook / lesson) instead of by l
 | `models/qwen35/accuracy.md` | Qwen3.5 HF bf16 logits goldens, size-keyed (0.8b/2b/4b/9b/27b all committed), through `past_key_values`: short replay covers sequential graph, bucket-straddling batched graph, and slot-compaction; long replay covers 4097/8192-token prompts; full GSM8K 8-shot now matches the HF baseline within 0.15 percentage points. |
 | `models/qwen35/model-crate.md` | `pegainfer-qwen35` owns Qwen3.5 model/scheduler/recurrent ops/tests/benches; feature-gated behind `qwen35` (Triton AOT is the only Python build dependency); root loads it through `EngineHandle`. Build/check/clippy, root bench sanity check, historical Qwen3.5 e2e, and scheduler e2e records live here. |
 | `models/qwen35/batched-step-tail.md` | Qwen3.5 issue #353 implementation record: final prefill tail is batched, decode/unified sample from batched logits, host full-vocab copies are logprobs-only, HF + scheduler e2e pass, and final serving A/B supports only the first-token/short-output TTFT claim. |
-| `models/qwen35/tp-design.md` | Qwen3.5 TP design: Phase 1 is eager dense TP on Qwen3's controller/worker runtime; validate TP2 first, fail closed for indivisible degrees and TP+CUDA Graph, shard dense full-attention/MLP, and leave sharded linear/GDR state to follow-up. |
-| `models/qwen35/tp-implementation.md` | Qwen3.5 TP Phase 1 and P2A are complete: TP2 has start-gated eager unified prefill+decode, strict ID-aligned artifacts, fail-closed lifecycle recovery, and pre-load ordinal validation; P2B GDR state sharding is next. |
+| `models/qwen35/tp-design.md` | Qwen3.5 TP design: Phase 1 eager dense TP on Qwen3's controller/worker runtime, P2a mixed-step protocol, P2b rank-local GDR sharding, P2c CUDA Graph under TP gated on the compiled decode GQA group (27B group-6 stays eager). |
+| `models/qwen35/tp-implementation.md` | Qwen3.5 TP landed through P2c on #870 (2026-08, 2× RTX 4090): Phase 1/P2A lifecycle and ID contracts kept; GDR state sharded per rank (27B TP2 fits 48 GB pairs); batched eager decode (27B: 292 tok/s ×16); TP decode CUDA Graphs for 4B/9B (9B: 767 vs 706 tok/s ×16 eager). 9B/27B TP2 HF + e2e gates pass. |
 | `models/qwen35/mixed-load-itl-470.md` | Issue #470: full cold `--max-batch 8/bg=4` matrix on RTX 4090 (24/24 valid) + starvation negative control. Qwen3.5 is not immune; chunking bounds max/per-step stall but raises p99 at low QPS (~14→~80–92ms) and pulls p99/max back from the prefill wall to the chunk wall at high load; `qps·prefill_s≳1` is a throughput wall (chunking can't fix it, and ON's +15% TTFT can trip it earlier). The old "p99 immunity" was a slot-starvation artifact. |
 | `models/qwen35/adaptive-scheduler-policy.md` | Issue #727 adaptive scheduler policy record: default `off`, opt-in `auto`, hard `--max-prefill-tokens` cap, TP `auto` rejection, and pre-review whole-prefill benchmark tradeoff retained as non-default evidence. |
 | `models/qwen35/unified-prefill-overlap.md` | Issue #715 implementation record: opt-in single-GPU shared-SM overlap keeps one prefill chunk in flight while active decode continues; default serial policy and unsupported-combination guards remain explicit. |
@@ -234,6 +234,7 @@ Organized by domain (model line / subsystem / playbook / lesson) instead of by l
 | `benchmarks/bs1-4k64-vllm-pegainfer.md` | RTX 5090 single-concurrency probe: `input_len=4096`, `output_len=64`, no vLLM prefix cache. PegaInfer TTFT median `177ms` vs vLLM `198ms`; TPOT median `6.47ms` vs `6.36ms`; corrected output throughput `+6%` for PegaInfer. |
 | `benchmarks/mixed-load-itl.md` | Qwen3-4B + Qwen3.5 mixed-load ITL (#244, #375): chunking-off sweeps via `bench_serving mixed`. Both freeze active decode for the full prefill. Qwen3 p99 blows up with prompt/QPS; the old Qwen3.5 “p99-immune” table is a **measurement artifact** (primary: hardcoded `max_batch=4` slot starvation — see #470 / `models/qwen35/mixed-load-itl-470.md`; secondary: short `bg_output_len`). Prefix reuse defeats it on Qwen3. |
 | `benchmarks/accuracy-eval-results.md` | Phase 1 GSM8K: Qwen3-4B PASS (pegainfer 85.37% vs HF 85.82%, delta -0.45 pp). Qwen3.5-4B historical FAIL recovered by #250 (strict 79.38%, flexible 79.30% vs HF 79.45%). |
+| `benchmarks/qwen35-27b-tp2-knowledge-eval.md` | Qwen3.5-27B TP2 (2× RTX 4090, batched eager decode) knowledge benchmarks vs official: MMLU-Redux 94.09 vs 93.2 (full 5330), C-Eval 88.11 vs 90.5 (full 1346, thinking-cap truncation rerun-merged) — both in cross-harness band; MMLU-Pro/SuperGPQA partial smokes only. |
 | `benchmarks/qwen3-8b-pd-vs-mix-h200.md` | Qwen3-8B 多轮负载三方 A/B（2×H200）：P/D 1P+1D vs mixed×2（会话亲和 LB）vs mixed×1。吞吐持平（47.8k vs 47.0k tok/s），P/D 赢在 decode 稳定性（TPOT p99 10.08 vs 12.77ms，turn2+ TTFT 恒定 ~107ms vs 爬升 71→132ms），冷 turn1 多付 ~200ms（M3 目标）。含 vllm-bench 命令与 `max_completion_tokens` 坑。 |
 
 ## conventions

docs/models/qwen35/roadmap.md

@@ -47,7 +47,7 @@ out:
 | Fault isolation | Open risk: batch-level execution errors can still fail multiple active requests | #654 |
 | Prefix reuse | Open: bounded joint KV/recurrent/conv snapshot design and implementation | #257 |
 | DFlash | In flight and opt-in: correctness-first work must stay default-off until gates pass | #434, PR #626, #654 |
-| Tensor parallel | Phase 1 complete: eager dense TP2 worker/scheduler execution; Phase 2 still needs mixed-step execution and sharded linear-attention/GDR state. | `docs/models/qwen35/tp-implementation.md`, #446 |
+| Tensor parallel | Done through Phase 2b: eager dense TP2, mixed-step unified execution, and sharded linear/GDR state, verified on 2× RTX 4090 for 9B and 27B; TP CUDA Graph + perf gates remain | `docs/models/qwen35/tp-implementation.md`, #446 |
 
 ## Active Contract
 

docs/models/qwen35/tp-design.md

@@ -1,6 +1,6 @@
 # Qwen3.5 Tensor Parallelism Design
 
-> **TL;DR:** Qwen3.5 TP Phase 2 is two separately delivered correctness milestones: P2a adds eager `RunUnifiedStep` with a shared ordered `RequestId` plan while retaining Phase 1 replicated GDR; P2b shards the head-indexed linear-attention/GDR surface and adds only the hidden all-reduce after local `out_proj`.
+> **TL;DR:** Qwen3.5 tensor parallelism should reuse Qwen3's controller/worker TP runtime and stay degree-parametric. Phases 1, 2a, and 2b are implemented (see `tp-implementation.md` for the landing record, including the rebase onto #870): eager dense TP, TP mixed-step unified execution, and sharded linear-attention/GDR state. Remaining design work: TP CUDA Graph capture.
 >
 > **Last touched:** 2026-08
 
@@ -110,7 +110,7 @@ For any candidate `tp`, require:
 - `num_attention_heads % tp == 0`
 - `num_key_value_heads % tp == 0`
 - `intermediate_size % tp == 0`
-- Phase 2 additionally requires `linear_num_key_heads % tp == 0` and `linear_num_value_heads % tp == 0`
+- Phase 2 requires `linear_num_key_heads % tp == 0`; `linear_num_value_heads % tp == 0` then follows from the checkpoint invariant `linear_num_value_heads % linear_num_key_heads == 0`, so no second runtime guard
 
 Full attention local dimensions:
 
@@ -233,7 +233,7 @@ Lifecycle observability, cancellation ordering, fail-closed cleanup, and unified
 
 ## P2b: Local-Head Linear Attention / GDR
 
-P2b converts the 24 linear-attention layers from replicated execution to true TP execution. It additionally requires `linear_num_key_heads % tp == 0` and `linear_num_value_heads % tp == 0`; unsupported degrees and unsupported local kernel shapes fail before model loading.
+P2b converts the 24 linear-attention layers from replicated execution to true TP execution. It requires `linear_num_key_heads % tp == 0` (value-head divisibility follows from the checkpoint invariant `linear_num_value_heads % linear_num_key_heads == 0`); unsupported degrees and unsupported local kernel shapes fail before model loading.
 
 Shard every head-indexed linear-attention/GDR surface by the local key/value-head ranges:
 
@@ -269,6 +269,30 @@ Validation scope:
 - recurrent-state cleanup on finish/drop/cancellation
 - no stale local recurrent state after a new `RequestId` is admitted
 
+## P2c: CUDA Graph under TP
+
+Status: landed (2026-08-20) on `feat/qwen35-tp2-rebased`, gated on
+`Config35::decode_group_is_compiled` — 4B/9B TP2 capture and replay decode graphs;
+27B TP2 (group 6) stays on the batched eager path byte-for-byte until group-6
+batch-decode kernels are compiled. Execution record: `tp-implementation.md`
+section "P2c — CUDA Graph under TP".
+
+**Gate**: graph mode active iff `enable_cuda_graph && config.decode_group_is_compiled()`. 27B TP2 is group-6 (`SUPPORTED_GQA_GROUP_SIZES = [1,2,3,4,8]`, group ratio is TP-invariant), so 27B TP2 keeps the batched eager path byte-for-byte until group-6 batch-decode kernels are compiled; 4B/9B TP2 capture graphs. Startup logs once when graph was requested but the group gate keeps decode eager.
+
+**State model**: scheduler owns slot semantics (TP1 mirror); workers execute slot copies on command, never infer slots worker-side.
+
+- KV paged state unchanged (pool stable; page tables are per-step H2D via `sync_paged_meta`).
+- Per rank: `BatchDecodeGraphState`-equivalent at `bucket_for(effective_max_batch)` slots — fixed-address `slot_states: Vec<RecurrentState>` + one persistent `LinearStatePointerTables` built once over slots (contents stable → replay-safe).
+- Admission: decode command rows carry explicit `slot_idx` (`slot_for_new_request`); first decode row D2D-copies prefill `RecurrentState` into the slot (`copy_state_to_slot`), drops the per-request allocation.
+- Retirement: `DropRequest` gains `compaction: Option<(RequestId, from, to)>`; worker D2D-moves slot state (`move_slot_within`), asserts occupancy, poisons on mismatch.
+- Decode rows arrive dense slot order `0..bs`; padding rows clobber free slots (benign — admission overwrites).
+
+**Capture/replay**: startup pre-capture sweep ported from qwen3 (`executor.rs:1424`): `Warmup` (port `warmup_tp_collective`, one all-reduce per bucket message size — lazy NCCL connect inside capture wedges), `Capture`/`Launch` per bucket `[1,2,4,8,16,32,64]` with synthetic rows, `Finalize` asserts all captured; dedicated 600 s abort watchdog (60 s startup timeout too small). New `TpWorkerCommand::Precapture { phase }` via existing exact-rank dispatch. Serve time: replay-only (`ensure is_captured` + `launch_captured`), never capture mid-serving. Sampling/logprobs stay rank-0 host-side outside the graph. Mixed ticks: prefill eager + decode replay; collective order canonical per plan. `TpWorkerState` declares graph state before `model` so graphs drop before the NCCL comm (teardown hang precedent qwen3 `executor.rs:3076`).
+
+**Memory** (27B TP2/rank): weights ~17.5 GB + KV pool ~5.9 GiB + slot state reserve ~6.1 GiB + buffers/graphs ~0.3 + scratch/NCCL ~2.5 ≈ 32 GiB → fits 48 GB. 9B TP2 slot state ~1.6 GiB. Loader already reserves `2 × max_batch × bytes_per_request` before sizing KV.
+
+**Validation ladder**: CPU lib suite → TP2 graph HF gate (9B: sequential + bucket-straddling + post-compaction replay vs eager stats) → e2e scheduler graph variant → serving_tp2 graph smoke → 27B TP2 regression unchanged (group-6 stays eager) → per-bucket eager-vs-graph decode benchmark recorded in `bench_snapshots/`.
+
 ## References
 
 - `docs/models/qwen3/tp-design.md`