feat(qwen35): cuda graph decode under TP (split 3/4 of #946) - #1005
Open
Ma1oneZhang wants to merge 4 commits into
Open
feat(qwen35): cuda graph decode under TP (split 3/4 of #946)#1005Ma1oneZhang wants to merge 4 commits into
Ma1oneZhang wants to merge 4 commits into
Conversation
Phase 2b ported onto pegainfer-project#870: recurrent/conv state, GDR scratch, and the linear-attention weight surface are allocated and addressed per rank instead of replicated, which is what makes 27B TP2 fit on 2x48 GB cards. - weight_loader: additive stitch/shard loaders (2D row stitch for the fused qkv [q|k|v] segments, 1D element stitch for conv1d channels, bf16/f32 1D shards for dt_bias/A_log) - config: local_linear_* accessors mirroring the full-attn TP style; linear head divisibility fails closed in TensorParallelConfig - weights: per-rank stitched shards for in_proj_qkv/conv1d (head-local slices per segment), row shards for z/b/a, col shard for out_proj, dt_bias/A_log sliced, norm_weight kept replicated (head-shared); loader reserve uses TP-aware estimates - recurrent_state/decode_buffers/prefill_buffers: state and GDR scratch at local value-head/qkv sizes; capacity math derives from local allocation_bytes - batch_decode/prefill: local head counts into the GDR decode/conv/ Triton-AOT prefill chains, gated RMSNorm at local v heads, all-reduce after linear out_proj; batch_decode_full_attention_via_prefill is now TP-local so eager decode routes 27B TP2 group-6 full attention through prefill (was FlashInfer Unsupported group_size: 6) - tp_executor: worker capacity math and per-request state use the rank-local sizes; decode rows still run as a per-request bs=1 loop (batched in a follow-up) Recurrent/conv state is never all-reduced. Signed-off-by: Ziyang Zhang <hafuhafu@qq.com>
Port of the batched eager TP decode step onto pegainfer-project#870's worker structure: decode rows in one command now run as ONE batched forward per step on every rank plus one batched rank-0 sampling pass, instead of a per-request bs=1 loop. - run_decode_batch resolves every decode row's worker state in command order, builds a step-scoped LinearStatePointerTables over the whole batch (from_recurrent_refs(..., bs, ...)), runs one batch_decode_eager_logits forward, then rank 0 snapshots all requested logprob rows before one batched select_batch over per-row params - execute_decode_rows (used by both decode-only and unified steps) calls run_decode_batch once; per-row results fan out in command order - TpRequestState.linear_pointer_tables (capacity-1, decode-only) removed; ensure_prefill_state no longer builds it. The step-scoped table is rebuilt every step, so swap_remove retirement can't stale it Seeded rows keep per-row semantics: select_batch isolates each seeded row into its own single-row philox call keyed on (request seed, step 0), so seeded output stays independent of batch composition. Unseeded rows decorrelate via the per-step command seed, same as the single-GPU batched path. Reference (27B TP2, 2x RTX 4090, eager): 16 concurrent 256-token completions aggregate 24.9 -> 292.3 tok/s; single-request unchanged. Signed-off-by: Ziyang Zhang <hafuhafu@qq.com>
P2c from docs/models/qwen35/tp-design.md. TP decode captures/replays per-bucket CUDA Graphs when --cuda-graph is set AND the TP-local decode GQA group has a compiled kernel (4B/9B TP2); 27B TP2 (group 6) keeps the batched eager path byte-for-byte under the gate. - Gate: drop the fail-closed TP+graph rejections in config.rs/lib.rs/ tp_executor.rs; log once when graph was requested but the group gate keeps decode eager. - State: scheduler owns dense decode slots (slot_idx on TP decode rows, slot_for_new_request at promote, compaction_after_retire on retire); workers hold a fixed-address BatchDecodeGraphState plus slot_map, D2D copy prefill state into the slot on the first decode row, and apply DropRequest compactions via move_slot_within with occupancy assertions (poison on mismatch). - Capture/replay: startup pre-capture sweep ported from qwen3 (Warmup via new warmup_tp_collective, Capture/Launch per bucket, Finalize) behind a 600 s abort watchdog; batch_decode_graph gains DecodeGraphUse (Serve/CaptureOnly/Replay); TP serving is replay-only; graphs drop before the NCCL comm. - Convenience executor API (model-local tests) keeps a slot tracker mirroring Qwen35Executor; scheduler flows pass explicit slots. Validation (2x RTX 4090, venv NCCL): lib 105/105; 9B TP2 HF gates eager + graph (sequential replay, bucket-straddling, post-compaction) pass within existing tolerances; 9B TP2 scheduler e2e eager+graph pass; serving_tp2 now launches with graph on; 27B TP2 HF+e2e pass unchanged (group-6 gate keeps eager, graph variant self-skips). Serving benchmark 9B TP2, 16 concurrent x 256 out: 767.15 tok/s graph vs 705.86 eager (+8.7% steady output, TPOT 20.04 vs 21.99 ms). Signed-off-by: Ziyang Zhang <hafuhafu@qq.com>
The graph-slot pre-reserve used bucket_for(requested_max_batch), but the graph state is later allocated as bucket_for(effective_max_batch). On a tight-memory rank the oversized reserve could starve the effective recurrent capacity to zero and refuse startup. Iterate the reserve bucket downward against the fitted capacity until it stabilises (the bucket only shrinks, so it converges), and clamp the fitted batch to the reserved bucket so the later bucket_for(effective) allocation never exceeds the reserve. Absorbs the codex review comment on Ma1oneZhang/pegainfer PR pegainfer-project#946 (tp_executor.rs graph_slot_reserve). Signed-off-by: Ziyang Zhang <hafuhafu@qq.com>
This was referenced Aug 30, 2026
Contributor
Author
|
@codex please review |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Split 3/4 of #946, tracked in #1001. Stacked on #1003 and #1004 (contains their commits). Merge last of the three.
Enables CUDA Graph decode under TP for rank-local GQA groups that have a compiled batch-decode kernel, gated at load time via
LocalGeometry::local_decode_group_is_compiled(); uncompiled groups (e.g. 27B group 6) stay on the batched eager path byte-for-byte. Graph bucket state pre-allocates one fixed-address recurrent slot per bucket position.Also absorbs a codex review fix from #946: the graph-slot pre-reserve is now sized by iterating the effective batch bucket instead of
bucket_for(requested_max_batch), so a tight-memory rank is no longer over-reserved into zero effective capacity, and the laterbucket_for(effective)allocation can never exceed the reserve.Type of Change
Evidence
cargo check -p pegainfer-qwen35 --features qwen35 --all-targets: clean.cargo test -p pegainfer-qwen35 --features qwen35 --lib: 109/109 pass (7 GPU-gated skipped), includingunified_step_decode_matches_graph_decodeand the graph-gate tests.cargo fmt --all -- --check: clean.The graph on/off A/B (+8.7% output tok/s, −8.8% TPOT @ 16 concurrent) was measured 2026-08-20 on the pre-rebase #946 branch; GPU rerun on this rebased stack is tracked in #1001 and not re-claimed here.
Checklist
docs/conventions/coding-style.md).CLAUDE.md).