perf(qwen35): batch eager decode rows under TP (split 2/4 of #946) - #1004
Open
Ma1oneZhang wants to merge 4 commits into
Open
perf(qwen35): batch eager decode rows under TP (split 2/4 of #946)#1004Ma1oneZhang wants to merge 4 commits into
Ma1oneZhang wants to merge 4 commits into
Conversation
Phase 2b ported onto pegainfer-project#870: recurrent/conv state, GDR scratch, and the linear-attention weight surface are allocated and addressed per rank instead of replicated, which is what makes 27B TP2 fit on 2x48 GB cards. - weight_loader: additive stitch/shard loaders (2D row stitch for the fused qkv [q|k|v] segments, 1D element stitch for conv1d channels, bf16/f32 1D shards for dt_bias/A_log) - config: local_linear_* accessors mirroring the full-attn TP style; linear head divisibility fails closed in TensorParallelConfig - weights: per-rank stitched shards for in_proj_qkv/conv1d (head-local slices per segment), row shards for z/b/a, col shard for out_proj, dt_bias/A_log sliced, norm_weight kept replicated (head-shared); loader reserve uses TP-aware estimates - recurrent_state/decode_buffers/prefill_buffers: state and GDR scratch at local value-head/qkv sizes; capacity math derives from local allocation_bytes - batch_decode/prefill: local head counts into the GDR decode/conv/ Triton-AOT prefill chains, gated RMSNorm at local v heads, all-reduce after linear out_proj; batch_decode_full_attention_via_prefill is now TP-local so eager decode routes 27B TP2 group-6 full attention through prefill (was FlashInfer Unsupported group_size: 6) - tp_executor: worker capacity math and per-request state use the rank-local sizes; decode rows still run as a per-request bs=1 loop (batched in a follow-up) Recurrent/conv state is never all-reduced. Signed-off-by: Ziyang Zhang <hafuhafu@qq.com>
This was referenced Aug 30, 2026
Contributor
Author
|
@codex please review |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Ma1oneZhang
force-pushed
the
feat/qwen35-tp-batched-decode
branch
2 times, most recently
from
August 30, 2026 14:20
1049aa4 to
07bde36
Compare
The TP rank-sliced 1D f32 loader casts the safetensors byte span to f32 exactly like the whole-tensor loader next to it (which already carries the allow); the missed attribute trips clippy::cast-ptr-alignment under the workspace's -D warnings gates. Signed-off-by: Ziyang Zhang <hafuhafu@qq.com>
…f32s via as_chunks if_not_else and chunks_exact_to_as_chunks (pedantic/default) break the Qwen3.5 clippy gate under -D warnings. Signed-off-by: Ziyang Zhang <hafuhafu@qq.com>
Port of the batched eager TP decode step onto pegainfer-project#870's worker structure: decode rows in one command now run as ONE batched forward per step on every rank plus one batched rank-0 sampling pass, instead of a per-request bs=1 loop. - run_decode_batch resolves every decode row's worker state in command order, builds a step-scoped LinearStatePointerTables over the whole batch (from_recurrent_refs(..., bs, ...)), runs one batch_decode_eager_logits forward, then rank 0 snapshots all requested logprob rows before one batched select_batch over per-row params - execute_decode_rows (used by both decode-only and unified steps) calls run_decode_batch once; per-row results fan out in command order - TpRequestState.linear_pointer_tables (capacity-1, decode-only) removed; ensure_prefill_state no longer builds it. The step-scoped table is rebuilt every step, so swap_remove retirement can't stale it Seeded rows keep per-row semantics: select_batch isolates each seeded row into its own single-row philox call keyed on (request seed, step 0), so seeded output stays independent of batch composition. Unseeded rows decorrelate via the per-step command seed, same as the single-GPU batched path. Reference (27B TP2, 2x RTX 4090, eager): 16 concurrent 256-token completions aggregate 24.9 -> 292.3 tok/s; single-request unchanged. Signed-off-by: Ziyang Zhang <hafuhafu@qq.com>
Ma1oneZhang
force-pushed
the
feat/qwen35-tp-batched-decode
branch
from
August 30, 2026 14:33
07bde36 to
1135791
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Split 2/4 of #946, tracked in #1001. Stacked on #1003 (contains its commit); followed by #1005. Merge order: PR1 → this PR → PR3.
Batches the eager TP decode path: one batched decode step across ready rows per TP rank instead of per-request steps, including the paged full-attention fallback for TP-local GQA groups without a compiled batch-decode kernel (27B group-6 case).
Type of Change
Evidence
cargo check -p pegainfer-qwen35 --features qwen35 --all-targets: clean.cargo test -p pegainfer-qwen35 --features qwen35 --lib: 106/106 pass (7 GPU-gated skipped).cargo fmt --all -- --check: clean.Throughput A/B numbers cited in #946 come from its 2026-08-20 pre-rebase measurement; rerun on this stack is tracked in #1001.
Checklist
docs/conventions/coding-style.md).CLAUDE.md).