[None][feat] Nano-v3 stack PRs v2 #9062
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: William Zhang <[email protected]>
Signed-off-by: Lucas Liebenwein <[email protected]>
Signed-off-by: Lucas Liebenwein <[email protected]>
Signed-off-by: Chenghao Zhang <[email protected]> Add conv act fusion Signed-off-by: Chenghao Zhang <[email protected]> fix unit tests Signed-off-by: Suyog Gupta <[email protected]> fix tests Signed-off-by: Suyog Gupta <[email protected]> Address reviewer's comments Signed-off-by: Chenghao Zhang <[email protected]>
Signed-off-by: Suyog Gupta <[email protected]> fix typo Signed-off-by: Suyog Gupta <[email protected]>
Signed-off-by: Neta Zmora <[email protected]> Fixes and UT Signed-off-by: Neta Zmora <[email protected]> Use trtllm moe for relu2 mlp case Signed-off-by: Chenghao Zhang <[email protected]> Fix the runGemmProfile Signed-off-by: Chenghao Zhang <[email protected]> Replace the FP8 fused MoE backend Before: torch.ops.auto_deploy.triton_quant_fp8_moe After: torch.ops.auto_deploy.trtllm_quant_fp8moe_fused Signed-off-by: Neta Zmora <[email protected]> Code refactoring Signed-off-by: Neta Zmora <[email protected]> syntax error fixes Signed-off-by: Neta Zmora <[email protected]> remove dead code Signed-off-by: Neta Zmora <[email protected]> fix moe operator function name Signed-off-by: Neta Zmora <[email protected]> Add skips if not hopper+ Signed-off-by: Neta Zmora <[email protected]> remove unused code Signed-off-by: Neta Zmora <[email protected]>
1 task
Signed-off-by: Wanli Jiang <[email protected]>
Wanli-Jiang
force-pushed
the
user/williamj/nano-v3-fp8-stack
branch
from
49a0c40 to
7664abb
Compare
Member
|
fyi, just merged #8812 --> so you can drop the PR from the list next time you rebase |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Features:
This PR stacked the following PRs on top of TRTLLM main.
[None][fixes] Add tool call parsing fixes and Qwen3 coder parser #8817 Qwen3 code tool parser
[#8763][feature] AutoDeploy: configurable dtype for caching #8812 mamba cache dtype config (Can be bf16 or fp32)
[None][feat] AutoDeploy: Perf improvement for mamba layers #8991 fuse silu to causal conv1d
fix prefill nv-auto-deploy/TensorRT-LLM#156 fix prefill
[#8732][feat] Update TRTLLM Cutlass MoE kernels with ReLU2 #9011 TRTLLM cutlass MoE kernel
Added nano_v3_bench.yaml and nano_v3_accuracy.yaml for bench and accuracy checking.
CMD to launch trtllm-serve for bench: