-
Notifications
You must be signed in to change notification settings - Fork 197
Fix BMM style MoE export in fp8_pc_pt recipe #515
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #515 +/- ##
=======================================
Coverage 74.45% 74.45%
=======================================
Files 182 182
Lines 18250 18250
=======================================
Hits 13588 13588
Misses 4662 4662 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
4bb85d1 to
fbd5417
Compare
|
Could you share what models have you tested? |
Signed-off-by: Zhiyu Cheng <[email protected]>
Signed-off-by: Zhiyu Cheng <[email protected]>
8d7fe0b to
b4b6d9c
Compare
Signed-off-by: Zhiyu Cheng <[email protected]>
Signed-off-by: Zhiyu Cheng <[email protected]>
Llama4 scout is tested |
meenchen
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Will defer to @cjluo-nv for approval.
| return (weight / weights_scaling_factor[:, None, None]).to(torch.float8_e4m3fn) | ||
| elif weights_scaling_factor.dim() == 2: | ||
| # Per-channel scaling: check which dimension matches | ||
| if weights_scaling_factor.shape[0] != weight.shape[0]: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can we just do assert here instead for simplification? Same for line 794
What does this PR do?
Type of change: Bug fix
Overview: The Llama-4-Scout-17B-16E-Instruct model uses Llama4TextExperts, which stores expert weights in a BMM (batch matrix multiply) layout: (num_experts, input_dim, output_dim). This is different from standard MoE models. The FP8_PC_PT (FP8 per-channel per-token) quantization code didn't handle this layout properly, causing shape mismatches.
Usage
Testing
Before your PR is "Ready for review"
Additional Information