Skip to content

perf(custom_ops): compute FP8 per-token scale via quantscalepertokenfp8 - #814

Open
prashantgupta24 wants to merge 7 commits into
torch-spyre:mainfrom
prashantgupta24:fp8-quantscalepertokenfp8
Open

perf(custom_ops): compute FP8 per-token scale via quantscalepertokenfp8#814
prashantgupta24 wants to merge 7 commits into
torch-spyre:mainfrom
prashantgupta24:fp8-quantscalepertokenfp8

Conversation

@prashantgupta24

@prashantgupta24 prashantgupta24 commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator

Description

Replace the eager per-token FP8 activation scale computation in SpyreFp8LinearKernel with torch.ops.spyre.quantscalepertokenfp8. The op performs the amax reduction, scale division, and hardware clip inside its own compiled Inductor graph, avoiding an eager amax/division boundary and matching the deeptools operator contract. It is kept separate from the FP8 matmul graph because fusing it with _scaled_mm, quantize_fp8_with_scale, and quantize_weight_fp8_with_scale causes the SDSC compiler to fail on the combined kernel. Per-tensor activations keep the existing eager scalar scale path because quantscalepertokenfp8 always reduces over the hidden dimension.

Related Issues

Relates to the torch-spyre quantscalepertokenfp8 op introduced in torch-spyre/torch-spyre#3457.

Test Plan

  • bash format.sh — passed
  • uv run pytest tests/test_fp8_linear.py -m "not upstream" — 17 passed
  • Microbenchmark on Spyre hardware comparing old eager-scale path to new in-graph path showed identical outputs for tested shapes.

Checklist

  • I have read the contributing guidelines
  • My code follows the project's code style (run bash format.sh)
  • I have added tests for my changes (if applicable)
  • I have updated the documentation (if applicable)
  • My commits include a Signed-off-by: line (DCO compliance)

Signed-off-by: Prashant Gupta <prashantgupta24@gmail.com>
@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

👋 Hi! Thank you for contributing.
Just a reminder: Make sure that your code passes all the linting checks, otherwise your PR won't be able to be merged. To do so, run ./format.sh.
Now you are good to go 🚀.

We also recommend installing prek and configuring it to check your code before every local commit.

Signed-off-by: Prashant Gupta <prashantgupta24@gmail.com>
@prashantgupta24
prashantgupta24 marked this pull request as ready for review September 9, 2026 17:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant