[GAUDISW-245117] add b2b matmul #770
Conversation
linoybu
requested review from
adobrzyn,
afierka-intel,
iboiko-habana,
kamil-kaczor,
ksmusz,
kzawora-intel,
mgawarkiewicz-intel,
michalkuligowski and
xuechendi
as code owners
🚧 CI BlockedThe main CI workflow was not started for the following reason:
|
There was a problem hiding this comment.
Pull request overview
This PR introduces a B2BMatmul class for batch-to-block matrix multiplication operations. The change replaces the generic Matmul class with the new B2BMatmul class for batch2block_matmul and block2batch_matmul operations in the attention backend.
- Added new
B2BMatmulclass that inherits fromMatmul - Updated attention backend to use
B2BMatmulfor batch-to-block operations - Modified import statement to include the new class
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| vllm_gaudi/extension/utils.py | Defines the new B2BMatmul class inheriting from Matmul |
| vllm_gaudi/attention/backends/hpu_attn.py | Updates batch2block_matmul and block2batch_matmul to use B2BMatmul instead of Matmul |
| vllm_gaudi/extension/ops.py | Contains an apparent typo in variable name |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
🚧 CI BlockedThe main CI workflow was not started for the following reason:
|
Signed-off-by: linoy buchnik <[email protected]>
linoybu
force-pushed
the
add_b2b_matmul
branch
from
7cd4a46 to
bbebcc1
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
@copilot open a new pull request to apply changes based on the comments in this thread |
Co-authored-by: Copilot <[email protected]> Signed-off-by: Linoy Buchnik <[email protected]>
|
@copilot open a new pull request to apply changes based on the comments in this thread |
| This class is intentionally kept functionally identical to ``Matmul``. | ||
| It exists to provide semantic distinction in the codebase (e.g., for | ||
| patterns that specifically require back-to-back matmul) and to allow | ||
| future customization without changing call sites. |
There was a problem hiding this comment.
maybe edit the comment to be more specific, change back-to-back to batch2block/block2batch and explain the reasoning for it, that it is used by the INC to adjust the scale to the needed values of the input tensor as some of them are discarded by the 2nd input which is kind of a mask mapping
Signed-off-by: Linoy Buchnik <[email protected]>
Signed-off-by: Linoy Buchnik <[email protected]>
Signed-off-by: Linoy Buchnik <[email protected]>
2 participants
No description provided.