For LLM forward passes, it is necessary to support matmul with a transposed B matrix. This is how the weights are stored, and so supporting a transposed B in the kernels avoids having to materialize the transpose at either load time or forward time.
The RFC is vague on how to do this. It seems to suggest that linalg.matmul_transpose_b is a possibility (it leaves the details of linalg rather open to interpretation). But also suggests that one should separate compute from addressing via indexing maps.
I believe MLIR does support matmul_transpose_b, but seems to wants to deprecate it in favor of indexing maps.
So I wanted to check with the rest of the team on this. What is the preferred approach?
if I understand correctly, the Python emulator supports neither: no matmul_transpose_b, and while it does parse indexing maps matmul currently does not use them? hence my questions here thanks!!
Note: for the rust work #117 I have opted for the indexing map approach.
For LLM forward passes, it is necessary to support matmul with a transposed B matrix. This is how the weights are stored, and so supporting a transposed B in the kernels avoids having to materialize the transpose at either load time or forward time.
The RFC is vague on how to do this. It seems to suggest that
linalg.matmul_transpose_bis a possibility (it leaves the details oflinalgrather open to interpretation). But also suggests that one should separate compute from addressing via indexing maps.I believe MLIR does support
matmul_transpose_b, but seems to wants to deprecate it in favor of indexing maps.So I wanted to check with the rest of the team on this. What is the preferred approach?
if I understand correctly, the Python emulator supports neither: no
matmul_transpose_b, and while it does parse indexing mapsmatmulcurrently does not use them? hence my questions here thanks!!Note: for the rust work #117 I have opted for the indexing map approach.