practical implementation difference between elementwise multiplication and matrix multiplication

[meetu30]

Hi,
I understand how matrix multiplication and element wise multiplication is done. My question is practically, if i have 2 matrices 3*3, when should I use matrix multiplication or element wise multiplication? Kindly guide. Thanks in advance

[mrdbourke]

Hi @meetu30 ,

Generally all operations in deep learning are matrix multiplications rather than element-wise operations.

For example:

import torch

X = torch.randn(1, 10)
y = torch.randn(1, 10)

z = torch.matmul(X, y.T) # transpose required to satisfy matrix multiplication rules

And see the math behind the code of torch.nn.Linear:

For a more thorough guide, see this blog post: https://mkang32.github.io/python/2020/08/30/numpy-matmul.html

It's about numpy but the same operations can be applied with PyTorch.