What does the multi_lora_a and multi_lora_b mean in the code?

[CaffreyR]

Hi, may I ask that what does the multi_lora_a mean ? Is there any paper that has explained it ? Many thanks!

t-few/src/models/lora.py

Line 22 in 43fdb51

if self.scaling_rank:

[HaokunLiu]

It's a notation from LoRA. LoRA uses the product of two matrices BA to create a low-rank matrix. In the original LoRA design, BA should be added to W. Multi means instead of adding to W, we use elementwise multiplication. In fact, if we (i) change add to multiply, (ii) set the rank is set to 1, and (iii) only keep the three internal parameters, we can derive IA3 from LoRA.

[CaffreyR]

Hi @HaokunLiu , is there any work that shows the equation of this method? What is the meaning of elementwise multiplication? Thanks!

[HaokunLiu]

Wait, somehow I didn't see your reply. Sorry.
For the equation of LoRA, you can refer to their paper. https://arxiv.org/abs/2106.09685
Element-wise multiplication simply means when you have two tensors of the same shape (or one tensor is expandable to have the same shape as the other), for instance X = (5, 3, 16) 3D-tensor, Y = (3, 16) 2D-matrix. you do multiplication at every location. So the output Z = (5, 3, 16) 3D-tensor will be Z_(i,j,k) = X(i,j,k) * Y(j,k), for all the i in {0,1,...,3}, j in {0, 1,2}, and k in {0, 1, ..., 16).