Thank you for the good work!
When I train with normal LoRA, the sparse_indices is a list sorted by importance. However, when I use QLoRA with 4-bit training, I noticed that in spft/modules/linear4bit.py at line 125, the sparse_indices is just a sequential list from 0 to 142, which is abnormal. Could you please let me know if there are any additional requirements for the 4-bit configuration in order to fix this issue?
Thank you for the good work!
When I train with normal LoRA, the
sparse_indicesis a list sorted by importance. However, when I use QLoRA with 4-bit training, I noticed that inspft/modules/linear4bit.pyat line 125, thesparse_indicesis just a sequential list from 0 to 142, which is abnormal. Could you please let me know if there are any additional requirements for the 4-bit configuration in order to fix this issue?