Reproduction

[dorsa-zeinali]

Hi,
I tried reproducing your code for the 2 bit quantization of llama-2-7b , and it took 4 hours and 12 minutes with 2 RTX 3090 gpus. Is this expected?. Also, for wiki text2 ppl I get 14.59, which is close to your reported value of 16.73. However, for c4, I get a ppl of 56.73, which is not close at all to your reported ppl of 23.73. Is this expected? Could there be a bug on my end? I did not get any errors during the process. Thank you and please let me know.

[AozhongZhang]

@dorsa-zeinali I am not sure about the runtime using the RTX 3090 GPUs. We used A100. For c4, maybe you could try sampling data from c4?

[dorsa-zeinali]

you mean as part of the training data ?

[AozhongZhang]

Yes. But not part of training data. I mean that the whole training data is sampling from c4. The default setting is wikitext2. The running time of 2bit is indeed longer than that of 4bit and 3bit. And I think maybe one RTX 3090 GPU is enough. @dorsa-zeinali

[dorsa-zeinali]

Ok I did that and it worked! I was able to get your numbers. thank you so much.

[dorsa-zeinali]

sorry, just one more question. for the experiments with group size 128, the CD_iter parameter should be 1 or 30?

[AozhongZhang]

CD_iter only depends on bit. If you use 2bit and 128 group size, you should use 30 CD_iter. Otherwise you can use 1 CD_iter. Of course, more CD_iter will lead better performance for 3 and 4 bit. BTW, thanks for following our work. @dorsa-zeinali

[dorsa-zeinali]

ok got it! thank you and thanks for the quick and helpful responses!

[dorsa-zeinali]

when I set group size to 128 I get this error : torch._C._LinAlgError: linalg.cholesky: The factorization could not be completed because the input is not positive-definite (the leading minor of order 3943 is not positive-definite).

[AozhongZhang]

This repo is just for per-channel quantization. For per-group quantization, the setting of the hyper-parameter beta (should be 0.95) is different from per-channel quantization. We describe the details in the experimental section of our paper. And I will update new version about per-group quantization.

[dorsa-zeinali]

oh I see. When do you think the update will happen? also, does a lower "error" that you report during the quantization process correspond to better quantization? thank you.

[AozhongZhang]

I have updated it. BTW, you can change the beta manually. I think lower "error" will lead better performance.

[dorsa-zeinali]

Hi, thank you so much. so I had one more question. in the proximal preprocess, it seems like X is only one batch of the training data. so you do the 200 iterations of linfty proximal gradient descent on each batch and at the end of all batches the final W is produced?

[AozhongZhang]

We only use one batch of training data X for 200 iterations of linfty proximal gradient descent. And each layer just uses one batch training data.

[dorsa-zeinali]

I see! thanks for letting me know

…

________________________________ From: Aozhong Zhang ***@***.***> Sent: Friday, January 3, 2025 9:49:13 PM To: AozhongZhang/MagR ***@***.***> Cc: Dorsa Zeinali ***@***.***>; Mention ***@***.***> Subject: Re: [AozhongZhang/MagR] Reproduction (Issue #1) We only use one batch of training data X for 200 iterations of linfty proximal gradient descent. And each layer just uses one batch training data. — Reply to this email directly, view it on GitHub<#1 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BCOLFSCIAHI43ABW2PM3E3L2I5K3TAVCNFSM6AAAAABULOXZWKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNZQGAZTKNJYGA>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[dorsa-zeinali]

one batch is like 1 example from the wikitext2 right. and is it a different batch for each layer?