Add autoquant support for torchao quantizer #35503
Conversation
|
README can be updated after #35490 is landed |
|
cc @SunMarc can you help review? |
|
@SunMarc please take a look again, thanks |
There was a problem hiding this comment.
Nice, thanks for adding this ! Thanks for updating the docstring. It would be even better if you can add a paragraph on autoquant inside torchao docs there : https://huggingface.co/docs/transformers/main/en/quantization/torchao ?
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
| output = quantized_model.generate(**input_ids, max_new_tokens=self.max_new_tokens) | ||
| quantized_model.finalize_autoquant() |
There was a problem hiding this comment.
Make sure to include this in the doc, otherwise it will be hard for the user to understand how to use autoquant. Instead of generate, can we just call the forward of the model or it makes a difference ?
There was a problem hiding this comment.
I think we should call the codepath that is finally being tested, in this case generate
There was a problem hiding this comment.
If you are calling generate you need to make sure the static cache is used otherwise compilation will happen at each steps
|
gentle ping @ArthurZucker as @jerryzh168 will soon be in PTO. |
|
@ArthurZucker can you help land this? |
|
Can you update the documentation @jerryzh168 ? After that, I will merge the PR |
There was a problem hiding this comment.
LGTM, this needs some default generation config as well :
- static cache
- disable compile add disable compile option #36161 will bring it!
| output = quantized_model.generate(**input_ids, max_new_tokens=self.max_new_tokens) | ||
| quantized_model.finalize_autoquant() |
There was a problem hiding this comment.
If you are calling generate you need to make sure the static cache is used otherwise compilation will happen at each steps
Summary: att, also verified that autoquantized model can be saved and loaded: save: https://gist.github.com/jerryzh168/01d367aaf44dbbbfd4068a4a10a00061 load: https://gist.github.com/jerryzh168/d5c6c401b2abdf18e0b6771341f1525c Test Plan: tested locally with above script model uploaded to https://huggingface.co/jerryzh168/llama3-8b-autoquant Reviewers: Subscribers: Tasks: Tags:
jerryzh168
force-pushed
the
add-autoquant
branch
from
88ef86d to
dfbb0a0
Compare
|
@ArthurZucker added the cache_implementation settings, I didn't use disable_compile right now, is it required as well? @SunMarc added autoquant to doc |
Summary:
att, also verified that autoquantized model can be saved and loaded:
save: https://gist.github.com/jerryzh168/01d367aaf44dbbbfd4068a4a10a00061
load: https://gist.github.com/jerryzh168/d5c6c401b2abdf18e0b6771341f1525c
Test Plan:
tested locally with above script
model uploaded to https://huggingface.co/jerryzh168/llama3-8b-autoquant
Reviewers:
Subscribers:
Tasks:
Tags: