Add fp8 calibration procedure #309
Conversation
Signed-off-by: Artur Fierka <[email protected]>
🚧 CI BlockedThe main CI workflow was not started for the following reason:
|
🚧 CI BlockedThe main CI workflow was not started for the following reason:
|
🚧 CI BlockedThe main CI workflow was not started for the following reason:
|
Signed-off-by: Artur Fierka <[email protected]>
🚧 CI BlockedThe main CI workflow was not started for the following reason:
|
🚧 CI BlockedThe main CI workflow was not started for the following reason:
|
|
Thank you @skavulya for all proposed fixes! Code updated and validated with Llama3.1-8B-Instruce. Accuracy on quantized FP8 model is marginally lower than BF16. |
afierka-intel
requested review from
adobrzyn,
kzawora-intel,
mgawarkiewicz-intel,
michalkuligowski,
mswiniarsk,
vivekgoe and
xuechendi
as code owners
✅ CI PassedAll checks passed successfully against the following vllm commit: |
✅ CI PassedAll checks passed successfully against the following vllm commit: |
michalkuligowski
left a comment
michalkuligowski
left a comment
There was a problem hiding this comment.
Please change extension mentions
Signed-off-by: Artur Fierka <[email protected]>
🚧 CI BlockedThe main CI workflow was not started for the following reason:
|
✅ CI PassedAll checks passed successfully against the following vllm commit: |
Signed-off-by: Artur Fierka <[email protected]>
|
/skip-gaudi-tests |
|
Only codeowners and testowners can request to run Gaudi tests. Contact list: kzawora-intel, xuechendi, mswiniarsk, adobrzyn, mgawarkiewicz-intel, vivekgoe, afierka-intel, michalkuligowski, iboiko-habana, PatrykWo, kamil-kaczor, kfojcik-intel, ksmusz, wuxun-zhang, xuechendi, attafosu, ulivne, Kacper-Pietkun, iboiko-habana, jkaniecki |
✅ CI PassedAll checks passed successfully against the following vllm commit: |
Porting the FP8 calibration procedure from vllm-hpu-extension: https://github.com/HabanaAI/vllm-hpu-extension/tree/main/calibration --------- Signed-off-by: Artur Fierka <[email protected]>
4 participants
Porting the FP8 calibration procedure from vllm-hpu-extension: https://github.com/HabanaAI/vllm-hpu-extension/tree/main/calibration