Better HF-to-CT2 conversion for Whisper model #1546
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
please rebase for tests to go through |
| config.suppress_ids_begin = model.config.begin_suppress_tokens | ||
| config.alignment_heads = _WHISPER_ALIGNMENT_HEADS.get(model.name_or_path) | ||
|
|
||
| non_lang_special_tokens = [ |
There was a problem hiding this comment.
Can you make a method for this logic ?
If the generation_config.json is used, then what do you thing of also using it to get the lang_ids (or filter the lang tokens if the ids are not aligned), and fallback with your method if the generation_config is not available ?
flyingleafe
force-pushed
the
fix-whisper-converter
branch
from
2de8b2b to
b1d21db
Compare
|
@vince62s rebased |
|
can you fix the tests ? |
|
@vince62s just seen the error, should be fixed now |
|
@vince62s all fixed, good to merge? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
On attempt to convert Whisper-v3 model from HF to ct2 format I encountered two issues:
additional_tokens_idslists resulting from the tokenizer checkpoints inlarge-v3and previous versions, the first language token<|en|>was not included in thelang_idslist, which resulted in the inability of the model to transcribe English text - every such transcription became a translation to some other language;generation_config.jsonof the HF checkpoint, which resulted in auto-generated alignment heads and non-present set of suppressed tokens.Those issues are fixed in this PR. The logic of selecting language IDs got more complicated, but less prone to bugs stemming from any future updates of HF checkpoints.