Repository navigation
Commit b009389
fix: register g9v3 on BasicLLMProcessor for HF Fast tokenizer
G9v3 ships tokenizer.json (PreTrainedTokenizerFast / GPT-2 BPE). The
SentencePiece decode patch was leaking Ġ/Ċ and mojibake in smoke output.
Co-authored-by: Cursor <cursoragent@cursor.com>1 parent 51ff3a8 commit b009389
1 file changed
Lines changed: 7 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
79 | 79 | | |
80 | 80 | | |
81 | 81 | | |
82 | | - | |
83 | 82 | | |
84 | 83 | | |
85 | 84 | | |
86 | 85 | | |
87 | 86 | | |
88 | 87 | | |
89 | 88 | | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
0 commit comments