Skip to content

Commit b009389

Browse files
fix: register g9v3 on BasicLLMProcessor for HF Fast tokenizer
G9v3 ships tokenizer.json (PreTrainedTokenizerFast / GPT-2 BPE). The SentencePiece decode patch was leaking Ġ/Ċ and mojibake in smoke output. Co-authored-by: Cursor <cursoragent@cursor.com>
1 parent 51ff3a8 commit b009389

1 file changed

Lines changed: 7 additions & 1 deletion

File tree

‎python/infinilm/processors/sentencepiece_processor.py‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -79,11 +79,17 @@ class MistralProcessor(SentencePieceProcessor):
7979
@register_processor("minicpm")
8080
@register_processor("minicpm4")
8181
@register_processor("minicpm5_moe")
82-
@register_processor("g9v3")
8382
class MiniCPMProcessor(SentencePieceProcessor):
8483
pass
8584

8685

8786
@register_processor("deepseek_v2")
8887
class DeepSeekV2Processor(BasicLLMProcessor):
8988
pass
89+
90+
91+
# G9v3 HF checkpoint uses PreTrainedTokenizerFast + tokenizer.json (GPT-2 BPE),
92+
# not SentencePiece — use default HF decode (Ġ/Ċ), not the ▁ SP patch.
93+
@register_processor("g9v3")
94+
class G9v3Processor(BasicLLMProcessor):
95+
pass

0 commit comments

Comments
 (0)