Commit 33b2a6e
committed
Update tokenizer pin
1 parent 7436438 commit 33b2a6e
1 file changed
Lines changed: 1 addition & 1 deletion
Submodule tokenizers updated 21 files
- TARGETS-5
- include/pytorch/tokenizers/bpe_tokenizer_base.h+2
- include/pytorch/tokenizers/llama2c_tokenizer.h+2
- include/pytorch/tokenizers/sentencepiece.h+2
- include/pytorch/tokenizers/tokenizer.h+15
- pyproject.toml+2-1
- setup.py+2-1
- src/bpe_tokenizer_base.cpp+18
- src/hf_tokenizer.cpp+25-3
- src/llama2c_tokenizer.cpp+5
- src/sentencepiece.cpp+11
- targets.bzl-218
- test/resources/test_llama2c_tokenizer.bin
- test/targets.bzl+2-6
- test/test_hf_tokenizer.cpp+253
- test/test_llama2c_tokenizer.cpp+43-6
- test/test_sentencepiece.cpp+36
- test/test_tekken.cpp+32
- test/test_tiktoken.cpp+29
- third-party/TARGETS-5
- third-party/targets.bzl-17
0 commit comments