Releases: niieani/gpt-tokenizer
Releases · niieani/gpt-tokenizer
2.0.0-beta.1
Features
- complete rewrite to support different models (eedd944), closes #5 #6
BREAKING CHANGES
- default encoder is now GPT3.5 / GPT4
1.0.5
1.0.5 (2023-05-16)
Bug Fixes
1.0.4
1.0.4 (2023-05-01)
Bug Fixes
- eslint and jest complaints (8b1cb01)
- importing/requiring under certain setups was not working (8b71131), closes #4
1.0.3
1.0.3 (2023-04-18)
Bug Fixes
1.0.2
1.0.2 (2023-04-18)
Bug Fixes
1.0.1
1.0.1 (2023-04-16)
Bug Fixes
1.0.0
1.0.0 (2023-04-16)
Bug Fixes
- remove global cache memory leak (40a2985), closes #35
Features
- add a way to count tokens without encoding the whole text (88c8590)
- add decodeAsyncGenerator and support any iterable input (41175f4)