Skip to content

Releases: niieani/gpt-tokenizer

2.0.0-beta.1

23 May 16:59
Compare
Choose a tag to compare
2.0.0-beta.1 Pre-release
Pre-release

2.0.0-beta.1 (2023-05-23)

Features

  • complete rewrite to support different models (eedd944), closes #5 #6

BREAKING CHANGES

  • default encoder is now GPT3.5 / GPT4

1.0.5

16 May 07:19
f7df7e9
Compare
Choose a tag to compare

1.0.5 (2023-05-16)

Bug Fixes

1.0.4

01 May 05:44
Compare
Choose a tag to compare

1.0.4 (2023-05-01)

Bug Fixes

  • eslint and jest complaints (8b1cb01)
  • importing/requiring under certain setups was not working (8b71131), closes #4

1.0.3

18 Apr 08:35
7b9e7fd
Compare
Choose a tag to compare

1.0.3 (2023-04-18)

Bug Fixes

1.0.2

18 Apr 08:16
Compare
Choose a tag to compare

1.0.2 (2023-04-18)

Bug Fixes

1.0.1

16 Apr 06:44
Compare
Choose a tag to compare

1.0.1 (2023-04-16)

Bug Fixes

1.0.0

16 Apr 06:35
Compare
Choose a tag to compare

1.0.0 (2023-04-16)

Bug Fixes

  • remove global cache memory leak (40a2985), closes #35

Features

  • add a way to count tokens without encoding the whole text (88c8590)
  • add decodeAsyncGenerator and support any iterable input (41175f4)