Tiktoken

Elixir bindings for OpenAI's Tiktoken.

Built on @zurawiki's tiktoken-rs.

Installation

If available in Hex, the package can be installed by adding tiktoken to your list of dependencies in mix.exs:

def deps do
  [
    {:tiktoken, "~> 0.3"}
  ]
end

Required dependencies to compile the shared library

You will need to have a Rust compiler and Python installed.

For example on debian/ubuntu this means:

apt install cargo libpython3-dev

To install more up-to-date versions of Rust check rustup.

The libraries the bindings are built on use PyO3 so you may need to set the PyO3 flag before compiling:

export PYO3_PYTHON=python
# or depending on your OS
export PYO3_PYTHON=python3

Force compilation of the shared library

If the precompiled shared library are not available, you can force rustler_precompiled to compile it by adding the following to you config.exs:

config :rustler_precompiled, :force_build, tiktoken: true

Documentation can be generated with ExDoc and published on HexDocs. Once published, the docs can be found at https://hexdocs.pm/tiktoken.