CharBench

Character-level benchmark and analysis suite for LLMs.

The dataset is available through HuggingFace.

For complete details on data curation and evaluation, see the paper.

Installation

1. Use Python 3.10 or newer.

2. From the repo root, install dependencies:

   python -m pip install -r requirements.txt

3. Create a .env file with your credentials:

   OPENAI_API_KEY=your_openai_key
   TOGETHER_API_KEY=your_together_key
   HUGGINGFACE_TOKEN=your_huggingface_token
   

OpenAI and Together require valid API keys. Hugging Face tokenization requires an access token.

Full reproducibility

Run the complete pipeline:

python main.py

This builds the benchmark, evaluates all models used in the paper, runs tokenization with each tokenizer, and produces analysis plots.

Expected outputs:

1. benchmark.csv
2. evaluation_results.csv
3. plots/ (directory with figures)

Modular reproducibility

Run individual steps as needed:

1. Construct the benchmark

   python benchmark/benchmark.py

2. Evaluate OpenAI models via API

   python evaluation/eval_openai.py

3. Evaluate Together.ai models via API

   python evaluation/eval_together.py

4. Tokenize benchmark words with Hugging Face and TikToken

   python evaluation/tokenize_results.py

5. Run analysis and generate plots

   python analysis/analyze.py

Modifications

These are the possible args and flags for each script:

main.py

python main.py [--steps ...] [-c CORPUS] [-n NUM] [-b PATH] [-s SEED] [-e PATH] [--task TASK] [-p DIR]

Args: --steps default "create_benchmark eval_openai eval_together tokenize analyze", -c/--corpus JeanKaddour/minipile, -n/--num_unique_words 100000, -b/--benchmark_file benchmark.csv, -s/--seed 42, -e/--evaluation_file evaluation_results.csv, --task all, -p/--output_plots_folder plots.

evaluation/eval_openai.py

python evaluation/eval_openai.py [-m MODELS...] [-i PATH] [-o PATH] [--batch_size N] [--task TASK]

Defaults: -m gpt-4o gpt-4o-mini gpt-3.5-turbo, -i benchmark.csv, -o evaluation_results.csv, --batch_size 100, --task all.

evaluation/eval_together.py

python evaluation/eval_together.py [-m MODELS...] [-i PATH] [-o PATH] [--task TASK] [--batch_size N]

Defaults: -m meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo meta-llama/Llama-3.3-70B-Instruct-Turbo deepseek-ai/DeepSeek-V3 mistralai/Mistral-7B-Instruct-v0.2, -i benchmark.csv, -o evaluation_results.csv, --task all, --batch_size 20.

evaluation/tokenize_results.py

python evaluation/tokenize_results.py [-i PATH]

Defaults: -i/--input valuation_results.csv.

analysis/analyze.py

python analysis/analyze.py [-i PATH] [-o DIR] [-m MODELS...]

Defaults: -i/--input evaluation_results.csv, -o/--output_folder plots, -m/--models meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo, meta-llama/Llama-3.3-70B-Instruct-Turbo, deepseek-ai/DeepSeek-V3, mistralai/Mistral-7B-Instruct-v0.2, gpt-4o, gpt-4o-mini, gpt-3.5-turbo.

Citation

If you use CharBench, please cite:

@inproceedings{uzan2026charbench,
  title         = {CharBench: Evaluating the Role of Tokenization in Character-Level Tasks},
  author        = {Uzan, Omri and Pinter, Yuval},
  booktitle     = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year          = {2026},
  address       = {Singapore},
  note          = {AAAI-26},
  eprint        = {2508.02591},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2508.02591}
}