build: bump tokenizers to >=0.21.0 for Windows ARM64 support

[Komatlakarthik]

Problem

tokenizers==0.15.2 has no pre-built wheel for Windows ARM64 (Snapdragon X Elite). This causes link.exe build errors and blocks all Windows ARM64 contributors from setting up the project.

Fix

Relaxed the pin from ==0.15.2 to >=0.21.0,<1.0.0. Version 0.21.0 and above ship official ARM64 Windows wheels on PyPI, enabling seamless installation on newer hardware.

Verified on

• Windows 11 ARM64 (Snapdragon X Elite)
• 113/113 tests passing after change (verified with uv run pytest tests/ -v)

Impact

Unblocks Windows ARM64 contributors — an increasingly common hardware platform (Microsoft Copilot+ PCs). This improves the project's accessibility and developer experience for modern Windows environments.

Checklist

• My code follows the project's code style and conventions
• I have made corresponding changes to the documentation
• My changes generate no new warnings or errors
• I have joined the Discord server and I will share a link to this PR with the project maintainers there
• I have read the Contributing Guidelines

Summary by CodeRabbit

• Chores
  • Updated tokenizers dependency constraints to support a wider range of versions (0.21.0 and above, below 1.0.0).

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 023bcc7f-83ff-4112-ac3d-4ba7f4296ac0

📥 Commits

Reviewing files that changed from the base of the PR and between 9fa9789 and 6f67592.

📒 Files selected for processing (1)

• pyproject.toml

---

Walkthrough

The pyproject.toml dependency for tokenizers was loosened from an exact pin (==0.15.2) to a range (>=0.21.0,<1.0.0), allowing installations with newer tokenizers releases within that range.

Changes

Cohort / File(s)	Summary
Dependency Update pyproject.toml	Changed tokenizers dependency from ==0.15.2 to >=0.21.0,<1.0.0 (one-line change).

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~2 minutes

Suggested labels

Python Lang

Poem

🐰 I nudged a pin and set it free,
Tokenizers now roam, wild and glee,
Versions newer, bright as day,
Hop along — the code can play! ✨

🚥 Pre-merge checks | ✅ 2

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly describes the main change: updating the tokenizers dependency version to enable Windows ARM64 support, which directly aligns with the single-line modification in pyproject.toml.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

📝 Coding Plan

• Generate coding plan for human review comments

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@pyproject.toml`:
- Line 17: Update the tokenizers dependency constraint in pyproject.toml to
include an upper bound for stability: change the current "tokenizers>=0.21.0"
entry to a bounded spec such as "tokenizers>=0.21.0,<0.22.0" (or
"tokenizers>=0.21.0,<1.0.0" if you prefer a wider but still safe range) so
future breaking releases won't be pulled automatically; edit the tokenizers line
in pyproject.toml accordingly.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: c887b97a-46cc-4052-a73a-1f90b7e03859

📥 Commits

Reviewing files that changed from the base of the PR and between 578bc79 and 9fa9789.

📒 Files selected for processing (1)

• pyproject.toml