Skip to content

Update tokenizer pin #16323

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
mergennachin merged 3 commits into from
Jan 13, 2026
Merged

Update tokenizer pin #16323

mergennachin merged 3 commits into from
Jan 13, 2026

Conversation

@mergennachin
Copy link
Contributor

@mergennachin mergennachin commented Dec 18, 2025 •
edited
Loading

Here are the new commits:

meta-pytorch/tokenizers@3aada3f...37e1c7e

Copilot AI review requested due to automatic review settings December 18, 2025 18:49
@pytorch-bot
Copy link

pytorch-bot bot commented Dec 18, 2025 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/16323

Note: Links to docs will display an error until the docs builds have been completed.

⏳ No Failures, 27 Pending

As of commit 459c6dc with merge base 35eb01a (image):
💚 Looks good so far! There are no failures yet. 💚

UNSTABLE - The following job is marked as unstable, possibly due to flakiness on trunk:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Dec 18, 2025
@github-actions
Copy link

github-actions bot commented Dec 18, 2025

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

Copilot AI reviewed Dec 18, 2025
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the git submodule reference for the tokenizers component to a newer commit.

Key Changes:

  • Updates the tokenizers submodule commit hash from 3aada3fe28c945d14d5ec62254eb56ccdf10eb11 to 7b26e65a22f70dc0021612b12873fa0af818b51e

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@mergennachin mergennachin force-pushed the update_tokenizer_pin_2 branch from 460d953 to 14c6958 Compare January 12, 2026 17:44
Copilot AI review requested due to automatic review settings January 12, 2026 18:21
Copilot AI reviewed Jan 12, 2026
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@mergennachin mergennachin force-pushed the update_tokenizer_pin_2 branch from 1736310 to d2d0249 Compare January 12, 2026 20:14
Copilot AI review requested due to automatic review settings January 13, 2026 00:43
@mergennachin mergennachin force-pushed the update_tokenizer_pin_2 branch from d2d0249 to bf12149 Compare January 13, 2026 00:43
@mergennachin mergennachin requested review from lucylq and removed request for jackzhxng January 13, 2026 00:43
Copilot AI reviewed Jan 13, 2026
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

mergennachin
mergennachin commented Jan 13, 2026
View reviewed changes
.buckconfig

[buildfile]
name = TARGETS
name_v2 = TARGETS,BUCK
Copy link
Contributor Author

@mergennachin mergennachin Jan 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bigfootjon

Is this correct?

One of the git submodules of ET, i.e., tokenizers, now contains BUCK file instead of TARGET or targets.bzl file.

Copy link
Contributor

@lucylq lucylq Jan 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks good based on D52947777, from https://fb.workplace.com/groups/fbcode.fyi/permalink/7176797622355977/

Copy link
Member

@bigfootjon bigfootjon Jan 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah that's the correct value

@mergennachin mergennachin merged commit b91e6e2 into main Jan 13, 2026
307 of 309 checks passed
@mergennachin mergennachin deleted the update_tokenizer_pin_2 branch January 13, 2026 02:32
@mergennachin mergennachin mentioned this pull request Jan 13, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lucylq lucylq lucylq approved these changes

Copilot code review Copilot Copilot left review comments

@larryliu0820 larryliu0820 Awaiting requested review from larryliu0820 larryliu0820 is a code owner

@bigfootjon bigfootjon Awaiting requested review from bigfootjon

Assignees

No one assigned

Labels

ciflow/trunk CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@mergennachin @bigfootjon @lucylq