[WIP] feat: Add UDS-based external tokenizer service #137
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
delavet
requested review from
vMaroon,
dannyharnik,
elevran and
kfirtoledo
as code owners
Signed-off-by: Hang Yin <[email protected]>
delavet
force-pushed
the
external-tokenizer
branch
from
ec44bd5 to
d29dd1d
Compare
Signed-off-by: Hang Yin <[email protected]>
vMaroon
reviewed
Oct 6, 2025
| @@ -0,0 +1,129 @@ | |||
| # Model Caching in Tokenizer Service | |||
There was a problem hiding this comment.
I think that the uds_tokenizer package would be better housed in a new services directory.
| 1. Once from the chat template itself (which may include BOS token) | ||
| 2. Once from the add_special_tokens parameter | ||
|
|
||
| vLLM handles this by setting add_special_tokens=False when using chat templates. |
There was a problem hiding this comment.
Do you think it would make sense to extract and reuse vLLM preprocessing code as-is? Serving as a lightweight vLLM sub-component for disaggregated tokenization. Its maintenance would be syncing up versions and dependencies.
Not a blocker, for this open PR, but I think we should aim towards this path. What do you think?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces a new Unix Domain Socket (UDS) based tokenizer service that can be used as an external tokenizer for the KV Cache Manager, related to #126. The changes include:
Added a new tokenizer mode in the KV Cache Manager, which communicate with an external tokenizer service via http over UDS.
Added an example external tokenizer service with:
Updated the Helm chart to deploy the external tokenizer as a sidecar container alongside the KV Cache Manager
Added configuration options to enable the external tokenizer in the kv events online example.