Description
Every request through the serving frontend asks the tokenizer wrapper for its vocabulary size: TextRequestProcessor::prepare fills SamplingLimits::tokenizer_vocab_size from HuggingFaceTokenizer::vocab_size(). At the pinned vLLM revision that call answers get_vocab_size(true) from the tokenizers backend, which clones the merged base-plus-added vocabulary into a fresh map to count it. On Gemma 4's 262,144-token vocabulary that is tens of milliseconds of host time per request, spent before the prompt is tokenized, and it is most of the first-token time of a short prompt at low concurrency.
Found while gating the 26B-A4B serving path for the routed-kernel slice: the first-token time of short prompts did not follow the engine. The count never changes after load; a cached count in the wrapper removes the cost. Measured on this server with that cache applied to the pinned crate (a local vendored copy, otherwise byte-identical), single sm_89 GPU, Gemma 4 26B-A4B NVFP4, one cooled round per arm:
|
tiny prompt end-to-end, median |
c1 first token, median / p99 |
c4 first token, median / p99 |
c16 first token, median / p99 |
| pinned wrapper |
48.2 ms |
117.8 / 129.3 ms |
266.4 / 424.7 ms |
435.9 / 1613.1 ms |
| cached count |
12.5 ms |
84.3 / 92.2 ms |
112.1 / 394.9 ms |
333.7 / 1586.4 ms |
|
c1 out tok/s / TPOT |
c4 out tok/s / TPOT |
c16 out tok/s / TPOT |
| pinned wrapper |
96.9 / 9.48 ms |
259.9 / 13.34 ms |
545.1 / 25.98 ms |
| cached count |
99.6 / 9.44 ms |
261.8 / 14.30 ms |
550.7 / 26.35 ms |
Tiny prompt: <bos>hi, one output token, 30 timed public requests after 5 warm ones. Concurrency rows: vllm bench serve, random 1024 in / 128 out, 8 / 24 / 64 prompts at c1 / c4 / c16, greedy, ignore_eos, cooled between runs. A second round on the cached arm read 12.2 ms for the tiny prompt and 88.6 ms for the c1 median before the run was stopped; the throughput and TPOT columns are one round each and their few-percent differences are unresolved. The change touches nothing after tokenization.
Description
Every request through the serving frontend asks the tokenizer wrapper for its vocabulary size:
TextRequestProcessor::preparefillsSamplingLimits::tokenizer_vocab_sizefromHuggingFaceTokenizer::vocab_size(). At the pinned vLLM revision that call answersget_vocab_size(true)from thetokenizersbackend, which clones the merged base-plus-added vocabulary into a fresh map to count it. On Gemma 4's 262,144-token vocabulary that is tens of milliseconds of host time per request, spent before the prompt is tokenized, and it is most of the first-token time of a short prompt at low concurrency.Found while gating the 26B-A4B serving path for the routed-kernel slice: the first-token time of short prompts did not follow the engine. The count never changes after load; a cached count in the wrapper removes the cost. Measured on this server with that cache applied to the pinned crate (a local vendored copy, otherwise byte-identical), single
sm_89GPU, Gemma 4 26B-A4B NVFP4, one cooled round per arm:Tiny prompt:
<bos>hi, one output token, 30 timed public requests after 5 warm ones. Concurrency rows:vllm bench serve, random 1024 in / 128 out, 8 / 24 / 64 prompts at c1 / c4 / c16, greedy,ignore_eos, cooled between runs. A second round on the cached arm read 12.2 ms for the tiny prompt and 88.6 ms for the c1 median before the run was stopped; the throughput and TPOT columns are one round each and their few-percent differences are unresolved. The change touches nothing after tokenization.