Skip to content

Commit 95d2d2b

Browse files
authored
Allow bounded concurrent prefix captures in LLM serving (#24303)
### Summary This PR improves prompt reuse when several agents run in parallel, reducing how often the model processes the same context again. ExecuTorch already supports running multiple requests at once. However, only one request at a time can save its processed prompt for future reuse. While that request is busy, other requests continue running but may miss the opportunity to save their prompts. For example, when the main agent and three subagents overlap: ```text Before With this PR One save at a time Multiple saves allowed Main agent Saves its prompt Saves its prompt Subagent A Save skipped Saves its prompt Subagent B Save skipped Saves its prompt Subagent C Save skipped Saves its prompt All four can run concurrently in both cases. ``` The difference matters on the **next turn**. Suppose a subagent processes 4,000 tokens of context, then sends a follow-up containing that context plus 200 new tokens: ```text Follow-up prompt: +--------------------------------------+-------------+ | Existing context: 4,000 tokens | New: 200 | +--------------------------------------+-------------+ Without a saved prompt: |<-------------- Process all 4,200 ----------------->| With a saved prompt: |<--------- Reuse previous work ------>| Process 200 | ``` Repeated across many subagent turns, these missed saves can create substantial extra work. The PR allows multiple requests to save their processed prompts concurrently: - The limit adjusts automatically to the server's configured request capacity, with an explicit override available. - Temporary storage is accounted for, and reservations are released on completion, cancellation, or failure. - If the limit is reached, requests continue normally; saving the prompt remains optional. The default `--prefix-capture-slots 0` means **automatic sizing**. With prefix caching enabled and intermediate checkpoints disabled: ```text capture slots = min(max_sessions, max_inflight_requests) ``` | Configuration | Automatic capture slots | |---|---:| | Our benchmark: 8 sessions, 64 requests | 8 | | Muse launcher defaults: 4 sessions, 4 requests | 4 | | Generic launcher defaults: 16 sessions, 64 requests | 16 | | Prefix caching disabled | 0 | The rationale is that **if the server permits several requests to run together, it should normally let each save its processed prompt**. Otherwise, increasing request concurrency can introduce a bottleneck in saving reusable context. Request capacity is the appropriate bound because a capture slot remains occupied across the request's lifetime, which can span many GPU batches. Intermediate-checkpoint mode uses at most two slots in automatic mode: checkpoint collection is serialized, while one named-session replacement can overlap. A positive `--prefix-capture-slots` value overrides automatic sizing. Allowing more saves at once requires additional temporary memory. MLX grows storage as requests execute; CUDA's fixed pool can lose per-session context capacity as the slot count increases. Users with tight memory budgets can lower the capture limit explicitly. In our five-session Muse Glimmer benchmark (248 requests, 14,201 output tokens; one pass per configuration): ```text Before: 1 slot After: 8 slots Processed input tokens 510,838 175,201 Total replay time 53.83 minutes 25.23 minutes ``` This comparison demonstrates the benefit of eight slots versus one; it does not establish whether four would perform equally well for this workload. Authored with OpenAI Codex. ### Test plan Python server and Muse-Glimmer suites: 902 passed, 33 skipped, with the rebuilt native worker. Native serving/batching/protocol suites: 227/219/37 passed. Repository lint passed. Tests cover automatic sizing, explicit overrides, disabled caching, row-overflow validation, repeated bursts at limits one/two/four, automatic overlapping captures, replacement sharing the budget, and concurrent checkpoint collection at limit two. Seven concurrency tests passed 50 repetitions.
1 parent a311131 commit 95d2d2b

14 files changed

Lines changed: 437 additions & 65 deletions

File tree

‎examples/llm_server/cpp/module_worker_main.cpp‎

Lines changed: 18 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,10 @@ DEFINE_int32(
3535
prefix_checkpoint_interval,
3636
0,
3737
"Checkpoint interval in tokens; 0: initial prompt only. Uses global LRU.");
38+
DEFINE_int32(
39+
prefix_capture_slots,
40+
0,
41+
"Transient capture rows; 0: size automatically from session/request limits");
3842
DEFINE_int32(max_session_tokens, 2048, "Maximum retained tokens per session");
3943
DEFINE_int32(max_decode_sequences, 2, "Decode slots in one packed forward");
4044
DEFINE_int32(
@@ -47,16 +51,26 @@ int main(int argc, char** argv) {
4751
if (FLAGS_pte.empty() || FLAGS_tokenizer.empty() || FLAGS_max_sessions <= 0 ||
4852
FLAGS_max_session_tokens <= 0 || FLAGS_max_decode_sequences <= 0 ||
4953
FLAGS_max_inflight_requests <= 0 || FLAGS_prefix_cache_entries < 0 ||
50-
FLAGS_prefix_checkpoint_interval < 0 ||
54+
FLAGS_prefix_capture_slots < 0 || FLAGS_prefix_checkpoint_interval < 0 ||
5155
(FLAGS_prefix_checkpoint_interval > 0 &&
5256
FLAGS_prefix_cache_entries == 0)) {
5357
std::cerr
54-
<< "--pte and --tokenizer are required; capacity limits must be positive "
55-
"and prefix-cache limits nonnegative; checkpoints require prefix_cache_entries > 0\n";
58+
<< "--pte and --tokenizer are required; capacity limits must be positive"
59+
"; prefix-cache limits must be nonnegative; "
60+
"checkpoints require prefix_cache_entries > 0\n";
5661
return 1;
5762
}
63+
executorch::extension::llm::serving::ServingRuntimeConfig config;
64+
config.max_sessions = FLAGS_max_sessions;
65+
config.prefix_cache_capacity = FLAGS_prefix_cache_entries;
66+
config.max_prefix_captures = FLAGS_prefix_capture_slots;
67+
config.prefix_checkpoint_interval = FLAGS_prefix_checkpoint_interval;
68+
config.max_context_length = FLAGS_max_session_tokens;
69+
config.max_requests = FLAGS_max_inflight_requests;
70+
config.max_pending_operations = FLAGS_max_inflight_requests;
5871
const auto physical_sessions = static_cast<std::int64_t>(FLAGS_max_sessions) +
59-
FLAGS_prefix_cache_entries + (FLAGS_prefix_cache_entries > 0 ? 1 : 0);
72+
FLAGS_prefix_cache_entries +
73+
static_cast<int64_t>(config.resolved_prefix_captures());
6074
if (physical_sessions > std::numeric_limits<int>::max()) {
6175
std::cerr
6276
<< "session and prefix-cache capacity exceeds executor row limit\n";
@@ -106,13 +120,6 @@ int main(int argc, char** argv) {
106120
width, decode_slots, width - decode_slots);
107121
if (!scheduler)
108122
throw std::runtime_error("invalid scheduler limits");
109-
llm::serving::ServingRuntimeConfig config;
110-
config.max_sessions = FLAGS_max_sessions;
111-
config.prefix_cache_capacity = FLAGS_prefix_cache_entries;
112-
config.prefix_checkpoint_interval = FLAGS_prefix_checkpoint_interval;
113-
config.max_context_length = FLAGS_max_session_tokens;
114-
config.max_requests = FLAGS_max_inflight_requests;
115-
config.max_pending_operations = FLAGS_max_inflight_requests;
116123
config.default_stop_tokens.assign(eos.begin(), eos.end());
117124
llm::serving::ServingRuntime runtime(
118125
**executor, std::move(scheduler), *tokenizer, config);

‎examples/llm_server/cpp/test_batching_worker_main.cpp‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -239,7 +239,9 @@ int main(int argc, char** argv) {
239239
config.prefix_cache_capacity = prefix_cache ? 2 : 0;
240240
executor.capacity = static_cast<int>(
241241
config.max_sessions +
242-
(prefix_cache ? config.prefix_cache_capacity + 1 : 0));
242+
(prefix_cache
243+
? config.prefix_cache_capacity + config.resolved_prefix_captures()
244+
: 0));
243245
if (stop_immediately) {
244246
config.default_stop_tokens = {9};
245247
}

‎examples/llm_server/python/README.md‎

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -63,10 +63,16 @@ fallback is not enabled by this launcher.
6363
`--max-context` is required and sets both the HTTP limit and native session-token
6464
limit; choose a value supported by the export. Capacity defaults are
6565
`--max-sessions 16`, `--max-decode-sequences 8`, `--max-inflight-requests 64`, and
66-
`--prefix-cache-entries 0`. Logical sessions, packed decode width, and in-flight
67-
request capacity are independent limits. Python concurrency follows advertised
68-
request capacity, not the single native delivery thread. Python reads native
69-
async streams directly through bounded per-request mailboxes, without a thread
66+
`--prefix-cache-entries 0`. With prefix caching enabled, `--prefix-capture-slots 0`
67+
(the default) automatically allows one pending prompt capture per admitted request,
68+
bounded by `min(max_sessions, max_inflight_requests)`. A positive value overrides
69+
this allowance. With checkpoints enabled, automatic sizing uses at most two rows:
70+
checkpoint collection is serialized, but one replacement can overlap. Explicit
71+
values above two add unused row capacity in that mode.
72+
Logical sessions, packed decode width, and in-flight request capacity are independent
73+
limits. Python concurrency follows advertised request capacity, not the single
74+
native delivery thread. Python reads native async streams directly through bounded
75+
per-request mailboxes, without a thread
7076
pool or a second token queue. A paused consumer does not stall peers, but consumers
7177
must not block the event loop; native callbacks must also do short, bounded,
7278
nonblocking work. Prefix snapshots require extra native capacity, which the worker

‎examples/llm_server/python/serve.py‎

Lines changed: 24 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@
2424
_MAX_INT32 = (1 << 31) - 1
2525

2626

27-
def _validate_limits(parser, args) -> None:
27+
def _validate_limits(parser, args, *, prefix_capture_slots=0) -> int:
2828
for flag, value in (
2929
("--max-context", args.max_context),
3030
("--max-sessions", args.max_sessions),
@@ -39,15 +39,23 @@ def _validate_limits(parser, args) -> None:
3939
parser.error(f"--prefix-checkpoint-interval must be between 0 and {_MAX_INT32}")
4040
if args.prefix_checkpoint_interval and not args.prefix_cache_entries:
4141
parser.error("--prefix-checkpoint-interval requires --prefix-cache-entries > 0")
42+
if not 0 <= prefix_capture_slots <= _MAX_INT32:
43+
parser.error(f"--prefix-capture-slots must be between 0 and {_MAX_INT32}")
44+
# Mirrors ServingRuntimeConfig::resolved_prefix_captures for row validation.
45+
if prefix_capture_slots == 0:
46+
prefix_capture_slots = min(args.max_sessions, args.max_inflight_requests)
47+
if args.prefix_checkpoint_interval:
48+
prefix_capture_slots = min(prefix_capture_slots, 2)
49+
if not args.prefix_cache_entries:
50+
prefix_capture_slots = 0
4251
physical_sessions = (
43-
args.max_sessions
44-
+ args.prefix_cache_entries
45-
+ (1 if args.prefix_cache_entries else 0)
52+
args.max_sessions + args.prefix_cache_entries + prefix_capture_slots
4653
)
4754
if physical_sessions > _MAX_INT32:
4855
parser.error("session and prefix-cache capacity exceeds the row limit")
4956
if not 1 <= args.port <= 65535:
5057
parser.error("--port must be between 1 and 65535")
58+
return prefix_capture_slots
5159

5260

5361
def _parse_args(argv=None):
@@ -81,6 +89,12 @@ def _parse_args(argv=None):
8189
parser.add_argument("--max-decode-sequences", type=int, default=8)
8290
parser.add_argument("--max-inflight-requests", type=int, default=64)
8391
parser.add_argument("--prefix-cache-entries", type=int, default=0)
92+
parser.add_argument(
93+
"--prefix-capture-slots",
94+
type=int,
95+
default=0,
96+
help="Transient capture rows; 0 (default): size from session/request limits",
97+
)
8498
parser.add_argument(
8599
"--prefix-checkpoint-interval",
86100
type=int,
@@ -91,7 +105,7 @@ def _parse_args(argv=None):
91105
parser.add_argument("--port", type=int, default=8000)
92106
args = parser.parse_args(argv)
93107

94-
_validate_limits(parser, args)
108+
_validate_limits(parser, args, prefix_capture_slots=args.prefix_capture_slots)
95109

96110
args.worker_bin = os.path.expanduser(args.worker_bin)
97111
if shutil.which(args.worker_bin) is None:
@@ -153,6 +167,11 @@ async def serving_factory():
153167
"--prefix_cache_entries",
154168
str(args.prefix_cache_entries),
155169
]
170+
+ (
171+
["--prefix_capture_slots", str(args.prefix_capture_slots)]
172+
if args.prefix_cache_entries
173+
else []
174+
)
156175
+ (
157176
["--prefix_checkpoint_interval", str(args.prefix_checkpoint_interval)]
158177
if args.prefix_checkpoint_interval

‎examples/llm_server/python/tests/test_serve_launcher.py‎

Lines changed: 68 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,6 @@
1212

1313
import pytest
1414
import uvicorn
15-
1615
from executorch.examples.llm_server.python import serve
1716
from executorch.examples.llm_server.python.chat_template import ChatTemplate
1817
from executorch.examples.llm_server.python.multiplexed_worker_client import (
@@ -158,6 +157,57 @@ def test_checkpoint_options_reach_worker(argv, launch):
158157
assert command[command.index("--prefix_checkpoint_interval") + 1] == "512"
159158

160159

160+
@pytest.mark.parametrize(
161+
"sessions,requests,interval,override,expected",
162+
[
163+
(16, 8, 0, 0, 8),
164+
(4, 8, 0, 0, 4),
165+
(16, 8, 64, 0, 2),
166+
(1, 8, 64, 0, 1),
167+
(16, 8, 0, 1, 1),
168+
(16, 8, 64, 3, 3),
169+
],
170+
)
171+
def test_capture_capacity_validation_and_worker_override(
172+
argv, launch, sessions, requests, interval, override, expected
173+
):
174+
serve.main(
175+
argv
176+
+ [
177+
"--max-sessions",
178+
str(sessions),
179+
"--max-inflight-requests",
180+
str(requests),
181+
"--prefix-cache-entries",
182+
"1",
183+
"--prefix-checkpoint-interval",
184+
str(interval),
185+
"--prefix-capture-slots",
186+
str(override),
187+
]
188+
)
189+
command = launch.stages["spawn"].call_args.args[0]
190+
assert command[command.index("--prefix_capture_slots") + 1] == str(override)
191+
args = SimpleNamespace(
192+
max_context=1024,
193+
max_sessions=sessions,
194+
max_decode_sequences=8,
195+
max_inflight_requests=requests,
196+
prefix_cache_entries=1,
197+
prefix_checkpoint_interval=interval,
198+
port=8000,
199+
)
200+
assert (
201+
serve._validate_limits(Mock(), args, prefix_capture_slots=override) == expected
202+
)
203+
204+
205+
def test_capture_capacity_defaults_to_automatic(argv, launch):
206+
serve.main(argv + ["--prefix-cache-entries", "4"])
207+
command = launch.stages["spawn"].call_args.args[0]
208+
assert command[command.index("--prefix_capture_slots") + 1] == "0"
209+
210+
161211
def test_default_launch_requires_multiplexing_and_shares_context(argv, launch):
162212
args = serve._parse_args(argv)
163213
serve.main(argv)
@@ -355,6 +405,8 @@ def test_overrides_and_capability_based_concurrency(argv, native_launch):
355405
"23",
356406
"--prefix-cache-entries",
357407
"5",
408+
"--prefix-capture-slots",
409+
"8",
358410
"--hf-tokenizer",
359411
"organization/model",
360412
"--assistant-header",
@@ -379,6 +431,8 @@ def test_overrides_and_capability_based_concurrency(argv, native_launch):
379431
"23",
380432
"--prefix_cache_entries",
381433
"5",
434+
"--prefix_capture_slots",
435+
"8",
382436
]
383437
launch.stages["template"].assert_called_once_with(
384438
hf_tokenizer_path="organization/model", assistant_header=header
@@ -441,10 +495,20 @@ def test_invalid_positive_capacities_fail_before_template_or_spawn(
441495
@pytest.mark.parametrize(
442496
"options",
443497
[
498+
["--prefix-capture-slots", "-1"],
499+
["--prefix-capture-slots", str(1 << 31)],
444500
["--prefix-cache-entries", "-1"],
445501
["--prefix-cache-entries", str(1 << 31)],
446502
["--max-sessions", str((1 << 31) - 1), "--prefix-cache-entries", "1"],
447-
["--max-sessions", "1", "--prefix-cache-entries", str((1 << 31) - 2)],
503+
["--max-sessions", "1", "--prefix-cache-entries", str((1 << 31) - 1)],
504+
[
505+
"--max-sessions",
506+
str((1 << 31) - 3),
507+
"--prefix-cache-entries",
508+
"1",
509+
"--prefix-capture-slots",
510+
"2",
511+
],
448512
["--port", "0"],
449513
["--port", "65536"],
450514
],
@@ -471,6 +535,8 @@ def test_numeric_limit_boundaries_are_inclusive(argv, port):
471535
str(max_int32),
472536
"--prefix-cache-entries",
473537
"1",
538+
"--prefix-capture-slots",
539+
"1",
474540
"--port",
475541
str(port),
476542
]

‎examples/llm_server/spec/README.md‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -277,8 +277,14 @@ batched cache via `cache::kind::kBatched`, derives scheduler width from
277277
one `ServingRuntime`. It does not preload `forward`, run token-step loops, or use
278278
Glimmer's legacy session implementation. Prefix caching is disabled by default.
279279
`--prefix_cache_entries=N` enables creation-only reuse for both greedy and
280-
sampled requests. It adds `N + 1` physical rows for retained snapshots and one
281-
transient capture; logical `--max_sessions` remains unchanged.
280+
sampled requests. With `--prefix_capture_slots=C`, it adds `N + C` physical rows
281+
for retained snapshots and transient captures; logical `--max_sessions` remains
282+
unchanged. The default `--prefix_capture_slots=0` resolves `C` automatically to
283+
`min(max_sessions, max_inflight_requests)`, so overlapping requests can each capture
284+
their initial prompt. With checkpoints enabled, automatic sizing uses at most two
285+
rows because checkpoint collection remains serialized and one replacement can
286+
overlap. A positive value overrides automatic sizing; values above two add unused
287+
row capacity in checkpoint mode. Disabled prefix caching reserves no capture rows.
282288

283289
With an existing MLX-enabled ExecuTorch installation and matching gflags package,
284290
configure the MLX LLM examples and build `llm_worker`. Launch with `--pte`,

‎examples/models/muse-glimmer/BATCHING_SERVING.md‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -223,9 +223,14 @@ python -m executorch.examples.models.muse_glimmer.serving.serve_batching \
223223

224224
`--data-path` is required and `--pos-embed-path` is rejected; image requests
225225
fail preparation. Every resident session reserves `--max-context` cells of the
226-
pool: requests, `--prefix-cache-entries` snapshots, and one capture lane when
227-
snapshots are enabled. Their total reservation must leave one cell for padded
228-
writes within the exported `--max-cells`.
226+
pool: requests, `--prefix-cache-entries` snapshots, and the resolved number of
227+
capture slots when snapshots are enabled. `--prefix-capture-slots 0` (the default)
228+
matches request concurrency, capped at two when checkpoints are enabled; a
229+
positive value sets an explicit limit. Their total reservation must leave one
230+
cell for padded writes within the exported `--max-cells`. More capture slots
231+
reduce the context available per session in the native worker's automatic
232+
context mode. This Python launcher requires an explicit `--max-context`;
233+
reduce the context or slot counts if they do not fit.
229234
The cell pool tracks at most 64 resident sessions. Prefix snapshots clone a
230235
session's cells by reference, sharing the prompt's KV rather than copying it.
231236

‎examples/models/muse-glimmer/runtime/runners/batching_bootstrap.cpp‎

Lines changed: 14 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,10 @@ DEFINE_int32(
4040
prefix_checkpoint_interval,
4141
0,
4242
"Checkpoint interval in tokens; 0: initial prompt only. Uses global LRU.");
43+
DEFINE_int32(
44+
prefix_capture_slots,
45+
0,
46+
"Transient capture rows; 0: size automatically from session/request limits");
4347
DEFINE_uint64(max_image_bytes, 20 * 1024 * 1024, "Maximum encoded image bytes");
4448
DEFINE_uint64(
4549
max_input_frame_bytes,
@@ -62,7 +66,7 @@ create_muse_glimmer_batching_runtime() {
6266
if (FLAGS_pte.empty() || FLAGS_tokenizer.empty() || FLAGS_max_sessions <= 0 ||
6367
FLAGS_max_inflight_requests <= 0 || FLAGS_max_decode_sequences <= 0 ||
6468
FLAGS_prefix_cache_entries < 0 || FLAGS_max_image_bytes == 0 ||
65-
FLAGS_prefix_checkpoint_interval < 0 ||
69+
FLAGS_prefix_checkpoint_interval < 0 || FLAGS_prefix_capture_slots < 0 ||
6670
(FLAGS_prefix_checkpoint_interval > 0 &&
6771
FLAGS_prefix_cache_entries == 0) ||
6872
FLAGS_max_image_bytes > 20 * 1024 * 1024 ||
@@ -73,8 +77,16 @@ create_muse_glimmer_batching_runtime() {
7377
throw std::invalid_argument(
7478
"required model/tokenizer or bounded batching limits are invalid");
7579
}
80+
serving::ServingRuntimeConfig serving;
81+
serving.max_sessions = FLAGS_max_sessions;
82+
serving.max_requests = FLAGS_max_inflight_requests;
83+
serving.max_pending_operations = FLAGS_max_inflight_requests;
84+
serving.prefix_cache_capacity = FLAGS_prefix_cache_entries;
85+
serving.max_prefix_captures = FLAGS_prefix_capture_slots;
86+
serving.prefix_checkpoint_interval = FLAGS_prefix_checkpoint_interval;
7687
const int64_t physical = static_cast<int64_t>(FLAGS_max_sessions) +
77-
FLAGS_prefix_cache_entries + (FLAGS_prefix_cache_entries > 0 ? 1 : 0);
88+
FLAGS_prefix_cache_entries +
89+
static_cast<int64_t>(serving.resolved_prefix_captures());
7890
if (physical > std::numeric_limits<int>::max())
7991
throw std::invalid_argument("too many physical sessions");
8092
auto result = std::make_unique<MuseGlimmerBatchingRuntime>();
@@ -114,13 +126,7 @@ create_muse_glimmer_batching_runtime() {
114126
batching::DecodeFirstScheduler::create(budget, decodes, budget - decodes);
115127
if (!scheduler)
116128
throw std::runtime_error("invalid scheduler limits");
117-
serving::ServingRuntimeConfig serving;
118-
serving.max_sessions = FLAGS_max_sessions;
119129
serving.max_context_length = result->backend.preparation->max_context_length;
120-
serving.max_requests = FLAGS_max_inflight_requests;
121-
serving.max_pending_operations = FLAGS_max_inflight_requests;
122-
serving.prefix_cache_capacity = FLAGS_prefix_cache_entries;
123-
serving.prefix_checkpoint_interval = FLAGS_prefix_checkpoint_interval;
124130
serving.default_stop_tokens = {FLAGS_eos_id};
125131
for (const char* name : {"<|eot|>", "<|end_of_text|>"}) {
126132
auto encoded = result->tokenizer->encode(name, 0, 0);

‎examples/models/muse-glimmer/serving/serve_batching.py‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,12 @@ def _parse_args(argv=None):
6767
help="Opt-in prefix reuse for new sessions and prompt replacements; "
6868
"requires batching executor cache-clone support.",
6969
)
70+
parser.add_argument(
71+
"--prefix-capture-slots",
72+
type=int,
73+
default=0,
74+
help="Transient capture rows; 0 (default): size from session/request limits",
75+
)
7076
parser.add_argument("--max-vision-patches", type=int, default=4096)
7177
parser.add_argument(
7278
"--prefix-checkpoint-interval",
@@ -87,7 +93,7 @@ def _parse_args(argv=None):
8793
parser.add_argument("--host", default="127.0.0.1")
8894
parser.add_argument("--port", type=int, default=8000)
8995
args = parser.parse_args(argv)
90-
_validate_limits(parser, args)
96+
_validate_limits(parser, args, prefix_capture_slots=args.prefix_capture_slots)
9197
if not 0 <= args.dflash_n_draft < _INT32_MAX:
9298
parser.error("--dflash-n-draft must be nonnegative and below INT32_MAX")
9399
if args.dflash_n_draft and not args.dflash:
@@ -158,6 +164,10 @@ def _worker_command(args):
158164
("max_decode_sequences", args.max_decode_sequences),
159165
("max_inflight_requests", args.max_inflight_requests),
160166
("prefix_cache_entries", args.prefix_cache_entries),
167+
(
168+
"prefix_capture_slots",
169+
args.prefix_capture_slots if args.prefix_cache_entries else None,
170+
),
161171
("prefix_checkpoint_interval", args.prefix_checkpoint_interval or None),
162172
("max_vision_patches", args.max_vision_patches),
163173
("max_image_bytes", args.max_image_bytes),

0 commit comments

Comments
 (0)