Repository navigation
Commit 95d2d2b
authored
Allow bounded concurrent prefix captures in LLM serving (#24303)
### Summary
This PR improves prompt reuse when several agents run in parallel,
reducing how often the model processes the same context again.
ExecuTorch already supports running multiple requests at once. However,
only one request at a time can save its processed prompt for future
reuse. While that request is busy, other requests continue running but
may miss the opportunity to save their prompts.
For example, when the main agent and three subagents overlap:
```text
Before With this PR
One save at a time Multiple saves allowed
Main agent Saves its prompt Saves its prompt
Subagent A Save skipped Saves its prompt
Subagent B Save skipped Saves its prompt
Subagent C Save skipped Saves its prompt
All four can run concurrently in both cases.
```
The difference matters on the **next turn**. Suppose a subagent
processes 4,000 tokens of context, then sends a follow-up containing
that context plus 200 new tokens:
```text
Follow-up prompt:
+--------------------------------------+-------------+
| Existing context: 4,000 tokens | New: 200 |
+--------------------------------------+-------------+
Without a saved prompt:
|<-------------- Process all 4,200 ----------------->|
With a saved prompt:
|<--------- Reuse previous work ------>| Process 200 |
```
Repeated across many subagent turns, these missed saves can create
substantial extra work.
The PR allows multiple requests to save their processed prompts
concurrently:
- The limit adjusts automatically to the server's configured request
capacity, with an explicit override available.
- Temporary storage is accounted for, and reservations are released on
completion, cancellation, or failure.
- If the limit is reached, requests continue normally; saving the prompt
remains optional.
The default `--prefix-capture-slots 0` means **automatic sizing**. With
prefix caching enabled and intermediate checkpoints disabled:
```text
capture slots = min(max_sessions, max_inflight_requests)
```
| Configuration | Automatic capture slots |
|---|---:|
| Our benchmark: 8 sessions, 64 requests | 8 |
| Muse launcher defaults: 4 sessions, 4 requests | 4 |
| Generic launcher defaults: 16 sessions, 64 requests | 16 |
| Prefix caching disabled | 0 |
The rationale is that **if the server permits several requests to run
together, it should normally let each save its processed prompt**.
Otherwise, increasing request concurrency can introduce a bottleneck in
saving reusable context. Request capacity is the appropriate bound
because a capture slot remains occupied across the request's lifetime,
which can span many GPU batches.
Intermediate-checkpoint mode uses at most two slots in automatic mode:
checkpoint collection is serialized, while one named-session replacement
can overlap. A positive `--prefix-capture-slots` value overrides
automatic sizing.
Allowing more saves at once requires additional temporary memory. MLX
grows storage as requests execute; CUDA's fixed pool can lose
per-session context capacity as the slot count increases. Users with
tight memory budgets can lower the capture limit explicitly.
In our five-session Muse Glimmer benchmark (248 requests, 14,201 output
tokens; one pass per configuration):
```text
Before: 1 slot After: 8 slots
Processed input tokens 510,838 175,201
Total replay time 53.83 minutes 25.23 minutes
```
This comparison demonstrates the benefit of eight slots versus one; it
does not establish whether four would perform equally well for this
workload.
Authored with OpenAI Codex.
### Test plan
Python server and Muse-Glimmer suites: 902 passed, 33 skipped, with the
rebuilt native worker. Native serving/batching/protocol suites:
227/219/37 passed. Repository lint passed.
Tests cover automatic sizing, explicit overrides, disabled caching,
row-overflow validation, repeated bursts at limits one/two/four,
automatic overlapping captures, replacement sharing the budget, and
concurrent checkpoint collection at limit two. Seven concurrency tests
passed 50 repetitions.1 parent a311131 commit 95d2d2b
14 files changed
Lines changed: 437 additions & 65 deletions
File tree
- examples
- llm_server
- cpp
- python
- tests
- spec
- models/muse-glimmer
- runtime/runners
- serving
- tests
- extension/llm/serving
- test
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
35 | 35 | | |
36 | 36 | | |
37 | 37 | | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
38 | 42 | | |
39 | 43 | | |
40 | 44 | | |
| |||
47 | 51 | | |
48 | 52 | | |
49 | 53 | | |
50 | | - | |
| 54 | + | |
51 | 55 | | |
52 | 56 | | |
53 | 57 | | |
54 | | - | |
55 | | - | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
56 | 61 | | |
57 | 62 | | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
58 | 71 | | |
59 | | - | |
| 72 | + | |
| 73 | + | |
60 | 74 | | |
61 | 75 | | |
62 | 76 | | |
| |||
106 | 120 | | |
107 | 121 | | |
108 | 122 | | |
109 | | - | |
110 | | - | |
111 | | - | |
112 | | - | |
113 | | - | |
114 | | - | |
115 | | - | |
116 | 123 | | |
117 | 124 | | |
118 | 125 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
239 | 239 | | |
240 | 240 | | |
241 | 241 | | |
242 | | - | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
243 | 245 | | |
244 | 246 | | |
245 | 247 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
63 | 63 | | |
64 | 64 | | |
65 | 65 | | |
66 | | - | |
67 | | - | |
68 | | - | |
69 | | - | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
70 | 76 | | |
71 | 77 | | |
72 | 78 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
27 | | - | |
| 27 | + | |
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
| |||
39 | 39 | | |
40 | 40 | | |
41 | 41 | | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
42 | 51 | | |
43 | | - | |
44 | | - | |
45 | | - | |
| 52 | + | |
46 | 53 | | |
47 | 54 | | |
48 | 55 | | |
49 | 56 | | |
50 | 57 | | |
| 58 | + | |
51 | 59 | | |
52 | 60 | | |
53 | 61 | | |
| |||
81 | 89 | | |
82 | 90 | | |
83 | 91 | | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
84 | 98 | | |
85 | 99 | | |
86 | 100 | | |
| |||
91 | 105 | | |
92 | 106 | | |
93 | 107 | | |
94 | | - | |
| 108 | + | |
95 | 109 | | |
96 | 110 | | |
97 | 111 | | |
| |||
153 | 167 | | |
154 | 168 | | |
155 | 169 | | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
156 | 175 | | |
157 | 176 | | |
158 | 177 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
15 | | - | |
16 | 15 | | |
17 | 16 | | |
18 | 17 | | |
| |||
158 | 157 | | |
159 | 158 | | |
160 | 159 | | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
161 | 211 | | |
162 | 212 | | |
163 | 213 | | |
| |||
355 | 405 | | |
356 | 406 | | |
357 | 407 | | |
| 408 | + | |
| 409 | + | |
358 | 410 | | |
359 | 411 | | |
360 | 412 | | |
| |||
379 | 431 | | |
380 | 432 | | |
381 | 433 | | |
| 434 | + | |
| 435 | + | |
382 | 436 | | |
383 | 437 | | |
384 | 438 | | |
| |||
441 | 495 | | |
442 | 496 | | |
443 | 497 | | |
| 498 | + | |
| 499 | + | |
444 | 500 | | |
445 | 501 | | |
446 | 502 | | |
447 | | - | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
448 | 512 | | |
449 | 513 | | |
450 | 514 | | |
| |||
471 | 535 | | |
472 | 536 | | |
473 | 537 | | |
| 538 | + | |
| 539 | + | |
474 | 540 | | |
475 | 541 | | |
476 | 542 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
277 | 277 | | |
278 | 278 | | |
279 | 279 | | |
280 | | - | |
281 | | - | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
282 | 288 | | |
283 | 289 | | |
284 | 290 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
223 | 223 | | |
224 | 224 | | |
225 | 225 | | |
226 | | - | |
227 | | - | |
228 | | - | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
229 | 234 | | |
230 | 235 | | |
231 | 236 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
40 | 40 | | |
41 | 41 | | |
42 | 42 | | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
43 | 47 | | |
44 | 48 | | |
45 | 49 | | |
| |||
62 | 66 | | |
63 | 67 | | |
64 | 68 | | |
65 | | - | |
| 69 | + | |
66 | 70 | | |
67 | 71 | | |
68 | 72 | | |
| |||
73 | 77 | | |
74 | 78 | | |
75 | 79 | | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
76 | 87 | | |
77 | | - | |
| 88 | + | |
| 89 | + | |
78 | 90 | | |
79 | 91 | | |
80 | 92 | | |
| |||
114 | 126 | | |
115 | 127 | | |
116 | 128 | | |
117 | | - | |
118 | | - | |
119 | 129 | | |
120 | | - | |
121 | | - | |
122 | | - | |
123 | | - | |
124 | 130 | | |
125 | 131 | | |
126 | 132 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
67 | 67 | | |
68 | 68 | | |
69 | 69 | | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
70 | 76 | | |
71 | 77 | | |
72 | 78 | | |
| |||
87 | 93 | | |
88 | 94 | | |
89 | 95 | | |
90 | | - | |
| 96 | + | |
91 | 97 | | |
92 | 98 | | |
93 | 99 | | |
| |||
158 | 164 | | |
159 | 165 | | |
160 | 166 | | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
161 | 171 | | |
162 | 172 | | |
163 | 173 | | |
| |||
0 commit comments