You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit a311131
Browse filesBrowse the repository at this point in the historyBrowse files
Add per-request ignore_eos to LLM serving (#24302)
### Summary
AgentX replay needs to reproduce recorded output-token budgets even when
the model emits an end-of-sequence token early. This PR adds
**per-request `ignore_eos` support** to ExecuTorch's LLM server.
For example, a request can specify:
```json
{
"max_tokens": 128,
"ignore_eos": true
}
```
If the model emits EOS early, generation continues toward 128 tokens.
Explicit request stops, cancellation, and context limits still apply.
The implementation has three main parts:
1. **Pass the flag through every layer:** HTTP request → Python serving
code → multiplexed worker → native generation. It defaults to `false`,
and concurrent requests can use different values.
2. **Change generation termination:** native generation ignores the
model's default EOS stops, and Python stops forwarding automatic
template terminators as generation stops. EOS remains sampleable; the
sampler itself is unchanged.
3. **Keep the returned response coherent:** the server shows only the
first visible assistant turn. Tokens generated after its terminator are
hidden from text, reasoning, and tool-call parsing, but still count
toward completion-token usage. The request finishes when generation
finishes.
```text
Native generation: Paris. <EOS> ...continues to 128 tokens...
Visible response: Paris.
Completion usage: 128 tokens
```
The worker advertises support and its decoded EOS markers. Unsupported
workers, or workers unable to report those markers, reject the option
instead of silently ignoring it. Matching model/tokenizer assets are
required. The generic and Muse-Glimmer adapters share this behavior.
Hidden post-EOS tokens are **not inserted into later conversation
messages**. A later prompt rebuilt from the visible answer may therefore
require rebuilding the session; prefix caching can recover matching
retained context. AgentX already synthesizes subsequent prompts
independently.
Authored with OpenAI Codex.
### Test plan
Python server and Muse-Glimmer suites: 963 passed, 33 skipped, with the
rebuilt native test worker. Native serving/batching/protocol suites:
219/219/39 passed. Tests cover stream/nonstream tools, reasoning, split
and model-specific native terminators, malformed readiness cleanup, all
plain-chat special boundaries, explicit stops inside terminators and
native-truncated marker fragments, explicit stops after hidden EOS,
session echo, legacy capability rejection, and Muse-Glimmer stop
forwarding. Repository lint passed.
A native deferred-capture test uses a one-token limit to avoid racing
its bounded output queue; it still verifies capture release before
execution.
0 commit comments