Skip to content

Commit a311131

Browse files
authored
Add per-request ignore_eos to LLM serving (#24302)
### Summary AgentX replay needs to reproduce recorded output-token budgets even when the model emits an end-of-sequence token early. This PR adds **per-request `ignore_eos` support** to ExecuTorch's LLM server. For example, a request can specify: ```json { "max_tokens": 128, "ignore_eos": true } ``` If the model emits EOS early, generation continues toward 128 tokens. Explicit request stops, cancellation, and context limits still apply. The implementation has three main parts: 1. **Pass the flag through every layer:** HTTP request → Python serving code → multiplexed worker → native generation. It defaults to `false`, and concurrent requests can use different values. 2. **Change generation termination:** native generation ignores the model's default EOS stops, and Python stops forwarding automatic template terminators as generation stops. EOS remains sampleable; the sampler itself is unchanged. 3. **Keep the returned response coherent:** the server shows only the first visible assistant turn. Tokens generated after its terminator are hidden from text, reasoning, and tool-call parsing, but still count toward completion-token usage. The request finishes when generation finishes. ```text Native generation: Paris. <EOS> ...continues to 128 tokens... Visible response: Paris. Completion usage: 128 tokens ``` The worker advertises support and its decoded EOS markers. Unsupported workers, or workers unable to report those markers, reject the option instead of silently ignoring it. Matching model/tokenizer assets are required. The generic and Muse-Glimmer adapters share this behavior. Hidden post-EOS tokens are **not inserted into later conversation messages**. A later prompt rebuilt from the visible answer may therefore require rebuilding the session; prefix caching can recover matching retained context. AgentX already synthesizes subsequent prompts independently. Authored with OpenAI Codex. ### Test plan Python server and Muse-Glimmer suites: 963 passed, 33 skipped, with the rebuilt native test worker. Native serving/batching/protocol suites: 219/219/39 passed. Tests cover stream/nonstream tools, reasoning, split and model-specific native terminators, malformed readiness cleanup, all plain-chat special boundaries, explicit stops inside terminators and native-truncated marker fragments, explicit stops after hidden EOS, session echo, legacy capability rejection, and Muse-Glimmer stop forwarding. Repository lint passed. A native deferred-capture test uses a one-token limit to avoid racing its bounded output queue; it still verifies capture release before execution.
1 parent 51b5c73 commit a311131

21 files changed

Lines changed: 892 additions & 39 deletions

‎examples/llm_server/README.md‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -112,3 +112,35 @@ Reliability guidance:
112112

113113
See [Terminal-Bench](evals/terminal_bench/README.md) for setup, configuration,
114114
and evaluation commands on macOS.
115+
116+
## Benchmark with agentx-mini
117+
118+
[agentx-mini](https://huggingface.co/datasets/SocialLocalMobile/agentx-mini) replays
119+
shortened agent traces, including parallel subagent calls. Follow its README to
120+
install AgentX Harness, download the dataset, and configure its 8-token blocks.
121+
122+
Build the MLX `llm_worker` and export a packed off-graph-cache model following the
123+
[MLX LLM guide](../../backends/mlx/examples/llm/README.md). Start the batching server:
124+
125+
```bash
126+
python -m executorch.examples.llm_server.python.serve \
127+
--worker-bin cmake-out/backends/mlx/examples/llm/llm_worker \
128+
--model-path /path/to/model.pte \
129+
--tokenizer-path /path/to/hf-tokenizer-directory \
130+
--model-id Qwen/Qwen3-0.6B \
131+
--max-context 40960 --max-sessions 8 --max-decode-sequences 8 \
132+
--prefix-cache-entries 8 --port 8000
133+
```
134+
135+
Use the model ID and tokenizer matching your export, and its context limit. Point
136+
AgentX at `http://127.0.0.1:8000`, enable streaming, and set
137+
`benchmark.endpoint.extra: {ignore_eos: true}`. Keep AgentX's root concurrency at
138+
`1` and `trace_idle_gap_cap_seconds: 1`; the worker's eight decode slots let
139+
subagents within a session overlap. Run `agentx profile --config benchmark.yaml`
140+
using the dataset README's configuration.
141+
142+
`ignore_eos` is per request: normal chat requests still stop at EOS, and explicit
143+
request stop strings remain active. Rebuild the multiplexed worker to include this
144+
support; no chat-template changes are needed. Generation continues after EOS, but
145+
the HTTP response only parses and displays the first assistant turn. Usage counts
146+
include all generated tokens.

‎examples/llm_server/cpp/multiplexed_worker.cpp‎

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -134,6 +134,13 @@ serving::PromptInput prompt_input(const Json& request) {
134134

135135
serving::GenerationOptions generation_options(const Json& request) {
136136
serving::GenerationOptions options;
137+
if (request.contains("ignore_eos")) {
138+
const auto& value = request.at("ignore_eos");
139+
if (!value.is_boolean()) {
140+
throw std::invalid_argument("ignore_eos must be a boolean");
141+
}
142+
options.ignore_eos = value.get<bool>();
143+
}
137144
if (request.contains("max_new_tokens")) {
138145
const auto& value = request.at("max_new_tokens");
139146
const bool in_range = value.is_number_unsigned()
@@ -268,9 +275,12 @@ class Worker {
268275
}
269276
try {
270277
const auto info = runtime_.info();
278+
const auto stops = runtime_.default_stop_strings();
271279
queue_.push_back(
272280
{nullptr,
273281
Json({{"ready", true},
282+
{"supports_ignore_eos", true},
283+
{"default_stop_strings", stops ? Json(*stops) : Json(nullptr)},
274284
{"multiplexed", true},
275285
{"max_named_sessions", info.max_sessions},
276286
{"max_inflight_requests", config_.max_inflight_requests}})
@@ -413,7 +423,8 @@ class Worker {
413423
"top_p",
414424
"top_k",
415425
"seed",
416-
"stop"});
426+
"stop",
427+
"ignore_eos"});
417428
std::optional<std::string> key;
418429
if (message.contains("session_id"))
419430
key = session_key(message);

‎examples/llm_server/cpp/test/test_multiplexed_worker.cpp‎

Lines changed: 40 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -122,7 +122,8 @@ class ProtocolTest : public ::testing::Test {
122122
MultiplexedWorkerConfig config = {},
123123
std::size_t max_sessions = 4,
124124
std::size_t max_requests = 8,
125-
std::size_t max_pending_operations = 64) {
125+
std::size_t max_pending_operations = 64,
126+
std::vector<batching::Token> default_stop_tokens = {}) {
126127
signal(SIGPIPE, SIG_IGN);
127128
ASSERT_EQ(pipe(input_), 0);
128129
ASSERT_EQ(pipe(output_), 0);
@@ -133,6 +134,10 @@ class ProtocolTest : public ::testing::Test {
133134
runtime_config.max_pending_operations = max_pending_operations;
134135
runtime_config.max_events_per_request = 128;
135136
runtime_config.max_tokens_per_request = 8192;
137+
const Json decoded_stops = default_stop_tokens.empty() ? Json::array()
138+
: tokenizer_.piece.empty() ? Json(nullptr)
139+
: Json::array({tokenizer_.piece});
140+
runtime_config.default_stop_tokens = std::move(default_stop_tokens);
136141
runtime_ = std::make_unique<serving::ServingRuntime>(
137142
executor_,
138143
batching::DecodeFirstScheduler::create(16, 8, 8),
@@ -145,9 +150,11 @@ class ProtocolTest : public ::testing::Test {
145150
auto ready = receive();
146151
ASSERT_TRUE(ready.value("ready", false));
147152
ASSERT_TRUE(ready.value("multiplexed", false));
153+
ASSERT_TRUE(ready.value("supports_ignore_eos", false));
148154
ASSERT_FALSE(ready.contains("request_id"));
149155
ASSERT_EQ(ready.at("max_named_sessions"), max_sessions);
150156
ASSERT_EQ(ready.at("max_inflight_requests"), config.max_inflight_requests);
157+
EXPECT_EQ(ready.at("default_stop_strings"), decoded_stops);
151158
}
152159

153160
std::shared_ptr<CheckpointGate> gate() {
@@ -575,6 +582,12 @@ TEST_F(ProtocolTest, StrictFieldsAndUnsupportedModalitiesAreIsolated) {
575582
request["max_new_tokens"] = value;
576583
invalid.push_back(request);
577584
}
585+
for (const auto& value :
586+
{Json(), Json(0), Json(1), Json("true"), Json::array()}) {
587+
auto request = generate(id++);
588+
request["ignore_eos"] = value;
589+
invalid.push_back(request);
590+
}
578591
auto image = generate(id++);
579592
image["image"] = "ignored?";
580593
invalid.push_back(image);
@@ -1320,6 +1333,32 @@ TEST_F(ProtocolTest, OperationCapacityReservesCancellationResponses) {
13201333
EXPECT_TRUE(done.value("cancelled", false));
13211334
EXPECT_EQ(finish(), 0);
13221335
}
1336+
TEST_F(ProtocolTest, AcceptsBothIgnoreEosValues) {
1337+
executor_.stop_token = 999;
1338+
executor_.emit_before_stop = 1;
1339+
start({}, 4, 8, 64, {999});
1340+
for (uint64_t id = 1; id <= 2; ++id) {
1341+
auto request = generate(id, 4);
1342+
request["ignore_eos"] = id == 2;
1343+
send(request);
1344+
Json done;
1345+
do {
1346+
done = receive();
1347+
ASSERT_FALSE(done.contains("error")) << done;
1348+
} while (done.contains("token"));
1349+
EXPECT_TRUE(done.value("done", false));
1350+
EXPECT_EQ(done.at("completion_tokens"), id == 1 ? 1 : 4);
1351+
EXPECT_EQ(done.at("finish_reason"), id == 1 ? "stop" : "length");
1352+
}
1353+
EXPECT_EQ(finish(), 0);
1354+
}
1355+
1356+
TEST_F(ProtocolTest, UndecodableDefaultStopsAdvertiseNull) {
1357+
tokenizer_.piece.clear();
1358+
start({}, 4, 8, 64, {999});
1359+
EXPECT_EQ(finish(), 0);
1360+
}
1361+
13231362
// Subprocess mode for Python validation regressions. The inherited socket is
13241363
// separate from JSONL: 'A' witnesses active execution, and 'R' releases it.
13251364
int run_validation_worker(int gate_fd) {

‎examples/llm_server/python/chat_template.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@
3434
"<|im_end|>",
3535
"<|endoftext|>",
3636
"<|eot_id|>",
37+
"<|eot|>",
3738
"<|end|>",
3839
"<|end_of_text|>",
3940
"<end_of_turn>",

‎examples/llm_server/python/multiplexed_worker_client.py‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -238,6 +238,8 @@ def __init__(
238238
max_buffered_chars: int = 1024 * 1024,
239239
max_message_bytes: int = _MAX_MESSAGE_BYTES,
240240
max_request_bytes: int = _MAX_REQUEST_BYTES,
241+
supports_ignore_eos: bool = False,
242+
default_stop_strings: list[str] | None = None,
241243
):
242244
_validate_limits(
243245
max_named_sessions=max_named_sessions,
@@ -250,6 +252,15 @@ def __init__(
250252
if proc.stdin is None or proc.stdout is None:
251253
raise WorkerError("worker requires stdin and stdout pipes")
252254
self._loop = asyncio.get_running_loop()
255+
if default_stop_strings is not None and (
256+
not isinstance(default_stop_strings, list)
257+
or any(not isinstance(s, str) or not s for s in default_stop_strings)
258+
):
259+
raise WorkerError(
260+
"default_stop_strings must be null or a list of nonempty strings"
261+
)
262+
self.default_stop_strings = default_stop_strings
263+
self.supports_ignore_eos = supports_ignore_eos
253264
self.max_named_sessions = max_named_sessions
254265
self.max_inflight_requests = max_inflight_requests
255266
self._mailbox_capacity = mailbox_capacity
@@ -717,6 +728,8 @@ def generate(self, prompt, config, request_id=None) -> WorkerGeneration:
717728
"seed": getattr(config, "seed", 0),
718729
"stop": list(getattr(config, "stop", []) or []),
719730
}
731+
if getattr(config, "ignore_eos", False):
732+
request["ignore_eos"] = config.ignore_eos
720733
segments = getattr(config, "prompt_segments", None)
721734
request["prompt_segments" if segments is not None else "prompt"] = (
722735
segments if segments is not None else prompt
@@ -823,6 +836,8 @@ async def spawn_multiplexed_worker(
823836
return MultiplexedWorkerClient(
824837
proc,
825838
max_named_sessions=msg.get("max_named_sessions", 0),
839+
supports_ignore_eos=msg.get("supports_ignore_eos") is True,
840+
default_stop_strings=msg.get("default_stop_strings"),
826841
max_inflight_requests=msg.get("max_inflight_requests", 64),
827842
mailbox_capacity=mailbox_capacity,
828843
max_buffered_chars=max_buffered_chars,

‎examples/llm_server/python/protocol.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,8 @@ class ChatCompletionRequest(BaseModel):
6767
frequency_penalty: Optional[float] = None
6868
presence_penalty: Optional[float] = None
6969
top_k: Optional[int] = None
70+
# Disable model/template termination; explicit request stops still apply.
71+
ignore_eos: bool = Field(default=False, strict=True)
7072
logit_bias: Optional[dict[str, float]] = None
7173
# Output-contract fields: modeled (not dropped) so we reject the ones we
7274
# can't honor rather than returning an output that violates what was asked.

0 commit comments

Comments
 (0)